Using GitOps with the agent for Kubernetes (deprecated)

Version history
  • Introduced in GitLab 13.7.
  • Introduced in GitLab 14.0, the resource_inclusions and resource_exclusions attributes were removed and reconcile_timeout, dry_run_strategy, prune, prune_timeout, prune_propagation_policy, and inventory_policy attributes were added.
  • Moved from GitLab Premium to GitLab Free in 15.3.
  • Changed to make the id attribute optional in GitLab 15.7.
  • Specifying a branch, tag, or commit reference to fetch the Kubernetes manifest files introduced in GitLab 15.7.
caution
This feature was deprecated in GitLab 16.2. You should use the Flux integration for GitOps.

This diagram shows the repositories and main actors in a GitOps deployment:

sequenceDiagram participant D as Developer participant A as Application code repository participant M as Manifest repository participant K as GitLab agent participant C as Agent configuration repository loop Regularly K-->>C: Grab the configuration end D->>+A: Pushing code changes A->>M: Updating manifest loop Regularly K-->>M: Watching changes M-->>K: Pulling and applying changes end

For details, view the architecture documentation.

Migrate to Flux

You should migrate your legacy GitOps deployments so they can be deployed with Flux. To migrate, configure Flux to deploy manifests with Kustomize. If you don’t have a kustomization.yaml file in the given path, it is generated automatically.

Prerequisites:

  • A configuration like:

    manifest_projects:
    - id: my-group/my-project
      default_namespace: production
      paths:
      - glob: 'environments/production/**/*.yaml'
    
  • A Flux installation with manifests in environments/flux-system.
  • You use a deploy token to access GitLab.
  • Your cluster can access GitLab over HTTPS.

To migrate:

  1. Create a file called environments/flux-system/production.yaml with the following contents:

    # This manifest was generated by flux. DO NOT EDIT.
    ---
    apiVersion: source.toolkit.fluxcd.io/v1
    kind: GitRepository
    metadata:
      name: production
      namespace: flux-system
    spec:
      interval: 1m0s
      ref:
        branch: main
      secretRef:
        name: gitlab-deploy-token
      url: https://gitlab.example.com/my-group/my-project.git
    ---
    apiVersion: kustomize.toolkit.fluxcd.io/v1
    kind: Kustomization
    metadata:
      name: production
      namespace: flux-system
    spec:
      interval: 10m0s
      path: ./environments/production
      prune: true
      sourceRef:
        kind: GitRepository
        name: production
    
  2. Optional. Because agentk uses the default_namespace by default, you might need to add a kustomization.yaml file to /environments/production/ and list the relevant resources. For example:

    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    namespace: default
    resources:
    - relative/path/to-your/resource-1.yaml
    - relative/path/to-your/resource-2.yaml
    

    When you commit the kustomization.yaml file to the repository, a reconciliation with Flux and agentk is triggered. Because agentk can’t handle the Kustomization file, it logs errors when you commit the file.

  3. To ensure Flux has taken over the management of the resource, check for the resource in the status.inventory value of the production Flux Kustomization object:

    kubectl get kustomization production -n flux-system -o json | jq '.status.inventory.entries'
    
  4. Remove the entry from the manifest_projects list.

After you migrate, your GitOps deployments deploy with Flux. To get the most out of your Flux integration, see the Flux Kustomization CRD and GitLab Flux documentation.

GitOps workflow steps

To update a Kubernetes cluster by using GitOps, complete the following steps.

  1. Ensure you have a working Kubernetes cluster, and that the manifests are in a GitLab project.
  2. In the same project, register and install the GitLab agent.
  3. Configure the agent configuration file so that the agent monitors the project for changes to the Kubernetes manifests. Use the GitOps configuration reference for guidance.

Any time you commit updates to your Kubernetes manifests, the agent updates the cluster.

GitOps configuration reference

The following snippet shows an example of the possible keys and values for the GitOps section of an agent configuration file (config.yaml).

gitops:
  manifest_projects:
  - id: gitlab-org/cluster-integration/gitlab-agent
    ref: # either `branch`, `tag` or `commit` can be specified
      branch: production
      # commit: <mysha>
      # tag: v1.0
    default_namespace: my-ns
    paths:
      # Read all YAML files from this directory.
    - glob: '/team1/app1/*.yaml'
      # Read all .yaml files from team2/apps and all subdirectories.
    - glob: '/team2/apps/**/*.yaml'
      # If 'paths' is not specified or is an empty list, the configuration below is used.
    - glob: '/**/*.{yaml,yml,json}'
    reconcile_timeout: 3600s
    dry_run_strategy: none
    prune: true
    prune_timeout: 3600s
    prune_propagation_policy: foreground
    inventory_policy: must_match
KeywordDescription
manifest_projectsProjects where your Kubernetes manifests are stored. The agent monitors the files in the repositories in these projects. When manifest files change, the agent deploys the changes to the cluster.
idPath to a Git repository that has Kubernetes manifests in YAML or JSON format. No authentication mechanisms are supported. Default is the agent configuration repository.
refOptional. Git reference in the configured Git repository to fetch the Kubernetes manifest files from. If not specified or empty, the default branch is used. If specified, it must contain either branch, tag, or commit.
ref.branchBranch name in the configured Git repository to fetch the Kubernetes manifest files from.
ref.tagTag name in the configured Git repository to fetch the Kubernetes manifest files from.
ref.commitCommit SHA in the configured Git repository to fetch the Kubernetes manifest files from.
default_namespaceNamespace to use if not set explicitly in object manifest. Also used for inventory ConfigMap objects.
pathsRepository paths to scan for manifest files. Directories with names that start with a dot (.) are ignored.
paths[].globRequired. See doublestar and the match function for globbing rules.
reconcile_timeoutDetermines whether the applier should wait until all applied resources have been reconciled, and if so, how long to wait. Default is 3600 seconds (1 hour).
dry_run_strategyDetermines whether changes should be performed. Can be: none, client, or server. Default is none.
pruneDetermines whether pruning of previously applied objects should happen after apply. Default is true.
prune_timeoutDetermines whether to wait for all resources to be fully deleted after pruning, and if so, how long to wait. Default is 3600 seconds (1 hour).
prune_propagation_policyThe deletion propagation policy that should be used for pruning. Can be: orphan, background, or foreground. Default is foreground.
inventory_policyDetermines whether an inventory object can take over objects that belong to another inventory object or don’t belong to any inventory object. This is done by determining if the apply/prune operation can go through for a resource based on comparison of the inventory-id value in the package and the owning-inventory annotation (config.k8s.io/owning-inventory) in the live object. Can be: must_match, adopt_if_no_inventory, or adopt_all. Default is must_match.

GitOps annotations

The GitLab agent for Kubernetes has annotations you can use to:

  • Sort resources: Apply or delete resources in a specific order.
  • Use apply-time mutation: Dynamically substitute fields from one resource configuration to another.

The agent has default sorting, but with annotations, you can fine-tune the order and apply time-value injection.

To provide the GitOps functionality, the GitLab agent for Kubernetes uses the cli-utils library, a Kubernetes SIG project. For more information, see the available annotations in the cli-utils documentation.

Automatic drift remediation

Drift happens when the current configuration of an infrastructure resource differs from its desired configuration. Typically, this is caused by manually editing resources directly rather than via the used infrastructure-as-code mechanism. Minimizing the risk of drift helps to ensure configuration consistency and successful operations.

In GitLab, the agent for Kubernetes regularly compares the desired state from the git repository with the actual state from the Kubernetes cluster. Deviations from the git state are fixed at every check. These checks happen automatically every 5 minutes. They are not configurable.

The agent uses server-side applies. As a result, every field in a resource can have different managers. Only fields managed by git are checked for drift. This facilitates the use of in-cluster controllers to modify resources like Horizontal Pod Autoscalers.

Troubleshooting

Avoiding conflicts when you have multiple projects

The agent watches each glob pattern set under a project’s paths section independently, and makes updates to the cluster concurrently. If changes are found at multiple paths, when the agent attempts to update the cluster, a conflict can occur.

To prevent this from happening, consider storing a logical group of manifests in a single place and reference them only once to avoid overlapping globs.

For example, both of these globs match *.yaml files in the root directory and could cause conflicts:

gitops:
  manifest_projects:
  - id: project1
    paths:
    - glob: '/**/*.yaml'
    - glob: '/*.yaml'

Instead, specify a single glob that matches all *.yaml files recursively:

gitops:
  manifest_projects:
  - id: project1
    paths:
    - glob: '/**/*.yaml'

Use multiple agents or projects

If you store your Kubernetes manifests in separate GitLab projects, update your agent configuration file with the location of these projects.

caution
The project with the agent’s configuration file can be private or public. Other projects with Kubernetes manifests must be public. Support for private manifest projects is tracked in this epic.