CI/CD and declarative deployments

One versioned pipeline template in GitLab CI, immutable image tags carrying the version and commit hash, and deployment through a reconciliation loop in Argo CD instead of a script.

CI/CDKubernetes and GitOps

A pipeline copied between repositories diverges into many files with a common ancestor, the latest tag removes repeatability and the ability to roll back from a deployment, and a script running kubectl apply does not see changes made on the cluster after it finishes. This document describes three elements of one delivery chain: a shared pipeline template in GitLab CI, the image tag as an immutable version, and deployment through a GitOps controller (Argo CD) instead of a script.

Pipeline template: include and variables

Pipeline logic lives in one templates repository, pulled in through include, versioned with tags and tested before every release. A project’s .gitlab-ci.yml contains an include key with the fields project (path of the templates repository), ref (branch, tag or commit SHA) and file (file path, may be a list), plus a variables: section with four values: the application name (APP_NAME), the directory in a monorepo (APP_PATH), the runtime image (RUNTIME_IMAGE) and the test command (TEST_COMMAND). GitLab merges the included configuration with the local one, and values defined in the project take precedence, so project variables override template defaults of the same name. The templates repository must be readable by the people who run pipelines; the simplest option is internal visibility on the instance.

A project defines no jobs of its own. A script: appearing in its file means the template lacks an extension point. GitLab does allow a project to override any job key from the template, but that is the beginning of another copy. The variables: section is visible to anyone who reads the repository, so tokens and passwords do not belong in it: they go into project or group variables marked as masked and protected, or into an external secrets manager. The same pattern can be built on CI/CD components with spec: inputs parameters, which validate types and permitted values (components are generally available from GitLab 17.0). A component version is given after the @ sign as a tag, SHA or branch, and shortened versions such as @1 work only for components published in the CI/CD catalog, so they partly replace the manual version pointer described below.

The template stages always have the same order: lint, test, build, publish, manifest update. A project can disable a stage with a variable (for example SKIP_TESTS) but does not change its content. The two required variables are checked by a separate job in the .pre stage, which always runs first, so a missing variable ends the pipeline with a readable message instead of an image pull error. Building takes place without a Docker daemon, in the hidden .kaniko-build job inherited by the build-* jobs, so the runner does not need the privileged mode. The GitLab documentation page on kaniko is marked as removed, and kaniko is no longer a maintained project; the alternatives are Docker, Buildah and Podman. Publishing uses the tag carrying the version (described in the next section), and the last stage changes the tag in the manifests repository from which the GitOps controller deploys. Extension points are described in the template documentation: a variable outside it is not part of the contract and may disappear in the next version.

stages: [lint, test, build, publish, deploy]

variables:
  APP_NAME: ""            # required in the project
  APP_PATH: "."
  RUNTIME_IMAGE: ""       # required in the project
  LINT_COMMAND: "make lint"
  TEST_COMMAND: "make test"
  SKIP_TESTS: "false"
  DOCKERFILE: Dockerfile

check-variables:
  stage: .pre
  image: busybox:1.36
  script:
    - test -n "$APP_NAME" || { echo "APP_NAME is not set"; exit 1; }
    - test -n "$RUNTIME_IMAGE" || { echo "RUNTIME_IMAGE is not set"; exit 1; }

# Image build without a Docker daemon: build-* jobs inherit the image and the registry login
.kaniko-build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.23.2-debug
    entrypoint: [""]
  before_script:
    - mkdir -p /kaniko/.docker
    - printf '{"auths":{"%s":{"auth":"%s"}}}' "$CI_REGISTRY" "$(printf '%s:%s' "$CI_REGISTRY_USER" "$CI_REGISTRY_PASSWORD" | base64 | tr -d '\n')" > /kaniko/.docker/config.json

test:
  stage: test
  image: $RUNTIME_IMAGE
  script:
    - cd "$APP_PATH"
    - sh -c "$TEST_COMMAND"
  rules:
    - if: $SKIP_TESTS == "true"
      when: never
    - when: on_success

Versioning: release tags and the major-version pointer

Template releases are vX.Y.Z tags under protection (Settings › Repository › Protected tags), which restricts who can create or delete a release tag; it is available in the Free, Premium and Ultimate plans. Projects, however, do not reference a specific tag but the major-version pointer: a protected branch v2, moved after every backward-compatible release. A fix then reaches all projects without separate merge requests, while a backward-incompatible change means a new v3 branch and migration at the projects’ own pace. The pointer is moved with git push origin 'v2.4.0^{commit}:refs/heads/v2' after pushing the v2.4.0 tag; without --force, Git rejects any change other than a fast-forward, so the pointer only moves forward.

Fig. 1. Projects include a versioned template and supply only variables; a fix reaches all of them when the v2 pointer moves, and the drift check catches projects on old versions or with their own logic.
ref in the projectWhat the project receivesUse
mainevery merged change, including faulty onesreference project only
v2 (pointer branch)successive backward-compatible releasesdefault for projects
v2.4.0 (tag)exactly that releaseprojects requiring full repeatability
commit SHAexactly that statediagnosing a problem

Template change process and project migration

The main branch of the templates repository is protected, and changes enter it only through a merge request approved by the team responsible for the platform. Enforcing approvals by designated people (approval rules and the CODEOWNERS file) is a feature of the Premium and Ultimate plans; in the Free plan, branch protection and a team agreement remain. A change goes through four steps:

  1. A change on a template branch and a merge request describing what changes, why, and whether it is backward-compatible.
  2. A check on a pilot project: in its test branch ref points at the change branch, and the pipeline must pass in full.
  3. After the merge to main, the templates repository pipeline runs the pipeline of the reference project, a minimal application that exists only to go through the whole path to a deployment on the test environment. A job with the trigger key (project, branch: main) and strategy: mirror does this; it adopts the status of the reference project pipeline, so an error in the template stops the template pipeline before anyone assigns a tag.
  4. The release tag, moving the pointer and an entry in the changelog.

Migration of existing projects starts with the pilot: its CI file is replaced by include and variables, and every difference from the template signals a missing extension point. After it is added in a backward-compatible release, further projects migrate, starting with those that deviate least from the pattern. One project is one merge request, and its pipeline proves that the stages and the image tag scheme are the same as before the change. Projects with logic that variables cannot express either get a new extension point in the template or remain an exception with a named owner and a review date.

Drift control

A report run weekly as a scheduled pipeline (Build › Pipeline schedules) goes through the group’s projects via the API and reads their .gitlab-ci.yml. For each project it prints the version of the included template and the number of its own script: sections.

#!/bin/sh
# CI variables: GITLAB_API (https://gitlab.example.com/api/v4), GROUP_ID, REPORT_TOKEN (read_api scope); the list covers only the first page, max. 100 projects
auth="PRIVATE-TOKEN: $REPORT_TOKEN"
curl -sf -H "$auth" "$GITLAB_API/groups/$GROUP_ID/projects?include_subgroups=true&simple=true&per_page=100" |
  jq -r '.[] | "\(.id) \(.path_with_namespace)"' |
  while read -r id path; do
    ci=$(curl -sf -H "$auth" "$GITLAB_API/projects/$id/repository/files/.gitlab-ci.yml/raw?ref=HEAD") ||
      { echo "$path: no .gitlab-ci.yml"; continue; }
    ref=$(printf '%s\n' "$ci" | sed -n 's/^ *ref: *//p' | head -n 1)
    own=$(printf '%s\n' "$ci" | grep -c '^ *script:')
    echo "$path: template ${ref:-NONE}, own script: sections: $own"
  done

The result requires a decision for three groups. A project on the previous major version needs a migration date. A project with its own sections requires a conversation with the team: the template gets a new extension point or the override disappears. A project without the template is outside the arrangement and goes onto the migration list. The list is short if the rules apply from the start.

Image tag as a version

latest does not mean “the latest stable version”. It is an ordinary tag applied by default when no other is given, and it can be moved to any image at any time. There are three consequences. The same manifest on two nodes can run two different images, depending on when each node pulled it. There is no record of which image latest pointed at a week ago, so there is nothing to return to. Finally, with latest Kubernetes sets imagePullPolicy: Always by default, and the runtime then contacts the registry on every container start, so a registry outage or an exhausted rate limit can prevent a pod from restarting even though the image is on the node. Setting IfNotPresent with latest does not solve the problem but hides it: each node stays with the version it pulled first, and different nodes run different code under the same name.

Reference in the manifestDefault imagePullPolicyEffect
app:latestAlwaysevery container start queries the registry; if the registry does not respond, the pod does not start
app (no tag)Alwaysas above, because no tag means latest
app:1.4.2-3f9c2a1bIfNotPresentthe image from the node, if already there
app@sha256:…IfNotPresentimage identified by content

Tag scheme

A tag answers two questions: which version this is and which code it was built from. The format 1.4.2-3f9c2a1b combines a semantic version (order and compatibility readable by humans) with a commit hash (an unambiguous link to the code). In GitLab CI the version comes from the Git tag v1.4.2 and the hash from the CI_COMMIT_SHORT_SHA variable, which has eight characters. The hash stays in the name despite the Git tag: a Git tag can be moved, while the image tag records which code the image was actually built from, visible in kubectl describe without consulting Git.

The build-release job in the template runs for Git tags matching ^v\d+\.\d+\.\d+$, derives VERSION="${CI_COMMIT_TAG#v}" and builds the image with two destinations: ${VERSION}-${CI_COMMIT_SHORT_SHA} and ${VERSION}. The --digest-file option saves the digest of the pushed image as an artefact for the job that changes the manifest. The build-branch job, for working branches other than the default one, publishes only the tag sha-3f9c2a1b, so the tag alone shows whether an image is a release, and the cleanup policy distinguishes them easily.

The separator is a hyphen, because semver writes build metadata after a plus sign (1.4.2+3f9c2a1b), while an image tag permits only letters, digits, underscore, dot and hyphen, up to 128 characters. In semver the part after a hyphen denotes a pre-release version, so if a tool sorts tags by semver, releases should also carry a plain 1.4.2 tag.

Tag immutability in the registry

A tag scheme achieves nothing if someone can push a different image under an existing tag. Container tag protection rules in GitLab restrict pushing and deleting tags matching a pattern to selected roles; they are available in the Free, Premium and Ultimate plans, generally available from GitLab 17.11. Full immutability, independent of role, is an Ultimate feature, generally available from GitLab 18.10. Harbor has tag immutability rules in project settings, and Amazon ECR has the repository setting imageTagMutability set to IMMUTABLE. Regardless of the registry, the pipeline can refuse to overwrite on its own: a job run before the build checks whether the tag already exists and aborts the build.

Pinning by digest

A digest (sha256:…) is a hash of the image manifest content. A tag can be moved, a digest cannot, because a different image has a different digest. Kubernetes accepts a reference with a tag and a digest at the same time, for example registry.example.com/shop/app:1.4.2-3f9c2a1b@sha256:6f1c9e…: the image is then identified by the digest, and the tag remains readable information for humans. Pinning is mandatory where there is no control over tag immutability: for base images, CI tool images and images from public Helm charts. For own images in a registry with tag protection, a tag with the version and commit is sufficient; a digest is added for the most important services.

Registry cleanup policy

Immutable tags mean the registry grows, and cleanup has to distinguish releases from working-branch images. In GitLab this is handled by the tag cleanup policy in the project’s container registry settings (Settings › Packages and registries).

Tag groupPatternRule
Releases\d+\.\d+\.\d+(-[0-9a-f]{8})?always kept
Working imagessha-[0-9a-f]{8}removed after 14 days, the 10 latest kept
Layer cacheseparate cache repositoryreproducible, lifetime as in the build tool

GitLab wraps patterns in start and end anchors itself, so ^ and $ are not entered in the field. The policy does not remove tags covered by a protection rule or an immutability rule. Removing tags does not free disk space immediately, because layers are released only by registry garbage collection. On a self-managed instance without the registry metadata database it is run manually in a maintenance window; a registry with the metadata database removes untagged layers automatically after 24 hours.

GitOps versus a deployment script

A deployment script, whether Bash, a pipeline job or an Ansible playbook, performs steps and finishes, and it knows only what it did at the moment it ran. A typical example is a deploy job in the pipeline with the commands kubectl apply -f k8s/ and kubectl rollout status. A manual change on the cluster remains until the next pipeline, often until the next code change. A resource removed from the repository stays on the cluster, because kubectl apply removes nothing by default (removal requires --prune). The pipeline needs a kubeconfig with write access to the cluster in CI variables, and a rollback requires re-running an old pipeline, which does not always reproduce the previous state.

In GitOps the repository is the single source of truth for the desired state, and a controller in the cluster compares that state with reality in a loop and removes differences. In Argo CD the unit is the Application.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: orders
  source:
    repoURL: https://git.example.com/platform/deploy.git
    targetRevision: main
    path: apps/orders/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: orders
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
SettingWhat it doesDefault
automatedsynchronises after detecting a change in the repositoryoff: Argo CD only shows OutOfSync
prune: trueremoves from the cluster resources removed from the repositoryoff
selfHeal: truereverts manual changes made on the clusteroff
CreateNamespace=truecreates the target namespaceoff
PruneLast=trueremoves surplus resources at the end of synchronisationoff
resources-finalizer finalizerwhen the Application is deleted, also deletes its resourcesnone

Without prune and selfHeal, automated synchronisation adds and changes resources but does not remove orphaned ones and does not revert manual changes. With both options the cluster continuously matches the repository.

AspectScript in a pipelineGitOps (Argo CD)
When a change reaches the clusterwhen the pipeline runsafter a change in the repository, within minutes
Manual change on the clusterstays until the next deploymentvisible as OutOfSync, reverted with selfHeal
Resource removed from the repositorystays on the clusterremoved with prune
Rollbackre-running an old pipelinegit revert (without reverting a database migration)
Pipeline permissionswrite access to the cluster APIwrite access to the manifests repository
Rebuilding a cluster from scratchrunning all pipelines in turnafter bootstrap: pointing at the parent application repository

Bootstrap is the installation of Argo CD and manual creation of a parent application (the app of apps pattern) that points at the others. Secrets, which must not be stored in Git, remain outside it.

Ordering: sync waves and hooks

Argo CD applies resources in waves, from the lowest number, and moves to the next only when the previous one is healthy. The wave number is the argocd.argoproj.io/sync-wave annotation on a resource, for example "-1" for a ConfigMap that must exist before the Deployment. A database migration before a new version is a PreSync hook: a Job with the annotations argocd.argoproj.io/hook: PreSync and argocd.argoproj.io/hook-delete-policy: BeforeHookCreation, run before the remaining resources are synchronised. Its failure blocks the deployment.

Drift and emergency fixes

By default Argo CD checks the repository every 120 seconds (timeout.reconciliation) plus a random delay of up to 60 seconds (timeout.reconciliation.jitter), that is, up to about three minutes; a webhook from the Git server shortens this to seconds. Every difference between the repository and the cluster is an OutOfSync state, and with selfHeal the change is reverted automatically. Fields that are meant to be changed by someone else, for example the replica count with a HorizontalPodAutoscaler, are excluded from the comparison through ignoreDifferences (the /spec/replicas pointer for a Deployment) together with the RespectIgnoreDifferences=true option.

The same loop works against a manual emergency fix. When someone manually raises a memory limit in a Deployment during an outage, with selfHeal the controller reverts the change almost immediately (the default self-heal retry interval is 5 seconds, --self-heal-timeout-seconds), before it can help. Without self-healing the change lives until the next deployment, which three weeks later restores the outage with no obvious cause. The incident procedure accounts for this with two paths: a quick commit to the deployment repository, which leaves a trace, or a deliberate suspension of automated synchronisation of the application (the automated field, for example enabled: false) with an owner and a deadline. After the incident the change goes into the repository and synchronisation is restored.

Rollback

When deployment goes through GitOps, every version change is a commit in the manifests repository, so a rollback is a git revert of the commit that changed the version, for example from 1.4.2-3f9c2a1b to 1.4.3-7d2e91f0, pushed to the main branch. The controller restores the previous image, and because the tag 1.4.2-3f9c2a1b still points at the same image, exactly the same bits that ran before return. The Git history is complete, because nothing happened outside Git. A version is not rolled back with kubectl rollout undo or kubectl set image: a controller with self-healing restores the state from the repository, and the history keeps no trace of the rollback. argocd app rollback does not work for an application with automated synchronisation enabled.

Repository structure and promotion between environments

The manifests repository has a base and overlays layout: apps/orders/base holds the shared manifests, while overlays/test and overlays/prod hold only the differences (image tag, replica count, addresses). Each environment has its own Application pointing at its overlay. Promotion from test to production is a commit that sets the same image tag in the production overlay as in the test one: the image is not built a second time, and the release leaves a single trace in Git (who, when and in which merge request).

Boundary of the pipeline’s role

GitOps does not eliminate CI. The pipeline still builds the image, runs the tests and publishes it with the tag carrying the version, but it does not touch the cluster: it modifies the manifest in the deployment repository, for example with kustomize edit set image registry.example.com/shop/orders:1.4.2-3f9c2a1b, pushes a commit and finishes. It therefore needs only write access to the manifests repository, not a cluster kubeconfig. Compromising the pipeline remains dangerous, because whoever writes to the manifests repository effectively decides what is deployed.

Two mechanisms narrow the reach of that influence. The first is protection of the deployment repository branch: production changes go through a merge request with human review. The second is a dedicated Argo CD project (AppProject). The default project permits any repositories, clusters and resource kinds, including cluster-scoped ones, while the orders project restricts them: sourceRepos narrows the source to one deployment repository, destinations to the orders namespace on the local cluster, and namespaceResourceBlacklist excludes ResourceQuota, LimitRange and NetworkPolicy. An application from such a project deploys manifests only from one repository and only into one namespace, while resource limits and network policies remain under the platform team’s control. Cluster-scoped resources are permitted by the clusterResourceWhitelist list, kept as short as possible.

Decisions and alternatives

ApproachWhen it makes senseLimitation
Shared template through include with a major-version pointer branchmany projects with the same delivery patterna template change reaches all projects; requires a reference project and drift control
include with a tag or SHAprojects requiring full repeatability, diagnosisfixes do not arrive without a change to the project file
CI/CD components with spec: inputsvalidation of parameter types and permitted valuesshortened versions (@1) only for components from the CI/CD catalog
A copy of the CI file in every projecta single projectcopies diverge, every change is a separate merge request
version-hash tagown images in a registry with tag protectionwithout tag protection overwriting is still possible
Pinning by digestbase images, CI tools, images from public charts, the most important servicesunreadable for humans without a tag alongside
latest tagno justified use in deploymentsno repeatability or rollback, registry dependency on every pod start
Deployment scripta single server, a few deployments a year, access only for the pipelinedoes not detect drift or remove orphaned resources; pipeline with write access to the cluster API
GitOps (Argo CD)more than one person changes the environment, frequent or multi-component deployments, change audit, reproducibilitycontroller maintenance; secrets outside Git; self-healing reverts manual fixes

GitOps pays off when at least one of the conditions from the table row is met; otherwise a script is simpler and sufficient.

Implementation

  1. Templates repository: protected main branch, protected release tags, a template with stages and extension points, and a reference project run by a trigger job with strategy: mirror. Check: an error introduced deliberately into the template stops its pipeline before a tag is assigned.
  2. Project migration from the pilot to those deviating least from the pattern, one merge request per project. Check: the pipeline of a migrated project has the same stages and the same image tag scheme as before the change. Exceptions have an owner and a review date.
  3. The tag scheme in the template (build-release, build-branch), tag protection or immutability rules in the registry, and a job rejecting an overwrite of an existing tag. Check: an attempt to push an image under an existing tag ends in refusal.
  4. Inventory of images without a tag or with the latest tag: a list of images from the containers and initContainers of all pods (kubectl get pods -A with -o jsonpath), filtered for a missing tag or :latest. The expected result is empty, and each line is a namespace and an image to correct.
  5. Registry cleanup policy with patterns for releases and working images, enabled after comparison with the list of images running in the clusters.
  6. Deployment through Argo CD: a separate AppProject and Application for each environment (prune, selfHeal), bootstrap of the parent application, branch protection on the manifests repository, and, in the pipeline, a tag change in the manifest instead of kubectl apply. Check: a manual change on the cluster is visible as OutOfSync and reverted, and git revert restores the previous version.
  7. A weekly template drift report with a decision for each of the three project groups.

Limitations and pitfalls

  • A tag named like the pointer branch. With two v2 references GitLab fetches the file from the tag, and the project file does not show which one the author meant. A tag with the same name as the pointer branch is not created; an unambiguous form is the full name, refs/heads/v2 or refs/tags/v2.
  • A revert does not undo a database migration. Reverting the manifest restores the image, but not changes made by the new version or the PreSync hook, so the previous version must work on the post-migration schema. Migrations are designed to be backward-compatible: a column is added first and removed only in a later release.
  • HPA and replicas in the manifest. A replicas field in a Deployment together with an HPA on the same Deployment are two controllers fighting over one value: the HPA scales, Argo CD reverts. The simplest fix is to remove replicas from the manifest; ignoreDifferences is for cases where that cannot be done.
  • Cleanup removes a running image. A cleanup rule that removes the tag of an image running in a cluster causes a pull error on the next pod restart on another node. The rule’s patterns are compared with the running images, and releases stay outside the removal scope.

Sources