An image tag is a version: why never :latest, and how to roll back with one commit

The application had been running unchanged for a week. Overnight a node was restarted after an operating system update, the pod started on another node and pulled the app:latest image, which since yesterday had pointed to an unreleased version from the main branch. In the morning the application behaves differently, there is no new commit in the deployment repository, and the question “what do we go back to” has no answer, because the previous latest no longer exists.

An image tag should be a version: an immutable identifier of a single build. The manifest in Git then says exactly what is running, all nodes run the same code, and a rollback is a one-line change.

Why never :latest

latest does not mean “the newest stable version”. It is an ordinary tag that tools assign by default when no other is given, and it can be moved to any image at any time. Three problems follow:

• No repeatability. The same manifest on two nodes can run two different images, depending on when each node pulled it.
• No rollback. There is no record of which image latest pointed to a week ago, so there is nothing to return to.
• A dependency on the registry at every pod start. Kubernetes sets the default pull policy based on the tag.

Reference in the manifest	Default imagePullPolicy	Effect
app:latest	Always	every container start queries the registry; if the registry does not answer, the pod does not start
app (no tag)	Always	as above, because no tag means latest
app:1.4.2-3f9c2a1b	IfNotPresent	the image from the node, if it is already there
app@sha256:…	IfNotPresent	the image identified by its content

With Always, the kubelet checks with the registry which digest the tag points to every time it starts a container. A registry outage or an exhausted rate limit then means a pod restart ends in ErrImagePull, even though the image is on the node.

The tag scheme: version and commit hash

A good tag answers two questions: which version it is and what code it was built from. The 1.4.2-3f9c2a1b format combines a semantic version (order and compatibility readable by people) with the commit hash (an unambiguous link to the code). In GitLab CI the version comes from the Git tag v1.4.2, and the hash from the CI_COMMIT_SHORT_SHA variable, which is eight characters long.

build-image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.24.0-debug
    entrypoint: [""]
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+\.\d+\.\d+$/
  script:
    # registry login as in the previous part of the series
    - VERSION="${CI_COMMIT_TAG#v}"
    - /kaniko/executor
      --context "${CI_PROJECT_DIR}"
      --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${VERSION}-${CI_COMMIT_SHORT_SHA}"
      --destination "${CI_REGISTRY_IMAGE}:${VERSION}"
      --digest-file "${CI_PROJECT_DIR}/image-digest.txt"
  artifacts:
    paths: [image-digest.txt]

Builds from working branches, which nobody deploys to production, get only a sha-3f9c2a1b tag. The tag alone then shows whether an image is a release, and the clean-up policy can tell them apart easily. The --digest-file output contains the digest of the pushed image and is useful in the next job, the one that changes the manifest.

Tag immutability in the registry

A tag scheme achieves nothing if someone can push a different image under an existing tag. Registries have mechanisms for this:

• GitLab: protection rules for container tags (also available in the free tier) restrict pushing and deleting tags that match a pattern to selected roles. Full immutability, independent of role, is in the Ultimate tier.
• Harbor: tag immutability rules in the project settings.
• Amazon ECR: the repository setting imageTagMutability set to IMMUTABLE.

Whatever the registry, the pipeline can refuse to overwrite on its own. A fragment of a job that runs before the build, in an image that contains skopeo:

TAG="${CI_COMMIT_TAG#v}-${CI_COMMIT_SHORT_SHA}"
if skopeo inspect --creds "${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD}" \
     "docker://${CI_REGISTRY_IMAGE}:${TAG}" > /dev/null 2>&1; then
  echo "Tag ${TAG} already exists in the registry, stopping."
  exit 1
fi

Pinning by digest

A digest (sha256:…) is a hash of the image manifest’s content. A tag can be moved, a digest cannot: a different image has a different digest. Kubernetes accepts a reference with both a tag and a digest; the image is then identified by the digest, and the tag remains readable information for people.

spec:
  template:
    spec:
      containers:
        - name: app
          image: registry.example.com/shop/app:1.4.2-3f9c2a1b@sha256:6f1c9e…

The digest can be read without pulling the image:

skopeo inspect --format '{{.Digest}}' docker://registry.example.com/shop/app:1.4.2-3f9c2a1b

Pinning by digest is mandatory wherever you do not control tag immutability: base images, tool images in CI, images from public Helm charts. For your own images in a registry with tag protection, a tag with version and commit is enough; add the digest for the most critical services.

Rollback: revert the manifest

When deployment goes through GitOps, every version change is a commit in the manifest repository. A rollback is a revert of that commit:

git log --oneline -3 -- apps/shop/overlays/prod/
# a91c0de shop: 1.4.3-7d2e91f0
# 5be2f13 shop: 1.4.2-3f9c2a1b
git revert --no-edit a91c0de
git push origin main

The GitOps controller sees the change and restores the previous image. Because the 1.4.2-3f9c2a1b tag still exists and points to the same image, the rollback restores exactly what was running before.

Registry clean-up policy

Immutable tags mean the registry grows. Clean-up has to distinguish releases from working builds. In GitLab this is the tag clean-up policy in the project’s container registry settings (Settings → Packages and registries):

Tag group	Pattern	Rule
Releases	\d+\.\d+\.\d+(-[0-9a-f]{8})?	always kept
Working builds	sha-[0-9a-f]{8}	removed after 14 days, the last 10 kept
Layer cache	separate cache repository	reproducible, lifetime as set in the build tool

GitLab wraps the patterns in start and end anchors itself, so you do not type ^ or $ in the field. Removing tags does not free disk space straight away: on a self-managed instance that only happens with registry garbage collection, run in a maintenance window. Before a rule takes effect, compare it with the list of images running in your clusters: an image that is running must not disappear from the registry, because the next pod restart on another node will end in a pull error.

Check your own environment

Images with the latest tag or no tag running in the cluster:

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.spec.containers[*].image}{"\n"}{end}' \
  | awk '{for (i = 2; i <= NF; i++) { n = split($i, p, "/"); if (p[n] !~ /[:@]/ || p[n] ~ /:latest$/) print $1, $i }}' \
  | sort -u
# Good: no output. Each line is a namespace and an image to fix.

The image in the manifest and the digest that is actually running:

kubectl get pods -n shop -o custom-columns='POD:.metadata.name,IMAGE:.spec.containers[*].image,IMAGEID:.status.containerStatuses[*].imageID'
# Good: all replicas have the same imageID.
# Bad: different digests for the same tag, meaning the tag was overwritten between pulls.

The digest and creation date of the image in the registry:

skopeo inspect --format '{{.Digest}} {{.Created}}' docker://registry.example.com/shop/app:1.4.2-3f9c2a1b
# For a single-architecture image the digest should match the end of imageID from the previous command.

The project’s clean-up policy settings in GitLab:

glab api "projects/shop%2Fapp" | jq '.container_expiration_policy'
# Good: enabled true and a name_regex_keep that covers release tags.
# Bad: enabled false while the registry keeps growing, or no pattern that keeps releases.