The argument “we do not need cluster backups because everything is in Git” is half true. Manifests, configuration, image versions: yes, they are in the repository and the cluster can be rebuilt by syncing again. The other half is everything that is created while the cluster runs and is not in the repository. That half is what hurts after an outage.

What is in the repository

Definitions of deployments, services, network policies, operator configurations, Helm chart values, image versions. If the cluster is managed declaratively, a new cluster connected to the repository converges to the same state with no manual steps. The repository itself needs a backup: a mirror somewhere other than the Git server, which may well run in the same cluster or on the same host.

What is not in the repository

Data on persistent volumes. Databases, user files, indices, queues. A cluster rebuilt from the repository has empty volumes.

Secrets. Passwords, keys, certificates. They should not be in the repository in plain text, so they are either in an external secrets manager, or encrypted in the repository with a key held elsewhere, or typed in by hand and undocumented. The third case is more common than anyone admits.

Certificates and the certificate authority. An internal authority issuing certificates to cluster services. Losing it means replacing certificates everywhere they are trusted, including outside the cluster.

Cluster database state. Resources created dynamically by operators, annotations, controller state, objects created by hand “just for a moment”. In a cluster run strictly through GitOps there is little of this; in practice there is always something.

Images in the registry. If the registry runs in the cluster and has no backup, the rebuilt cluster has nothing to run until the images are rebuilt, which requires a working CI, which may also live in the cluster.

The strategy

Volumes: two tiers. Volume snapshots through the storage driver interface, scheduled and replicated to another location, for fast restores. Separately, application-consistent backups for databases: the database operator taking a dump or archiving the transaction log to object storage outside the cluster. A volume snapshot of a running database does not guarantee consistency; an operator-driven dump does.

Secrets: one source. An external secrets manager with its own backup, or encryption in the repository with the key stored in two independent places outside the cluster. A secrets key stored only inside the cluster is a backup that cannot be read after an outage.

The certificate authority. A copy of the authority’s key in a password vault or on media outside the cluster, with a restore procedure.

Cluster state. A snapshot of the cluster database from the control plane nodes, daily, to storage outside the cluster. Additionally, a resource export by a cluster backup tool, which captures what was created outside the repository and shows what should be moved into it.

The registry. Replication to a second registry outside the cluster, or a procedure for rebuilding images from a CI that runs outside the cluster. One of the two, chosen deliberately.

The restore test

A new, empty cluster in an isolated environment. Connect to the repository and sync. Restore secrets. Restore volumes or databases from backup. Verify that applications start and see their data. Measure the time from zero to a working application.

This test, run once a quarter, answers whether “everything is in Git” is true. Usually it is not the first time, and the list of gaps becomes the task list.

Summary

A GitOps-managed cluster is rebuilt from the repository, but applications are rebuilt from data, secrets and certificates that are not there. Snapshot and database dump backups outside the cluster, secrets with a key outside the cluster, a copy of the certificate authority, a cluster database snapshot, a plan for the registry, and a from-scratch restore test every quarter. The repository is the building’s blueprint; the backups are what lives inside it.