Backups and outage handling

RPO and RTO, the 3-2-1 rule with an immutable copy, a restore test protocol, Kubernetes cluster backups (etcd in k3s, Velero) and the first-hour outage procedure: lead, messages, rollback and escalation.

Backup and recoveryObservability

A green backup job status does not prove that the data can be restored: the job may cover too few volumes, and the key to the repository may have been saved on the very server that has to be rebuilt. In a Kubernetes cluster managed through GitOps there is the added assumption that “everything is in Git”, which holds only for definitions. This document covers setting RPO and RTO, backups beyond the reach of production, a restore test protocol, k3s cluster backups and the first-hour outage procedure, in which roles, communication and rollback decide the outcome.

RPO and RTO for each system

A “completed successfully” status only means that data was written. A backup exists once someone has restored it, measured the time and compared the result with the assumptions. Each system has two figures, agreed with its owner rather than set by the IT department alone:

  • RPO (Recovery Point Objective): how much data can be lost, measured in time since the last backup.
  • RTO (Recovery Time Objective): how long the system can be down, from the decision to restore to users confirming that it works.

The simplest form is a table in the documentation, reviewed once a year:

SystemRPORTOBackupRestore test
Domain controllers24 h4 hsystem state daily, two controllersonce a year, in isolation
Accounting system database15 min4 hfull daily, transaction log every 15 minquarterly, to a point in time
File server24 h8 h (single file: 1 h)daily, 30 days of versionsa folder monthly, the whole server once a year
Git and CI repository24 h8 hnightly mirror off the Git serverevery six months, on a clean machine
Kubernetes cluster24 h8 hetcd snapshot and volume backupsquarterly, to a separate cluster

RPO sets backup frequency: with an RPO of 15 minutes a daily backup does not meet the requirement, however well it is performed. RTO sets the restore method: restoring 2 TB from off-site storage over a 200 Mb/s link takes more than 20 hours, so an RTO of 8 hours requires a local copy.

A copy beyond the reach of production

The 3-2-1 rule means three copies of the data, counting the production one, on two different media, one off-site. The 3-2-1-1-0 extension adds an immutable or offline copy that cannot be modified or deleted from production, even by a domain administrator account, and zero errors in the restore test.

Nothing that shares the fate of the original is a backup:

  • a Hyper-V checkpoint: the same storage, a chain that depends on the parent disk,
  • an array or volume snapshot: protects against a logical error, not against failure of the array,
  • RAID and synchronous replication: also replicate deletion and encryption,
  • a folder synced with OneDrive or SharePoint: a local deletion deletes in the cloud, and the recycle bin has a time limit,
  • a CSI snapshot in a cluster on the same storage.

A backup is only what lives elsewhere and can be restored without the original.

Form of copyWhat it protects againstWhat to watch
Object storage with object locking (S3 Object Lock with versioning enabled, or an equivalent) in compliance modedeletion and overwriting during the lock period, even by the account administratora lock period longer than an attack can stay unnoticed, e.g. at least 30 days
A repository on a server outside the domain, with separate credentials and MFA for the consoletakeover of domain accountsno passwords shared with the domain, no RDP access from the office network
A network share in the same domainfailure of production hardwaredoes not protect against ransomware holding domain privileges; write access belongs only to the dedicated backup system account, not to Domain Admins or Domain Users

In compliance mode, S3 Object Lock does not allow a protected object version to be overwritten or deleted by any user, including the root account, whereas GOVERNANCE mode can be bypassed by users with a special permission. The Azure equivalent is immutable blob storage, which matches compliance mode only once the retention policy is locked.

Ransomware holding domain administrator privileges encrypts or deletes any backups it can reach. The same applies to keys: a backup encrypted with a key stored only on the backup server or in the domain shares a single point of failure with production. Without a working domain the following must remain available: documentation with the restore order (directory and DNS usually first), passwords and keys for the backup system, installation media and network device configuration.

Restore test protocol

A test without a protocol is an anecdote. The protocol defines what is restored, how often, by whom and what counts as success.

ScenarioFrequencyWhoSuccess criterion
A single file or folder from a given datemonthlythe on-call administrator, in rotationcontent matches the version from that date, time under 1 h
A virtual machine in an isolated networkquarterlya person who did not configure the backupthe system starts, the service answers a test request, time within the RTO
A database to a point in timequarterlyan administrator with the application ownerconsistency check without errors, the application sees the data from the given moment
A domain controller in isolationonce a yeartwo people, following a written procedurelogon and LDAP queries work, the DSRM password is known

Three rules decide the value of a test:

  • The test is performed by someone other than the author of the backup configuration, using only the documentation; every question to the author is a gap in the documentation.
  • The restored machine starts in a network with no connection to production (on Hyper-V a Private switch is enough), because a server backup has the same name and address as the original.
  • Time is measured from the decision to restore to confirmation that the service works, not from starting the job.

Every test ends with a completed protocol, stored with the system documentation. The protocol contains: the date, the person performing it (noting that they did not configure the backup), the system, the backup (date and repository), the target (RPO and RTO), the scope, start and end times with the total duration, the verification method (e.g. 20 random files opened, file count matching the original), the result, deviations (e.g. the repository password is known to only one person) and tasks with an owner and a deadline.

Age of the last successful backup as a metric

The job status in the console says little: a job may not start at all because someone disabled the schedule or changed the service account password, and the console shows no error because no job has failed. What counts is therefore the age of the last successful backup of each system, read from the backup system’s report or API (in Windows Server Backup, the Get-WBSummary cmdlet, field LastSuccessfulBackupTime). The value goes to monitoring as a metric with a threshold, not into an email report nobody reads. Alert thresholds follow backup frequency:

Backup frequencyWarningCritical alert
every 15 minutes (transaction log)1 h2 h
daily26 h50 h
weekly8 days15 days

Backups in Kubernetes

A Git repository restores definitions. After a failure of the array that ran all control plane nodes of a k3s cluster, Argo CD syncs the repository and the deployments turn green, but the applications start with empty databases: the data lived on the local volumes of the failed nodes, and a secret entered by hand with kubectl create secret never made it into the repository. Data, secrets and state created while the cluster runs need their own backups, stored outside the cluster.

Three layers, three sources of recovery

A cluster has three layers restored from different places: definitions from the repository, API state from an etcd snapshot, application data from volume backups and database dumps. Added to these are the keys without which none of these backups will work.

WhatRecovery sourceToolWhere to storeFrequency
Manifests, Helm values, Argo CD application definitionsGit repositoryre-syncGit server and a mirror outside the clusteron every change
API state: objects outside Git, Secrets, operator resourcesetcd snapshotk3s etcd-snapshotnode disk and S3 storage outside the clusterevery 6 hours
Namespaces with persistent volumesVelero backupvelero scheduleS3 storage outside the clusterdaily
Databaseslogical dumpCronJob with pg_dump or an equivalentS3 storage outside the clusterper RPO, e.g. hourly
k3s server token, secrets encryption key, CA keypassword vaultmanual copyvault outside the clusterafter installation and every change

The repository itself also needs a backup if the Git server runs in the same cluster or on the same host. A mirror (git clone --mirror) on another machine, refreshed periodically, is enough.

etcd snapshots in k3s

Embedded etcd in k3s (a cluster initialised with --cluster-init, or servers joined to an HA cluster) takes snapshots automatically: by default every 12 hours, retaining five snapshots, in /var/lib/rancher/k3s/server/db/snapshots on each server. By default nothing leaves the node, so a disk failure takes the cluster together with its snapshots. The schedule, retention and upload to S3 storage are set in the configuration file on every server, and a change requires restarting the k3s service:

etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 28
etcd-s3: true
etcd-s3-config-secret: k3s-etcd-s3

With snapshots every 6 hours, a local retention of 28 gives a week of history on each node. In S3 the limit is shared across the cluster and defaults to etcd-snapshot-retention, so with three servers 28 objects is about 9 cycles; the documentation advises setting etcd-s3-retention to the number of servers times the number of cycles (3 × 28 = 84).

Newer k3s releases make it possible not to keep the key in the file: etcd-s3-config-secret points to a Secret of type etcd.k3s.cattle.io/s3-config-secret in kube-system with the keys etcd-s3-endpoint, etcd-s3-bucket, etcd-s3-folder, etcd-s3-access-key and etcd-s3-secret-key, and RBAC restricts access to the Secret. The Secret works only if the file contains nothing but etcd-s3 and etcd-s3-config-secret (other S3 options, including etcd-s3-retention, cause it to be ignored), and it is not used when restoring from S3, where the settings are given in the command. In older releases the key sits in config.yaml, so the file must be readable by root only. The key itself has write permission to a single bucket only. A bucket with object locking protects snapshots from deletion by someone who takes over a node; the lock period has to be matched to retention, because locked snapshots are not removed before it expires, regardless of etcd-s3-retention. Before a risky change, such as a k3s version upgrade, a snapshot is taken manually with k3s etcd-snapshot save.

Restoring from a snapshot replaces the state of the whole cluster. It is done on one server, with k3s stopped on all servers, using k3s server --cluster-reset --cluster-reset-restore-path=<path to the snapshot>, and the remaining servers are rejoined after their local /var/lib/rancher/k3s/server/db directory is removed. It is a tool for corrupted API state, not for a single deleted application. A single k3s server without --cluster-init uses SQLite by default and the etcd-snapshot commands do not apply to it; the backup is the /var/lib/rancher/k3s/server/db directory together with the token, taken while k3s is stopped.

Velero: namespaces and volumes

Velero exports the API objects of selected namespaces to object storage and, with the node agent enabled, also copies volume contents at file level. It restores at namespace level, including under a different name (--namespace-mappings) or in another cluster, which an etcd snapshot cannot do. Local volumes (local-path) are bound to a node, so restoring them depends on that node being available or on a file-level backup.

velero schedule create app-prod-daily --schedule="0 2 * * *" --include-namespaces app-prod --default-volumes-to-fs-backup --ttl 720h

The schedule runs daily at 02:00 and keeps backups for 30 days. The --default-volumes-to-fs-backup option exists since Velero 1.10; in earlier versions it was called --default-volumes-to-restic.

A file-level copy of a running database does not guarantee consistency. For databases the right backup is a logical dump taken by a CronJob to storage outside the cluster, or a Velero hook (the pre.hook.backup.velero.io/command annotation on the pod) that dumps to a volume before the backup. CSI volume snapshots are fast, but without moving the data to object storage they stay in the same storage system as the original, so they protect against an application error, not against failure of the storage.

Restore test to a separate cluster

The test checks whether the assumption “everything is in Git” is true. It is run quarterly, on a separate cluster in an isolated network:

  1. A fresh k3s in the same version as production, on one or several test machines.
  2. Velero with the same storage plugin and the backup location in read-only mode, so that the test cluster cannot delete or overwrite production backups.
  3. Sync from the repository, restoring secrets from the vault or secrets manager.
  4. Restoring namespaces from the Velero backup and databases from dumps.
  5. Verifying that the applications start and see their data, and measuring the time from an empty cluster to a working application.

The first test usually fails, and the list of gaps becomes the task list: a secret entered by hand, a volume outside the schedule, a token nobody copied.

The first hour of an outage

Outage duration often depends less on how hard the problem is than on whether someone is leading, whether users know what is happening, and whether the first question was “what changed”. The procedure organises the first hour into stages: leading, scope, message, recent changes, rollback, notes and escalation.

Minute 0: confirming the outage and the lead

The first step is to confirm that the problem is real and wider than a single workstation: a second user, a second site, a monitoring alert. Then the outage is declared explicitly and a lead is named.

The lead does not have to be the most experienced person on the team. The role is coordination, communication and notes, not diagnosis. The first person to confirm the outage leads until explicitly handing the role to someone else. With two people on call, one leads and the other diagnoses: establishes scope, checks recent changes, rolls back and tests. User reports are collected by the helpdesk or the lead.

From this moment everything about the outage happens in one place: one thread in the team chat or one document. The lead copies findings from phone calls there straight away and freezes other deployments and changes in the affected systems, because every additional change makes establishing the cause harder.

Minute 5: establishing scope

Before repair begins, the team answers four questions: who is affected (everyone, one site, one group), which systems, since exactly when (the time from monitoring or a log, not from the first report) and what still works. What works narrows the search as effectively as what does not.

SymptomFirst suspicion
One application is down for everyonelatest deployment, expiring certificate, database, no disk space
Many systems at once at one sitelink, switch, DNS as a single point of failure, power
Logon fails for many servicesdomain controllers, DNS, system time, expired service account password
Everything works, but slowlydata storage, a running backup job, a saturated link

Minute 15: the first message

The first message goes out after 15 minutes at the latest, even without knowing the cause. Users need to know that the problem is known, who is handling it and when the next message will come, not the diagnosis.

[OUTAGE] <system>, since <time>

What is not working: <symptom from the user’s perspective, e.g. error after logging in to the ERP>
Who is affected: <everyone / site / department>; not affected: <if known>
Workaround: <e.g. invoices issued manually following the intranet instructions> or: none
Handled by: the IT team, led by <name>
Next update: at <time, e.g. 09:30>, even if nothing has changed

Please do not report this problem separately.

Subsequent messages go out at the announced rhythm, usually every 30 minutes. A message saying “no change, next update at 10:00” is better than silence, because silence generates phone calls that consume the diagnosing team’s time. The recipient list and a fallback channel are set before an outage: if the outage affects email or the messaging platform, the message goes out another way, for example by SMS to managers.

Minutes 15–30: what changed

According to the Google SRE book, roughly 70% of outages result from changes to a live system. A change may be a deployment, an update, a configuration change, or the expiry of a certificate or password. The diagnosing person’s first question is therefore “what changed in the last few hours”, not “why is this not working”.

On Windows servers, changes leave traces in the System log: event 19 is an installed update, 1074 a restart initiated by a process or user, 7040 a change of service start type, and 7045 a new service. In a Kubernetes cluster the first source is events sorted from oldest to newest (kubectl get events -A --sort-by=.lastTimestamp); they are kept for an hour by default. What was deployed and when is visible in the infrastructure repository history (git log --since="6 hours ago") and in the sync history in Argo CD (argocd app history <application>). If a revision from a few minutes earlier precedes the first report, the hypothesis is ready.

Rollback first, then the search for the cause

If there was a change in the window before the outage, rolling it back is the first action, not a last resort. Restoring service is the goal of the first hour; understanding the cause is the goal of the post-incident review.

ChangeRollbackNotes
A GitOps deploymentgit revert <commit> and syncthe repository history stays complete
A Windows updatewusa /uninstall /kb:<number>packages bundled with a servicing stack update are removed with DISM /Online /Remove-Package
A Group Policy changeRestore-GPO from a backup, or disabling the GPO linkworkstations see the change after a policy refresh
A change on a machine with a checkpointrestoring the checkpointwrites since the checkpoint was created are lost
A firewall rule, a DNS recordrestoring the previous valuerequires the previous value to have been recorded

In the GitOps model the rollback is a revert in the repository. argocd app rollback cannot be run for an application with automatic sync enabled and leaves the application out of line with the repository, so it suits a situation with no access to the repository. After it, a revert in the repository is mandatory: the next sync, manual or after automatic sync is re-enabled, will deploy the faulty version again. Exceptions to the “rollback first” rule are irreversible changes (a database schema migration, data already processed by the new version) and situations where a rollback is riskier than the outage; the lead then decides and records the decision in the notes.

Notes and escalation

The lead records every finding and every action with the time and the initials of the person. The notes serve messages, handing over the lead role and the post-incident review; with this kind of leadership the hypothesis pointing at the latest deployment appears after a dozen or so minutes instead of several dozen. Escalation thresholds are set before an outage, so that nobody debates during it whether “it is time yet”.

SituationWhoWhen
No hypothesis after 30 minutesa second person from the team, the system vendorwhen the threshold passes
Outage of a vendor or cloud servicea report to the vendor at the highest priority, in parallel with diagnosiswithin the first 15 minutes
Suspected security incident: file encryption, unknown accounts, unusual logonsmanagement and the incident response procedure; disconnect machines from the network, do not power them offimmediately
Possible personal data breachthe data protection officer (advises) and the data controller (decides on notifying the supervisory authority)immediately
Outage time exceeds the system’s RTOmanagement, decision on emergency modeonce the RTO is exceeded

Notification of the supervisory authority is made without undue delay and, where feasible, within 72 hours of becoming aware of the breach, and applies to breaches likely to result in a risk to the rights and freedoms of individuals (Article 33 GDPR). Entities covered by the NIS2 directive have their own, separate incident reporting deadlines.

A short post-incident review

Within a few working days the team meets for 30–45 minutes. The aim is not to find someone to blame, but to establish what made the outage last as long as it did and what will shorten the next one. The review record contains: the outage (system, date, from–to, total duration), impact (who was affected, what did not work, whether data was lost), a timeline from the notes (detection, declaration, message, hypothesis, rollback, confirmation), the cause (the direct one and the one that allowed it), what worked and what did not, and tasks with an owner and a deadline. The tasks go onto the team’s regular task list. A review without tasks is only a conversation.

Decisions and alternatives

ApproachWhen it makes senseLimitation
Object locking in compliance mode (S3 Object Lock, immutable blob storage with a locked retention policy)a copy resistant to deletion by the account administratorGOVERNANCE mode can be bypassed with a permission; the lock also holds snapshots that should expire
A backup repository outside the domain with separate credentialsno storage with object lockingprotection depends on credential isolation and network access
etcd snapshot (k3s etcd-snapshot)corrupted API state of a cluster with embedded etcdreplaces the state of the whole cluster; useless without the server token; SQLite on a single server needs a directory backup
Velero with file-level volume backuprestoring a namespace, including under another name or in another clustera file-level copy of a running database does not guarantee consistency
Logical database dump (CronJob or a Velero hook)databasesneeds storage outside the cluster and its own frequency per RPO
CSI snapshot without moving the dataa quick undo of an application errorstays in the same storage as the original, does not protect against its failure
git revert and syncrolling back a GitOps deploymentrequires access to the repository
argocd app rollbackno access to the repositoryunavailable with automatic sync; leaves the application out of line with the repository
Rollback of a change before diagnosisthere was a change in the window before the outageirreversible changes and rollbacks riskier than the outage need the lead’s decision

Implementation

  1. Write down RPO and RTO for each system with its owner. Check: backup frequency follows from RPO, and the restore method (local or remote copy) from RTO.
  2. Provide a copy beyond the reach of production (3-2-1-1-0): object locking or a repository outside the domain, keys and passwords in a vault outside the domain, documentation of the restore order. Check: a domain administrator account has no permission to delete the backups.
  3. In k3s: an etcd snapshot schedule with upload to S3 on every server and a service restart, and a copy of the server token, the secrets encryption key and the CA key in a vault outside the cluster. Check: sudo k3s etcd-snapshot ls shows a latest snapshot no older than the schedule interval and entries with an s3:// location; file:// entries alone mean all snapshots sit on the same node.
  4. Velero schedules for namespaces with volumes, logical database dumps per RPO and a Git repository mirror on another machine.
  5. A restore test protocol with frequency and performers, and an alert on the age of the last successful backup with the thresholds from the table. Check: disabling the schedule on a trial system raises an alert within the warning threshold.
  6. A first restore test of the cluster to a separate cluster in an isolated network; the list of gaps becomes the task list, and the test is repeated quarterly.
  7. The outage procedure: the message template, the recipient list with a fallback channel, escalation thresholds and the review template set before the first outage.

Limitations and pitfalls

  • Keys and privileges shared with production. A backup encrypted with a key stored only on the backup server or in the domain is lost together with the domain, and ransomware holding domain administrator privileges deletes any backups it reaches. Keys and passwords for the backup system are kept outside the domain, and the repository has separate credentials.
  • An etcd snapshot without the server token. The token encrypts confidential data in the cluster datastore, so a snapshot without the token in force when it was taken is useless. The token (/var/lib/rancher/k3s/server/token) goes to a vault outside the cluster, labelled with the cluster it belongs to.
  • A test cluster behaves like production. A cluster restored from a snapshot or a Velero backup runs CronJobs, connects to external systems, sends email and writes to the databases it can reach; a test without network isolation can send customers duplicate invoices. The test network has no access to production systems.
  • Restoring from backup when an attack is suspected. Rebuilding the environment destroys evidence and may bring it back together with the attacker’s access. A security incident stops being an ordinary outage the moment it is suspected, and a separate response procedure applies from then on.

Sources