Backups and outage handling
RPO and RTO, the 3-2-1 rule with an immutable copy, a restore test protocol, Kubernetes cluster backups (etcd in k3s, Velero) and the first-hour outage procedure: lead, messages, rollback and escalation.
A green backup job status does not prove that the data can be restored: the job may cover too few volumes, and the key to the repository may have been saved on the very server that has to be rebuilt. In a Kubernetes cluster managed through GitOps there is the added assumption that “everything is in Git”, which holds only for definitions. This document covers setting RPO and RTO, backups beyond the reach of production, a restore test protocol, k3s cluster backups and the first-hour outage procedure, in which roles, communication and rollback decide the outcome.
RPO and RTO for each system
A “completed successfully” status only means that data was written. A backup exists once someone has restored it, measured the time and compared the result with the assumptions. Each system has two figures, agreed with its owner rather than set by the IT department alone:
- RPO (Recovery Point Objective): how much data can be lost, measured in time since the last backup.
- RTO (Recovery Time Objective): how long the system can be down, from the decision to restore to users confirming that it works.
The simplest form is a table in the documentation, reviewed once a year:
| System | RPO | RTO | Backup | Restore test |
|---|---|---|---|---|
| Domain controllers | 24 h | 4 h | system state daily, two controllers | once a year, in isolation |
| Accounting system database | 15 min | 4 h | full daily, transaction log every 15 min | quarterly, to a point in time |
| File server | 24 h | 8 h (single file: 1 h) | daily, 30 days of versions | a folder monthly, the whole server once a year |
| Git and CI repository | 24 h | 8 h | nightly mirror off the Git server | every six months, on a clean machine |
| Kubernetes cluster | 24 h | 8 h | etcd snapshot and volume backups | quarterly, to a separate cluster |
RPO sets backup frequency: with an RPO of 15 minutes a daily backup does not meet the requirement, however well it is performed. RTO sets the restore method: restoring 2 TB from off-site storage over a 200 Mb/s link takes more than 20 hours, so an RTO of 8 hours requires a local copy.
A copy beyond the reach of production
The 3-2-1 rule means three copies of the data, counting the production one, on two different media, one off-site. The 3-2-1-1-0 extension adds an immutable or offline copy that cannot be modified or deleted from production, even by a domain administrator account, and zero errors in the restore test.
Nothing that shares the fate of the original is a backup:
- a Hyper-V checkpoint: the same storage, a chain that depends on the parent disk,
- an array or volume snapshot: protects against a logical error, not against failure of the array,
- RAID and synchronous replication: also replicate deletion and encryption,
- a folder synced with OneDrive or SharePoint: a local deletion deletes in the cloud, and the recycle bin has a time limit,
- a CSI snapshot in a cluster on the same storage.
A backup is only what lives elsewhere and can be restored without the original.
| Form of copy | What it protects against | What to watch |
|---|---|---|
| Object storage with object locking (S3 Object Lock with versioning enabled, or an equivalent) in compliance mode | deletion and overwriting during the lock period, even by the account administrator | a lock period longer than an attack can stay unnoticed, e.g. at least 30 days |
| A repository on a server outside the domain, with separate credentials and MFA for the console | takeover of domain accounts | no passwords shared with the domain, no RDP access from the office network |
| A network share in the same domain | failure of production hardware | does not protect against ransomware holding domain privileges; write access belongs only to the dedicated backup system account, not to Domain Admins or Domain Users |
In compliance mode, S3 Object Lock does not allow a protected object version to be overwritten or deleted by any user, including the root account, whereas GOVERNANCE mode can be bypassed by users with a special permission. The Azure equivalent is immutable blob storage, which matches compliance mode only once the retention policy is locked.
Ransomware holding domain administrator privileges encrypts or deletes any backups it can reach. The same applies to keys: a backup encrypted with a key stored only on the backup server or in the domain shares a single point of failure with production. Without a working domain the following must remain available: documentation with the restore order (directory and DNS usually first), passwords and keys for the backup system, installation media and network device configuration.
Restore test protocol
A test without a protocol is an anecdote. The protocol defines what is restored, how often, by whom and what counts as success.
| Scenario | Frequency | Who | Success criterion |
|---|---|---|---|
| A single file or folder from a given date | monthly | the on-call administrator, in rotation | content matches the version from that date, time under 1 h |
| A virtual machine in an isolated network | quarterly | a person who did not configure the backup | the system starts, the service answers a test request, time within the RTO |
| A database to a point in time | quarterly | an administrator with the application owner | consistency check without errors, the application sees the data from the given moment |
| A domain controller in isolation | once a year | two people, following a written procedure | logon and LDAP queries work, the DSRM password is known |
Three rules decide the value of a test:
- The test is performed by someone other than the author of the backup configuration, using only the documentation; every question to the author is a gap in the documentation.
- The restored machine starts in a network with no connection to production (on Hyper-V a Private switch is enough), because a server backup has the same name and address as the original.
- Time is measured from the decision to restore to confirmation that the service works, not from starting the job.
Every test ends with a completed protocol, stored with the system documentation. The protocol contains: the date, the person performing it (noting that they did not configure the backup), the system, the backup (date and repository), the target (RPO and RTO), the scope, start and end times with the total duration, the verification method (e.g. 20 random files opened, file count matching the original), the result, deviations (e.g. the repository password is known to only one person) and tasks with an owner and a deadline.
Age of the last successful backup as a metric
The job status in the console says little: a job may not start at all because someone disabled the schedule or changed the service account password, and the console shows no error because no job has failed. What counts is therefore the age of the last successful backup of each system, read from the backup system’s report or API (in Windows Server Backup, the Get-WBSummary cmdlet, field LastSuccessfulBackupTime). The value goes to monitoring as a metric with a threshold, not into an email report nobody reads. Alert thresholds follow backup frequency:
| Backup frequency | Warning | Critical alert |
|---|---|---|
| every 15 minutes (transaction log) | 1 h | 2 h |
| daily | 26 h | 50 h |
| weekly | 8 days | 15 days |
Backups in Kubernetes
A Git repository restores definitions. After a failure of the array that ran all control plane nodes of a k3s cluster, Argo CD syncs the repository and the deployments turn green, but the applications start with empty databases: the data lived on the local volumes of the failed nodes, and a secret entered by hand with kubectl create secret never made it into the repository. Data, secrets and state created while the cluster runs need their own backups, stored outside the cluster.
Three layers, three sources of recovery
A cluster has three layers restored from different places: definitions from the repository, API state from an etcd snapshot, application data from volume backups and database dumps. Added to these are the keys without which none of these backups will work.
| What | Recovery source | Tool | Where to store | Frequency |
|---|---|---|---|---|
| Manifests, Helm values, Argo CD application definitions | Git repository | re-sync | Git server and a mirror outside the cluster | on every change |
| API state: objects outside Git, Secrets, operator resources | etcd snapshot | k3s etcd-snapshot | node disk and S3 storage outside the cluster | every 6 hours |
| Namespaces with persistent volumes | Velero backup | velero schedule | S3 storage outside the cluster | daily |
| Databases | logical dump | CronJob with pg_dump or an equivalent | S3 storage outside the cluster | per RPO, e.g. hourly |
| k3s server token, secrets encryption key, CA key | password vault | manual copy | vault outside the cluster | after installation and every change |
The repository itself also needs a backup if the Git server runs in the same cluster or on the same host. A mirror (git clone --mirror) on another machine, refreshed periodically, is enough.
etcd snapshots in k3s
Embedded etcd in k3s (a cluster initialised with --cluster-init, or servers joined to an HA cluster) takes snapshots automatically: by default every 12 hours, retaining five snapshots, in /var/lib/rancher/k3s/server/db/snapshots on each server. By default nothing leaves the node, so a disk failure takes the cluster together with its snapshots. The schedule, retention and upload to S3 storage are set in the configuration file on every server, and a change requires restarting the k3s service:
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 28
etcd-s3: true
etcd-s3-config-secret: k3s-etcd-s3
With snapshots every 6 hours, a local retention of 28 gives a week of history on each node. In S3 the limit is shared across the cluster and defaults to etcd-snapshot-retention, so with three servers 28 objects is about 9 cycles; the documentation advises setting etcd-s3-retention to the number of servers times the number of cycles (3 × 28 = 84).
Newer k3s releases make it possible not to keep the key in the file: etcd-s3-config-secret points to a Secret of type etcd.k3s.cattle.io/s3-config-secret in kube-system with the keys etcd-s3-endpoint, etcd-s3-bucket, etcd-s3-folder, etcd-s3-access-key and etcd-s3-secret-key, and RBAC restricts access to the Secret. The Secret works only if the file contains nothing but etcd-s3 and etcd-s3-config-secret (other S3 options, including etcd-s3-retention, cause it to be ignored), and it is not used when restoring from S3, where the settings are given in the command. In older releases the key sits in config.yaml, so the file must be readable by root only. The key itself has write permission to a single bucket only. A bucket with object locking protects snapshots from deletion by someone who takes over a node; the lock period has to be matched to retention, because locked snapshots are not removed before it expires, regardless of etcd-s3-retention. Before a risky change, such as a k3s version upgrade, a snapshot is taken manually with k3s etcd-snapshot save.
Restoring from a snapshot replaces the state of the whole cluster. It is done on one server, with k3s stopped on all servers, using k3s server --cluster-reset --cluster-reset-restore-path=<path to the snapshot>, and the remaining servers are rejoined after their local /var/lib/rancher/k3s/server/db directory is removed. It is a tool for corrupted API state, not for a single deleted application. A single k3s server without --cluster-init uses SQLite by default and the etcd-snapshot commands do not apply to it; the backup is the /var/lib/rancher/k3s/server/db directory together with the token, taken while k3s is stopped.
Velero: namespaces and volumes
Velero exports the API objects of selected namespaces to object storage and, with the node agent enabled, also copies volume contents at file level. It restores at namespace level, including under a different name (--namespace-mappings) or in another cluster, which an etcd snapshot cannot do. Local volumes (local-path) are bound to a node, so restoring them depends on that node being available or on a file-level backup.
velero schedule create app-prod-daily --schedule="0 2 * * *" --include-namespaces app-prod --default-volumes-to-fs-backup --ttl 720h
The schedule runs daily at 02:00 and keeps backups for 30 days. The --default-volumes-to-fs-backup option exists since Velero 1.10; in earlier versions it was called --default-volumes-to-restic.
A file-level copy of a running database does not guarantee consistency. For databases the right backup is a logical dump taken by a CronJob to storage outside the cluster, or a Velero hook (the pre.hook.backup.velero.io/command annotation on the pod) that dumps to a volume before the backup. CSI volume snapshots are fast, but without moving the data to object storage they stay in the same storage system as the original, so they protect against an application error, not against failure of the storage.
Restore test to a separate cluster
The test checks whether the assumption “everything is in Git” is true. It is run quarterly, on a separate cluster in an isolated network:
- A fresh k3s in the same version as production, on one or several test machines.
- Velero with the same storage plugin and the backup location in read-only mode, so that the test cluster cannot delete or overwrite production backups.
- Sync from the repository, restoring secrets from the vault or secrets manager.
- Restoring namespaces from the Velero backup and databases from dumps.
- Verifying that the applications start and see their data, and measuring the time from an empty cluster to a working application.
The first test usually fails, and the list of gaps becomes the task list: a secret entered by hand, a volume outside the schedule, a token nobody copied.
The first hour of an outage
Outage duration often depends less on how hard the problem is than on whether someone is leading, whether users know what is happening, and whether the first question was “what changed”. The procedure organises the first hour into stages: leading, scope, message, recent changes, rollback, notes and escalation.
Minute 0: confirming the outage and the lead
The first step is to confirm that the problem is real and wider than a single workstation: a second user, a second site, a monitoring alert. Then the outage is declared explicitly and a lead is named.
The lead does not have to be the most experienced person on the team. The role is coordination, communication and notes, not diagnosis. The first person to confirm the outage leads until explicitly handing the role to someone else. With two people on call, one leads and the other diagnoses: establishes scope, checks recent changes, rolls back and tests. User reports are collected by the helpdesk or the lead.
From this moment everything about the outage happens in one place: one thread in the team chat or one document. The lead copies findings from phone calls there straight away and freezes other deployments and changes in the affected systems, because every additional change makes establishing the cause harder.
Minute 5: establishing scope
Before repair begins, the team answers four questions: who is affected (everyone, one site, one group), which systems, since exactly when (the time from monitoring or a log, not from the first report) and what still works. What works narrows the search as effectively as what does not.
| Symptom | First suspicion |
|---|---|
| One application is down for everyone | latest deployment, expiring certificate, database, no disk space |
| Many systems at once at one site | link, switch, DNS as a single point of failure, power |
| Logon fails for many services | domain controllers, DNS, system time, expired service account password |
| Everything works, but slowly | data storage, a running backup job, a saturated link |
Minute 15: the first message
The first message goes out after 15 minutes at the latest, even without knowing the cause. Users need to know that the problem is known, who is handling it and when the next message will come, not the diagnosis.
[OUTAGE] <system>, since <time>
What is not working: <symptom from the user’s perspective, e.g. error after logging in to the ERP>
Who is affected: <everyone / site / department>; not affected: <if known>
Workaround: <e.g. invoices issued manually following the intranet instructions> or: none
Handled by: the IT team, led by <name>
Next update: at <time, e.g. 09:30>, even if nothing has changed
Please do not report this problem separately.
Subsequent messages go out at the announced rhythm, usually every 30 minutes. A message saying “no change, next update at 10:00” is better than silence, because silence generates phone calls that consume the diagnosing team’s time. The recipient list and a fallback channel are set before an outage: if the outage affects email or the messaging platform, the message goes out another way, for example by SMS to managers.
Minutes 15–30: what changed
According to the Google SRE book, roughly 70% of outages result from changes to a live system. A change may be a deployment, an update, a configuration change, or the expiry of a certificate or password. The diagnosing person’s first question is therefore “what changed in the last few hours”, not “why is this not working”.
On Windows servers, changes leave traces in the System log: event 19 is an installed update, 1074 a restart initiated by a process or user, 7040 a change of service start type, and 7045 a new service. In a Kubernetes cluster the first source is events sorted from oldest to newest (kubectl get events -A --sort-by=.lastTimestamp); they are kept for an hour by default. What was deployed and when is visible in the infrastructure repository history (git log --since="6 hours ago") and in the sync history in Argo CD (argocd app history <application>). If a revision from a few minutes earlier precedes the first report, the hypothesis is ready.
Rollback first, then the search for the cause
If there was a change in the window before the outage, rolling it back is the first action, not a last resort. Restoring service is the goal of the first hour; understanding the cause is the goal of the post-incident review.
| Change | Rollback | Notes |
|---|---|---|
| A GitOps deployment | git revert <commit> and sync | the repository history stays complete |
| A Windows update | wusa /uninstall /kb:<number> | packages bundled with a servicing stack update are removed with DISM /Online /Remove-Package |
| A Group Policy change | Restore-GPO from a backup, or disabling the GPO link | workstations see the change after a policy refresh |
| A change on a machine with a checkpoint | restoring the checkpoint | writes since the checkpoint was created are lost |
| A firewall rule, a DNS record | restoring the previous value | requires the previous value to have been recorded |
In the GitOps model the rollback is a revert in the repository. argocd app rollback cannot be run for an application with automatic sync enabled and leaves the application out of line with the repository, so it suits a situation with no access to the repository. After it, a revert in the repository is mandatory: the next sync, manual or after automatic sync is re-enabled, will deploy the faulty version again. Exceptions to the “rollback first” rule are irreversible changes (a database schema migration, data already processed by the new version) and situations where a rollback is riskier than the outage; the lead then decides and records the decision in the notes.
Notes and escalation
The lead records every finding and every action with the time and the initials of the person. The notes serve messages, handing over the lead role and the post-incident review; with this kind of leadership the hypothesis pointing at the latest deployment appears after a dozen or so minutes instead of several dozen. Escalation thresholds are set before an outage, so that nobody debates during it whether “it is time yet”.
| Situation | Who | When |
|---|---|---|
| No hypothesis after 30 minutes | a second person from the team, the system vendor | when the threshold passes |
| Outage of a vendor or cloud service | a report to the vendor at the highest priority, in parallel with diagnosis | within the first 15 minutes |
| Suspected security incident: file encryption, unknown accounts, unusual logons | management and the incident response procedure; disconnect machines from the network, do not power them off | immediately |
| Possible personal data breach | the data protection officer (advises) and the data controller (decides on notifying the supervisory authority) | immediately |
| Outage time exceeds the system’s RTO | management, decision on emergency mode | once the RTO is exceeded |
Notification of the supervisory authority is made without undue delay and, where feasible, within 72 hours of becoming aware of the breach, and applies to breaches likely to result in a risk to the rights and freedoms of individuals (Article 33 GDPR). Entities covered by the NIS2 directive have their own, separate incident reporting deadlines.
A short post-incident review
Within a few working days the team meets for 30–45 minutes. The aim is not to find someone to blame, but to establish what made the outage last as long as it did and what will shorten the next one. The review record contains: the outage (system, date, from–to, total duration), impact (who was affected, what did not work, whether data was lost), a timeline from the notes (detection, declaration, message, hypothesis, rollback, confirmation), the cause (the direct one and the one that allowed it), what worked and what did not, and tasks with an owner and a deadline. The tasks go onto the team’s regular task list. A review without tasks is only a conversation.
Decisions and alternatives
| Approach | When it makes sense | Limitation |
|---|---|---|
| Object locking in compliance mode (S3 Object Lock, immutable blob storage with a locked retention policy) | a copy resistant to deletion by the account administrator | GOVERNANCE mode can be bypassed with a permission; the lock also holds snapshots that should expire |
| A backup repository outside the domain with separate credentials | no storage with object locking | protection depends on credential isolation and network access |
etcd snapshot (k3s etcd-snapshot) | corrupted API state of a cluster with embedded etcd | replaces the state of the whole cluster; useless without the server token; SQLite on a single server needs a directory backup |
| Velero with file-level volume backup | restoring a namespace, including under another name or in another cluster | a file-level copy of a running database does not guarantee consistency |
| Logical database dump (CronJob or a Velero hook) | databases | needs storage outside the cluster and its own frequency per RPO |
| CSI snapshot without moving the data | a quick undo of an application error | stays in the same storage as the original, does not protect against its failure |
git revert and sync | rolling back a GitOps deployment | requires access to the repository |
argocd app rollback | no access to the repository | unavailable with automatic sync; leaves the application out of line with the repository |
| Rollback of a change before diagnosis | there was a change in the window before the outage | irreversible changes and rollbacks riskier than the outage need the lead’s decision |
Implementation
- Write down RPO and RTO for each system with its owner. Check: backup frequency follows from RPO, and the restore method (local or remote copy) from RTO.
- Provide a copy beyond the reach of production (3-2-1-1-0): object locking or a repository outside the domain, keys and passwords in a vault outside the domain, documentation of the restore order. Check: a domain administrator account has no permission to delete the backups.
- In k3s: an etcd snapshot schedule with upload to S3 on every server and a service restart, and a copy of the server token, the secrets encryption key and the CA key in a vault outside the cluster. Check:
sudo k3s etcd-snapshot lsshows a latest snapshot no older than the schedule interval and entries with ans3://location;file://entries alone mean all snapshots sit on the same node. - Velero schedules for namespaces with volumes, logical database dumps per RPO and a Git repository mirror on another machine.
- A restore test protocol with frequency and performers, and an alert on the age of the last successful backup with the thresholds from the table. Check: disabling the schedule on a trial system raises an alert within the warning threshold.
- A first restore test of the cluster to a separate cluster in an isolated network; the list of gaps becomes the task list, and the test is repeated quarterly.
- The outage procedure: the message template, the recipient list with a fallback channel, escalation thresholds and the review template set before the first outage.
Limitations and pitfalls
- Keys and privileges shared with production. A backup encrypted with a key stored only on the backup server or in the domain is lost together with the domain, and ransomware holding domain administrator privileges deletes any backups it reaches. Keys and passwords for the backup system are kept outside the domain, and the repository has separate credentials.
- An etcd snapshot without the server token. The token encrypts confidential data in the cluster datastore, so a snapshot without the token in force when it was taken is useless. The token (
/var/lib/rancher/k3s/server/token) goes to a vault outside the cluster, labelled with the cluster it belongs to. - A test cluster behaves like production. A cluster restored from a snapshot or a Velero backup runs CronJobs, connects to external systems, sends email and writes to the databases it can reach; a test without network isolation can send customers duplicate invoices. The test network has no access to production systems.
- Restoring from backup when an attack is suspected. Rebuilding the environment destroys evidence and may bring it back together with the attacker’s access. A security incident stops being an ordinary outage the moment it is suspected, and a separate response procedure applies from then on.