The places where a small environment fails most often, and how to design them before they do: DNS and links, the cluster and certificates, backups, snapshots and directory recovery, and finally the first hour of an outage.
The ticket says “the network is down”, yet the network is fine and only names fail to resolve. Where DNS hides single points of failure, and how to check clients, DHCP, forwarders and in-cluster DNS.
The second link is often bought, plugged in and never tested. How to set up link health checks, what happens to VPN, telephony and exposed services during failover, and how to test it.
Three nodes and etcd protect the cluster state, not the applications. What happens to pods, volumes and the API address after a node failure, and how to design replicas, spreading and disruption budgets in k3s.
Three control plane nodes achieve nothing if clients connect to the API at the address of one of them. Three ways to keep a shared address, their configuration, a certificate with the right names, and how to check who really uses the address.
A certificate’s expiry date is known from the day it is issued, and yet expired certificates keep stopping services. How to build an inventory from three sources, read the dates with commands and set alert thresholds matched to the renewal method.
The backup job finishes with a “success” status every night, and the restore can still fail. How to write down RPO and RTO, run restore tests to a protocol and monitor the age of the last successful backup.
A virtual machine checkpoint sits on the same storage as the machine, grows with every write and depends on its parent disk. How the AVHDX chain works, when a checkpoint makes sense, and how to find the ones that stayed too long.
A snapshot of a domain controller's virtual machine is not a directory backup. System state backups, the deleted object lifetime, the AD Recycle Bin, four ways to restore, and the stages of forest recovery when no working controller is left.
In a small team the first hour of an outage decides how long it lasts. A lead, a first message within 15 minutes, establishing scope, checking recent changes, and rolling back before looking for the cause.