Highly available Kubernetes cluster
k3s etcd quorum, pod and data behaviour after a node failure, disruption budgets and a shared API address (kube-vip, keepalived, external load balancer) with diagnosis of address flapping.
Three k3s server nodes with etcd in quorum protect the cluster state, but they protect neither the applications nor access to the API. When one node fails, an application with a single replica stops responding for at least five minutes, a database with a volume on the local disk of the powered-off machine does not start anywhere else, and every client configured with the address of the failed node loses connectivity to the cluster even though the two remaining API servers are running. This document covers etcd quorum, pod and data behaviour after a node failure, disruption budgets and three ways of keeping a shared API address, together with diagnosis of its flapping.
etcd quorum
etcd accepts writes as long as a majority of its members (the quorum) responds. The number of nodes determines how many failures the cluster survives.
| etcd nodes | Quorum | Failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
Two nodes are worse than one, because the failure of either stops writes. A fourth node adds no resilience, and the k3s documentation requires an odd number of server nodes. Three etcd nodes are usually right, and five only when the cluster has to survive two simultaneous failures.
Three virtual machines with etcd on one virtualisation host form a single failure domain. Each server node therefore runs on a different physical host, and an anti-affinity rule keeps migration from changing that.
Pods after a node failure
Kubernetes does not move pods immediately.
| Time since the failure | Event |
|---|---|
| a few tens of seconds | the node goes NotReady, new pods are no longer scheduled on it |
| about 5 minutes | pods without their own tolerations are evicted and recreated on the remaining nodes (default tolerationSeconds: 300) |
| no limit | StatefulSet pods and pods with a local volume wait for the node to return |
An application with a single replica is therefore unavailable for at least five minutes. Two replicas are the minimum for any application whose unavailability someone would notice. The default spreading in kube-scheduler is soft and allows both replicas on one node, so a spread rule (topologySpreadConstraints) makes the placement explicit. For critical stateless applications the tolerations for an unreachable node can additionally be shortened from the default 300 seconds.
spec:
replicas: 2
template:
metadata:
labels:
app: portal
spec:
# shorter toleration than the default 300 s
tolerations:
- key: node.kubernetes.io/unreachable
operator: Exists
effect: NoExecute
tolerationSeconds: 60
- key: node.kubernetes.io/not-ready
operator: Exists
effect: NoExecute
tolerationSeconds: 60
# replicas on different nodes
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: portal
When there is no choice, ScheduleAnyway starts the replicas on the same node, which in a three-node cluster is safer than DoNotSchedule, where the replacement replica may stay Pending after a node failure.
A StatefulSet pod from an unavailable node stays in Terminating, and the controller does not create a replacement with the same identity. When the machine is certainly powered off, this can be unblocked with the taint node.kubernetes.io/out-of-service=nodeshutdown:NoExecute (stable since Kubernetes 1.28). It forces the deletion of the pods and the detaching of the volumes, so it is applied only to a machine that is powered off and removed once the machine returns.
Disruption budget
A PodDisruptionBudget protects against planned work (draining a node, upgrading the cluster), not against failures. It limits how many pods of an application can be unavailable at the same time because of evictions. A policy/v1 object in the application’s namespace selects the pods with a label selector (for example app: portal) and sets maxUnavailable: 1. For applications with several replicas this is safer than minAvailable.
Data and images after a failure
The default k3s storage class, local-path, creates volumes on the node’s disk, so a pod can start only there. Every application with a local volume needs a deliberate decision: application-level replication (a database with replicas on different nodes), replicated block storage, or accepting the outage and relying on a backup.
A database with an operator usually elects a new primary within several tens of seconds, but replicas that lost the connection to the old one do not always attach to the new one, depending on the operator. Replication lag is therefore monitored, not only the number of running instances.
After a failure the cluster starts dozens of pods on the remaining nodes, and they pull missing images from the registry, so public registry rate limits can slow recovery. The IfNotPresent policy with version tags (not latest) helps only partly, because pods landing on another node do not have the image there. A registry mirror is the real safeguard. k3s has a built-in one (embedded-registry, an experimental option according to the flag reference), which additionally requires mirroring to be enabled for selected registries. The nodes also need a DNS resolver independent of the cluster, because otherwise a DNS failure inside the cluster blocks image pulls.
The API address
In k3s the kubelets on servers connect to the local API, and agents connect to the servers through a built-in load balancer, so a node failure does not cut off the kubelets themselves. A shared API address is needed by clients outside the cluster (administrators, CI), by the network plugin (especially one replacing kube-proxy), by operators and by the registration of new nodes. It protects the cluster only if the mechanism keeping it survives a node failure and every client actually uses it.
There are three ways of keeping the address:
- kube-vip runs as a pod on the control plane nodes and elects a leader through a lease (a Lease object) in the Kubernetes API.
- Keepalived, as a system service, elects the node holding the address with VRRP, and a script checking the API decides whether the node is fit.
- An external load balancer, a pair of machines with HAProxy and keepalived or a network appliance, checks every API server and distributes connections; the address then belongs to the balancer, not to the nodes.
Dependence on the API and etcd has the greatest impact: slow etcd hits the mechanism that keeps access to the API. The kube-vip manifest is generated without the --services parameter, so that it handles only the API address.
Keepalived: checking API readiness
The check script queries /readyz instead of probing port 6443, because the port only shows that a process is listening. In k3s anonymous access to the API is disabled by default, so the script uses the local kubeconfig through k3s kubectl and needs read permission on /etc/rancher/k3s/k3s.yaml, which by default only root can read. Without it the address is never taken over.
vrrp_script chk_apiserver {
script "/usr/local/bin/k3s kubectl get --raw=/readyz --request-timeout=2s"
interval 3
timeout 3
fall 3
rise 2
}
vrrp_instance K8S_API {
state BACKUP
nopreempt
interface eth0
virtual_router_id 51
# 100, 90 and 80 on successive nodes
priority 100
advert_int 1
virtual_ipaddress {
10.10.0.50/24
}
track_script {
chk_apiserver
}
}
fall 3 with a 3-second interval means failover after about 10 seconds without readiness, and nopreempt spares clients a second failover when a node with a higher priority returns.
External load balancer
A single HAProxy is a new point of failure, so the balancer is a pair with keepalived; on the standby machine HAProxy has to listen on an address that does not exist yet (net.ipv4.ip_nonlocal_bind = 1). A frontend in tcp mode on port 6443 forwards traffic to the three API servers with the check option, and timeout client and timeout server must not cut long-lived watch connections. A port check alone does not exclude a server that is listening but not ready; option httpchk GET /readyz is more precise, provided the API allows anonymous requests (the default in kubeadm, not in k3s).
frontend k8s-api
bind 192.0.2.10:6443
mode tcp
timeout client 1h
default_backend k8s-api
backend k8s-api
mode tcp
timeout server 1h
option httpchk GET /readyz
http-check expect status 401
server cp1 10.10.0.21:6443 check check-ssl verify none inter 3s fall 3 rise 2
server cp2 10.10.0.22:6443 check check-ssl verify none inter 3s fall 3 rise 2
server cp3 10.10.0.23:6443 check check-ssl verify none inter 3s fall 3 rise 2
In k3s without anonymous access a ready server answers 401, so expect status 401 proves only that the API responds over TLS; in kubeadm expect status 200 checks readiness. A server that does not respond at all is removed after three failed probes, that is after about 10 seconds.
API certificate and clients bypassing the address
The virtual address and the DNS name under which clients know it must be in the SAN field of the API server certificate. In k3s they are added with the tls-san option in /etc/rancher/k3s/config.yaml on every server node (for example k8s-api.corp.example.com and 10.10.0.50), and a later change requires restarting k3s on each of them. A missing entry causes the error “x509: certificate is valid for …, not 10.10.0.50” only at the first connection. Clients know the DNS name, not the IP address: changing the address is then one record and one certificate entry, not every kubeconfig.
The k3s.yaml file points to 127.0.0.1, so after copying it to a workstation or to CI the address has to be changed to the virtual name. k3s agents register at the address from server, so when it points to a single server, joining a new node during that server’s failure fails. The same problem affects operators with an explicitly entered API address and kubeconfigs used by monitoring, GitOps and pipelines.
Address flapping: causes and diagnosis
Losing the address for a few seconds is enough for the agent of a network plugin replacing kube-proxy to lose the API, for new pods to get no networking and for CoreDNS to fail to report readiness. Flapping has several typical causes:
- Dependence on the API. A kube-vip leader that has not renewed its lease removes the address and restarts (kube-vip 1.3), so the restart count is a rough counter of flaps. The logs of the previous pod instance (
kubectl -n kube-system logs <kube-vip-pod> --previous) show “error retrieving resource lock” and “failed to renew lease”. If the leases of kube-vip, kube-controller-manager and kube-scheduler changed holders within a few seconds, the common cause is the API or etcd; if onlyplndr-cp-lockgrows, the cause is closer to kube-vip. The holders and the number of changes are shown bykubectl -n kube-system get leasewith the columnsspec.holderIdentityandspec.leaseTransitions. - etcd latency. A write is committed after a synchronous write to disk (fsync), which a disk shared with backups, an overloaded CPU, time taken by the hypervisor or defragmentation stretch from milliseconds to seconds. In the
k3sservice log (in kubeadm, in the etcd pod) series of “slow fdatasync” point to the disk, while “apply request took too long” and “leaderelection lost” point to write latency; a persistently highstorwainvmstatmeans waiting for the host or the disk. - Lease too short. The current kube-vip defaults are a 15 s lease, a 10 s renew deadline and a 2 s retry period; in versions up to 1.0 they were 5, 3 and 1 s.
- Pod resources. A kube-vip pod without resource requests loses the contest for CPU, and with a CPU limit it is throttled; the renewal is late even though the API responds.
- ARP. A router or firewall that ignores the gratuitous ARP of the new owner sends traffic to the MAC address of the old node until the entry expires.
Remedies
The variables vip_leaseduration, vip_renewdeadline and vip_retryperiod are set in the kube-vip container; client-go requires the lease to exceed the renew deadline, and the deadline to exceed 1.2 × the retry period. The 15 / 10 / 2 s profile means the address is lost after an API stall longer than about 10 s and taken over after a node failure in about 15–17 s (the 5 / 3 / 1 s profile: about 3 s and 5–6 s). A longer lease gives headroom but does not remove the cause.
The kube-vip pod should have resource requests (for example 100m CPU and 64 MiB of memory), the system-node-critical class and no CPU limit. The etcd data directory (k3s: /var/lib/rancher/k3s/server/db/etcd, kubeadm: /var/lib/etcd) should sit on an SSD or NVMe not loaded by backups. Alerts cover kube-vip restarts and the 99th percentile of etcd fsync above 10 ms; collecting etcd metrics from outside the node requires the k3s option etcd-expose-metrics: true (default false).
Decisions and alternatives
| Approach | When it makes sense | Limitation |
|---|---|---|
| kube-vip in the cluster (ARP mode) | three nodes in one network, no extra machines, monitored restarts and a lease with headroom | depends on the API and etcd; needs a shared layer 2; one leader, no traffic distribution |
| Keepalived on the nodes (VRRP) | unavailability has real costs and dependence on the API and etcd is unacceptable | needs a shared layer 2; one address owner, no traffic distribution; the check script needs access to the kubeconfig |
| External load balancer (HAProxy with keepalived or an appliance) | nodes in different networks or traffic distribution across API servers is needed | two extra machines or an appliance; the balancer itself is a point of failure unless paired |
In a cluster of three nodes in one network, kube-vip is a reasonable start, provided restarts are monitored and the lease has headroom. Keepalived removes the most dangerous dependency, and load balancers make sense for nodes in different networks or when traffic distribution is needed.
Implementation
- Three etcd nodes on different physical hosts, with an anti-affinity rule. Check: the failure of one host does not take down two server nodes.
tls-sanwith the virtual address and the DNS name inconfig.yamlof every server node, then a restart of k3s one node at a time. Check: a connection by name without an x509 error.- The chosen API address mechanism started (kube-vip, keepalived or a balancer). Check:
/readyzresponds through the virtual address from another machine (in k3s with a 401). - All components moved to the virtual name: administrator and CI kubeconfigs, the network plugin, operators, monitoring, GitOps and the agents’
serverentry. Check: no component points to the address of a single node. - At least two replicas per application spread across nodes, shorter tolerations for critical stateless applications and a PodDisruptionBudget. Check: no Deployment other than those deliberately accepted has a single replica, including the default k3s components in
kube-system(among them CoreDNS and Traefik). - A recorded decision for every application with a local volume: replication, replicated storage or a backup with an accepted outage.
- Resource requests and the
system-node-criticalclass for the kube-vip pod, and alerts on restarts and etcd fsync. The change restarts the pods, so the address will flap; it is made in a maintenance window. - The failure test described below.
Failure test
The most reliable verification is a hard power-off of a node for at least twenty minutes, that is longer than the five-minute pod toleration, in an agreed maintenance window. It is best to power off the node holding the API address and to measure the gap with a loop querying /readyz every second from another machine. After the test, not only the application responses are checked, but also pods in Pending, restarts of components in kube-system, database replication lag and the network plugin logs. The test is repeated for each node separately.
Limitations and pitfalls
- Budget without headroom.
minAvailable: 1for an application with a single replica means zero allowed disruptions:kubectl drainretries the eviction indefinitely with the message “Cannot evict pod as it would violate the pod’s disruption budget”, and the node upgrade stalls. For an application with a single replica the budget is omitted or the number of replicas is raised. - Component pointing at a node address. A single component configured with the address of a specific node makes high availability disappear when that node fails: the network plugin stops programming the network, DNS in the cluster stops responding, and the failure of one node becomes a failure of the whole cluster. In Cilium the address is set by
k8sServiceHostandk8sServicePort; both must point to the virtual address. - Restart during flapping. Restarting a node in the middle of a series of address flaps usually makes things worse: etcd loses its quorum margin, the returning member loads the disks of the others, and the address flaps again. First the failing layer is identified (leases, etcd, ARP).
- The out-of-service taint. It forces the deletion of pods and the detaching of volumes, so on a machine that is still running it risks two copies of the same instance running at once; it is applied after confirming the machine is powered off and removed when it returns.
- Apparent readiness in k3s. A
/readyzcheck through HAProxy without anonymous access returns 401 also for a server that is not ready; full readiness checking is provided by the keepalived script with the local kubeconfig.