The documentation of every Kubernetes distribution promises high availability once you run three control plane nodes. That is true in a very narrow sense: the etcd database will survive losing one node. Everything above etcd has to be designed separately, and in small clusters, where each node plays several roles at once, there are more failure points than the diagram suggests.
The API server address
Cluster clients, including the kubelet on every node and the CNI itself, need to reach the API server at some address. If that address belongs to the first node, a failure of that node stops the whole cluster even though etcd keeps running. The solution is a virtual IP floating between control plane nodes, or an external load balancer.
A common mistake: the virtual address exists, but not every component uses it. A network plugin configured during installation with the address of a specific node will work for months, until that specific node goes down. Then the plugin loses its connection to the API, stops programming the network, in-cluster DNS stops answering, and every application that resolves names starts throwing errors. A single node failure turns into a whole-cluster outage, and the root cause is far away from the symptoms.
The rule: every component that talks to the API uses the virtual address only. Verify it after installation rather than assuming it.
Stability of the virtual address itself
The component that maintains the virtual address, whatever it is, can itself be unstable. If it restarts frequently, the address disappears for a moment and reappears on another node. A single failover is harmless; a series of them in a short time can trigger a cascade of reconnections that some components do not handle well.
Restarts of that component deserve the same monitoring attention as etcd health. A restart count that grows week after week is a sign that something is misconfigured, even if the cluster “works”.
Data storage
A database in a container with an operator promises automatic failover. Usually it delivers: a new primary is elected within tens of seconds. The question is what happens to the replicas. A replica that lost its connection to the old primary does not always reattach to the new one automatically. It can keep running, answer queries, and replicate nothing until the failed node comes back.
Conclusions:
- monitor replication lag, not just the number of running instances,
- test the scenario where the failed node does not return for several hours,
- know how to force a replica to reattach before you need to.
The image registry
After a node failure Kubernetes reschedules pods onto the remaining nodes. If the image pull policy forces a registry check on every start, the cluster makes dozens of registry requests within a few minutes. Public registries have rate limits, and a cluster in the middle of an outage starts receiving refusals.
Two remedies: a pull policy that uses a locally present image when available, and a private registry or a caching proxy on the local network.
Dependence on DNS outside the cluster
Nodes need to resolve names: the registry, repositories, update sources. If the only DNS server is a service running inside the same cluster or on one of its nodes, every cluster network failure takes away the nodes’ ability to recover. Nodes should have at least one DNS server independent of the cluster.
The test that tells the truth
The only credible verification is switching a node off. Not stopping a service, not unplugging a cable for ten seconds, but a hard stop of the machine for longer than every timeout in the system, which in practice means at least twenty minutes. After the test, check not only whether applications respond, but also replication state, restart counts of system components, and the network plugin logs.
A cluster that has passed this test for each node individually can be called highly available. A cluster nobody has dared to test this way is highly available only in the documentation.