A cluster virtual IP that keeps moving: effects, causes and diagnosis
The API address jumps between nodes, and the network plugin, controllers and DNS restart in its wake. How to tell a kube-vip problem from slow etcd, the network or node resources, and what to change so the address stops moving.
Every few days the deployment pipeline fails with “Unable to connect to the server: dial tcp 192.0.2.10:6443: i/o timeout”, and a rerun succeeds. The kube-vip restart counter grows by a few each week. One night, during the virtual machine backup, the API address moves three times in ten minutes: the network plugin restarts on every node, CoreDNS stops reporting ready, and applications return name resolution errors.
A virtual address that keeps moving is rarely the problem in itself. It is usually the first visible symptom of slow etcd, a lease that is too short, missing resources or ARP trouble. Choosing the address mechanism and configuring clients is covered in A virtual IP for the cluster API; this post shows how to establish which layer is failing.
Symptoms
kubectland CI jobs occasionally return “The connection to the server 192.0.2.10:6443 was refused” or a timeout.- kube-vip restart counters grow on several nodes, and the address changes node although no node was restarted.
- In the same minutes, network plugin agents, controllers or operators restart.
After losing its lease, kube-vip removes the address and exits, so its restart count is a rough counter of moves.
The cascade: why a few seconds is too long
Pods reach the API through the kubernetes service (ClusterIP), which leads to the real addresses of the API servers. The virtual address is used by clients outside the cluster and by components that have it configured explicitly: the network plugin agent (especially one that replaces kube-proxy) and, in kubeadm, also the kubelet and kube-proxy.
- The kube-vip leader fails to renew its lease in time, removes the address and restarts. Another node takes over the address after roughly the lease duration.
- During that time, and longer with ARP problems, the API does not answer at the virtual address.
- The network plugin agent loses the API. Depending on the plugin, it fails its liveness probe, is restarted and waits for the API on start-up. New pods get no network.
- CoreDNS pods starting during this time do not report ready without synchronising with the API. Applications get name and connection errors.
A second branch runs in parallel: if the cause is slow etcd, kube-controller-manager, kube-scheduler and operators with leader election also fail to renew their leases, even though they do not use the virtual address.
Causes
Dependency on the API itself. The lease (a Lease object) is written by the API to etcd, so any slowdown of the API or etcd hits the mechanism meant to keep the API reachable. In --inCluster mode kube-vip connects to the API through the kubernetes service, which means also through kube-proxy or network plugin rules.
A lease that is too short. By default kube-vip uses a 5 s lease duration, a 3 s renew deadline and 1 s between retries, so the leader loses the address after about 3 seconds without a successful renewal. kube-controller-manager and kube-scheduler default to 15, 10 and 2 seconds.
etcd latency. etcd commits a write after a synchronous write to disk (fsync). A disk shared with backups, an overloaded CPU or CPU time taken from the virtual machine by the hypervisor stretch this from milliseconds to seconds. etcd defragmentation blocks the member for its duration.
Missing resources and priority for the pod. A kube-vip pod without resource requests loses the competition for CPU, and with a CPU limit it is throttled. The renewal is late even though the API responds.
ARP. The new owner announces the address with a gratuitous ARP packet. A router or firewall that ignores it sends traffic to the old node’s MAC address until the entry expires. An address left behind on the old node results in two machines with the same address.
The clock. Leader election measures elapsed time locally, so a constant offset between clocks does it no harm. What harms it is a virtual machine pause (saved state, migration) longer than the lease.
Step-by-step diagnosis
Is kube-vip losing its lease
kubectl -n kube-system get pods -o wide | grep kube-vip
kubectl -n kube-system logs <kube-vip-pod> --previous --timestamps | grep -iE 'lease|lock|leader' | tail -n 20
kubectl -n kube-system get pod <kube-vip-pod> \
-o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{" "}{.status.containerStatuses[0].lastState.terminated.finishedAt}{"\n"}'The messages “error retrieving resource lock” and “failed to renew lease” (from the client-go library) mean the API did not handle the renewal in time. OOMKilled points to the memory limit.
Is the problem limited to kube-vip
The leases of components with leader election live in kube-system, and the leaseTransitions field counts changes of owner.
kubectl -n kube-system get lease \
-o custom-columns=NAME:.metadata.name,HOLDER:.spec.holderIdentity,TRANSITIONS:.spec.leaseTransitions,ACQUIRED:.spec.acquireTimeNAME HOLDER TRANSITIONS ACQUIRED
kube-controller-manager k8s-cp-02_4f6c0e1a-2b7d-4c1e-9a55-0d3e8f61b2c4 37 2026-10-14T02:07:41.512033Z
kube-scheduler k8s-cp-01_9b2d77c3-5e10-4a8f-b6d2-71c9a0e4f3d8 35 2026-10-14T02:07:43.208761Z
plndr-cp-lock k8s-cp-03 52 2026-10-14T02:07:39.917420Zplndr-cp-lock is the default name of the API address lease in kube-vip. In the example all three leases changed owner within four seconds: the common cause is the API or etcd, and kube-vip is one of the victims. If only the plndr-cp-lock counter grows, the cause lies closer to kube-vip: in the lease, the pod’s resources or the network.
etcd, the API and the node
In k3s the API, controllers and embedded etcd write to the journal of the k3s service; in kubeadm etcd runs as a static pod.
# k3s, on every server
journalctl -u k3s --since "2 hours ago" --no-pager \
| grep -E 'apply request took too long|slow fdatasync|failed to send out heartbeat|leaderelection lost|failed to renew lease'
# kubeadm
kubectl -n kube-system logs etcd-k8s-cp-01 --since=2h | grep -E 'apply request took too long|slow fdatasync'
journalctl -u kubelet --since "2 hours ago" --no-pager | grep -i 'lease'An occasional “apply request took too long” happens with large writes. Series of “slow fdatasync” in the minutes of the moves point to the disk.
kubectl get --raw='/readyz/etcd'; echo
time kubectl get --raw='/readyz' >/dev/null
vmstat 1 10 # columns wa (waiting for disk) and st (time taken by the hypervisor)
timedatectl | grep -E 'synchronized|NTP service'st consistently above a few per cent (if the platform reports it) or wa above the low teens during the moves means the node is waiting for the host or the disk.
The network and ARP
After a move, a client should see the new node’s MAC address within seconds (ip -br link show dev eth0 on that node). An old MAC address visible for minutes means the gratuitous ARP is being ignored.
ip neigh show 192.0.2.10Get-NetNeighbor -IPAddress 192.0.2.10Symptom, cause, check
| Symptom | Likely cause | What to check |
|---|---|---|
| kube-vip restarts, “failed to renew lease” in the logs | slow API or etcd, short lease | leases of other components, etcd warnings |
| All leases change owner in the same seconds | etcd or the API, most often the disk | “slow fdatasync”, backup schedule |
Only plndr-cp-lock changes | pod resources or priority, short lease | requests, priorityClassName, vip_* variables |
| After a move the address stays unreachable for long | gratuitous ARP ignored | ip neigh, router ARP table |
arping shows two MAC addresses | the address on two machines | ip -br addr on the nodes |
| Moves after a VM saved state or migration | leader paused for longer than the lease | hypervisor event log |
High st or wa during the moves | overloaded host or disk | vmstat, host load |
Remedies
A lease with headroom
The vip_leaseduration, vip_renewdeadline and vip_retryperiod variables are set in the kube-vip container. client-go requires the lease duration to exceed the renew deadline, and the renew deadline to exceed 1.2 × the retry period.
| Profile (duration / renew / retry) | Leader loses the address when the API stalls for longer than | Address takeover after a node failure (approximate) |
|---|---|---|
| 5 / 3 / 1 s (default) | about 3 s | about 5–6 s |
| 15 / 10 / 2 s (as kube-controller-manager) | about 10 s | about 15–17 s |
kubectl -n kube-system get pod <kube-vip-pod> \
-o jsonpath='{range .spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' | grep -E '^vip_(leaseduration|renewdeadline|retryperiod)='A longer lease does not remove the cause; it only provides headroom for short etcd stalls.
Priority and resources for the pod
Resource requests guarantee kube-vip CPU when the node is overloaded, and the priority class reduces the risk of the pod being evicted. The CPU limit is deliberately omitted, because throttling delays the renewal.
spec:
template:
spec:
priorityClassName: system-node-critical
containers:
- name: kube-vip
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
memory: 256MiIn a static pod the same fields go into the pod manifest. kubectl -n kube-system get pod <kube-vip-pod> -o jsonpath='{.spec.priorityClassName}{" "}{.status.qosClass}' should return system-node-critical Burstable, not BestEffort.
etcd on a fast disk
The etcd data directory (k3s: /var/lib/rancher/k3s/server/db/etcd, kubeadm: /var/lib/etcd) should sit on an SSD or NVMe disk that is not loaded by backups. Spread backups and snapshots of the control plane machines over time, so they never cover all nodes at once.
Separate roles and a mechanism independent of the API
When kube-vip with the --services parameter also handles LoadBalancer service addresses, a restart after losing the API lease interrupts those too. Service addresses are better entrusted to another mechanism, for example L2 announcements in Cilium or MetalLB.
When etcd cannot be made faster quickly, keepalived on the nodes or an HAProxy pair with keepalived removes the dependency on the API: slow etcd then slows the API but does not move the address (both variants). Configuring the network plugin with the address of a single node is not a solution: the moves disappear, but the failure of that node becomes a cluster-wide network outage.
Monitoring changes of owner
Alerts cover kube-vip restarts (kube-state-metrics) and etcd write latency (in kubeadm and k3s the etcd metrics are available locally at http://127.0.0.1:2381/metrics):
increase(kube_pod_container_status_restarts_total{namespace="kube-system", pod=~"kube-vip.*"}[1h]) > 0
histogram_quantile(0.99, sum by (instance, le) (rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m]))) > 0.01Add a probe from outside the cluster against https://192.0.2.10:6443/readyz every 10–15 seconds, and a periodic read of .spec.leaseTransitions of the plndr-cp-lock lease with an alert on every increase.
Check your own environment
The first block lists the pods in kube-system, the most frequently restarted last.
kubectl -n kube-system get pods -o wide --sort-by='.status.containerStatuses[0].restartCount'A good result: kube-vip and network plugin agent restarts match planned work. A bad one: kube-vip restarts on several nodes and plugin agent restarts with a similar “… ago” time.
The second block shows the owner of the address lease and renewal errors from the last 24 hours in every kube-vip pod.
kubectl -n kube-system get lease plndr-cp-lock \
-o jsonpath='{.spec.holderIdentity}{" "}{.spec.leaseTransitions}{" "}{.spec.renewTime}{"\n"}'
for p in $(kubectl -n kube-system get pods -o name | grep kube-vip); do
printf '%s: ' "$p"; kubectl -n kube-system logs "$p" --since=24h | grep -ciE 'error retrieving resource lock|failed to renew lease'
donerenewTime should be a few seconds old, leaseTransitions should not have grown by the next day without maintenance, and the error counters should be 0. NotFound means a different lease name (list: kubectl -n kube-system get lease).
The third block checks which node holds the address and who answers for it on the network.
# On the control plane nodes: the address should be on exactly one
for n in k8s-cp-01 k8s-cp-02 k8s-cp-03; do
printf '%s: ' "$n"; ssh "$n" "ip -br -4 addr show | grep -F '192.0.2.10/'" || echo 'no address'
done
# From another machine in the same segment (iputils-arping); only sends ARP requests, changes nothing
sudo arping -c 3 -I eth0 192.0.2.10A good result: one node with a line such as eth0 UP 192.0.2.11/24 192.0.2.10/32, the others “no address”, and arping replies from a single MAC address matching that node’s interface. Two different MAC addresses mean the address is on two machines.
The fourth block measures the API response at the virtual address for one minute.
for i in $(seq 1 60); do
curl -k -s -o /dev/null --max-time 3 -w "$(date +%T) %{http_code} %{time_total}s\n" https://192.0.2.10:6443/readyz
sleep 1
doneA good result: a steady code (200, or 401 when anonymous access is disabled, as in k3s) and times in the tens of milliseconds. 000 means no answer, and times in seconds point to a slow API or etcd.