A virtual IP for the cluster API: kube-vip, keepalived or an external load balancer

The cluster has three control plane nodes and a virtual API address. After the first node restarts, networking stops across the whole cluster: pods cannot resolve names, new ones do not start, and applications report connection errors. An hour later it turns out that the network plugin was installed with that node’s API address rather than the virtual address. For a year it had worked perfectly.

Cluster clients need one API address that survives the failure of any node. The mechanism that maintains it, and whether every client really uses it, decide whether the failure of one node stays the failure of one node.

Three approaches

An address managed from inside the cluster. kube-vip runs as a static pod or DaemonSet on the control plane nodes, elects a leader through a lease (a Lease object) in the Kubernetes API and announces the address on the network: via ARP at layer 2 or via BGP to a router. It needs no extra machines.

An address managed on the nodes, outside Kubernetes. Keepalived runs as a system service on the control plane nodes and uses VRRP to choose the node that holds the address. A script checking the API server decides whether a node is fit for that role.

An external load balancer. A pair of machines with HAProxy and keepalived, or a network appliance, sits in front of the nodes, checks each API server and distributes connections. The virtual address belongs to the load balancer, not to the nodes.

Criterion	kube-vip in the cluster	Keepalived on nodes	External load balancer
Extra machines	none	none	two, or an appliance
Depends on the API and etcd	yes	no	no
Needs a shared layer 2	in ARP mode, yes	yes	not for the nodes
Spreads traffic across nodes	no, one leader	no, one owner	yes
Operational complexity	lowest	medium	highest

kube-vip: the lease and its parameters

An address managed from inside the cluster depends on the API it exposes. The leader must renew its lease in the API regularly, and the API needs etcd. When etcd responds slowly, for example on an overloaded disk, the renewal misses its deadline, the address jumps to another node, and every client drops its connection and adds even more load to the API. A kube-vip restart counter that grows week after week is the most common symptom of this loop.

kube-vip generates its own manifest. Without the --services parameter it handles only the API address, which separates that role from addresses for LoadBalancer services:

kube-vip manifest daemonset \
  --interface eth0 \
  --address 10.10.0.50 \
  --inCluster \
  --taint \
  --controlplane \
  --arp \
  --leaderElection

The default lease parameters (5, 3 and 1 second) are tuned for fast failover. Where etcd is sometimes slower, it is worth giving them headroom in the container’s environment variables:

env:
  - name: vip_leaseduration
    value: "15"
  - name: vip_renewdeadline
    value: "10"
  - name: vip_retryperiod
    value: "2"

Keepalived: checking API readiness

Keepalived depends on neither etcd nor the API. The address moves when the check script on a node stops returning success. The script should query /readyz rather than check that port 6443 is open, because an open port only says the process is listening. In k3s anonymous access to the API is disabled by default, so the script uses the local kubeconfig through k3s kubectl:

vrrp_script chk_apiserver {
    script "/usr/local/bin/k3s kubectl get --raw=/readyz --request-timeout=2s"
    interval 3
    timeout 3
    fall 3
    rise 2
}

vrrp_instance K8S_API {
    state BACKUP
    nopreempt
    interface eth0
    virtual_router_id 51
    # 100, 90 and 80 on the successive nodes
    priority 100
    advert_int 1
    virtual_ipaddress {
        10.10.0.50/24
    }
    track_script {
        chk_apiserver
    }
}

fall 3 with a 3-second interval means failover after about 10 seconds without readiness. nopreempt means a higher-priority node does not take the address back immediately when it returns, which spares clients a second failover. The script must run with permissions that allow it to read /etc/rancher/k3s/k3s.yaml.

An external load balancer

A single HAProxy machine is a new single point of failure, so the load balancer is a pair with keepalived moving the address between them. On the standby machine HAProxy must be able to listen on an address it does not hold yet, which the setting net.ipv4.ip_nonlocal_bind = 1 allows.

frontend k8s_api
    bind 10.10.0.50:6443
    mode tcp
    default_backend k8s_api_servers

backend k8s_api_servers
    mode tcp
    balance roundrobin
    default-server inter 3s fall 3 rise 2
    server cp1 10.10.0.11:6443 check
    server cp2 10.10.0.12:6443 check
    server cp3 10.10.0.13:6443 check

The API certificate and --tls-san

The virtual address and the DNS name clients know it by must be in the SAN field of the API server certificate. In k3s they are added with the --tls-san parameter or in the configuration file, on every server node:

tls-san:
  - k8s-api.corp.example.com
  - 10.10.0.50

The list is best settled before installation. Adding an entry later requires restarting the k3s service on every server and checking that the certificate contains the new name. A missing entry shows up as the error “x509: certificate is valid for …, not 10.10.0.50” only when somebody first connects through the virtual address.

Clients that bypass the virtual address

The k3s.yaml file on a k3s server points at https://127.0.0.1:6443. When copying it to an administrator’s workstation or into CI, change the address to the virtual name, not to the address of the node the file came from. k3s agents use the address from the server parameter when registering, and after joining they connect to the servers through a built-in load balancer. If server points at one specific server, however, joining a new node while that server is down will fail.

What to choose

In a small cluster on one network, kube-vip is a reasonable start, provided its restarts are monitored and the lease has headroom. When the cluster runs something whose unavailability hurts, keepalived on the nodes removes the most dangerous dependency at the cost of one system service. An external load balancer pair makes sense when the nodes are on different networks, when traffic distribution is needed, or when the same load balancer serves other services.

The test

Switch off the node currently holding the address for at least twenty minutes. A loop run from another machine measures the interruption: code 000 means no answer, any HTTP code means the address responds.

while true; do
  printf '%s %s\n' "$(date +%T)" \
    "$(curl -sk -o /dev/null -w '%{http_code}' --max-time 1 https://10.10.0.50:6443/readyz)"
  sleep 1
done

After the test, check the restart counts of components in kube-system and whether the address jumped back to the node needlessly when it returned.

Check your own environment

The first command shows which address the current kubeconfig connects to.

kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}'

A good result is the virtual name or address, for example https://k8s-api.corp.example.com:6443. A specific node’s address means this client loses access together with that node. Run the command on every administrator workstation and in every pipeline.

The second block checks whether the API answers through the virtual address.

curl -k --max-time 3 https://10.10.0.50:6443/readyz; echo
kubectl --server=https://10.10.0.50:6443 get --raw='/readyz?verbose' | tail -n 1

An ok response from curl means the API is ready. A JSON response with code 401 means the address works but anonymous access is disabled (as in k3s); the second command then decides, and it should return readyz check passed. No response means the address does not work, and an x509 error from the second command means the address is missing from the certificate.

The third block shows the names in the API server certificate (OpenSSL 1.1.1 or later).

openssl s_client -connect 10.10.0.50:6443 -servername k8s-api.corp.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

The list should contain both DNS:k8s-api.corp.example.com and IP Address:10.10.0.50.

The fourth block searches the configuration of system components for specific node addresses.

# Control plane node addresses instead of the virtual address
kubectl -n kube-system get daemonset,deployment,configmap -o yaml | grep -nE '10\.10\.0\.1[123]'

A good result is empty. Inspect every match: if it is the API address, the component depends on a single node.