Core services: DNS and certificates

DNS resilience against the failure of a single resolver, forwarder or Kubernetes cluster network, and certificate inventory and monitoring with alert thresholds matched to the renewal method.

NetworkingSecurity

A DNS outage looks like an outage of everything else: mail does not work, signing in takes minutes, applications “cannot see the server”, and monitoring reports hundreds of unavailable services, most of which are in fact running. An expired certificate stops services just as effectively, although its date is known from the day it is issued. This document describes how to remove single points of failure in DNS (clients, DHCP, forwarders, the Kubernetes cluster) and how to inventory and monitor certificates.

Where single points of failure hide in DNS

Two DNS addresses in a client’s configuration do not mean two independent resolvers: it is enough for both domain controllers with the DNS role to sit on the same virtualisation host, which did not come back after overnight updates, while workstations receive only their addresses from DHCP.

What failsSymptomSafeguard
The virtualisation host running both domain controllersNo name resolution anywhere in the organisation, sign-in and mail stopResolvers on different hosts, an anti-affinity rule in the virtualisation cluster
The first resolver in the client’s listEvery query delayed by seconds, applications report timeoutsA working second resolver, a shorter timeout in resolv.conf, an alert on response time
The provider both forwarders point toInternal names work, external ones do not, despite a working backup linkForwarders from two providers, root hints as a last resort
The Kubernetes cluster networkPods cannot resolve names; nodes that query in-cluster DNS cannot pull imagesNodes on resolvers outside the cluster, CoreDNS forwarding to them
DHCP with a single DNS addressWorkstations know only one resolverOption 6 with two addresses in every scope

Two resolvers that really are independent

The two resolvers run on different virtualisation hosts, ideally also on different storage arrays and power circuits. In an Active Directory environment these are usually two domain controllers with the DNS role. Separating them on installation day is not enough: live migration can place both machines on one host for months. In a Hyper-V failover cluster the separation is maintained by the AntiAffinityClassNames property of the virtual machine groups; the enforced variant is enabled by the cluster property ClusterEnforcedAntiAffinity.

A domain controller points to another controller as its first DNS server, and to itself (127.0.0.1) only as a later one; once replication has been verified, Microsoft also allows the reverse order. A controller that queries only itself may wait a long time for its own DNS service after a restart, and when its zone is stale because of replication problems it will not find the SRV records of its replication partners.

Clients receive both addresses from DHCP in option 6 and usually query the first address in the list first. Linux servers without systemd-resolved keep the addresses in /etc/resolv.conf. By default glibc waits 5 seconds for each server to answer, so with the first resolver down every name lookup takes more than 5 seconds. The timeout and attempts options shorten that wait.

search corp.example.com
nameserver 10.10.0.10
nameserver 10.10.0.11
# 2 seconds per server instead of the default 5
options timeout:2 attempts:2

Forwarders and conditional forwarders

Every resolver has at least two forwarders from different providers. Two addresses of the same provider are one point of failure, and forwarders from two providers make sense only with two independent internet links.

A Windows DNS server with UseRootHint enabled (the default) resolves names on its own, starting from the root servers, when the forwarders fail, which is the last line of defence.

Every conditional forwarder has an owner and a justification. A forwarder for a partner whose cooperation has ended points at servers that no longer answer, and every query for that domain waits until it times out.

DNS in a Kubernetes cluster

CoreDNS in the cluster serves service names (*.svc.cluster.local) and forwards queries for everything else. In the default k3s and kubeadm configuration this is the Corefile line forward . /etc/resolv.conf, so CoreDNS uses the resolvers configured on the node. If the node points at the company resolvers, the chain is correct. If the node lists the cluster DNS service address, or a resolver running as a pod in the same cluster, a dependency loop appears: a cluster network failure takes name resolution away from the nodes, and without it the nodes cannot pull the images they need to recover.

Cluster nodes and all infrastructure therefore use resolvers independent of the cluster, while DNS inside the cluster serves only cluster service names and forwards the rest to the external resolvers. In k3s the coredns ConfigMap is overwritten on upgrade, so custom entries go into the coredns-custom ConfigMap in kube-system: *.server keys add separate server blocks, and *.override keys append lines to the main block.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  corp.server: |
    corp.example.com:53 {
      forward . 10.10.0.11 10.10.0.12
    }

DNS monitoring from the client’s perspective

A “DNS service is running” check also passes when the process answers after ten seconds or returns SERVFAIL. Monitoring measures the service from the client’s perspective, separately for each resolver:

  • response time for an internal name (for example the SOA record of the corp.example.com zone) and an external one,
  • the share of responses with an error.

A reasonable alert threshold is a response time above 200 ms for 5 minutes, or no response in three consecutive checks. In Zabbix the response time is returned by the item net.dns.perf[10.10.0.10,corp.example.com,SOA,2,1], in seconds, so a 200 ms threshold is 0.2. A value of 0 means only the absence of a response, and the trigger covers both a time above the threshold and the value 0. The ip, timeout and count parameters work on Windows only with agent 2.

DNS test by switching a resolver off

The most reliable proof of redundancy is switching one resolver off for an hour during business hours, after warning the team. The test passes when there are no user reports and the response time of the second resolver stays within the alert threshold; a delay of a few seconds at the moment clients switch over is normal. A worse result means clients lack a second address, the second resolver is not working, or the switchover is too slow. A rollback plan is needed: a designated person who switches the resolver back on immediately. The test is repeated every six months and after any major network change.

Where certificates hide

Everyone remembers the website certificate; other places are forgotten more often. The expiry date is written into a certificate from the day it is issued, so downtime usually results from not knowing that a given certificate exists, not from an unknown deadline.

  • LDAPS on domain controllers. When a controller’s certificate expires, sign-in stops in every application that connects to the directory over TLS.
  • CRLs of the internal authority. A revocation list has its own validity date (the NextUpdate field). Once it passes, clients cannot verify the status of certificates, so services that check revocation stop working, for example smart card sign-in.
  • Kubernetes clusters. In k3s, component certificates are valid for 365 days and renewed when the service starts with fewer than 120 days of validity left; an approaching expiry is signalled by the CertificateExpirationWarning event.
  • Certificates pinned by clients. An application or device that stores a specific certificate or its fingerprint stops working after a replacement, even when the new certificate is valid.

CA/Browser Forum rules reduce in stages the maximum validity of publicly trusted TLS certificates (not certificates of an internal authority): to 200 days from March 2026, 100 days from March 2027 and 47 days from March 2029 (as of October 2026). The first step already applies. Replacing certificates by hand once a year stops being possible: every public certificate needs automatic renewal, and monitoring has to detect automation that has quietly stopped working.

A certificate inventory from three sources

Network scanning finds everything that presents a certificate on a TLS port, including devices nobody remembers. It is done with nmap and the ssl-cert script, run against the TLS service ports (443, 636, 993, 5986, 8443) in a given subnet. The output includes Subject and Not valid after lines for every port; Issuer appears only with the -v option. A scan has to be agreed with whoever is responsible for security, because intrusion detection systems will notice it.

Issuer records know the certificates a scan will not see because they are not presented on any port: client and signing certificates. The internal certificate authority shows them in its console under issued certificates, and public certificates for the organisation’s own domains are visible in the Certificate Transparency logs, for example through the crt.sh search.

System configuration fills in the rest: certificate stores on Windows servers, files on Linux servers, secrets of type kubernetes.io/tls and Certificate resources in the cluster.

Every certificate gets an entry with the same fields: name, service address, issuer, expiry date, renewal method, owner, action after renewal and dependencies (for example applications with a pinned certificate). Once a week the scan results are set against the inventory: a certificate that appears in the scan but not in the list is an entry to add, together with its owner, and an entry the scan did not find means a service that was switched off or moved. New services go into the inventory when they are deployed.

Alert thresholds and certificate renewal

The basic metric is the number of days to expiry, calculated daily for every entry; the thresholds depend on how the certificate is renewed. For a manually renewed certificate, 30 days means a ticket with a replacement date for the owner, 14 days an alert in the team channel, visible also to the owner’s manager, and 7 days an immediate page handled by the on-call engineer as an incident. For automatic renewal the same 14 and 7 day thresholds apply, and the 30 day threshold needs no action if the automation renews earlier.

For automatically renewed certificates there is an additional alert that is event-based rather than time-based: a certificate older than the usual renewal point means the automation did not run, and the team has to know immediately. The usual renewal point comes from the tool. Since version 4.0 certbot renews a certificate when less than one third of its lifetime remains (previously a fixed 30 days), and cert-manager by default when two thirds of the lifetime have passed, or at the time set in spec.renewBefore. The alert threshold is the renewal point minus one day: renewBefore minus one day, and for a 90-day certificate without that field, 29 days.

In Prometheus with blackbox_exporter the time to expiry comes from the probe_ssl_earliest_cert_expiry metric, and cert-manager exports certmanager_certificate_expiration_timestamp_seconds. In Zabbix agent 2 the same reading is provided by the web.certificate.get item.

# services probed by blackbox_exporter: fewer than 14 days
(probe_ssl_earliest_cert_expiry - time()) / 86400 < 14

# cert-manager certificates: fewer than 29 days (renewBefore minus a day), i.e. the automation did not renew on time
(certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 29

The ACME protocol for public certificates and automatic enrolment in the internal authority remove most manual work. Certificates that cannot be automated have a written procedure and a deadline in the team’s calendar, not one person’s. Once a year, for certificates with a long validity period, such as intermediate authorities and federation certificates, a trial replacement is carried out in a test environment.

Decisions and alternatives

ApproachWhen it makes senseLimitation
Soft anti-affinity rule (AntiAffinityClassNames)default separation of resolvers across the hosts of a Hyper-V clusterwhen no other node is available, the cluster places both groups together
Enforced rule (ClusterEnforcedAntiAffinity)resolvers must never land on one hostwith two nodes, the failure of one prevents both groups from running on the remaining one
Forwarders from a single providera single internet linktwo addresses of the same provider are one point of failure
Forwarders from two providers plus root hintstwo independent linksroot hints need a firewall that lets DNS queries out to any address
Custom entries in the coredns-custom ConfigMapk3s, where coredns is overwritten on upgradespecific to k3s; other distributions handle Corefile changes differently
A resolver running as a pod in the same clusteronly for cluster service namesdependency on the cluster: its failure takes name resolution away from the nodes
Manual renewal with 30/14/7 day thresholdscertificates that cannot be automatedneeds an owner, a procedure and a deadline in the team’s calendar
Automatic renewal (ACME, automatic enrolment, cert-manager)public and internal certificatesneeds an event-based alert that detects automation that has stopped working

Order of work

  1. Inventory of DNS clients: the list of resolvers of every network adapter on workstations and servers (Get-DnsClientServerAddress -AddressFamily IPv4): two addresses of machines on different hosts everywhere, with no addresses of machines that no longer exist, no servers pointing only at themselves and no cluster DNS service address among a node’s resolvers.
  2. Correction of the DHCP scopes (option 6 with two addresses), the DNS order on domain controllers, the timeout in resolv.conf, forwarders and conditional forwarders, and the anti-affinity rule in the virtualisation cluster. Check: no client has a single address, and both resolvers sit on different hosts.
  3. Completion of the coredns-custom ConfigMap and verification that cluster nodes use resolvers outside the cluster.
  4. Addition of response time and error measurements for every resolver, and of alerts with the thresholds from the monitoring section.
  5. A test that switches one resolver off, repeated every six months and after any major network change.
  6. Certificate inventory from three sources. The Windows computer store is checked with Get-ChildItem Cert:\LocalMachine\My filtered on NotAfter for the next 30 days.
  7. Introduction of the daily days-to-expiry measurement, the 30/14/7 day thresholds for every entry, and the event-based alert for automatically renewed certificates. Check: the weekly comparison of the scan with the inventory shows no differences without an owner.
  8. Renewal: ACME and automatic enrolment wherever possible, manual procedures for the rest, and an annual trial replacement of certificates with a long validity period.

Limitations and pitfalls

  • Root hints and the firewall. UseRootHint works only if the firewall lets domain controllers send DNS queries to any address on the internet; if the rules allow only the forwarder addresses, root hints will not help. The firewall rules have to be tested with the forwarders disconnected.
  • A soft anti-affinity rule. A change to AntiAffinityClassNames takes effect only after the resources are recycled, and without another available node both groups end up on one host. The placement of resolvers after migrations and failures is checked separately.
  • Time measurement does not detect SERVFAIL. When the server answers with an error code, for example SERVFAIL or NXDOMAIN, the net.dns.perf item returns the time, not 0, so measuring time alone will not detect it. Responses with an error need a separate item.
  • A renewed file, an old certificate in the service’s memory. After renewal the service may still present the old certificate. The serial number and expiry date from the file (openssl x509 -noout -serial -enddate -in <file>) are compared with the certificate the service actually presents (openssl s_client -connect <host>:443 -servername <host> piped to openssl x509 -noout -serial -enddate).

Sources