A DNS outage has one peculiar property: it looks like an outage of everything else. Mail stops, pages do not load, domain logon takes minutes, monitoring reports hundreds of unavailable services that are working perfectly. Diagnosis takes long because the symptoms point at the network, the servers and the applications all at once, while the cause is one service nobody thought about.
Where single points of failure hide
One resolver. DHCP and server configuration list two addresses, but the second one is the same server under another address, a server decommissioned a year ago, or a domain controller sitting on the same virtualisation host as the first. Redundancy exists on paper.
A resolver that depends on what it serves. Container cluster nodes configured to use the DNS running inside that cluster. A cluster network failure takes away the nodes’ ability to resolve the image registry name they need in order to pull an image and recover.
Forwarders to a single provider. Both internal resolvers forward external queries to two addresses of the same ISP. An ISP outage means no internet by name, despite a working backup link.
Record lifetimes too long or too short. A long TTL makes moving a service to another address hard. A short TTL turns every resolver problem into an immediate outage, because the clients’ cache shields nothing.
Monitoring the process, not the service. A “DNS process is running” check passes when the process runs but answers after ten seconds or returns errors. Users have an outage; monitoring shows green.
A design that survives a failure
Two resolvers on independent virtualisation hosts, in separate network segments if the architecture allows. In an Active Directory environment these are usually two domain controllers with the DNS role, provided they do not share a host, a storage array or a power feed.
Every client, including servers and cluster nodes, knows both addresses. Order matters: some clients query the first resolver until it fails completely, others switch after a single failed query. It is worth splitting: half the clients with resolver A first, half with B.
Cluster nodes and infrastructure use resolvers independent of the cluster. DNS inside the cluster serves only cluster service names and forwards the rest to the external resolvers.
Every resolver has at least two forwarders from different providers: the ISP and a public server, or two public ones from different organisations.
Conditional forwarders and secondary zones are documented in one place, with a justification. A forwarder added five years ago for a partner that no longer exists is a delay on every query for that domain.
Monitoring that sees the failure
For each resolver separately: response time for an internal and an external name, share of error responses, queries per second. Alert on response time above a threshold for several minutes, not on process state. Additionally a test from the client’s perspective: a query from the user segment to the address handed out by DHCP.
The test
Switch off one resolver for an hour during business hours, after warning the team. If anyone notices, the design is not redundant: either clients do not have the second address, or the second resolver does not work, or the switchover takes longer than users’ patience. Repeat every six months and after every major network change.
Summary
DNS is the service every other one depends on, and at the same time the one least often designed. Two independent resolvers, two forwarders from different providers, clients that know both addresses, a cluster independent of its own DNS, and monitoring of response time instead of process state. A test by switching one off confirms that it works rather than merely looks like it works.