An expiring certificate is the most predictable outage there is: inventory and monitoring

On Monday morning no application can sign users in over LDAPS. The domain controllers are running, the accounts are fine, and the application logs repeat a certificate verification error. The controller’s certificate expired on Sunday evening. The date had been written in the certificate for two years, but it was not in any calendar, monitoring system or document.

Certificate expiry is the only outage whose exact date is known in advance. The cause of downtime is almost never not knowing that a certificate will expire, but not knowing that this particular certificate exists. The answer is an inventory from several sources, a daily measurement of days to expiry, and thresholds that turn the date into tasks with an owner.

Where certificates hide

Everyone remembers the website certificate. The places forgotten more often:

• LDAPS on domain controllers. When a controller’s certificate expires, sign-in stops in every application that connects to the directory over TLS.
• Federation and single sign-on. Certificates that sign SAML tokens have lifetimes measured in years, so nobody remembers how to replace them.
• The internal certificate authority. When an intermediate CA expires, everything it issued becomes invalid.
• Kubernetes clusters. In k3s, component certificates are valid for a year and renewed when the service starts as expiry approaches. A cluster nobody has restarted for a year can be a surprise.
• Devices. Management interfaces of servers, switches, storage arrays and UPS units.
• Integrations. Client certificates that applications use to authenticate to partner, bank and government APIs.
• Mail. TLS on relays and connectors, message signing certificates.

An inventory from three sources

No source is complete on its own, so the inventory combines three.

Network scanning finds everything that presents a certificate on a TLS port, including devices nobody remembers. Agree a scan of your own network with whoever is responsible for security, because intrusion detection systems will notice it.

# The output includes Subject, Issuer and Not valid after lines for every port
nmap -p 443,636,993,5986,8443 --script ssl-cert 10.10.0.0/24 -oN certificates-$(date +%F).txt

Issuer records know the certificates a scan will not see because they are not exposed on any port: client and signing certificates. The internal certificate authority shows them in its console under issued certificates, and public certificates for your own domains are visible in the Certificate Transparency logs, for example through the crt.sh search.

System configuration fills in the rest: certificate stores on Windows servers, files on Linux servers, secrets of type kubernetes.io/tls and Certificate resources in the cluster.

Every certificate gets an entry with the same fields. The format is secondary; what matters is that the inventory can be searched and compared with scan results:

- name: ldaps-dc01
  where: dc01.corp.example.com:636
  issuer: internal CA (Corp Issuing CA)
  expires: 2027-03-14
  renewal: auto-enrolment, Domain Controller template
  owner: infrastructure team
  after_renewal: check that LDAPS presents the new certificate
  dependencies: applications with a pinned certificate (list in the documentation)

An inventory is only reliable when it is regularly compared with reality. Once a week, the scan results are set against the list. A certificate that appears in the scan but not in the inventory is an entry to add, together with its owner. An entry the scan did not find means a service that was switched off or moved to another address. New services go into the inventory when they are deployed, as a step in the procedure, not at the first alert.

Thresholds and actions

The basic metric is the number of days to expiry, calculated daily for every entry. The thresholds depend on how the certificate is renewed.

Threshold	Manually renewed certificate	Automatically renewed certificate	Who responds
30 days	a ticket with a replacement date	no action if the automation renews earlier	owner
The usual renewal point has passed	not applicable	alert: the automation did not run	team
14 days	alert in the team channel	alert in the team channel	team and the owner’s manager
7 days	immediate page, handled as an incident	as for manual	on-call

The usual renewal point comes from the tool. By default certbot renews a certificate when 30 days of validity remain, and cert-manager when two thirds of the lifetime have passed, which for a 90-day certificate is also about 30 days before expiry. If 20 days remain and the certificate is still the old one, renewal has failed and you need to know immediately.

The measurement runs in whatever monitoring system the team already uses. In Prometheus with blackbox_exporter the time to expiry comes from the probe_ssl_earliest_cert_expiry metric, and cert-manager exports certmanager_certificate_expiration_timestamp_seconds. In Zabbix agent 2 the same reading is provided by the web.certificate.get item.

# services probed by blackbox_exporter: fewer than 14 days
(probe_ssl_earliest_cert_expiry - time()) / 86400 < 14

# cert-manager certificates: fewer than 21 days, i.e. the automation did not renew on time
(certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 21

openssl x509 -noout -serial -enddate -in /etc/ssl/certs/portal.crt
echo | openssl s_client -connect portal.example.com:443 -servername portal.example.com 2>/dev/null \
  | openssl x509 -noout -serial -enddate

Renewal

The ACME protocol for public certificates and auto-enrolment with the internal authority for domain certificates remove most manual work. Certificates that cannot be automated, for example ones uploaded to external portals, have a written procedure and a date in the team’s calendar, not one person’s.

Once a year, for long-lived certificates such as intermediate CAs and federation certificates: a trial replacement in a test environment. A procedure nobody has performed in three years is almost certainly out of date.

Check your own environment

The first block reads the expiry date of the certificate presented by a network service.

# HTTPS (SNI matters when several sites share one address)
echo | openssl s_client -connect portal.example.com:443 -servername portal.example.com 2>/dev/null \
  | openssl x509 -noout -subject -issuer -enddate
# LDAPS on a domain controller
echo | openssl s_client -connect dc01.corp.example.com:636 2>/dev/null | openssl x509 -noout -enddate
# SMTP with STARTTLS
echo | openssl s_client -connect mail.example.com:25 -starttls smtp 2>/dev/null | openssl x509 -noout -enddate
# Exit code 1 and the message Certificate will expire when fewer than 14 days remain
echo | openssl s_client -connect portal.example.com:443 -servername portal.example.com 2>/dev/null \
  | openssl x509 -noout -checkend 1209600

The notAfter date should be further away than the usual renewal point. The message Certificate will expire means the certificate has crossed the 14-day threshold.

The second block lists certificates in the Windows computer store that expire within 30 days, and the certificates on the domain controllers.

$limit = (Get-Date).AddDays(30)
Get-ChildItem Cert:\LocalMachine\My |
    Where-Object NotAfter -lt $limit |
    Sort-Object NotAfter |
    Select-Object Subject, NotAfter, Thumbprint,
        @{ n = 'DaysLeft'; e = { ($_.NotAfter - (Get-Date)).Days } }

# Certificates on the domain controllers (including LDAPS)
Invoke-Command -ComputerName dc01, dc02 -ScriptBlock {
    Get-ChildItem Cert:\LocalMachine\My | Select-Object Subject, NotAfter, Thumbprint
}

An empty result from the first command is good. Negative values in the DaysLeft column are certificates that have already expired: if the service uses a newer one, they are just leftovers; if it uses this one, the service has just stopped working.

The third block shows certificates managed by cert-manager and their renewal times.

kubectl get certificate -A
kubectl get certificate -A \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,EXPIRES:.status.notAfter,RENEWAL:.status.renewalTime

Every certificate should have READY set to True and RENEWAL in the future. A renewal time in the past with an unchanged expiry date means renewal is failing; kubectl describe certificate shows why.

The last block, run as root on a k3s server node, shows the validity of the cluster component certificates.

for f in /var/lib/rancher/k3s/server/tls/*.crt; do
  printf '%-60s %s\n' "$f" "$(openssl x509 -noout -enddate -in "$f")"
done

The cluster CA certificates (files with ca in the name) are valid for many years. The others should not end within the next few months; an expiry closer than 90 days means the service has not been restarted for a long time.