A monitoring system that sends two hundred notifications a day is as useful as one that sends none. The team learns to ignore the alert channel, and the alert that matters, when it arrives, gets lost between “CPU usage 81%” and “certificate expires in 89 days”. Alert fatigue is not a people problem; it is a design problem.

Rule one: an alert requires action

Every alert answers the question: what should the person who receives it do? If the answer is “nothing, just be aware”, it is not an alert but a dashboard item. If the answer is “check whether it is something serious”, the alert is too broad and needs narrowing.

A practical test: an alert that fired ten times in a month and nobody did anything any of those times gets removed or downgraded to information.

Rule two: symptoms, not causes

Users do not care that the CPU is at 90%. They care that the application responds after five seconds. Service-level alerts (response time, error rate, availability) catch every cause, including the one nobody predicted. Resource-level alerts (CPU, memory, disk) support diagnosis; they are not the main source of notifications.

The exception: resources whose exhaustion is irreversible or slow to fix. Disk space, certificates, licences, cluster quorum. Here a cause-based alert is right, because the symptom would appear too late.

Rule three: three levels

  • Immediately: the service is down for users or is about to be. A notification to a phone, at any hour. In a small organisation this should happen a few times a month.
  • During business hours: something needs attention within the day. A ticket in the ticketing system, not a text message.
  • Information: a trend, a warning, context. A dashboard, a weekly report. Zero notifications.

An alert without an assigned level defaults to the lowest.

Thresholds and duration

A round number is not a threshold. A threshold comes from history: if a disk fills up 1% per week, an alert at 80% gives four months and at 95% one month. If it fills up 5% a day, 80% is too late.

Every alert has a duration condition. A momentary CPU spike during a backup is not an incident. “CPU above 90% for 15 minutes” is. The duration condition removes most false alarms without losing the real ones.

Deduplication and grouping

A single switch failure generates a hundred unreachable-host alerts, one for every host behind it. The person on call needs one: “segment X unreachable, hosts affected: 100”. Grouping by dependency (host behind switch, service on host, application on service) reduces the noise to a single root cause.

Silences during maintenance windows are part of the change process: a planned restart wakes nobody.

Escalation

An “immediately” alert not acknowledged within an agreed time goes to the next person. Without escalation, on-call depends on whether one person heard their phone. The escalation rule is simple and written down; nobody wonders at three in the morning whom to wake.

Every alert includes a link to a short procedure: what to check first, what the typical cause looks like, how to restore the service temporarily. An alert without a procedure is a knowledge test for the person on call, taken under the worst possible conditions.

Review

Once a month: the list of alerts that fired, split into those followed by action and those ignored. The second group gets fixed or removed. The number of alerts goes down and trust in the channel goes up.

Summary

A good alerting system is small. An alert requires action, concerns a symptom visible to users, has a level, a threshold derived from data, a duration condition, dependency-based grouping, escalation and a procedure. Everything else goes on a dashboard, where it bothers nobody.