Metrics and alerts: what to measure and when to wake someone up

The monitoring agent collects several thousand metrics from every server, and the system sends two hundred notifications a day: “CPU 81%”, “certificate expires in 89 days”, “host unreachable” after every planned restart. A month later the team mutes the alert channel. When the database disk fills up during the night, the notification that matters is lost among the rest, and users report the outage in the morning.

Both symptoms share one source: metrics collected without a question they are meant to answer, and alerts set up without asking what the person who receives them should do. Below is a small set of signals to build alerts on, and the rules that make an alert at three in the morning genuinely worth it.

What to measure: four signals and three questions

For a user-facing service, regardless of technology, four signals are enough. For every physical or virtual resource, three questions are enough.

Object	Signal	Example metric	What it answers
Service	Latency	95th or 99th percentile of response time	are users waiting
Service	Traffic	requests per second, active sessions	context for the other signals
Service	Errors	share of 5xx responses, failed transactions	are requests succeeding
Service	Saturation	database connection pool usage, queue length	how close to its limit the narrowest resource is
Resource	Utilisation	CPU percentage, disk usage, interface throughput	what share of capacity is in use
Resource	Saturation	processor run queue, disk wait time, dropped packets	how much work is waiting for the resource
Resource	Errors	read and write errors, interface errors, retransmissions	is the resource working correctly

Latency is measured as a percentile, not an average. An average of 300 ms can hide that every twentieth user waits five seconds.

In a Windows environment a few metrics outside these schemes are added: replication latency and errors between domain controllers, failed authentications over time, clock skew, and the age of the last successful backup of each system.

What not to collect

Per-second metrics where decisions are made on a scale of minutes. An interval of 15–60 seconds is almost always enough.
Metrics for every process on every machine. Process details are collected only for services monitored as services.
Values that belong in logs: user IDs, IP addresses, request numbers. Every unique combination of labels is a separate time series, so such a metric blows up the database. A metric counts, a log describes; I cover the split in A logging strategy.

An alert requires action

Every alert answers one question: what should the person who receives it do? If the answer is “nothing, just be aware”, it is information for a dashboard. If it is “check whether it is something serious”, the alert is too broad and needs narrowing.

Alert on symptoms visible to users, not on causes. The error rate and response time catch every cause, including the one nobody predicted. CPU at 90% with normal response times is no reason to wake anyone; resource metrics are for diagnosis after an alert on a symptom has already arrived.

Alert levels and channels

Level	When	Channel	Expected response	Example
Wakes someone	the service is down for users or is about to be	phone or on-call app, at any hour	minutes	5xx error rate above 5% for 10 minutes
Ticket	something needs attention within the working day	a ticket in the ITSM system	same or next working day	disk will fill up within 7 days
Dashboard only	trend, context, warning	dashboard, weekly report	no notification	CPU above 85% at peak hours

An alert without an assigned level goes to the lowest one. In a small organisation, there should be a few “wakes someone” alerts a month, not a few a day.

Thresholds with a duration in Zabbix

A round number is not a threshold. A threshold comes from history: if a disk fills up by 1% a week, an alert at 80% leaves months of headroom; if it fills up by 5% a day, 80% is too late. On top of that, every alert has a duration condition: a momentary CPU spike during a backup is not an incident, 15 minutes above the threshold is.

Triggers in the syntax introduced in Zabbix 5.4 and used in 6.0 LTS and later:

# CPU above 90% for the whole 15 minutes (every value in the period)
min(/srv-app-01/system.cpu.util,15m)>90

# The “Portal” web scenario has been failing for 5 minutes
min(/srv-web-01/web.test.fail[Portal],5m)>0

# At the rate of the last day, /var will reach 95% within 7 days
timeleft(/srv-app-01/vfs.fs.size[/var,pused],24h,95)<7d

The first two triggers get the “wakes someone” or “ticket” level depending on the service; the third is always “ticket”, as it leaves a week to react during working hours. To stop a problem flapping around the threshold, a trigger gets a separate recovery expression, for example max(/srv-app-01/system.cpu.util,10m)<80. The problem then closes only once CPU has stayed below 80% for 10 minutes.

A Prometheus rule with a duration

In Prometheus, the duration is set by the for: field. The condition must hold at every evaluation for the given time; until then the alert is pending and does not reach Alertmanager. Metric and label names depend on the exporter.

groups:
  - name: portal
    rules:
      - alert: PortalHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="portal", status=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="portal"}[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "Portal: more than 5% of responses with a 5xx error for 10 minutes"
          runbook_url: "https://wiki.example.com/runbooks/portal-5xx"

      - alert: DiskFullIn7Days
        expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/var"}[6h], 7 * 24 * 3600) < 0
        for: 1h
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.instance }}: /var will fill up within 7 days"
          runbook_url: "https://wiki.example.com/runbooks/disk-full"

The severity label decides the channel. In Alertmanager the default route sends nothing, so an alert without a level stays on the dashboard:

route:
  receiver: dashboard-only        # no level = no notification
  group_by: [alertname, job]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['severity="page"']
      receiver: on-call
    - matchers: ['severity="ticket"']
      receiver: itsm
      repeat_interval: 24h

receivers:
  - name: dashboard-only
  - name: on-call                 # on-call notification settings go here
  - name: itsm                    # webhook that creates a ticket goes here

Grouping, maintenance windows and escalation

A single switch failure generates a hundred unreachable-host alerts, one for every host behind it. The person on call needs one. In Zabbix, trigger dependencies handle this: a host problem is not reported while the problem of the switch it depends on is active. In Alertmanager, group_by combines alerts into a single notification, and inhibit_rules silence service alerts while an alert about an unreachable node is active.

Planned work is part of the change process: a maintenance period in Zabbix or a silence in Alertmanager is set up before the restart, with the ticket number in the comment. A “wakes someone” alert not acknowledged within an agreed time, for example 15 minutes, goes to the next person. The escalation rule is written down, so that nobody wonders at three in the morning whom to wake.

Every alert with a runbook

An alert without a runbook is a knowledge test for the person on call, taken under the worst possible conditions. A runbook is short and has a fixed structure:

Meaning:     more than 5% of portal responses have ended in a 5xx error for 10 minutes
Impact:      some users cannot sign in or save data
Check:       1. the “Portal” dashboard: which endpoint returns errors
             2. application log for the last 15 minutes: exceptions, database timeouts
             3. whether there was a deployment in the last hour
Workaround:  roll back the last deployment or restart the application pool
Escalation:  the application owner, if there is no improvement after 30 minutes

The link to the runbook is part of the alert: in Prometheus as the runbook_url annotation, in Zabbix in the trigger’s URL field. That way it reaches the notification together with the alert text.

Two-tier retention

Tier	Resolution	Retention period	Used for
Raw data	collection interval, 15–60 s	7–14 days	incident diagnosis
Aggregates	hourly minimum, average and maximum	12 months or longer	capacity planning, trends

In Zabbix this split is built in: every item has a separate storage period for history (for example 14 days) and trends (for example 365 days), and trends are exactly those hourly aggregates. The local Prometheus database has a single retention, set with the --storage.tsdb.retention.time flag. Long-term data is therefore usually kept in a separate system that receives aggregated series from recording rules.

A monthly review

Once a month the team reviews the alerts from the last 30 days in two groups. The first is the loudest alerts: those that fired most often or lasted longest. The second is alerts nobody acted on: unacknowledged, with no ticket and no trace of action. An alert that fired ten times in a month with nobody doing anything any of those times gets fixed, downgraded to the dashboard or removed. The number of alerts goes down and trust in the channel goes up.

Check your own environment

Which alert rules have no duration, level or runbook:

curl -s http://prometheus.example.com:9090/api/v1/rules \
  | jq -r '.data.groups[].rules[]
      | select(.type == "alerting")
      | select(.duration == 0 or .labels.severity == null or .annotations.runbook_url == null)
      | .name'

Empty output is good. Every name on the list is a rule to complete: no for: means an alert on a momentary spike, no severity an alert that goes nowhere, no runbook_url an alert without a runbook.

Which alerts were the loudest in the last 30 days:

curl -sG http://prometheus.example.com:9090/api/v1/query \
  --data-urlencode 'query=topk(10, sum by (alertname) (count_over_time(ALERTS{alertstate="firing"}[30d])))' \
  | jq -r '.data.result[] | "\(.value[1])\t\(.metric.alertname)"'

The result is the number of evaluations in the firing state; with a one-minute evaluation interval it equals the number of minutes. An alert that was active for thousands of minutes in a month signals a permanent problem nobody is solving, or a badly set threshold. In Zabbix, the Reports → Top 100 triggers report for a 30-day period gives a similar list.

Which alerts are silenced, and since when:

amtool silence query --alertmanager.url=http://alertmanager.example.com:9093

Every silence should have a comment with a ticket number and an end date days away, not weeks. A silence that has been renewed for months is an alert nobody acts on, just better hidden.