Monitoring and event logs

Choosing metrics and building alerts that require action, and a strategy for Windows event logs: which events to collect, how to filter and process them, and how long to keep them.

ObservabilityWindows Server

Metrics collected without asking which question they answer, and alerts set without asking what the person who receives them is supposed to do, end in a muted notification channel in which the alert about a database disk filling up is lost. A central log system launched on the rule “collect everything” turns into an expensive warehouse of noise, and the event being looked for is often missing altogether because the system never generated it. This document describes a small set of signals and the rules for building alerts, together with a strategy for event logs: the purpose of a stream, the choice of Windows events, filtering, form, retention and the recipient.

Metrics: four signals and three questions

For a service visible to users, four signals are enough: latency, traffic, errors and saturation (the golden signals from Google’s book on site reliability engineering). For every resource, three questions from Brendan Gregg’s USE method are enough: utilisation, saturation, errors.

For a service, latency (a response-time percentile), traffic, the error rate and the saturation of the narrowest resource, e.g. a database connection pool, are measured. For a resource, e.g. a processor, disk or network interface, utilisation, saturation (queue length) and errors are measured.

Latency is measured with a percentile, not an average: a 300 ms average can hide that every twentieth user waits five seconds. Utilisation at 100% is not a problem with zero saturation, while saturation at low average utilisation is often caused by short bursts hidden in the average, or by a misconfiguration, e.g. a connection pool that is too small. In a Windows environment, replication latency and errors between domain controllers, failed authentications over time, clock skew and the age of the last successful backup of each system are added.

Alerts

Levels and action

Every alert answers the question of what the person who receives it is supposed to do. “Nothing, just know” means information for a dashboard, while “check whether it is serious” means the alert is too broad.

Alerts fire on symptoms visible to users, not on causes: the error rate and response time catch every cause, including unforeseen ones. CPU at 90% with a normal response time is no reason to wake anyone; resource metrics serve diagnosis after a symptom alert. The exception is resources whose exhaustion is irreversible or slow to fix: disk space, certificates, licences, cluster quorum and backup age. A cause alert is right there, because the symptom would appear too late.

LevelWhenChannelReactionExample
Wakes someonethe service is down for users or about to bephone or on-call app, any hourminutes5xx error share above 5% for 10 minutes
Ticketneeds attention within the working daya ticket in the ITSM systemsame or next working daythe disk will fill within 7 days
Dashboard onlytrend, context, warningdashboard, weekly reportno notificationCPU above 90% for 15 minutes

An alert without an assigned level goes to the lowest one. “Wakes someone” alerts should number a few a month, not a few a day.

A threshold with a duration in Zabbix

A round number is not a threshold; it comes from history. If a disk fills by 1% a week, an alert at 80% gives months of margin, and at 5% a day it is already too late. Every alert also has a time condition: a brief CPU spike during a backup is not an incident, 15 minutes above the threshold is. The triggers below use the syntax introduced in Zabbix 5.4, valid in 6.0 LTS and later.

# CPU above 90% for the whole 15 minutes (every value in the period); level: dashboard only
# The system.cpu.util key without parameters counts only user time on Linux, so the load is derived from idle time
100-max(/srv-app-01/system.cpu.util[,idle],15m)>90

# The “Portal” web scenario has been failing for 5 minutes
min(/srv-web-01/web.test.fail[Portal],5m)>0

# At the rate of the last day, /var will reach 95% within 7 days
timeleft(/srv-app-01/vfs.fs.size[/var,pused],24h,95)<7d

The first trigger, based on a cause, serves diagnosis. The second describes a symptom and gets the “wakes someone” or “ticket” level depending on the service. The third is always “ticket”: it leaves a week to react during working hours. To stop a problem flipping around the threshold, a trigger gets a recovery expression, e.g. 100-min(/srv-app-01/system.cpu.util[,idle],10m)<80.

In Prometheus the equivalent is the for: field of an alerting rule, and the route in Alertmanager is chosen by matching alert labels (e.g. severity, which the rule sets and which is not built in).

Grouping, maintenance windows and escalation

The failure of one switch generates a hundred unreachable-host alerts for everything behind it, while the on-call person needs one. In Zabbix trigger dependencies do this, in Alertmanager group_by and inhibit_rules. Planned work needs a maintenance period in Zabbix or a silence in Alertmanager, created before the restart, with the ticket number in the comment. A “wakes someone” alert not acknowledged within a set time (e.g. 15 minutes) goes to the next person; the escalation rule is written down so nobody has to wonder at three in the morning whom to wake.

Runbook

An alert without a runbook is a test of the on-call person’s knowledge under the worst conditions. A runbook is short and has a fixed structure of five fields:

  • meaning: what the alert signifies, e.g. more than 5% of portal responses have ended in a 5xx error for 10 minutes,
  • impact: who and what is affected, e.g. some users cannot sign in or save data,
  • checks: which endpoint returns errors (the service dashboard), which exceptions and database timeouts appear in the application log for the last 15 minutes, whether there was a deployment in the last hour,
  • workaround: roll back the last deployment or restart the application pool,
  • escalation: the application owner, if there is no improvement after 30 minutes.

The link to the runbook is part of the alert: in Prometheus an annotation (usually runbook_url), in Zabbix the trigger’s URL field.

Event logs

Purpose of a stream

Every log stream serves one of three purposes:

  1. Security and audit: logons, changes to permissions and groups, account creation. Complete, tamper-resistant events kept for a long time.
  2. Diagnostics: application and service errors. Needed quickly after a problem, rarely after a month.
  3. Metrics from logs: requests and errors per minute. Once counted, the counters remain and the events can be deleted.

A stream without a purpose does not go to the central system; it stays local with a short rotation.

Key Windows events

The whole Security log of a few domain controllers is usually a large share of the volume of all logs. Instead, the IDs that answer specific questions are collected:

IDEventWhy collect it
4624successful logonwho, from where and how; LogonType distinguishes interactive (2), network (3) and RDP (10)
4625failed logonpassword guessing, old passwords stored in services and tasks
4740account locked outthe source of the lockout in the Caller Computer Name field
4720user account createdaccounts created outside the agreed process
4728, 4732, 4756member added to a global, local or universal groupchanges to privileged groups
4672special privileges assigned to a logonuse of administrative accounts (only for their list)
1102audit log clearedan attempt to cover tracks
7045service installed (System log)malware persistence, remote command execution tools

The list is a minimum. Removals from groups (4729, 4733, 4757) and scheduled tasks (4698) have a low volume; process creation (4688) has a medium to high volume and needs a decision.

Events 4624 and 4672 on controllers have, according to Microsoft, a medium volume, mainly because of computer accounts and network logons. They are collected with a filter: 4624 only for RDP (LogonType 10) or accounts from the administrative list, 4672 only for accounts from that list.

In event 4625 the reason is in Status and SubStatus: 0xC000006A is a wrong password, 0xC0000064 a nonexistent user, 0xC0000072 a disabled account, 0xC0000234 a locked account; with the generic status 0xC000006D the detail is in SubStatus. Event 4740 is created on domain controllers, member servers and workstations. Controllers forward authentications with a bad password to the PDC Emulator, so lockouts are looked for on the controller that handled the logon and on the PDC Emulator.

Audit policy, log size and filtering

Events in the Security log appear only when the relevant advanced audit policy subcategory is enabled: Logon, Special Logon, User Account Management, Security Group Management; events 4698 additionally require Other Object Access Events, and 4688 requires Process Creation. “No Auditing” means no events regardless of the agent. The policies are deployed through GPO (Computer Configuration → Policies → Windows Settings → Security Settings → Advanced Audit Policy Configuration), and “Audit: Force audit policy subcategory settings…” enabled in Security Options protects them from being overridden by older category-level policies.

The state of the subcategories is shown by auditpol /get /category:* (it requires the “Manage auditing and security log” right, e.g. an administrator account). Logon and User Account Management should show “Success and Failure”, while “Success” is enough for Special Logon and Security Group Management, because those subcategories have no Failure events. “Success” alone on Logon means there will be no 4625 events. On a localised system the subcategory names are translated.

The local log is a buffer for when the central system or the link fails. If it holds a few hours, a weekend outage means lost events; the size is set by the GPO “Specify the maximum log file size (KB)” under Event Log Service → Security. On controllers 1 GB or more is usually needed, so the log covers a few days.

Filtering has to happen at the source: Windows agents and Windows Event Forwarding subscriptions accept an XPath query, which can be tested locally first (adm-nowak is an example administrative account).

$filter = @'
<QueryList>
  <Query Id="0" Path="Security">
    <Select Path="Security">*[System[(EventID=4625 or EventID=4740 or EventID=4720 or EventID=4728 or EventID=4732 or EventID=4756 or EventID=1102)]]</Select>
    <Select Path="Security">*[System[(EventID=4624)]] and *[EventData[Data[@Name='LogonType']='10' or Data[@Name='TargetUserName']='adm-nowak']]</Select>
    <Select Path="Security">*[System[(EventID=4672)]] and *[EventData[Data[@Name='SubjectUserName']='adm-nowak']]</Select>
  </Query>
  <Query Id="1" Path="System">
    <Select Path="System">*[System[(EventID=7045)]]</Select>
  </Query>
</QueryList>
'@
Get-WinEvent -FilterXml $filter -MaxEvents 20 |
    Select-Object TimeCreated, Id, MachineName

Sending everything and discarding it centrally wastes bandwidth and compute, and when the central system is overloaded, important events wait in the same queue as the noise.

Fig. 1. Filter and parse at the source, ship with acknowledgements, keep separate streams with their own retention.

Form: fields instead of text

A log as free text is easy to ship and hard to search. The agent sends the event with fields (EventID, TargetUserName, IpAddress, LogonType), and in the central system each is an attribute that can be filtered, counted and alerted on without regular expressions. In Graylog further processing is done by pipeline rules, added to a pipeline stage connected to the stream with Windows logs. Field names depend on the input and the agent, so one event has to be viewed in the search before a rule is written.

rule "AD: member added to a privileged group"
when
  has_field("EventID") AND
  (to_long($message.EventID) == 4728 OR
   to_long($message.EventID) == 4732 OR
   to_long($message.EventID) == 4756) AND
  contains(to_string($message.TargetUserName), "Admin", true)
then
  set_field("alert_category", "privileged_group_change");
  route_to_stream(name: "Security - critical");
end

In these events TargetUserName holds the group name and MemberName the member being added. A condition on part of the name is a simplification; protected groups are better kept in a lookup table.

Retention by category

Retention is a cost that grows linearly with time and volume, and categories need different periods:

CategoryExamplesCentral retentionRationale
Security and audit4624 (filtered), 4625, 4740, 4720, 4728/4732/4756, 4672, 1102, 7045365 days or as requiredinvestigations and audits reach months back
System and applicationservice errors, application logs30 daysdiagnosis and comparison with the previous month
High-volume diagnosticsdebug logs, proxy access logs7 days, or local onlyneeded right after an incident
Metrics from logsrequests and errors per minuteevents a few days, counters a yearthe number remains, not the event

In Graylog each category has its own stream writing to its own index set with its own rotation (e.g. 365 daily indices for security, 30 for applications).

Who looks

A log nobody looks at has only retrospective value. For security a few rules are enough: 1102 on any server always, 4728/4732/4756 for privileged groups outside the change window, 4740 e.g. more than 5 times for one account in an hour (usually an old password stored in a service or phone), 4625 e.g. more than 20 from one address in 10 minutes, and every 7045 on a domain controller. Notification levels and channels follow the table in the section on alerts.

Decisions and alternatives

ApproachWhen it makes senseLimitation
Alert on a symptom (error rate, response time)services visible to users; also catches unforeseen causesrequires a defined symptom and measurement on the user side
Alert on a causeirreversible resources or ones slow to fix: disk, certificates, licences, quorum, backup agefor elastic resources it produces noise (CPU at 90% with a normal response time)
Event filter at the source (XPath, Windows Event Forwarding)Security log of domain controllers and serversthe list of IDs and accounts needs maintenance
Send everything and filter centrallylow volume, no way to filter in the agentwastes bandwidth and compute; when overloaded, important events wait in the queue with the noise
Separate stream and index set for each categorydifferent retention periods for audit and diagnosticsmore objects to maintain
One shared index seta single, uniform retentiona year of storing noise, or audit events lost after a month
TCP input, preferably with TLS, or a protocol with acknowledgementssecurity eventsmore complex than UDP
UDP inputonly low-value datadrops events under load or when the receiver restarts, with no signal of the loss

Implementation

  1. Assign a purpose to every log stream; streams without one stay local with a short rotation.
  2. Choose metrics: four signals for every service visible to users, three questions for resources. Check: every collected metric has a question it answers, or goes on a dashboard.
  3. Enable the advanced audit policy subcategories through GPO together with “Audit: Force audit policy subcategory settings…” and set the Security log size so that it covers a few days (usually 1 GB or more on controllers). Check: auditpol /get /category:* shows the required values, and event 4625 appears on the controller within a day.
  4. Test the XPath query locally with Get-WinEvent -FilterXml, then deploy it in the agent or a Windows Event Forwarding subscription. Check: the volume of Security events from a controller after filtering is many times smaller than the whole log.
  5. A TCP input, preferably with TLS, fields instead of text, pipeline rules, separate streams and index sets with retention by category. Check: a single event has fields that can be filtered on and lands in the right stream.
  6. Alerts: a level for every alert, a threshold with a duration and a recovery expression, trigger dependencies or group_by and inhibit_rules, maintenance windows, an escalation rule and a link to the runbook. Check: “wakes someone” alerts number a few a month, and each has a runbook.
  7. Rules for the security events listed in the section “Who looks”, with a notification channel matching the level.

Limitations and pitfalls

  • Average instead of minimum in a trigger. 100-avg(/srv-app-01/system.cpu.util[,idle],15m)>90 does not mean “above 90% for 15 minutes”: the average exceeds the threshold when CPU sits at 100% for five minutes and at 88% for ten, and does not exceed it during an hour of steady 89%. “Always above” is written as min(…)>X, “always below” as max(…)<X.
  • UDP for security events. Under load or when the receiver restarts, events are lost with no signal of the loss. For security events a TCP input is used, preferably with TLS, or a protocol with acknowledgements.
  • Incomplete audit instead of no events. No 4625 events for a whole day on a controller usually means incomplete auditing (e.g. Logon with “Success” only), not that nobody mistyped a password; the same applies to “No Auditing”. The auditpol check precedes building alert rules.
  • Double storage of events in Graylog. A category stream that does not remove matched messages from the default stream stores every event twice. The option to remove matches from the default stream is enabled for every category stream.

Sources