Monitoring and event logs
Choosing metrics and building alerts that require action, and a strategy for Windows event logs: which events to collect, how to filter and process them, and how long to keep them.
Metrics collected without asking which question they answer, and alerts set without asking what the person who receives them is supposed to do, end in a muted notification channel in which the alert about a database disk filling up is lost. A central log system launched on the rule “collect everything” turns into an expensive warehouse of noise, and the event being looked for is often missing altogether because the system never generated it. This document describes a small set of signals and the rules for building alerts, together with a strategy for event logs: the purpose of a stream, the choice of Windows events, filtering, form, retention and the recipient.
Metrics: four signals and three questions
For a service visible to users, four signals are enough: latency, traffic, errors and saturation (the golden signals from Google’s book on site reliability engineering). For every resource, three questions from Brendan Gregg’s USE method are enough: utilisation, saturation, errors.
For a service, latency (a response-time percentile), traffic, the error rate and the saturation of the narrowest resource, e.g. a database connection pool, are measured. For a resource, e.g. a processor, disk or network interface, utilisation, saturation (queue length) and errors are measured.
Latency is measured with a percentile, not an average: a 300 ms average can hide that every twentieth user waits five seconds. Utilisation at 100% is not a problem with zero saturation, while saturation at low average utilisation is often caused by short bursts hidden in the average, or by a misconfiguration, e.g. a connection pool that is too small. In a Windows environment, replication latency and errors between domain controllers, failed authentications over time, clock skew and the age of the last successful backup of each system are added.
Alerts
Levels and action
Every alert answers the question of what the person who receives it is supposed to do. “Nothing, just know” means information for a dashboard, while “check whether it is serious” means the alert is too broad.
Alerts fire on symptoms visible to users, not on causes: the error rate and response time catch every cause, including unforeseen ones. CPU at 90% with a normal response time is no reason to wake anyone; resource metrics serve diagnosis after a symptom alert. The exception is resources whose exhaustion is irreversible or slow to fix: disk space, certificates, licences, cluster quorum and backup age. A cause alert is right there, because the symptom would appear too late.
| Level | When | Channel | Reaction | Example |
|---|---|---|---|---|
| Wakes someone | the service is down for users or about to be | phone or on-call app, any hour | minutes | 5xx error share above 5% for 10 minutes |
| Ticket | needs attention within the working day | a ticket in the ITSM system | same or next working day | the disk will fill within 7 days |
| Dashboard only | trend, context, warning | dashboard, weekly report | no notification | CPU above 90% for 15 minutes |
An alert without an assigned level goes to the lowest one. “Wakes someone” alerts should number a few a month, not a few a day.
A threshold with a duration in Zabbix
A round number is not a threshold; it comes from history. If a disk fills by 1% a week, an alert at 80% gives months of margin, and at 5% a day it is already too late. Every alert also has a time condition: a brief CPU spike during a backup is not an incident, 15 minutes above the threshold is. The triggers below use the syntax introduced in Zabbix 5.4, valid in 6.0 LTS and later.
# CPU above 90% for the whole 15 minutes (every value in the period); level: dashboard only
# The system.cpu.util key without parameters counts only user time on Linux, so the load is derived from idle time
100-max(/srv-app-01/system.cpu.util[,idle],15m)>90
# The “Portal” web scenario has been failing for 5 minutes
min(/srv-web-01/web.test.fail[Portal],5m)>0
# At the rate of the last day, /var will reach 95% within 7 days
timeleft(/srv-app-01/vfs.fs.size[/var,pused],24h,95)<7d
The first trigger, based on a cause, serves diagnosis. The second describes a symptom and gets the “wakes someone” or “ticket” level depending on the service. The third is always “ticket”: it leaves a week to react during working hours. To stop a problem flipping around the threshold, a trigger gets a recovery expression, e.g. 100-min(/srv-app-01/system.cpu.util[,idle],10m)<80.
In Prometheus the equivalent is the for: field of an alerting rule, and the route in Alertmanager is chosen by matching alert labels (e.g. severity, which the rule sets and which is not built in).
Grouping, maintenance windows and escalation
The failure of one switch generates a hundred unreachable-host alerts for everything behind it, while the on-call person needs one. In Zabbix trigger dependencies do this, in Alertmanager group_by and inhibit_rules. Planned work needs a maintenance period in Zabbix or a silence in Alertmanager, created before the restart, with the ticket number in the comment. A “wakes someone” alert not acknowledged within a set time (e.g. 15 minutes) goes to the next person; the escalation rule is written down so nobody has to wonder at three in the morning whom to wake.
Runbook
An alert without a runbook is a test of the on-call person’s knowledge under the worst conditions. A runbook is short and has a fixed structure of five fields:
- meaning: what the alert signifies, e.g. more than 5% of portal responses have ended in a 5xx error for 10 minutes,
- impact: who and what is affected, e.g. some users cannot sign in or save data,
- checks: which endpoint returns errors (the service dashboard), which exceptions and database timeouts appear in the application log for the last 15 minutes, whether there was a deployment in the last hour,
- workaround: roll back the last deployment or restart the application pool,
- escalation: the application owner, if there is no improvement after 30 minutes.
The link to the runbook is part of the alert: in Prometheus an annotation (usually runbook_url), in Zabbix the trigger’s URL field.
Event logs
Purpose of a stream
Every log stream serves one of three purposes:
- Security and audit: logons, changes to permissions and groups, account creation. Complete, tamper-resistant events kept for a long time.
- Diagnostics: application and service errors. Needed quickly after a problem, rarely after a month.
- Metrics from logs: requests and errors per minute. Once counted, the counters remain and the events can be deleted.
A stream without a purpose does not go to the central system; it stays local with a short rotation.
Key Windows events
The whole Security log of a few domain controllers is usually a large share of the volume of all logs. Instead, the IDs that answer specific questions are collected:
| ID | Event | Why collect it |
|---|---|---|
| 4624 | successful logon | who, from where and how; LogonType distinguishes interactive (2), network (3) and RDP (10) |
| 4625 | failed logon | password guessing, old passwords stored in services and tasks |
| 4740 | account locked out | the source of the lockout in the Caller Computer Name field |
| 4720 | user account created | accounts created outside the agreed process |
| 4728, 4732, 4756 | member added to a global, local or universal group | changes to privileged groups |
| 4672 | special privileges assigned to a logon | use of administrative accounts (only for their list) |
| 1102 | audit log cleared | an attempt to cover tracks |
| 7045 | service installed (System log) | malware persistence, remote command execution tools |
The list is a minimum. Removals from groups (4729, 4733, 4757) and scheduled tasks (4698) have a low volume; process creation (4688) has a medium to high volume and needs a decision.
Events 4624 and 4672 on controllers have, according to Microsoft, a medium volume, mainly because of computer accounts and network logons. They are collected with a filter: 4624 only for RDP (LogonType 10) or accounts from the administrative list, 4672 only for accounts from that list.
In event 4625 the reason is in Status and SubStatus: 0xC000006A is a wrong password, 0xC0000064 a nonexistent user, 0xC0000072 a disabled account, 0xC0000234 a locked account; with the generic status 0xC000006D the detail is in SubStatus. Event 4740 is created on domain controllers, member servers and workstations. Controllers forward authentications with a bad password to the PDC Emulator, so lockouts are looked for on the controller that handled the logon and on the PDC Emulator.
Audit policy, log size and filtering
Events in the Security log appear only when the relevant advanced audit policy subcategory is enabled: Logon, Special Logon, User Account Management, Security Group Management; events 4698 additionally require Other Object Access Events, and 4688 requires Process Creation. “No Auditing” means no events regardless of the agent. The policies are deployed through GPO (Computer Configuration → Policies → Windows Settings → Security Settings → Advanced Audit Policy Configuration), and “Audit: Force audit policy subcategory settings…” enabled in Security Options protects them from being overridden by older category-level policies.
The state of the subcategories is shown by auditpol /get /category:* (it requires the “Manage auditing and security log” right, e.g. an administrator account). Logon and User Account Management should show “Success and Failure”, while “Success” is enough for Special Logon and Security Group Management, because those subcategories have no Failure events. “Success” alone on Logon means there will be no 4625 events. On a localised system the subcategory names are translated.
The local log is a buffer for when the central system or the link fails. If it holds a few hours, a weekend outage means lost events; the size is set by the GPO “Specify the maximum log file size (KB)” under Event Log Service → Security. On controllers 1 GB or more is usually needed, so the log covers a few days.
Filtering has to happen at the source: Windows agents and Windows Event Forwarding subscriptions accept an XPath query, which can be tested locally first (adm-nowak is an example administrative account).
$filter = @'
<QueryList>
<Query Id="0" Path="Security">
<Select Path="Security">*[System[(EventID=4625 or EventID=4740 or EventID=4720 or EventID=4728 or EventID=4732 or EventID=4756 or EventID=1102)]]</Select>
<Select Path="Security">*[System[(EventID=4624)]] and *[EventData[Data[@Name='LogonType']='10' or Data[@Name='TargetUserName']='adm-nowak']]</Select>
<Select Path="Security">*[System[(EventID=4672)]] and *[EventData[Data[@Name='SubjectUserName']='adm-nowak']]</Select>
</Query>
<Query Id="1" Path="System">
<Select Path="System">*[System[(EventID=7045)]]</Select>
</Query>
</QueryList>
'@
Get-WinEvent -FilterXml $filter -MaxEvents 20 |
Select-Object TimeCreated, Id, MachineName
Sending everything and discarding it centrally wastes bandwidth and compute, and when the central system is overloaded, important events wait in the same queue as the noise.
Form: fields instead of text
A log as free text is easy to ship and hard to search. The agent sends the event with fields (EventID, TargetUserName, IpAddress, LogonType), and in the central system each is an attribute that can be filtered, counted and alerted on without regular expressions. In Graylog further processing is done by pipeline rules, added to a pipeline stage connected to the stream with Windows logs. Field names depend on the input and the agent, so one event has to be viewed in the search before a rule is written.
rule "AD: member added to a privileged group"
when
has_field("EventID") AND
(to_long($message.EventID) == 4728 OR
to_long($message.EventID) == 4732 OR
to_long($message.EventID) == 4756) AND
contains(to_string($message.TargetUserName), "Admin", true)
then
set_field("alert_category", "privileged_group_change");
route_to_stream(name: "Security - critical");
end
In these events TargetUserName holds the group name and MemberName the member being added. A condition on part of the name is a simplification; protected groups are better kept in a lookup table.
Retention by category
Retention is a cost that grows linearly with time and volume, and categories need different periods:
| Category | Examples | Central retention | Rationale |
|---|---|---|---|
| Security and audit | 4624 (filtered), 4625, 4740, 4720, 4728/4732/4756, 4672, 1102, 7045 | 365 days or as required | investigations and audits reach months back |
| System and application | service errors, application logs | 30 days | diagnosis and comparison with the previous month |
| High-volume diagnostics | debug logs, proxy access logs | 7 days, or local only | needed right after an incident |
| Metrics from logs | requests and errors per minute | events a few days, counters a year | the number remains, not the event |
In Graylog each category has its own stream writing to its own index set with its own rotation (e.g. 365 daily indices for security, 30 for applications).
Who looks
A log nobody looks at has only retrospective value. For security a few rules are enough: 1102 on any server always, 4728/4732/4756 for privileged groups outside the change window, 4740 e.g. more than 5 times for one account in an hour (usually an old password stored in a service or phone), 4625 e.g. more than 20 from one address in 10 minutes, and every 7045 on a domain controller. Notification levels and channels follow the table in the section on alerts.
Decisions and alternatives
| Approach | When it makes sense | Limitation |
|---|---|---|
| Alert on a symptom (error rate, response time) | services visible to users; also catches unforeseen causes | requires a defined symptom and measurement on the user side |
| Alert on a cause | irreversible resources or ones slow to fix: disk, certificates, licences, quorum, backup age | for elastic resources it produces noise (CPU at 90% with a normal response time) |
| Event filter at the source (XPath, Windows Event Forwarding) | Security log of domain controllers and servers | the list of IDs and accounts needs maintenance |
| Send everything and filter centrally | low volume, no way to filter in the agent | wastes bandwidth and compute; when overloaded, important events wait in the queue with the noise |
| Separate stream and index set for each category | different retention periods for audit and diagnostics | more objects to maintain |
| One shared index set | a single, uniform retention | a year of storing noise, or audit events lost after a month |
| TCP input, preferably with TLS, or a protocol with acknowledgements | security events | more complex than UDP |
| UDP input | only low-value data | drops events under load or when the receiver restarts, with no signal of the loss |
Implementation
- Assign a purpose to every log stream; streams without one stay local with a short rotation.
- Choose metrics: four signals for every service visible to users, three questions for resources. Check: every collected metric has a question it answers, or goes on a dashboard.
- Enable the advanced audit policy subcategories through GPO together with “Audit: Force audit policy subcategory settings…” and set the Security log size so that it covers a few days (usually 1 GB or more on controllers). Check:
auditpol /get /category:*shows the required values, and event 4625 appears on the controller within a day. - Test the XPath query locally with
Get-WinEvent -FilterXml, then deploy it in the agent or a Windows Event Forwarding subscription. Check: the volume of Security events from a controller after filtering is many times smaller than the whole log. - A TCP input, preferably with TLS, fields instead of text, pipeline rules, separate streams and index sets with retention by category. Check: a single event has fields that can be filtered on and lands in the right stream.
- Alerts: a level for every alert, a threshold with a duration and a recovery expression, trigger dependencies or
group_byandinhibit_rules, maintenance windows, an escalation rule and a link to the runbook. Check: “wakes someone” alerts number a few a month, and each has a runbook. - Rules for the security events listed in the section “Who looks”, with a notification channel matching the level.
Limitations and pitfalls
- Average instead of minimum in a trigger.
100-avg(/srv-app-01/system.cpu.util[,idle],15m)>90does not mean “above 90% for 15 minutes”: the average exceeds the threshold when CPU sits at 100% for five minutes and at 88% for ten, and does not exceed it during an hour of steady 89%. “Always above” is written asmin(…)>X, “always below” asmax(…)<X. - UDP for security events. Under load or when the receiver restarts, events are lost with no signal of the loss. For security events a TCP input is used, preferably with TLS, or a protocol with acknowledgements.
- Incomplete audit instead of no events. No 4625 events for a whole day on a controller usually means incomplete auditing (e.g. Logon with “Success” only), not that nobody mistyped a password; the same applies to “No Auditing”. The
auditpolcheck precedes building alert rules. - Double storage of events in Graylog. A category stream that does not remove matched messages from the default stream stores every event twice. The option to remove matches from the default stream is enabled for every category stream.