Deploying a central log system usually starts with the decision “collect everything, sort it out later”. A few months on, the disk is full, searches take minutes, and the answer to a simple question about one user’s logins has to be found among millions of Group Policy refresh events. A logging strategy is a set of decisions made before go-live, not after.
Question one: what for
Every log stream should have an answer to the question of what it will be used for. In practice there are three answers:
- Security and audit. Logins, permission changes, changes to privileged groups, account creation, access to sensitive resources. These events must be complete, tamper-proof and stored for a long time, often for formal compliance reasons.
- Diagnostics. Application errors, system events, service logs. Needed quickly after a problem occurs, rarely needed a month later.
- Metrics derived from logs. Request counts, response times, errors per minute. Once counted, the logs themselves can be discarded; the counters stay.
A stream with no assigned purpose should not go to the central system. It can stay local with a short rotation.
Question two: what exactly from domain controllers
Active Directory domain controllers generate enormous volumes of events, most of which have no value beyond local diagnostics. Collecting the whole Security log from three controllers can produce more data than all other servers combined.
Instead of the whole log, pick the event IDs that answer specific questions:
| Question | Events |
|---|---|
| Who logged in and from where | 4624, 4625, 4768, 4769, 4771 |
| Who changed group membership | 4728, 4729, 4732, 4733, 4756, 4757 |
| Who created or modified accounts | 4720, 4722, 4725, 4726, 4738, 4740 |
| Who changed passwords | 4723, 4724 |
| Policy and permission changes | 4670, 4719, 4739 |
The collecting agent should filter at the source. Sending everything and filtering centrally wastes bandwidth, storage and compute.
Question three: in what form
A log as free text is easy to ship and hard to search. A log with parsed fields, where user name, source address and outcome are separate attributes, allows queries, dashboards and alerts without regular expressions.
Parsing should happen as close to the source as possible, where the event structure is known. An agent on a Windows server knows that TargetUserName is the user name. A central system that receives text has to guess.
The transport format is secondary, but the protocol is not. UDP is simpler and cheaper, but it drops events under load or when the receiver restarts, and nobody finds out. For security events use TCP or a protocol with acknowledgements, even if it means more configuration.
Question four: for how long
Retention is a cost that grows linearly with time and volume. Different streams need different periods:
- security events: a year or longer, depending on requirements,
- application and system logs: two to four weeks,
- high-volume diagnostic logs: a few days, or local only.
This requires separate streams or indices in the central system, each with its own retention policy. One shared index with one retention means either paying for a year of stored noise or losing audit events after a month.
Question five: who is watching
A log nobody looks at and that raises no alerts has purely retrospective value. That may be sufficient for audit events, but diagnostics and security need rules: a series of failed logins, an account added to Domain Admins outside a maintenance window, a service account logging in from an unknown address. A handful of well-chosen alerts is worth more than a dashboard with twenty charts.
Summary
A logging strategy comes down to five decisions per stream: what for, what exactly, in what form, for how long, and who is watching. Made before deployment, they take one afternoon. Made afterwards, they require data migration and convincing the team that something that “worked” needs to be rebuilt.