The first hour of an outage: who leads, what to communicate and what to roll back

Monday, 8:40. Accounting reports that the ERP system returns an error after logon. Three administrators start looking for the cause independently, each in a different place. The IT phone rings non-stop, and management learns about the outage from users. After 40 minutes someone notices that a new application configuration was deployed at 7:55. Rolling the change back takes five minutes.

In a small team the length of an outage rarely depends on how hard the technical problem is. It depends on whether someone is leading, whether users know what is going on, and whether the first question was “what changed”. This post is a plan for the first hour for a team of two to a few people, with templates to copy.

Minute 0: confirm the outage and name a lead

The first step is to confirm that the problem is real and wider than one workstation: a second user, a second site, an alert in monitoring. Then comes one sentence that changes the mode of work: “I am declaring an outage, I am leading”.

The lead does not have to be the most experienced person on the team. Their job is coordination, communication and notes, not diagnosis. A rule that works in small teams: the first person to confirm the outage leads until they explicitly hand the role over to someone else.

Role	Tasks	In a two-person team
Lead	decisions, messages, notes, keeping time	person A
Diagnosis	scope, recent changes, rollback, tests	person B
User contact	collecting reports, answering with a link to the message	the lead or the helpdesk

From this moment on, everything about the outage happens in one place: a single thread in the team chat or a single document. The lead copies anything agreed on the phone there straight away.

Minute 5: establish the scope

Before anyone starts fixing, the team answers four questions: who is affected (everyone, one site, one group), which systems, since exactly when (the time from monitoring or a log, not from the first report), and what still works. What still works narrows the search as effectively as what does not.

Symptom	First suspicion
One application is down for everyone	recent deployment, certificate, database, full disk
Many systems at once at one site	link, switch, DNS, power
Logon fails across many services	domain controllers, DNS, system time, expired service account password
Everything works, but slowly	storage, a running backup job, a saturated link
The problem occurs only outside the office	VPN, public DNS, gateway certificate

Minute 15: the first message

The first message goes out within 15 minutes at the latest, even if the team knows nothing about the cause yet. Users do not need a diagnosis, only to know that the problem is known, who is handling it and when the next update will come.

[OUTAGE] <system>, since <time>

What is not working: <symptom from the user's point of view, e.g. error after logging on to ERP>
Who is affected: <everyone / site / department>; not affected: <if known>
Workaround: <e.g. invoices are issued manually following the intranet guide> or: none
Handled by: IT team, led by <full name>
Next update: at <time, e.g. 9:30>, even if nothing changes

Please do not report this problem separately.

Further messages go out at the announced rhythm, usually every 30 minutes. A message saying “no change, work continues, next update at 10:00” is better than silence, because silence generates phone calls that take time away from the people diagnosing.

Minutes 15–30: what changed

Most outages in a small environment are caused by a change: a deployment, an update, a configuration change, an expired certificate or password. That is why the first question for the person diagnosing is “what changed in the last few hours”, not “why is it not working”.

On Windows servers, changes leave traces in the System log:

# 19: update installed; 1074: restart or shutdown initiated by a process or user
# 7040: service start type changed; 7045: new service installed
Get-WinEvent -FilterHashtable @{
    LogName   = 'System'
    Id        = 19, 1074, 7040, 7045
    StartTime = (Get-Date).AddHours(-6)
} | Select-Object TimeCreated, Id, ProviderName, @{ Name = 'Summary'; Expression = { ($_.Message -split "`n")[0] } }

The same block run through Invoke-Command -ComputerName srv-app-01, srv-app-02 -ScriptBlock { ... } checks several servers at once.

In a Kubernetes cluster the first source is events, sorted from oldest to newest:

kubectl get events -A --sort-by=.lastTimestamp
kubectl rollout history deployment/erp-api -n erp

Events are kept for one hour by default, so after a longer time you have to look for them in the central logging system, if they are shipped there.

The infrastructure repository and Argo CD show what was deployed and when:

git log --all --since="6 hours ago" --date=iso --pretty=format:"%h %ad %an %s"
argocd app history erp

argocd app history shows the ID, date and revision of every sync. If the 7:55 revision precedes the first report by a few minutes, the hypothesis is ready.

Roll back first, look for the cause later

If there was a change in the window before the outage, rolling it back is the first action, not the last resort. Restoring the service is the goal of the first hour; understanding the cause is the goal of the post-incident review.

Change	Rollback	Notes
Deployment through GitOps	git revert <commit> and sync	the repository history stays complete
Windows update	wusa /uninstall /kb:<number>	packages combined with the servicing stack update are removed with DISM /Online /Remove-Package
Group Policy change	Restore-GPO from a backup or disabling the GPO link	workstations see the change after a policy refresh
Change on a machine with a checkpoint	apply the checkpoint	writes since the checkpoint are lost
Firewall rule, DNS record	restore the previous value	requires the previous value to have been recorded

In the GitOps model, a rollback is a revert in the repository. argocd app rollback works only with automated sync disabled and leaves the application out of sync with the repository, so it suits the situation in which the repository is unavailable.

There are two exceptions: changes that cannot be undone (a database schema migration, data already processed by the new version), and situations in which rolling back is riskier than the outage. Then the lead decides and records the decision in the notes.

Notes with timestamps

The lead records every finding and every action with the time and the person’s initials. The notes serve three purposes: messages, handing over the lead, and the post-incident review.

08:41 MK  Report from accounting: ERP returns error 502 after logon.
08:44 MK  Confirmed on two workstations and from the server network. Outage declared, MK leading.
08:52 MK  First message sent: email to all staff, text message to managers.
08:55 PW  argocd app history erp: sync at 07:55, connection pool change (a1b2c3d).
09:01 PW  git revert a1b2c3d pushed, sync finished 09:04.
09:07 MK  Accounting confirms: logon works. Observing until 09:40.
09:10 MK  Message: service restored, cause under analysis.

When to escalate

Escalation thresholds are set before the outage, so that nobody debates during it whether “it is time yet”.

Situation	To whom	When
No hypothesis after 30 minutes	a second team member, the system vendor	once the threshold passes
Outage of a vendor or cloud service	a ticket with the vendor at the highest priority, in parallel with diagnosis	within the first 15 minutes
Suspected security incident: encrypted files, unknown accounts, unusual logons	management and the incident response procedure; disconnect machines from the network, do not shut them down	immediately
Possible personal data breach	the data protection officer; notification to the supervisory authority within 72 hours	immediately
Outage longer than the system’s RTO	management, decision on contingency mode	once the RTO is exceeded

A short post-incident review

Within a few working days the team meets for 30–45 minutes. The goal is not to find someone to blame, but to establish what made the outage last as long as it did and what will shorten the next one.

Outage:             <system>, <date>, <from–to>, total <duration>
Impact:             <who was affected, what did not work, was data lost>
Timeline:           <from the notes: detection, declaration, message, hypothesis, rollback, confirmation>
Cause:              <the direct one and the one that allowed it>
What worked:        <e.g. revert in 6 minutes, message in 11 minutes>
What did not work:  <e.g. nobody knew about the deployment at 07:55>
Actions:            <action>, <owner>, <due date>

Actions from the review go onto the team’s normal task list, with a due date. A review without actions is just a conversation.

Check your own environment

The most recently installed updates on several servers, without logging on to each of them:

Invoke-Command -ComputerName srv-app-01, srv-app-02 -ScriptBlock {
    Get-HotFix | Sort-Object InstalledOn -Descending | Select-Object -First 3 HotFixID, InstalledOn
} | Format-Table PSComputerName, HotFixID, InstalledOn

A good result is a list that can be matched against the change calendar. If answering “what changed in the last day” takes more than a few minutes, during an outage it will take much longer.

Warnings in the cluster over the last hour:

kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp

On a quiet day the list should be short and understandable. A long list of recurring warnings that nobody explains will hide the one that matters during an outage.

Application sync state in Argo CD:

argocd app list

A good result: every application shows STATUS Synced and HEALTH Healthy. An application that is permanently OutOfSync indicates changes made outside the repository, which git log will not show during an outage.