System Design HLD-13.05
Incident response and postmortems
Running an incident: clear roles, one channel, mitigation before root cause, then blameless postmortems with concrete action items.
The big idea
At 13:45 the pager fires: error rates are spiking worldwide. Within minutes, fifteen engineers are in three chat channels, two are trying different fixes at once, support is asking for an update nobody has time to write, and the person who knows the system best is stuck explaining it on a call. Chaos makes outages longer.
IncidentsAn unplanned event that disrupts or degrades a service and needs an urgent, coordinated response. go better with a little structure decided in advance: clear roles so everyone knows who decides and who talks, one place for the response, and a firm priority to stop the bleeding first and understand it later. Afterwards, a blameless postmortemA written review after an incident covering what happened, its impact, how it was mitigated, contributing causes and follow-up actions, written without assigning blame to individuals. turns the painful hour into lasting improvements, by asking how the system allowed the failure rather than who caused it.
Think of firefighters at a scene. One incident commander directs; firefighters fight the fire; someone else handles the press and the neighbors. Nobody investigates the cause of the fire until it is out, and the investigation afterwards aims to improve codes and equipment, not to punish.
How it works
Step 1: Roles
- Incident commander (IC): owns the incident. Does not debug; coordinates, sets priorities, decides (roll back now or keep investigating?), and makes sure tasks have owners.
- Responders: engineers working the problem, reporting findings to the IC rather than acting unilaterally.
- Communications lead: writes status page updates, briefs support and leadership at a regular cadence, so the people fixing things are not interrupted.
- For long incidents, a scribe keeps a timeline of what was seen and done, which the postmortem will need.
Declare incidents early, with a severity level based on user impact. It is cheap to downgrade a false alarm and expensive to coordinate late.
Step 2: Mitigate first
The first goal is to stop user impact, not to find the root cause. The fastest mitigations are usually generic: roll back the last deploy ( Deployment strategies ), turn off a feature flag, fail over to another zone or region ( Redundancy, failover and multi-AZ ), shed load ( Load shedding and graceful degradation ), or block an abusive client. If something changed recently, undoing it is a good first bet even before you are sure it is the cause. Root cause analysis can happen once users are safe.
Step 3: Communicate
Users and internal teams need updates even when there is no news: what is affected, what is being done, when the next update is. Regular, honest updates on a status page reduce support load and keep trust. Inside the response, keep one incident channel and one call, and keep side discussions out of them.
Step 4: Blameless postmortems
After significant incidents (user-visible impact, data loss, a manual intervention, a monitoring failure), write a postmortem:
- Summary and impact: what users experienced, for how long, how many.
- Timeline: when it started, when it was detected, key decisions, when it was mitigated and resolved.
- Contributing causes: usually several. "An engineer pushed a bad rule" is not a root cause; why could one change reach every server at once, why did tests not catch it, why did detection take 15 minutes?
- Action items: specific, owned and tracked, such as "add staged rollout for rule changes", rather than "be more careful".
Blameless means assuming everyone acted reasonably with the information they had. People who fear blame hide mistakes, and hidden mistakes repeat. Systems that let one person's routine action cause an outage are the thing to fix.
Going deeper
Measure the response
Track time to detect, time to mitigate and time to resolve across incidents, and how many incidents were found by monitoring versus by customers. These numbers show whether investments in alerting, rollback speed and runbooks are paying off.
Practice
Incident response is a skill. Game days and simulated incidents ( Chaos engineering ) let people practice the IC and comms roles before a real outage, and postmortem reading sessions spread the lessons beyond the team that had the incident.
Trade-offs
| Option | Wins when | Loses when |
|---|---|---|
| Ad hoc response | Tiny teams and minor issues | Duplicate work, slow decisions and poor communication in big incidents |
| Formal roles (IC, comms, responders) | Any incident with significant user impact | Some overhead for small incidents; needs training |
| Mitigate first (rollback, flag off) | Ending user impact quickly | Can erase evidence; capture logs and state before rolling back when possible |
| Blameless postmortems | Honest analysis and systemic fixes | Time to write and review; action items must be followed through |
Common mistakes
- Debugging for an hour before trying the obvious rollback.
- The most senior engineer acting as IC and debugger at the same time.
- No status updates, so support and customers are left guessing.
- Postmortems that name a person as the root cause.
- Action items with no owner or due date, which never get done.
In the wild
Cloudflare
Details of the Cloudflare outage on July 2, 2019 (opens in a new tab)A new firewall rule containing a regular expression that backtracked enormously was deployed globally at once and exhausted CPU across Cloudflare's network, causing 502 errors and an 80 percent drop in traffic for 27 minutes. The postmortem gives a minute-by-minute timeline: the first page went out at 13:45, the WAF was identified at 14:00, a global WAF kill was executed at 14:07, and traffic was back to normal by 14:09. Its action items include re-adding CPU usage protection, profiling rule performance in tests, moving to a regex engine with run-time guarantees, and staged rollouts for rule changes.
This chapter defines a postmortem as a written record of an incident, its impact, the mitigation actions, root causes and follow-up actions, and lists common triggers such as user-visible downtime beyond a threshold, any data loss, on-call intervention like a rollback, and monitoring failures. It describes blameless postmortems as focusing on contributing causes without indicting any individual or team, assuming everyone acted with good intentions on the information they had. Google spreads lessons through reviews, a "postmortem of the month" and postmortem reading clubs.
Check yourself
-
Errors spiked right after a deploy 10 minutes ago. What is the best first move?
Show answer
B. Roll back the deploy to stop user impact, then investigate Mitigation comes first; undoing a recent change is the fastest likely fix.
-
What is the incident commander's main job?
Show answer
B. Coordinating the response, setting priorities and making decisions The IC coordinates; responders debug and the comms lead communicates.
-
Which is a blameless way to state a cause?
Show answer
B. A rule change could reach every server at once with no staged rollout or performance test It describes how the system allowed the failure, which points to fixes.
-
What makes a good action item?
Show answer
B. A specific change with an owner and a due date, such as adding staged rollout for rule deploys Concrete, owned, tracked changes are what prevent recurrence.
Recap
- Declare incidents early and assign roles: incident commander, responders, communications lead.
- Use one channel and regular updates to users and stakeholders.
- Mitigate first with rollback, flags, failover or shedding; find root cause afterwards.
- Write blameless postmortems with a timeline, contributing causes and owned action items.
- Measure detection and mitigation times, and practice the response.