Every organization running production systems has incidents. That's not negotiable. What is negotiable is whether they learn from them or repeat them until someone quits.
The postmortem is the mechanism that connects an incident to actual improvement. But in most companies I've worked with, postmortems are written for compliance, not for learning. They're drafted in a rush, filed in a wiki nobody visits, and the action items die in the backlog three sprints later.
Why most postmortems don't work
The pattern is always the same: a serious incident occurs, someone (usually whoever was on-call) writes a document describing what happened, a review meeting gets scheduled, "improvements" are identified, and two weeks later nobody remembers what was decided.
This fails for three reasons:
- Written for compliance, not understanding. The implicit goal is demonstrating that "something was done" after the incident. There's no real depth in the analysis.
- Generic action items. "Improve monitoring" is not an action item. It's a wish. Without an owner, a deadline and a definition of done, it's noise.
- No follow-up. Action items get created but nobody checks whether they were completed. Next time the same kind of incident hits, the team is in the exact same position.
Blameless culture: why blaming people destroys learning
When an engineer knows the postmortem will point at them as "the one who caused it," they stop being honest. They omit details, minimize their involvement, avoid mentioning decisions they made under pressure. The result is a document that doesn't reflect reality and therefore can't generate real improvements.
Blameless doesn't mean nobody is accountable. It means the analysis focuses on what failed in the system, not on who made the mistake. An engineer who deployed a change that caused an outage isn't "the one to blame." The right questions are: Why did the pipeline allow a broken deploy? Why was there no canary? Why wasn't the rollback automatic?
If your culture blames people, your postmortems are fiction. And fiction doesn't prevent incidents.
Anatomy of an effective postmortem
You don't need a 15-page template. You need to cover these elements with honesty and precision:
1. Timeline
A detailed chronology of the incident: when it was detected, what actions were taken, when it was resolved. No interpretations, just facts with timestamps. This is the foundation for all subsequent analysis.
2. Impact
Measured, not estimated. Number of users affected, duration of degradation, revenue loss if applicable, SLOs violated. If you can't measure the impact, you have an observability problem that's more serious than the incident itself.
3. Root cause
Not the surface-level cause ("the database went down") but the actual root ("the connection pool was configured for 50 connections on a service handling 200 concurrent requests, with no circuit breaker or backpressure"). Use the 5 whys technique if it helps, but don't stop at the first layer.
4. Contributing factors
Incidents rarely have a single cause. Contributing factors are the conditions that allowed the root cause to become a visible incident: missing alerts, outdated documentation, no runbooks, deploying on Friday at 5pm.
5. Action items
This is where most postmortems fall apart. Every action item needs:
- Specific description: not "improve monitoring" but "add an alert when the connection pool exceeds 80% capacity on service X."
- Owner: a person, not a team. If it belongs to the team, nobody does it.
- Deadline: a concrete date. No date means it doesn't exist.
- Priority: P0 (before the next deploy), P1 (this week), P2 (this sprint).
Common mistakes that kill effectiveness
- Generic action items. "Review the infrastructure" is a project, not an action item. Break it into concrete, executable tasks.
- No owner. "The platform team will handle it" means nobody will handle it. Assign a name.
- No follow-up. If you don't verify that action items were completed, the postmortem was a creative writing exercise.
- Writing too late. After 72 hours, details blur. Memory is unreliable. Write the timeline within the first 24 hours while facts are fresh.
- Confusing symptoms with causes. "The server ran out of memory" is a symptom. The cause is the memory leak in service Y that went undetected because there's no profiling in production.
Making action items actually happen
This is the step that separates organizations that learn from those that repeat. Two practices that work:
Weekly review: include postmortem action items in the team's weekly meeting. Not as the full agenda, but as a 5-minute checkpoint: What was completed? What's blocked? What needs re-prioritization?
Sprint integration: P0 and P1 action items enter the current sprint without negotiation. If they don't fit, something else gets pulled out. This requires management buy-in, and getting it is part of the tech lead's job. If your action items compete with features and always lose, the postmortem is decoration.
Some organizations go further: they don't allow a service involved in an incident to receive new features until all P0 action items are closed. It's aggressive, but it works.
A postmortem is not punishment. It's an investment in the system. Write it for compliance and it will fail. Write it for understanding and it will transform how your team responds to failure.
