post-mortem

**A post-mortem** (also called a retrospective or incident review) is a structured **after-incident analysis** conducted to understand what happened, why it happened, and what changes will prevent recurrence. It is the primary mechanism for **organizational learning** from production failures. **Post-Mortem Structure** - **Incident Summary**: What happened, when, and who was affected. Include duration, severity, and blast radius. - **Timeline**: Chronological sequence of events from detection through resolution. Include timestamps, actions taken, and who did what. - **Root Cause Analysis**: The underlying cause(s) — not just "the server crashed" but why it crashed and why safeguards didn't prevent impact. - **Impact Assessment**: Quantified impact — users affected, revenue lost, SLO budget consumed, safety implications. - **What Went Well**: Highlight things that worked — effective alerts, fast response, good runbooks. - **What Went Poorly**: Areas where the response was slow, confused, or ineffective. - **Action Items**: Specific, assigned, time-bound improvements to prevent recurrence. Each action item has an owner and deadline. **Core Principles** - **Blameless**: Focus on systemic issues, not individual mistakes. "Why did the system allow this to happen?" not "Who made the mistake?" - **Thorough**: Dig deep into root causes using the **"Five Whys"** technique or other root cause analysis methods. - **Actionable**: Every post-mortem produces concrete action items, not vague promises. - **Shared**: Post-mortems are shared widely to spread learning across the organization. **Post-Mortems for AI Systems** - **Model Regression Post-Mortem**: Why did the new model version perform worse? What evaluation gap allowed it through? - **Safety Incident Post-Mortem**: How did harmful content bypass safety filters? What guardrails need strengthening? - **Cost Post-Mortem**: What caused unexpected spending? How can cost controls prevent recurrence? **Best Practices** - **Schedule Within 48 Hours**: Conduct the post-mortem while details are fresh. - **Include All Participants**: Everyone involved in the incident response should attend. - **Track Action Items**: Use a tracking system to ensure action items are completed — unfinished action items from post-mortems undermine the entire process. Post-mortems are the **highest-leverage activity** for improving system reliability — each incident, properly analyzed, makes the system and team stronger.