incident response

**Incident response** is the structured process for **detecting, managing, resolving, and learning from** production outages, degradations, and security events. For AI systems, effective incident response is critical because model failures can impact users at scale and may involve safety concerns beyond typical software incidents. **Incident Response Phases** - **Detection**: Automated alerts, user reports, or monitoring dashboards identify a problem. The faster detection happens, the less user impact. - **Triage**: Assess severity and impact — how many users are affected? Is safety compromised? What's the blast radius? - **Mitigation**: Apply immediate fixes to restore service — rollback, restart, scale up, switch to fallback, disable problematic features. - **Root Cause Investigation**: While mitigation handles symptoms, investigate the underlying cause. - **Resolution**: Apply a permanent fix that addresses the root cause. - **Post-Mortem**: Document what happened, why, how it was resolved, and what changes will prevent recurrence. **Incident Severity Levels** - **SEV-1 (Critical)**: Complete service outage or major safety incident. All-hands response, executive communication. - **SEV-2 (Major)**: Significant degradation affecting many users. On-call team response with regular status updates. - **SEV-3 (Minor)**: Partial impact or non-critical degradation. Addressed during business hours. - **SEV-4 (Low)**: Cosmetic or minor issues. Tracked but not urgently addressed. **AI-Specific Incident Types** - **Model Quality Regression**: A deployed model produces worse outputs than its predecessor. - **Safety Failure**: The model generates harmful, toxic, or dangerous content that bypasses safety filters. - **Hallucination Spike**: Increased rate of factually incorrect responses. - **Provider Outage**: External LLM API provider is down or degraded. - **Cost Incident**: Unexpected spending spike due to prompt injection, loops, or abuse. - **Data Leak**: Model outputs contain sensitive information from training data. **Incident Communication** - **Internal**: Dedicated incident Slack channel, regular status updates (every 30 min for SEV-1). - **External**: Status page updates, customer communication for significant incidents. **Tools**: **PagerDuty**, **Incident.io**, **Rootly**, **Statuspage**, **Jira** (for tracking follow-up actions). Effective incident response is a **team discipline** — it requires practice, clear roles, and continuous improvement through honest post-mortems.