root cause analysis for systems

**Root cause analysis (RCA)** is a systematic investigation technique used to identify the **fundamental underlying cause(s)** of a system failure, rather than just addressing the immediate symptoms. In AI/ML systems, RCA is essential because failures often have complex, multi-layered causes. **RCA Methods** - **Five Whys**: Repeatedly ask "why?" to drill deeper into the cause chain. Example: "The model returned nonsense" → Why? "The prompt was malformed" → Why? "The template variable was null" → Why? "The user session expired" → Why? "The session timeout was too short for long-running queries." - **Fishbone Diagram (Ishikawa)**: Categorize potential causes into groups — People, Process, Technology, Data, Environment — and systematically analyze each branch. - **Fault Tree Analysis**: Build a tree of events that could lead to the failure, with AND/OR gates showing how causes combine. - **Timeline Analysis**: Reconstruct the exact sequence of events leading to the failure to identify the triggering change or condition. **Common Root Causes in AI Systems** - **Data Quality**: Training data issues (contamination, bias, distribution shift) that cascade into model behavior problems. - **Configuration Changes**: Updated system prompts, modified parameters, rotated API keys that inadvertently break functionality. - **Deployment Issues**: Incomplete rollouts, version mismatches, missing dependencies, incompatible model-tokenizer pairs. - **Capacity**: Insufficient GPU memory, exceeded rate limits, queue overflow under unexpected load. - **External Dependencies**: Third-party API changes, provider outages, upstream data source modifications. **RCA Best Practices** - **Look for Systemic Issues**: Individual errors are symptoms — the root cause is usually a **process or system gap** that allowed the error to have impact. - **Multiple Root Causes**: Complex incidents often have multiple contributing factors — don't stop at the first cause you find. - **Actionable Outcomes**: Every root cause should map to a specific preventive action — if you can't act on it, dig deeper. - **Avoiding Blame**: Focus on "what" and "how," not "who" — punishing individuals discourages honest reporting. Root cause analysis transforms every failure into an **improvement opportunity** — without it, organizations keep fighting the same fires repeatedly.