interpretability

Interpretability enables understanding of why models make specific predictions or decisions. **Motivation**: Trust, debugging, compliance (right to explanation), scientific understanding, safety verification. **Approaches**: **Feature attribution**: Which inputs influenced output (attention, gradients, SHAP, LIME). **Mechanistic interpretability**: Understand internal computations (circuits, neurons, features). **Concept-based**: Map representations to human-understandable concepts. **Probing**: What information is encoded in hidden layers. **Post-hoc vs intrinsic**: Explaining existing models vs designing interpretable architectures. **For transformers**: Attention visualization, layer-wise relevance propagation, probing classifiers, circuit analysis. **Challenges**: Faithfulness (explanations may not reflect actual reasoning), complexity of modern models, scalability. **Tools**: TransformerLens, Captum, Ecco, inseq. **Applications**: Understanding model failures, detecting spurious correlations, safety cases, model editing. **Trade-offs**: Interpretable models may sacrifice performance, post-hoc methods have faithfulness issues. **Current state**: Active research area, partial solutions exist, full mechanistic understanding distant. Critical for AI safety and trust.