model interpretability explainability

**Neural Network Interpretability and Explainability** is the **research and engineering discipline that develops methods to understand why neural networks make specific predictions — through attribution methods (gradients, SHAP, LIME) that identify which input features drive each prediction, attention visualization that reveals the model's focus, and concept-based explanations that map internal representations to human-understandable concepts, because deploying black-box models in safety-critical domains (healthcare, finance, autonomous driving) requires accountability, debugging capability, and regulatory compliance**. **Why Interpretability** - **Trust**: Clinicians won't follow an AI diagnosis they can't understand. Interpretable explanations build justified trust (or reveal when the model is wrong for the right reasons). - **Debugging**: A model achieving high accuracy might be relying on spurious correlations (watermarks, background context, dataset artifacts). Attribution reveals these shortcuts. - **Regulation**: EU AI Act, GDPR "right to explanation," FDA medical device requirements — all demand explainability for high-risk AI decisions. **Attribution Methods** **Gradient-Based**: - **Vanilla Gradients**: ∂output/∂input — which pixels most affect the prediction. Simple but noisy and suffers from saturation (low gradients in saturated ReLU regions). - **Gradient × Input**: Element-wise product of gradient and input. Reduces noise by weighting gradients by feature magnitude. - **Integrated Gradients (Sundararajan et al.)**: Average gradients along the path from a baseline (all zeros) to the input: IG_i = (x_i - x'_i) × ∫₀¹ (∂F/∂x_i)(x' + α(x - x')) dα. Satisfies completeness axiom — attributions sum to the model's output. Theoretically principled. - **GradCAM**: For CNNs — compute gradients of the target class score w.r.t. the last convolutional feature map. Weighted sum of feature channels → attention map highlighting important image regions. Coarse but effective. **Perturbation-Based**: - **LIME (Local Interpretable Model-agnostic Explanations)**: Perturb the input (mask features, modify pixels), observe prediction changes. Fit a simple interpretable model (linear model, decision tree) to the perturbation results. The simple model's coefficients are the feature importances for that specific prediction. - **SHAP (SHapley Additive exPlanations)**: Computes Shapley values — the game-theoretic fair allocation of the prediction to each feature. Each feature's SHAP value is its average marginal contribution across all possible feature subsets. Computationally expensive (exponential in feature count) — various approximations (KernelSHAP, TreeSHAP, DeepSHAP). **Concept-Based Explanations** - **TCAV (Testing with Concept Activation Vectors)**: Define human concepts (e.g., "striped texture," "wheels"). Find directions in the model's representation space corresponding to each concept. Test how much a concept influences the model's decision — "the model used 'striped' texture 78% of the time when classifying zebras." - **Probing Classifiers**: Train simple classifiers on intermediate representations to detect what information is encoded. If a linear classifier on layer 5 achieves 95% accuracy detecting part-of-speech, then layer 5 encodes syntactic information. Neural Network Interpretability is **the accountability infrastructure for AI deployment** — providing the explanations, debugging tools, and transparency mechanisms that responsible AI deployment demands, enabling human oversight of automated decisions that affect people's health, finances, and opportunities.