Neural Network Interpretability and Explainability is the research and engineering discipline that develops methods to understand why neural networks make specific predictions — through attribution methods (gradients, SHAP, LIME) that identify which input features drive each prediction, attention visualization that reveals the model's focus, and concept-based explanations that map internal representations to human-understandable concepts, because deploying black-box models in safety-critical domains (healthcare, finance, autonomous driving) requires accountability, debugging capability, and regulatory compliance.

Why Interpretability

• Trust: Clinicians won't follow an AI diagnosis they can't understand. Interpretable explanations build justified trust (or reveal when the model is wrong for the right reasons).
• Debugging: A model achieving high accuracy might be relying on spurious correlations (watermarks, background context, dataset artifacts). Attribution reveals these shortcuts.
• Regulation: EU AI Act, GDPR "right to explanation," FDA medical device requirements — all demand explainability for high-risk AI decisions.

Attribution Methods

Gradient-Based:

• Vanilla Gradients: ∂output/∂input — which pixels most affect the prediction. Simple but noisy and suffers from saturation (low gradients in saturated ReLU regions).
• Gradient × Input: Element-wise product of gradient and input. Reduces noise by weighting gradients by feature magnitude.
• Integrated Gradients (Sundararajan et al.): Average gradients along the path from a baseline (all zeros) to the input: IG_i = (x_i - x'_i) × ∫₀¹ (∂F/∂x_i)(x' + α(x - x')) dα. Satisfies completeness axiom — attributions sum to the model's output. Theoretically principled.
• GradCAM: For CNNs — compute gradients of the target class score w.r.t. the last convolutional feature map. Weighted sum of feature channels → attention map highlighting important image regions. Coarse but effective.

Perturbation-Based:

• LIME (Local Interpretable Model-agnostic Explanations): Perturb the input (mask features, modify pixels), observe prediction changes. Fit a simple interpretable model (linear model, decision tree) to the perturbation results. The simple model's coefficients are the feature importances for that specific prediction.
• SHAP (SHapley Additive exPlanations): Computes Shapley values — the game-theoretic fair allocation of the prediction to each feature. Each feature's SHAP value is its average marginal contribution across all possible feature subsets. Computationally expensive (exponential in feature count) — various approximations (KernelSHAP, TreeSHAP, DeepSHAP).

Concept-Based Explanations

• TCAV (Testing with Concept Activation Vectors): Define human concepts (e.g., "striped texture," "wheels"). Find directions in the model's representation space corresponding to each concept. Test how much a concept influences the model's decision — "the model used 'striped' texture 78% of the time when classifying zebras."
• Probing Classifiers: Train simple classifiers on intermediate representations to detect what information is encoded. If a linear classifier on layer 5 achieves 95% accuracy detecting part-of-speech, then layer 5 encodes syntactic information.

Neural Network Interpretability is the accountability infrastructure for AI deployment — providing the explanations, debugging tools, and transparency mechanisms that responsible AI deployment demands, enabling human oversight of automated decisions that affect people's health, finances, and opportunities.