saliency

**Saliency Maps and Gradient Attribution** are the **earliest and most widely used class of explainability methods that identify which input regions most influenced a neural network's prediction** — by computing gradients of the output with respect to input features to produce heatmaps highlighting the pixels, tokens, or features the model relied upon. **What Are Saliency Maps?** - **Definition**: Visualizations that assign an importance score to each input element (pixel for images, token for text) indicating how much that element influenced the model's prediction — produced by analyzing gradients from the output back to the input. - **Core Intuition**: If slightly changing a pixel causes a large change in the predicted class probability, that pixel is "salient" — the model's prediction depends on it. - **Output**: A heatmap overlaid on the input — warm colors (red/yellow) indicate high saliency, cool colors (blue) indicate low saliency. - **History**: Vanilla gradient saliency (Simonyan et al., 2014) is one of the oldest deep learning explainability methods, predating LIME and SHAP. **Why Saliency Maps Matter** - **Model Debugging**: Identify when models use wrong features — a skin cancer classifier highlighting the ruler in dermatology images rather than the lesion reveals a dangerous spurious correlation. - **Trust Building**: Show clinicians, radiologists, and domain experts what features drove the AI's decision — enabling validation of AI reasoning before clinical adoption. - **Bias Detection**: Reveal whether models attend to protected attributes (face color in images, gender-coded words in text) when making predictions on sensitive tasks. - **Scientific Discovery**: In scientific AI applications, saliency reveals which molecular features or genomic regions drive predictions — generating testable hypotheses. - **Regulatory Compliance**: Provide required explanations for automated decisions in regulated domains (credit, healthcare, hiring). **Saliency Methods Taxonomy** **Gradient-Based Methods**: **Vanilla Gradient (Sensitivity Map)**: - Compute ∂f(x)/∂x_i — gradient of predicted class score with respect to each input pixel/token. - Fast (single backward pass); noisy and sensitive to input perturbations. - Limitation: Saturated neurons have zero gradient even if very important. **Gradient × Input**: - Element-wise product of gradient and input value: (∂f/∂x_i) × x_i. - Reduces noise; captures both direction and magnitude of feature importance. **Guided Backpropagation**: - Modified gradient: zero out negative gradients during backpropagation. - Produces cleaner, visually appealing saliency maps. - Critical flaw: Springenberg et al. (2017) showed guided backprop produces the same map regardless of model parameters — it is effectively an edge detector on the input, not an explanation of the model. **SmoothGrad**: - Average gradients over N noisy versions of the input: E[∂f(x + ε)/∂x] where ε ~ N(0, σ). - Reduces gradient noise at the cost of N forward-backward passes (typically N=50). **GradCAM (Gradient-weighted Class Activation Mapping)**: - Weight feature map channels by the global average gradient, then average and ReLU. - Produces coarse but reliable class-discriminative visualizations at the final convolutional layer. - Widely adopted in medical imaging for showing which image regions drove classification. **GradCAM++ / EigenCAM / Score-CAM**: - Variants improving GradCAM accuracy, multi-target support, or removing gradient requirements. **Integrated Gradients**: - Axiomatic method satisfying sensitivity and completeness axioms — the gold standard gradient attribution method (see entry 834). **Applications by Domain** | Domain | Input | Method | What It Shows | |--------|-------|--------|---------------| | Medical imaging | X-ray, CT | GradCAM | Which lesion regions drove diagnosis | | NLP sentiment | Text tokens | Gradient × Input | Which words drove positive/negative | | Drug discovery | Molecular graph | Integrated Gradients | Which atoms contributed to toxicity | | Autonomous driving | Camera image | GradCAM | Which road features drove steering | | Cybersecurity | Network packets | SHAP | Which packet features indicate intrusion | **Critical Limitations** **Gradient Saturation**: - ReLU activations produce zero gradients for inputs beyond the saturation threshold — even highly important features may have zero gradient. - Solution: Integrated Gradients paths through the saturation region. **Input Sensitivity vs. Model Explanation**: - Saliency shows local gradient — not global feature importance or counterfactual explanation. - "This pixel is salient" means "changing this pixel slightly changes the output" — not "this pixel is the reason for the prediction." **Faithfulness**: - Guided backprop is demonstrably unfaithful — produces the same result regardless of model weights. - All gradient methods should be validated with faithfulness tests (feature deletion, pixel flipping). **Adversarial Vulnerability**: - Saliency maps can be adversarially manipulated — models can be trained to produce arbitrary saliency maps for any input without changing prediction accuracy. Saliency maps and gradient attribution are **the essential first vocabulary of neural network explanation** — despite their limitations, gradient-based methods provide fast, intuitive visualizations that have driven adoption of AI in medical imaging, scientific research, and safety-critical applications by giving human experts a starting point for validating model reasoning.