Explainability and Interpretability Shap Lime Attention and Saliency
# Explainability and Interpretability: SHAP, LIME, Attention, and Saliency
## 1. Introduction & Motivation
As machine learning models become more powerful, they often become less interpretable. Deep neural networks with millions of parameters and complex interactions are "black boxes" whose decisions are difficult to understand. This lack of interpretability poses problems:
- Accountability: Regulators (GDPR, Fair Lending) require explainable decisions
- Debugging: Understanding failure modes requires interpretability
- Trust: Users distrust unexplainable predictions
- Bias detection: Hard to identify discriminatory patterns in black-box models
Explainability research provides techniques to understand what models learn and why they make specific predictions. This article covers major approaches: local explanations (LIME, SHAP), attention mechanisms, gradient-based saliency, and global interpretability.
## 2. Core Concepts & Theory
### 2.1 Local vs Global Explanations
Local explanations: Why did the model make this specific prediction for this sample?
- Example: "Loan denied because credit score is 650"
- Useful for individual predictions, actionable
Global explanations: What patterns does the model learn overall?
- Example: "Model primarily uses credit score and income"
- Useful for model understanding, debugging
Most practical work focuses on local explanations (post-hoc, work with any model).
### 2.2 LIME: Local Interpretable Model-agnostic Explanations
LIME explains individual predictions by fitting interpretable surrogate models locally:
1. Sample perturbations of input:
$$ x' = x + \epsilon $$
2. Get model predictions: f(x')
3. Weight by proximity to original:
$$ w(x') = \exp(-d(x, x')^2 / \sigma^2) $$
4. Fit interpretable model:
$$ g(x') \approx f(x') $$
using weighted samples
For linear surrogate:
$$g(x') = w_0 + \sum_i w_i x'_i$$
Coefficients
$$ w_i $$
indicate feature importance for this prediction.
### 2.3 SHAP: SHapley Additive exPlanations
SHAP values based on Shapley values from game theory. For a model f and sample x:
$$\phi_i = \sum_{S \subseteq F \setminus \{i\}} \frac{|S|! (|F|-|S|-1)!}{|F|!} [f(S \cup \{i\}) - f(S)]$$
where S is feature subset,
$$ \phi_i $$
is contribution of feature i.
Interpretation: Feature i's contribution averaged over all possible coalitions.
Key properties:
- Local accuracy: Explanations sum to prediction
- Consistency: If model changes to use feature more, Shapley value increases
- Symmetry: Features with same impact get same weight
### 2.4 Attention as Explanation
In attention-based models (Transformers, RNNs with attention), attention weights indicate focus:
$$\alpha_{i,j} = \frac{\exp(e_{i,j})}{\sum_k \exp(e_{i,k})}$$
High
$$ \alpha_{i,j} $$
suggests token i attends to token j. Visualizing attention provides interpretability.
Caveat: Attention ≠ importance (gradient-based methods more reliable).
## 3. Mathematical Formulation
### 3.1 Shapley Value Computation
For finite player set, Shapley value of player i:
$$\phi_i(f) = \frac{1}{n!} \sum_{\pi} [f(\pi_i \cup \{i\}) - f(\pi_i)]$$
where sum is over all n! orderings
$$ \pi $$
,
$$ \pi_i $$
is set of players before i in ordering
$$ \pi $$
.
For computational tractability, approximations:
Kernel SHAP: Weight feature coalitions by Shapley kernel:
$$w_s = \frac{(m-1)}{s(m-s)}$$
where m is total features, s is features in coalition.
### 3.2 Gradient-Based Saliency
First-order gradient shows sensitivity to input perturbations:
$$S_i = \frac{\partial f(x)}{\partial x_i}$$
Magnitude indicates how much input i affects output.
Integrated gradients: Path integral from baseline to input:
$$ ext{IG}_i(x) = (x_i - x'_i) \int_0^1 \frac{\partial f(x' + \alpha(x - x'))}{\partial x_i} d\alpha$$
More faithful than single gradient (accounts for non-linearities).
### 3.3 CAM: Class Activation Maps
For convolutional networks, visualize which regions matter:
$$M_c(x, y) = \sum_k w_k^c \cdot A_k(x, y)$$
where
$$ A_k $$
is activation of filter k,
$$ w_k^c $$
is weight of filter k for class c.
Grad-CAM uses gradient instead of weights:
$$w_k^c = \frac{1}{Z} \sum_{x,y} \frac{\partial score_c}{\partial A_k(x,y)}$$
More general, works even without global average pooling.
### 3.4 Counterfactual Explanations
Explain by example: "If feature X changed to value Y, prediction would be Z."
Formulation: Find minimal input change producing different prediction:
$$\arg\min_x' d(x, x') \quad ext{s.t.} \quad f(x') eq f(x)$$
Trade-off: Proximity (small change) vs. validity (actually changes prediction).
## 4. Advanced Theory & Extensions
### 4.1 Feature Interaction Analysis
Beyond individual importance, understand feature interactions:
$$I(i, j) = \phi_{i,j}(f) - \phi_i(f) - \phi_j(f)$$
Interaction strength measured by gap between joint contribution and sum of individual contributions.
### 4.2 Influence Functions
Trace predictions back to training data:
$$I_{ ext{train}} = - abla_ heta \mathcal{L}( heta, z_{ ext{test}})^T H^{-1} abla_ heta \mathcal{L}( heta, z_{ ext{train}})$$
where H is Hessian. Identifies influential training samples.
High influence: Removing sample would significantly change prediction.
### 4.3 Concept Bottleneck Models
Intermediate layer learns human-interpretable concepts:
$$f(x) = g(h(x)), \quad h(x) = [c_1, c_2, \ldots, c_k]$$
where
$$ c_i $$
are interpretable concepts (edges, textures, object parts).
Benefits:
- Individual concept interpretability
- Can correct concept detection for robustness
- Allows human input on concept importance
### 4.4 Temporal Explanations
For sequence models, explain over time:
$$ ext{Importance}_t = \frac{\partial ext{output}}{\partial x_t}$$
Shows which time steps matter most. Visualization: heatmaps over time.
## 5. Computational Considerations
### 5.1 LIME Complexity
For each prediction:
- Generate perturbations: O(N) where N is sample count
- Get predictions: O(N) forward passes
- Fit surrogate model:
$$ O(d^3) $$
where d is feature dimension
Typical: 1-10s per explanation, acceptable for retrospective analysis.
### 5.2 SHAP Complexity
Exact Shapley computation: Exponential in feature count (
$$ O(2^d) $$
).
Approximations:
- Kernel SHAP:
$$ O(2^d \cdot d) $$
- still exponential, but tractable for
$$ d < 20 $$
- TreeSHAP:
$$ O(d \cdot T) $$
for tree models where T is tree depth
- Deep SHAP:
$$ O(d \cdot N) $$
approximate for neural networks
For 1000-dim inputs, exact SHAP infeasible; approximations necessary.
### 5.3 Gradient-Based Methods
Saliency computation very efficient:
- Single backward pass: O(d) parameters
- No additional models needed
- Minimal overhead compared to prediction
### 5.4 Memory Efficiency
Explanation methods can be memory-intensive:
- LIME: Stores perturbation samples (~10K)
- SHAP: Stores intermediate coalitions
- Gradient methods: Only need backward pass
## 6. Practical Implementation Strategies
### 6.1 Choosing Explanation Method
LIME:
- Pros: Simple, model-agnostic, intuitive
- Cons: Local only, doesn't guarantee fidelity
- Best for: Quick explanations, non-technical users
SHAP:
- Pros: Theoretically grounded, consistent
- Cons: Computational cost, harder to interpret
- Best for: Regulatory/legal requirements, feature importance
Saliency maps:
- Pros: Fast, faithful to model
- Cons: Hard to interpret (gradients ≠ importance)
- Best for: Vision models, identifying failure modes
Attention:
- Pros: Natural, visualizable
- Cons: Attention ≠ importance, only for attention models
- Best for: NLP, complementary to other methods
### 6.2 Local Explanation Workflow
Input: Sample x, trained model f
1. Generate perturbations: x' = x + noise (50-1000 samples)
2. Compute predictions: f(x') for all x'
3. Compute weights: w = exp(-distance(x, x')^2)
4. Fit linear model: minimize sum(w * (f(x') - g(x'))^2)
5. Extract coefficients as feature importance### 6.3 Handling High-Dimensional Inputs
For images/text with thousands of dimensions:
Feature grouping:
- Superpixels for images (group nearby pixels)
- Sentence embeddings for text
- Reduces effective dimensionality
Mask perturbations:
- Rather than noise, use masking (set to mean/default)
- Represents "feature absence"
### 6.4 Evaluating Explanation Quality
Fidelity: Does removing important features degrade performance?
$$ ext{Fidelity} = f(x) - f(x \setminus ext{important features})$$
Higher is better (removal hurts more).
Sensitivity: Do similar inputs get similar explanations?
$$ ext{Sensitivity} = 1 - \frac{\sum_i | ext{exp}(x_i) - ext{exp}(x_i')|}{d}$$
where
$$ x' \approx x $$
. Higher is better (stability).
## 7. Benchmark Datasets & Evaluation
### 7.1 Interpretability Benchmarks
ImageNet Classification:
- Model: ResNet-50
- Evaluation: Do saliency maps highlight correct objects?
- Metric: Pointing game (% of saliency in object bounding box)
- Results: Grad-CAM ~75%, attention ~70%, random ~20%
COMPAS Dataset (Recidivism Prediction):
- Model: Logistic regression baseline
- LIME explanations: Which features drive decisions?
- Evaluation: Expert agreement on explanation plausibility
- Results: ~80-85% agreement (reasonable)
VCR (Visual Commonsense Reasoning):
- Task: Answer visual questions requiring reasoning
- Explanations: Why did model choose this answer?
- Metric: Does explanation improve user understanding?
- Results: Users trust explanations 60% of the time
### 7.2 Faithfulness Metrics
Sufficiency: Removing unimportant features shouldn't hurt
$$ ext{Suff} = f(x) - f(x_{ ext{unimportant only}})$$
Small sufficiency score ≈ good explanation (unimportant features don't matter).
Comprehensiveness: Removing important features should hurt
$$ ext{Comp} = f(x) - f(x_{ ext{important only}})$$
Large comprehensiveness ≈ good explanation.
## 8. Key Challenges & Limitations
### 8.1 Attention Is Not Explanation
Research shows attention weights don't reliably indicate feature importance:
$$ ext{Attention}(x_i) ot\approx ext{Gradient-based importance}(x_i)$$
Attention trained for different objective (information routing, not importance). Can be misleading without validation.
### 8.2 Explanation Instability
Small input changes can drastically change explanations, even if predictions unchanged:
$$|| ext{explain}(x) - ext{explain}(x + \epsilon)|| ext{ can be large even if } ||f(x) - f(x+\epsilon)|| \approx 0$$
Explanations sometimes unstable (esp. LIME). Suggests explanations not fully reliable.
### 8.3 Problem of Multiple Valid Explanations
Different feature subsets can equally well predict:
- "High income + low debt" predicts loan approval
- "High savings + low risk" also predicts approval
- Both are "correct" but contradictory
Which explanation is "right"? No unique answer.
### 8.4 Computational Cost vs Utility Trade-off
Expensive explanations (SHAP) don't always better than cheap ones (gradients).
May not justify 100x computational cost for modest interpretability gain.
## 9. Hyperparameter Tuning & Optimization
### 9.1 LIME Parameters
Number of perturbations: 50-10000 typical
- More samples → more stable explanations
- Cost: Linear in number
- Typical: 1000-5000 for images
Kernel width:
$$ \sigma $$
in
$$ \exp(-d^2/\sigma^2) $$
- Controls locality of neighborhood
- Large
$$ \sigma $$
: Large neighborhood, less local
- Small
$$ \sigma $$
: Local but sparse
- Typical:
$$ \sigma = 0.25 $$
(normalized distance)
Regularization: Strength of regularization in surrogate fit
- Higher regularization: Simpler explanations
- Lower regularization: Fit data better
- Typical: L2 reg weight = 0.001
### 9.2 SHAP Parameters
Number of background samples: For computing expected values
- Larger: More stable estimates
- Cost: Quadratic in number
- Typical: 100-1000 samples
Feature permutation order: Varies Shapley estimates slightly
- More orderings: More stable
- Cost: Linear in orderings
- Typical: 100+ orderings for approximation
### 9.3 Saliency Map Parameters
Smoothing: Apply Gaussian blur to saliency
- Reduces noise
- Makes interpretable regions larger
- Typical:
$$ \sigma = 1-3 $$
pixels
Normalization: How to normalize saliency values
- Min-max normalization: Scale to [0, 1]
- Percentile clipping: Clip outliers
- Typical: Min-max
## 10. Real-World Applications & Case Studies
### 10.1 Credit Risk: Loan Approval Explanations
Problem: Bank must explain why loans approved/denied (Fair Lending Act)
Setup:
- Model: Gradient boosted tree (XGBoost)
- Input: Credit score, income, debt, assets (20 features total)
- Requirement: Explain each decision to applicants
Explanation Approach:
- Method: TreeSHAP (efficient for tree models)
- Format: "Loan approved. Credit score (+0.35), income (+0.20), debt (-0.15)"
- Computation: ~50ms per application
Results:
- Stakeholder satisfaction: 85%
- Complaint rate: Reduced 30% (from 15% → 10%)
- Regulatory compliance: Full compliance achieved
Technical details:
- Use base model value (marginal approval rate)
- Sum SHAP values to match prediction
- Display top 5 features to users
### 10.2 Medical Diagnosis with Saliency
Problem: Predict pneumonia from chest X-rays, explain to doctors
Setup:
- Model: ResNet-50 fine-tuned on medical images
- Input: 512×512 X-ray images
- Goal: Localize pneumonia regions for doctor confirmation
Explanation Approach:
- Method: Grad-CAM
- Output: Heatmap over X-ray image
- Interpretation: Bright regions = model focuses here
Results:
- Doctor agreement: 92% (localization matches radiologist reading)
- Confidence boost: 78% of doctors trust diagnosis more with explanation
- Clinical adoption: Currently deployed in 3 hospitals
Key insights:
- Spatial explanations crucial for medical domain
- Multiple explanation methods recommended (Grad-CAM + attention)
- Validation against expert consensus important
### 10.3 NLP: Sentiment Analysis Explanations
Problem: Explain sentiment classification decisions for content moderation
Setup:
- Model: BERT fine-tuned on sentiment classification
- Input: Text reviews
- Labels: Positive, negative, neutral
Explanation Approach:
- Method: Attention visualization + saliency
- Output: Highlight important words and phrases
- Example: "This product is amazing" → positive
Results:
- Content moderators trust: ~70% of cases
- Appeal rate reduction: 25% (fewer disputes on flagged content)
- Bias detection: Found model biased against certain demographics
Technical implementation:
- Average attention across heads (12 heads)
- Normalize by layer (lower layers: syntax, higher: semantics)
- Combine with gradient-based importance
### 10.4 Autonomous Driving Safety Analysis
Problem: Understand why autonomous vehicle made dangerous maneuvers
Setup:
- Model: Deep RL policy for driving
- Input: Camera, lidar, radar data
- Goal: Understand model decisions for safety analysis
Explanation Approach:
- Method: Saliency (which input regions matter)
- Output: Visualization of attended regions
- Analysis: Compare to human driver priorities
Results:
- Found model focuses on wrong regions in 5-10% of scenarios
- Identified systematic biases (e.g., ignores motorcycles)
- Led to model retraining, improved safety
Challenges:
- High-dimensional multimodal input
- Temporal aspect (decisions based on past frames)
- Safety-critical (wrong explanation can be dangerous)
## 11. Integration with Other Methods
### 11.1 Explanations for Model Debugging
Use explanations to identify and fix bugs:
- Wrong features used: Retrain or add constraints
- Distribution shift: Adapt model or add calibration
- Adversarial examples: Robust training
### 11.2 Explanations for Fairness Analysis
Identify bias via explanations:
- Feature importance shows over/under-reliance on protected attributes
- Compare explanations across demographic groups
- Apply bias mitigation if systematic disparities found
### 11.3 Explanations with Adversarial Training
Adversarially robust models often have different attention patterns:
- Standard model: Focuses on high-frequency artifacts
- Robust model: Focuses on object parts
- Use explanations to verify robustness properties
### 11.4 Human-in-the-Loop with Explanations
Use explanations to guide human review:
- High-uncertainty predictions: Require human approval
- Atypical explanation: Flag for review
- Enables selective human oversight
## 12. Future Research Directions
### 12.1 Unified Explanation Framework
Current: Multiple incompatible methods (LIME, SHAP, saliency, attention)
Goal: Single framework encompassing all approaches with clear trade-offs.
### 12.2 Causal Explanations
Current: Correlational (which features matter?)
Goal: Causal (what would change if we intervened on this feature?)
Requires causal models, counterfactual reasoning.
### 12.3 Efficient Explanations for Large Models
Current: Expensive for billion-parameter models
Goal: Fast explanations for foundation models (CLIP, GPT-3)
### 12.4 Interactive Explanations
Current: Static explanations
Goal: User can ask follow-up questions, drill down into details
Requires conversational AI + explanations.
## 13. Summary & Key Takeaways
Explanation Methods:
*Local (why this prediction):*
- LIME: Simple, agnostic, ~100s per sample, 70-80% fidelity
- SHAP: Principled, ~100s-1000s per sample, 85-90% fidelity
- Saliency: Fast (~1ms), but less reliable, needs validation
*Global (what model learns):*
- Feature importance: Average absolute SHAP or gradient
- Attention: Direct but unreliable
- Concept activation: Interpretable but requires concept learning
Practical Performance:
- Gradient methods: ~1ms, reasonable faithfulness
- LIME: ~100ms-1s, moderate faithfulness
- SHAP: ~1-10s, high faithfulness
- Trade-off: Speed vs interpretability
Choosing Method:
- Vision tasks: Grad-CAM or saliency maps
- Tabular data: SHAP or LIME
- NLP: Attention + gradient methods
- Regulatory setting: SHAP (theoretically grounded)
Validation:
- Fidelity: Removing important features hurts prediction
- Sensitivity: Stable explanations for similar inputs
- Faithfulness: Saliency correlates with gradients
- Human agreement: Domain expert validation
Key Limitations:
- Attention ≠ importance (validate independently)
- Multiple valid explanations possible (no unique answer)
- Explanations can be unstable (small input changes → big explanation changes)
- Computational cost-benefit trade-off (expensive ≠ better)
Current Applications:
- Financial: Credit decisions, fraud detection
- Medical: Diagnosis support, treatment recommendations
- Content: Moderation, recommendation justification
- Safety: Autonomous systems, safety-critical analysis
Explainability is increasingly important for deployed ML systems. No single best method; use multiple approaches and validate against domain experts and model behavior.
---
## Appendix: Practical Implementation Labs
### Lab 1: LIME Implementation
import numpy as np
from sklearn.linear_model import Ridge
import torch
class SimpleLIME:
def __init__(self, model, num_samples=1000, kernel_width=0.25):
self.model = model
self.num_samples = num_samples
self.kernel_width = kernel_width
def explain_instance(self, x, num_features=10):
"""Explain prediction for instance x"""
# Generate perturbed samples
perturbations = []
predictions = []
for _ in range(self.num_samples):
# Create perturbation (add Gaussian noise)
x_pert = x + np.random.normal(0, 1, x.shape)
perturbations.append(x_pert)
# Get prediction
x_pert_tensor = torch.from_numpy(x_pert).float().unsqueeze(0)
with torch.no_grad():
pred = self.model(x_pert_tensor).numpy()[0]
predictions.append(pred)
perturbations = np.array(perturbations)
predictions = np.array(predictions)
# Compute weights based on proximity
distances = np.linalg.norm(perturbations - x, axis=1)
weights = np.exp(-(distances ** 2) / (self.kernel_width ** 2))
# Fit linear surrogate model
surrogate = Ridge(alpha=1.0)
surrogate.fit(perturbations, predictions, sample_weight=weights)
# Extract feature importance from coefficients
importance = np.abs(surrogate.coef_)
top_features = np.argsort(importance)[-num_features:]
return {
'importance': importance[top_features],
'features': top_features,
'intercept': surrogate.intercept_
}### Lab 2: Gradient-Based Saliency
def compute_gradient_saliency(model, x, target_class=None):
"""Compute gradient-based saliency map"""
x = x.clone().detach().requires_grad_(True)
# Forward pass
output = model(x.unsqueeze(0))
# Target class
if target_class is None:
target_class = output.argmax(dim=1).item()
# Compute gradient
model.zero_grad()
target_score = output[0, target_class]
target_score.backward()
# Saliency = max absolute gradient across channels
saliency = torch.abs(x.grad).max(dim=0)[0].detach()
# Normalize to [0, 1]
saliency = (saliency - saliency.min()) / (saliency.max() - saliency.min() + 1e-8)
return saliency
# Test
# x = torch.randn(3, 224, 224) # Image tensor
# saliency = compute_gradient_saliency(model, x)
# visualize_saliency(saliency)### Lab 3: Attention Visualization
def visualize_attention(attention_weights, tokens, layers_to_show=None):
"""Visualize attention patterns from transformer"""
num_layers, num_heads, seq_len, seq_len = attention_weights.shape
if layers_to_show is None:
layers_to_show = [num_layers // 2] # Show middle layer
for layer_idx in layers_to_show:
# Average attention over all heads
layer_attention = attention_weights[layer_idx].mean(dim=0) # [seq_len, seq_len]
print(f"Layer {layer_idx} Attention Matrix:")
print("Tokens:", tokens)
print("Attention weights (rows = query, cols = key):")
# Print which tokens attend to which
for i, token_i in enumerate(tokens):
important_tokens = []
for j in range(len(tokens)):
if layer_attention[i, j] > 0.1: # Threshold
important_tokens.append((tokens[j], layer_attention[i, j].item()))
important_tokens.sort(key=lambda x: x[1], reverse=True)
print(f" {token_i} attends to: {important_tokens[:3]}")### Lab 4: Shapley Value Approximation (Kernel SHAP)
class KernelSHAP:
def __init__(self, model, X_background, num_samples=2000):
self.model = model
self.X_background = X_background
self.num_samples = num_samples
self.baseline = X_background.mean(axis=0)
def explain(self, x):
"""Compute approximate Shapley values"""
num_features = x.shape[0]
# Generate random coalitions
coalitions = []
for _ in range(self.num_samples):
mask = np.random.binomial(1, 0.5, num_features)
coalitions.append(mask)
coalitions = np.array(coalitions)
# Evaluate model on coalitions
predictions = []
for mask in coalitions:
# Mix: keep features in mask, use baseline for others
x_masked = x.copy()
x_masked[mask == 0] = self.baseline[mask == 0]
pred = self.model(x_masked)
predictions.append(pred)
predictions = np.array(predictions)
# Weight coalitions by Shapley kernel
m = num_features
s = coalitions.sum(axis=1) # Coalition sizes
weights = (m - 1) / (s * (m - s) + 1e-8)
# Fit weighted model
from sklearn.linear_model import Ridge
shap_model = Ridge(alpha=1.0)
shap_model.fit(coalitions, predictions, sample_weight=weights)
return shap_model.coef_