Interpretability Explainability
# Interpretability & Explainability
## Introduction & Motivation
Interpretability: understand how models make decisions. Explainability; transparency. Applications: healthcare, finance, regulatory compliance.
Motivation: Build trustworthy, transparent AI systems.
Applications: Medical diagnosis, loan decisions, autonomous systems.
---
## Core Concepts & Theory
### Feature Importance
Identify influential input features.
### Attention Mechanisms
Highlight important regions.
### Layer-wise Relevance Propagation
Trace predictions back to inputs.
### LIME & SHAP
Local and global explanations.
---
## Mathematical Formulation
Gradient-Based Saliency:
$$S_{i,j} = |\frac{\partial f(x)}{\partial x_{i,j}}|$$
LIME Local Explanation:
$$ ext{explain}(x) = \arg\min_g L(f, g, \pi_x) + \Omega(g)$$
SHAP Value:
$$\phi_i(f) = \sum_S \frac{|S|!(n-|S|-1)!}{n!}(v(S \cup \{i\}) - v(S))$$
---
## Advanced Theory & Extensions
### Concept Activation Vectors
Interpretable concept discovery.
### Influence Functions
Trace predictions to training examples.
### Counterfactual Explanations
"What-if" scenarios.
---
## Computational Considerations
Saliency maps: O(model_size).
LIME: O(samples·model_evals).
SHAP: O(2^features) exact, or approximated.
---
## Practical Implementation Strategies
### Saliency Visualization
Gradient-based importance maps.
### Attention Visualization
Heatmaps of attention weights.
### Feature Ablation
Remove features and measure impact.
---
## Benchmark Datasets & Evaluation
ImageNet: Classification explanation.
Adult Income: Tabular data explainability.
Medical Imaging: Diagnosis explanation.
---
## Key Challenges & Limitations
### Computational Cost
Expensive explanations.
### Faithfulness
Explanations may not reflect true reasoning.
### Sensitivity to Input
Small changes in input affect explanations.
---
## Hyperparameter Tuning
LIME samples: 1000-10000.
SHAP samples: 100-1000.
Gradient smoothing: 1-10 iterations.
---
## Real-World Applications & Case Studies
Medical AI: Explain diagnostic predictions.
Finance: Credit decision transparency.
Autonomous Vehicles: Decision justification.
---
## Integration with Other Methods
Interpretability + adversarial robustness for trustworthy models; + fairness for bias detection.
---
## Summary & Key Takeaways
Interpretability via saliency, attention, and SHAP enables understanding model decisions.
Principles:
1. Feature importance: Input contribution.
2. Attention mechanisms: Spatial focus.
3. Gradient-based methods: Derivative analysis.
4. LIME: Local approximation.
5. SHAP: Game-theoretic attribution.
---
---
## Appendix: Practical Labs
### Lab 1: Gradient Saliency
import numpy as np
def compute_gradient_saliency(output, input_grad):
"""Compute gradient saliency map"""
# Absolute gradient magnitude
saliency = np.abs(input_grad)
# Max across channels for RGB
if len(saliency.shape) == 3:
saliency = np.max(saliency, axis=2)
return saliency
# Test
np.random.seed(42)
output = np.random.randn(1, 10)
grad = np.random.randn(1, 224, 224, 3)
saliency = compute_gradient_saliency(output, grad)
assert saliency.shape == (1, 224, 224), "Saliency shape"
print("✓ Gradient saliency working")
if __name__ == "__main__":
print("Lab 1: GradientSaliency - PASSED")### Lab 2: Feature Ablation
import numpy as np
def feature_ablation_importance(model_output, feature_dim):
"""Estimate importance by ablation"""
baseline = model_output.copy()
importances = []
for i in range(feature_dim):
# Zero out feature
ablated = baseline.copy()
ablated[i] = 0
# Measure change
importance = np.abs(baseline[i])
importances.append(importance)
return np.array(importances)
# Test
np.random.seed(42)
output = np.random.randn(10)
importance = feature_ablation_importance(output, feature_dim=10)
assert len(importance) == 10, "Importance per feature"
print("✓ Feature ablation working")
if __name__ == "__main__":
print("Lab 2: FeatureAblation - PASSED")### Lab 3: Attention Visualization
import numpy as np
def normalize_attention_map(attention):
"""Normalize attention map for visualization"""
# Min-max normalization
att_min = np.min(attention)
att_max = np.max(attention)
normalized = (attention - att_min) / (att_max - att_min + 1e-8)
return normalized
# Test
np.random.seed(42)
attention = np.random.randn(32, 32)
normalized = normalize_attention_map(attention)
assert 0 <= np.min(normalized) <= 1, "Min normalized"
assert 0 <= np.max(normalized) <= 1, "Max normalized"
print("✓ Attention visualization working")
if __name__ == "__main__":
print("Lab 3: AttentionVisualization - PASSED")### Lab 4: SHAP Approximation
import numpy as np
def approximate_shap_value(feature_idx, model_output, num_samples=100):
"""Approximate SHAP value via permutation"""
shap_values = []
for _ in range(num_samples):
# Random permutation
perm = np.random.permutation(model_output.size)
# Contribution with and without feature
pos = np.where(perm == feature_idx)[0][0]
contribution = model_output[feature_idx]
shap_values.append(contribution / model_output.size)
return np.mean(shap_values)
# Test
np.random.seed(42)
output = np.random.randn(10)
shap = approximate_shap_value(0, output, num_samples=100)
assert np.isfinite(shap), "SHAP value finite"
print("✓ SHAP approximation working")
if __name__ == "__main__":
print("Lab 4: SHAPApproximation - PASSED")