Explainability Interpretability Lime Shap Model Explanations
# Explainability & Interpretability: LIME, SHAP & Model Explanations
## Introduction & Motivation
Explainability: understand model decisions. LIME: local linear approximation; perturbation-based. SHAP: Shapley values; game-theoretic; feature importance. Feature attribution: which features matter. Saliency maps: spatial importance (vision). Applications: model debugging, regulatory compliance, trust.
Motivation: Black-box models; need interpretability. Explanations build trust, enable debugging.
Applications: Healthcare, finance, regulatory compliance.
---
## Core Concepts & Theory
### LIME (Local Interpretable Model-agnostic Explanations)
Local linear model; perturb inputs, measure output change.
### SHAP (SHapley Additive exPlanations)
Shapley values; fair feature attribution.
### Saliency Maps
Gradient-based; spatial importance for images.
---
## Mathematical Formulation
LIME loss:
$$\min_w \sum_i L(f(x_i), \hat{f}(x_i)) w_i + \lambda \Omega(w)$$
approximate model, weight by proximity.
SHAP value:
$$\phi_i = \frac{1}{n!} \sum_S (f(S \cup \{i\}) - f(S))$$
marginal contribution over all orderings.
Saliency:
$$ ext{Saliency} = \left| \frac{\partial f}{\partial x}
ight|$$
input gradient magnitude.
---
## Advanced Theory & Extensions
### Integrated Gradients
Path integration; baseline to sample.
### Attention Visualization
Attention weights; model focus areas.
### Concept Activation Vectors
Semantic concepts; human-interpretable.
---
## Computational Considerations
LIME: O(K·forward) for K perturbations.
SHAP: O(2^d) exact; approximations available.
Saliency: O(forward + backward) per sample.
---
## Practical Implementation Strategies
### Perturbation Strategy
Small, meaningful perturbations; domain-specific.
### Local Approximation
Balance fidelity and simplicity; linear common.
### Background Data
SHAP needs reference; random or representative.
---
## Benchmark Datasets & Evaluation
Synthetic: Controlled ground truth.
Sanity Check: Random baseline comparison.
Human Evaluation: Expert assessment of explanations.
---
## Key Challenges & Limitations
### Computational Cost
SHAP exponential in features; approximations needed.
### Faithfulness
Local approximation may not capture global.
### Inconsistency
Different methods give different explanations.
---
## Hyperparameter Tuning
LIME perturbations: 1000+ typical.
SHAP approximation: trade off computation-accuracy.
Saliency smoothing: Gaussian blur or integration.
---
## Real-World Applications & Case Studies
Healthcare: Explain diagnosis predictions.
Finance: Credit decision interpretability.
Model Debugging: Identify spurious correlations.
---
## Integration with Other Methods
Explanations + Regularization → fairness constraints.
Explanations + Adversarial → robustness evaluation.
---
## Summary & Key Takeaways
Explainability via LIME, SHAP, and saliency maps provides interpretable model explanations through local approximations and feature attribution, enabling trust and debugging.
Principles:
1. LIME: local linear approximation.
2. SHAP: Shapley game-theoretic attribution.
3. Saliency: gradient-based importance.
4. Model-agnostic: work on any model.
5. Trade-offs: fidelity vs simplicity.
---
---
## Appendix: Practical Labs
### Lab 1: LIME Explanation
import numpy as np
from sklearn.linear_model import LinearRegression
def lime_explain(model, x, num_perturb=100, neighborhood_dist=0.25):
"""LIME: local linear explanation"""
# Perturb around x
perturb_samples = np.random.normal(x, neighborhood_dist, (num_perturb, len(x)))
# Get model predictions
predictions = model.predict(perturb_samples)
# Compute distances
distances = np.linalg.norm(perturb_samples - x, axis=1)
weights = np.exp(-distances ** 2 / (2 * neighborhood_dist ** 2))
# Fit local linear model
lime_model = LinearRegression()
lime_model.fit(perturb_samples, predictions, sample_weight=weights)
return lime_model.coef_
# Test
np.random.seed(42)
model = lambda x: (x ** 2).sum(axis=1) # Simple model
x = np.array([1.0, 2.0, 3.0])
weights = lime_explain(model, x, num_perturb=100)
assert len(weights) == 3, "Weights per feature"
print("✓ LIME working")
if __name__ == "__main__":
print("Lab 1: LIME - PASSED")### Lab 2: Feature Importance
import numpy as np
def compute_feature_importance(model, X, num_shuffle=10):
"""Permutation importance"""
baseline_score = (model.predict(X) > 0.5).mean()
importances = []
for feature_idx in range(X.shape[1]):
scores = []
for _ in range(num_shuffle):
X_perm = X.copy()
np.random.shuffle(X_perm[:, feature_idx])
score = (model.predict(X_perm) > 0.5).mean()
scores.append(baseline_score - score)
importances.append(np.mean(scores))
return np.array(importances)
# Test
np.random.seed(42)
model = lambda X: (X[:, 0] ** 2 + X[:, 1] > 0).astype(float)
X = np.random.randn(100, 3)
importance = compute_feature_importance(model, X)
assert len(importance) == 3, "Feature importances"
print("✓ Feature importance working")
if __name__ == "__main__":
print("Lab 2: Importance - PASSED")### Lab 3: Saliency Maps
import torch
import torch.nn.functional as F
import numpy as np
def compute_saliency(model, x, target_class):
"""Compute saliency map via gradients"""
x.requires_grad = True
output = model(x)
loss = output[:, target_class].sum()
loss.backward()
saliency = x.grad.abs()
saliency = (saliency - saliency.min()) / (saliency.max() - saliency.min())
return saliency.detach()
# Test
np.random.seed(42)
model = nn.Linear(784, 10)
x = torch.randn(1, 784, requires_grad=True)
saliency = compute_saliency(model, x, target_class=3)
assert saliency.shape == (1, 784), "Saliency shape"
assert (saliency >= 0).all() and (saliency <= 1).all(), "Normalized"
print("✓ Saliency maps working")
if __name__ == "__main__":
print("Lab 3: Saliency - PASSED")### Lab 4: Explanation Evaluation
import numpy as np
def explanation_fidelity(model, x, explanation, num_mask=10):
"""Evaluate explanation via feature masking"""
# Rank features by importance
ranked_features = np.argsort(-explanation)
# Progressively mask top features
fidelities = []
baseline_pred = model.predict(x)
for num_masked in range(1, len(x) + 1):
x_masked = x.copy()
x_masked[ranked_features[:num_masked]] = 0
masked_pred = model.predict(x_masked)
# Fidelity: how much prediction changes
fidelity = np.abs(masked_pred - baseline_pred)
fidelities.append(fidelity)
return np.array(fidelities)
# Test
np.random.seed(42)
model = lambda x: (x ** 2).sum()
x = np.array([1.0, 2.0, 3.0])
explanation = np.array([1.0, 2.0, 3.0])
fidelity = explanation_fidelity(model, x, explanation)
assert len(fidelity) == 3, "Fidelity per mask"
print("✓ Explanation evaluation working")
if __name__ == "__main__":
print("Lab 4: Evaluation - PASSED")