Out-of-Distribution Detection Robustness Generalization
# Out-of-Distribution Detection: Robustness & Generalization
## Introduction & Motivation
OOD Detection: identify samples from different distribution. Detect unfamiliar inputs; reject uncertain predictions. Applications: safety-critical systems, deployment robustness, model confidence.
Motivation: Prevent wrong predictions on unfamiliar data.
Applications: Safety, robustness, reliability.
---
## Core Concepts & Theory
### Distribution Shift
Training vs. deployment difference.
### Confidence Calibration
Ensure predicted confidence matches accuracy.
### Scoring Methods
Energy, maximum softmax, typicality.
---
## Mathematical Formulation
Maximum softmax score:
$$ ext{score}_{ ext{max}} = \max_c P(y=c | x)$$
Energy-based score:
$$E(x) = -\log \sum_c \exp(f_c(x))$$
Typicality:
$$ ext{typicality}(x) = - \log P(x | D_{ ext{train}})$$
---
## Advanced Theory & Extensions
### Mahalanobis Distance
Class-conditional Gaussian assumption.
### Energy-Based OOD
Energy scoring; improved performance.
### Contrastive Predictive Coding
Unsupervised OOD detection.
---
## Computational Considerations
Maximum softmax: O(num_classes).
Mahalanobis: O(num_classes·feature_dim²).
Energy-based: O(num_classes).
---
## Practical Implementation Strategies
### Threshold Selection
Validation set-based tuning.
### Ensemble Methods
Combine multiple OOD scores.
### Temperature Scaling
Improve confidence calibration.
---
## Benchmark Datasets & Evaluation
CIFAR-10/100 vs. STL-10: OOD benchmark.
ImageNet vs. iNaturalist: Scale OOD.
MNIST vs. Omniglot: Cross-dataset OOD.
---
## Key Challenges & Limitations
### Far vs. Near OOD
Semantically similar vs. different.
### Calibration
Overconfidence on OOD.
### Computational Cost
Real-time requirements.
---
## Hyperparameter Tuning
Confidence threshold: 0.7-0.95.
Temperature: 1-10 for scaling.
Validation split: Representative OOD.
---
## Real-World Applications & Case Studies
Autonomous Driving: Detect unfamiliar scenarios.
Medical Imaging: Flag unusual cases.
Chatbots: Reject out-of-domain queries.
---
## Integration with Other Methods
OOD + Uncertainty → confidence quantification.
OOD + Active Learning → query strategies.
---
## Summary & Key Takeaways
Out-of-Distribution Detection via scoring methods enables model robustness through identification of unfamiliar samples and distribution shift.
Principles:
1. Distribution shift: train-test mismatch.
2. Scoring: maximum softmax, energy.
3. Calibration: confidence accuracy.
4. Threshold: decision boundary.
5. Ensemble: multiple scores.
---
---
## Appendix: Practical Labs
### Lab 1: Maximum Softmax Confidence
import numpy as np
def maximum_softmax_confidence(logits):
"""Confidence score via maximum softmax"""
exp_logits = np.exp(logits - np.max(logits, axis=1, keepdims=True))
probs = exp_logits / exp_logits.sum(axis=1, keepdims=True)
# Maximum probability
confidence = np.max(probs, axis=1)
return confidence
# Test
np.random.seed(42)
in_dist_logits = np.random.randn(50, 10) + 2 # Higher logits
ood_logits = np.random.randn(50, 10) - 2 # Lower logits
in_conf = maximum_softmax_confidence(in_dist_logits)
ood_conf = maximum_softmax_confidence(ood_logits)
assert in_conf.mean() > ood_conf.mean(), "In-dist more confident"
print("✓ Maximum softmax working")
if __name__ == "__main__":
print("Lab 1: MaxSoftmax - PASSED")### Lab 2: Energy Score
import numpy as np
def energy_score(logits):
"""Energy-based OOD score"""
# Energy = -log(sum(exp(logits)))
max_logits = np.max(logits, axis=1, keepdims=True)
exp_logits = np.exp(logits - max_logits)
energy = -np.log(np.sum(exp_logits, axis=1) + 1e-8)
return energy
# Test
np.random.seed(42)
in_dist = np.random.randn(50, 10) + 3
ood = np.random.randn(50, 10) - 3
energy_in = energy_score(in_dist)
energy_ood = energy_score(ood)
assert energy_in.mean() < energy_ood.mean(), "In-dist lower energy"
print("✓ Energy score working")
if __name__ == "__main__":
print("Lab 2: EnergyScore - PASSED")### Lab 3: Mahalanobis Distance
import numpy as np
def mahalanobis_distance(features, class_mean, cov_matrix):
"""Mahalanobis distance for OOD detection"""
diff = features - class_mean
try:
cov_inv = np.linalg.inv(cov_matrix)
mahal_dist = np.sqrt(np.sum(diff @ cov_inv * diff, axis=1))
except:
mahal_dist = np.linalg.norm(diff, axis=1)
return mahal_dist
# Test
np.random.seed(42)
features = np.random.randn(50, 10)
class_mean = features.mean(axis=0)
cov = np.cov(features.T)
distances = mahalanobis_distance(features, class_mean, cov)
assert distances.shape == (50,), "Distance shape"
assert np.all(np.isfinite(distances)), "Finite distances"
print("✓ Mahalanobis working")
if __name__ == "__main__":
print("Lab 3: Mahalanobis - PASSED")### Lab 4: OOD Detection Evaluation
import numpy as np
def ood_detection_metrics(in_dist_scores, ood_scores, lower_is_ood=False):
"""Evaluate OOD detection performance"""
# Combine and create labels
scores = np.concatenate([in_dist_scores, ood_scores])
labels = np.concatenate([np.zeros(len(in_dist_scores)), np.ones(len(ood_scores))])
# AUROC-like metric via threshold sweep
best_tpr = 0
best_fpr = 1
for threshold in np.linspace(scores.min(), scores.max(), 100):
if lower_is_ood:
pred = (scores <= threshold).astype(int)
else:
pred = (scores >= threshold).astype(int)
tp = ((pred == 1) & (labels == 1)).sum()
fp = ((pred == 1) & (labels == 0)).sum()
tn = ((pred == 0) & (labels == 0)).sum()
fn = ((pred == 0) & (labels == 1)).sum()
tpr = tp / (tp + fn + 1e-8)
fpr = fp / (fp + tn + 1e-8)
if tpr > best_tpr and fpr < best_fpr:
best_tpr = tpr
best_fpr = fpr
return best_tpr, best_fpr
# Test
np.random.seed(42)
in_dist = np.random.randn(100) + 1
ood = np.random.randn(100)
tpr, fpr = ood_detection_metrics(in_dist, ood, lower_is_ood=False)
assert 0 <= tpr <= 1, "Valid TPR"
assert 0 <= fpr <= 1, "Valid FPR"
print("✓ OOD evaluation working")
if __name__ == "__main__":
print("Lab 4: OODEvaluation - PASSED")