Out-of-Distribution Detection Anomaly Likelihood Confidence Estimation

# Out-of-Distribution Detection: Anomaly Likelihood & Confidence Estimation

## Introduction & Motivation

OOD detection identifies when inputs differ from training distribution. Likelihood-based: lower density → OOD. Confidence-based: low model confidence → OOD. Distance-based: Mahalanobis distance in feature space. Critical for safe deployment, adversarial robustness.

Motivation: Models confident on OOD examples; dangerous in production. Detect and reject uncertain inputs or flag for human review.

Applications: Autonomous driving safety, medical imaging, fraud detection, adversarial robustness.

---

## Core Concepts & Theory

### Density-Based OOD

Estimate P(x); threshold on log-likelihood.

### Confidence-Based OOD

Use softmax entropy; high entropy → OOD.

### Distance-Based OOD

Mahalanobis distance in learned representation.

---

## Mathematical Formulation

Log-likelihood threshold:
$$ ext{OOD if } \log p(x) < au$$

Entropy-based:
$$H(\hat{p}) = -\sum_c \hat{p}_c \log \hat{p}_c$$
$$ ext{OOD if } H(\hat{p}) > au$$

Mahalanobis distance:
$$D_M(x) = \sqrt{(f(x) - \mu)^T \Sigma^{-1} (f(x) - \mu)}$$

where f(x) = feature representation, μ, Σ = class statistics.

---

## Advanced Theory & Extensions

### Energy-Based OOD Detection

Use unnormalized scores; more stable than softmax.

### Outlier Exposure

Train discriminator between ID and diverse OOD samples.

### ODIN (Temperature Scaling + Gradient)

Input perturbation; amplifies model confidence gap.

---

## Computational Considerations

Likelihood computation: O(d) for Gaussian; O(d³) for covariance.

Mahalanobis: O(d²) per-class; O(cd) total.

Online adaptation: Update μ, Σ incrementally.

---

## Practical Implementation Strategies

### Calibration

Train OOD detector threshold on validation OOD samples.

### Baseline Diversity

Use multiple baseline methods; ensemble for robustness.

### Computational Efficiency

Use efficient covariance estimates (diagonal, low-rank).

---

## Benchmark Datasets & Evaluation

ImageNet vs SVHN/Texture: Standard OOD benchmark.

MNIST vs Fashion-MNIST: Classic anomaly detection.

Metrics: AUROC, AUPR, FPR@95%TPR.

---

## Key Challenges & Limitations

### Diverse OOD

Single threshold doesn't work for all OOD types.

### Computational Cost

Mahalanobis requires O(d²) storage; scalability issues.

### Training-Test Mismatch

OOD characteristics differ at test time.

---

## Hyperparameter Tuning

Threshold τ: Grid search on validation OOD.

Temperature T: 0.1-5.0; higher = flatter probabilities.

Gaussian parameters: Diagonal vs. full covariance.

---

## Real-World Applications & Case Studies

Autonomous Driving: Detect unfamiliar road conditions; fallback to human.

Medical: Reject images outside training distribution; avoid errors.

Finance: Flag unusual transactions; prevent fraud.

---

## Integration with Other Methods

OOD + Uncertainty → epistemic vs. aleatoric uncertainty.

OOD + Active Learning → query unusual examples for labeling.

---

## Summary & Key Takeaways

OOD detection uses likelihood, confidence, or distance-based methods to identify inputs outside training distribution.

Principles:
1. Density-based: threshold on log-likelihood.
2. Confidence-based: softmax entropy or energy scores.
3. Distance-based: Mahalanobis in feature space.
4. Calibration essential; validate on diverse OOD.
5. Ensemble multiple methods for robustness.

---

---

## Appendix: Practical Labs

### Lab 1: Gaussian Likelihood OOD Detection

import numpy as np
from scipy.stats import multivariate_normal

def gaussian_likelihood_ood(X_train, X_test, threshold=None):
 """Fit Gaussian to train; score test samples"""
 mean = X_train.mean(axis=0)
 cov = np.cov(X_train.T)
 
 dist = multivariate_normal(mean=mean, cov=cov)
 log_probs = dist.logpdf(X_test)
 
 if threshold is None:
 threshold = np.percentile(dist.logpdf(X_train), 5)
 
 ood_scores = -log_probs
 return ood_scores, threshold

# Data
np.random.seed(42)
X_train = np.random.randn(100, 5)
X_id_test = np.random.randn(20, 5)
X_ood_test = np.random.randn(20, 5) * 3 # Shifted

scores_id, threshold = gaussian_likelihood_ood(X_train, X_id_test)
scores_ood, _ = gaussian_likelihood_ood(X_train, X_ood_test, threshold)

id_detected = (scores_id < threshold).sum()
ood_detected = (scores_ood > threshold).sum()

print(f"ID correctly detected as ID: {id_detected}/20")
print(f"OOD correctly detected as OOD: {ood_detected}/20")
assert id_detected > 5, "Should detect some ID samples"
assert ood_detected > 5, "Should detect some OOD samples"
print("✓ Gaussian likelihood OOD working")

if __name__ == "__main__":
 print("Lab 1: Gaussian Likelihood - PASSED")

### Lab 2: Entropy-Based OOD

import numpy as np

def entropy_ood(probs, threshold=None):
 """Detect OOD via softmax entropy"""
 entropy = -np.sum(probs * np.log(probs + 1e-8), axis=1)
 
 if threshold is None:
 threshold = np.mean(entropy)
 
 is_ood = entropy > threshold
 return entropy, threshold, is_ood

# Data: ID is confident, OOD is uncertain
np.random.seed(42)
probs_id = np.random.dirichlet([10, 1, 1], 20) # Confident
probs_ood = np.random.dirichlet([1, 1, 1], 20) # Uniform

entropy_id, threshold, _ = entropy_ood(probs_id)
entropy_ood_val, _, is_ood_detected = entropy_ood(probs_ood, threshold)

print(f"ID entropy: {entropy_id.mean():.4f}")
print(f"OOD entropy: {entropy_ood_val.mean():.4f}")
print(f"OOD detected: {is_ood_detected.sum()}/20")
assert entropy_id.mean() < entropy_ood_val.mean(), "OOD should have higher entropy"
print("✓ Entropy-based OOD working")

if __name__ == "__main__":
 print("Lab 2: Entropy OOD - PASSED")

### Lab 3: Mahalanobis Distance OOD

import numpy as np
from scipy.spatial.distance import mahalanobis

def mahalanobis_ood(X_train, X_test, y_train=None):
 """Compute Mahalanobis distance; lower = ID, higher = OOD"""
 if y_train is not None:
 # Per-class statistics
 classes = np.unique(y_train)
 min_distances = []
 for x in X_test:
 dists = []
 for c in classes:
 X_c = X_train[y_train == c]
 mu_c = X_c.mean(axis=0)
 cov_c = np.cov(X_c.T)
 d = mahalanobis(x, mu_c, np.linalg.inv(cov_c + np.eye(X_c.shape[1])*1e-6))
 dists.append(d)
 min_distances.append(min(dists))
 else:
 mu = X_train.mean(axis=0)
 cov = np.cov(X_train.T)
 cov_inv = np.linalg.inv(cov + np.eye(X_train.shape[1])*1e-6)
 min_distances = [mahalanobis(x, mu, cov_inv) for x in X_test]
 
 return np.array(min_distances)

# Data
np.random.seed(42)
X_train = np.random.randn(100, 5)
y_train = np.random.randint(0, 2, 100)
X_test_id = np.random.randn(20, 5)
X_test_ood = np.random.randn(20, 5) * 5

dist_id = mahalanobis_ood(X_train, X_test_id, y_train)
dist_ood = mahalanobis_ood(X_train, X_test_ood, y_train)

print(f"ID Mahalanobis: {dist_id.mean():.3f}")
print(f"OOD Mahalanobis: {dist_ood.mean():.3f}")
assert dist_id.mean() < dist_ood.mean(), "OOD should have larger distance"
print("✓ Mahalanobis OOD working")

if __name__ == "__main__":
 print("Lab 3: Mahalanobis - PASSED")

### Lab 4: OOD Detection Evaluation

import numpy as np
from sklearn.metrics import roc_auc_score, roc_curve

def evaluate_ood_detection(scores_id, scores_ood):
 """AUROC: higher score = OOD"""
 y_true = np.concatenate([np.zeros(len(scores_id)), np.ones(len(scores_ood))])
 scores = np.concatenate([scores_id, scores_ood])
 
 auroc = roc_auc_score(y_true, scores)
 fpr, tpr, _ = roc_curve(y_true, scores)
 
 # FPR@95%TPR
 idx_95 = np.argmin(np.abs(tpr - 0.95))
 fpr_95 = fpr[idx_95]
 
 return auroc, fpr_95

# Data
np.random.seed(42)
scores_id = np.random.randn(100) * 0.5
scores_ood = np.random.randn(100) * 0.5 + 2 # Higher scores

auroc, fpr95 = evaluate_ood_detection(scores_id, scores_ood)
print(f"AUROC: {auroc:.4f}, FPR@95TPR: {fpr95:.4f}")
assert 0 <= auroc <= 1, "AUROC should be in [0,1]"
assert 0 <= fpr95 <= 1, "FPR should be in [0,1]"
print("✓ OOD evaluation working")

if __name__ == "__main__":
 print("Lab 4: Evaluation - PASSED")