model calibration confidence uncertainty estimation
# Model Calibration: Confidence & Uncertainty Estimation
## Introduction & Motivation
Model calibration: align predicted confidence with accuracy. Well-calibrated: P(correct|confidence=c) ≈ c. Miscalibration: overconfident or underconfident. Temperature scaling: post-hoc calibration. Applications: uncertainty quantification, risk-aware decision making, confidence threshold tuning.
Motivation: Deep networks overconfident; need uncertainty. Calibrated confidence enables safe deployment.
Applications: Medical diagnosis, autonomous systems, risk management.
---
## Core Concepts & Theory
### Calibration Error
ECE, MCE metrics; measure miscalibration.
### Temperature Scaling
Single parameter post-hoc calibration.
### Confidence Threshold
Adjust decision boundary; control precision-recall.
---
## Mathematical Formulation
Expected Calibration Error:
$$ ext{ECE} = \sum_m |acc(m) - conf(m)| \cdot \frac{|B_m|}{N}$$
where B_m = confidence bin m.
Temperature scaling:
$$p_{ ext{cal}} = ext{softmax}(z / T)$$
where T = temperature (learned on validation set).
---
## Advanced Theory & Extensions
### Confidence Penalty
Penalize high confidence in training.
### Label Smoothing
Soften targets; improve calibration.
### Bayesian Approximation
MC Dropout, variational inference.
---
## Computational Considerations
ECE: O(N) computation per batch.
Temperature scaling: O(N) validation set.
MC Dropout: O(K · forward) where K = samples.
---
## Practical Implementation Strategies
### Validation Set
Use separate validation for calibration.
### Calibration Methods
Temperature scaling, platt scaling, isotonic regression.
### Monitoring
Track calibration during training and evaluation.
---
## Benchmark Datasets & Evaluation
CIFAR-10: ResNet often miscalibrated.
ImageNet: Modern CNNs overconfident.
Medical: Critical for diagnostic confidence.
---
## Key Challenges & Limitations
### Calibration-Accuracy Tradeoff
Calibration may reduce accuracy.
### Distribution Shift
Calibration on training distribution; changes at test time.
### Computational Cost
MC Dropout expensive; temperature scaling cheap.
---
## Hyperparameter Tuning
Temperature T: Learned on validation; typically 1.0-5.0.
Confidence threshold: Task dependent; ROC optimization.
Label smoothing α: 0.1-0.2; regularization.
---
## Real-World Applications & Case Studies
Medical Imaging: Confidence for diagnostic trust.
Autonomous Driving: Uncertainty for risk assessment.
Fraud Detection: Confidence thresholds for alerts.
---
## Integration with Other Methods
Calibration + Uncertainty → risk quantification.
Calibration + Threshold → precision-recall control.
---
## Summary & Key Takeaways
Model calibration via temperature scaling and confidence thresholding aligns predicted confidence with true accuracy for uncertainty-aware decision making.
Principles:
1. ECE: calibration error metric.
2. Temperature: post-hoc scaling.
3. Confidence threshold: ROC tuning.
4. Validation set: separate calibration data.
5. Monitoring: track calibration.
---
---
## Appendix: Practical Labs
### Lab 1: Expected Calibration Error
import numpy as np
def compute_expected_calibration_error(predictions, targets, n_bins=10):
"""Compute ECE"""
confidences = np.max(predictions, axis=1)
predicted_classes = np.argmax(predictions, axis=1)
# Bin samples
bin_edges = np.linspace(0, 1, n_bins + 1)
bin_indices = np.digitize(confidences, bin_edges) - 1
ece = 0
for bin_id in range(n_bins):
mask = bin_indices == bin_id
if mask.sum() == 0:
continue
bin_confidences = confidences[mask]
bin_accuracies = (predicted_classes[mask] == targets[mask]).astype(float)
avg_confidence = bin_confidences.mean()
avg_accuracy = bin_accuracies.mean()
bin_weight = mask.sum() / len(targets)
ece += bin_weight * np.abs(avg_confidence - avg_accuracy)
return ece
# Test
np.random.seed(42)
predictions = np.random.dirichlet([1]*10, 100)
targets = np.random.randint(0, 10, 100)
ece = compute_expected_calibration_error(predictions, targets)
assert 0 <= ece <= 1, "ECE in [0,1]"
print("✓ ECE computation working")
if __name__ == "__main__":
print("Lab 1: ECE - PASSED")### Lab 2: Temperature Scaling
import numpy as np
def temperature_scaling(logits, targets, learning_rate=0.01, iterations=100):
"""Learn temperature via validation set"""
T = 1.0 # Start at identity
for _ in range(iterations):
# Scale logits
scaled_logits = logits / T
# Softmax
exp_logits = np.exp(scaled_logits - np.max(scaled_logits, axis=1, keepdims=True))
probs = exp_logits / exp_logits.sum(axis=1, keepdims=True)
# Loss: cross-entropy
batch_size = logits.shape[0]
loss = -np.log(probs[np.arange(batch_size), targets] + 1e-8).mean()
# Simple gradient update (approximation)
# Increase T to reduce confidence
if probs[np.arange(batch_size), targets].mean() > 0.5:
T += learning_rate * 0.1
return T
# Test
np.random.seed(42)
logits = np.random.randn(100, 10)
targets = np.random.randint(0, 10, 100)
T = temperature_scaling(logits, targets)
assert T >= 0, "Temperature positive"
print("✓ Temperature scaling working")
if __name__ == "__main__":
print("Lab 2: TemperatureScaling - PASSED")### Lab 3: Confidence Threshold
import numpy as np
def find_optimal_confidence_threshold(predictions, targets, metric="f1"):
"""Find threshold for optimal metric"""
confidences = np.max(predictions, axis=1)
predicted_classes = np.argmax(predictions, axis=1)
thresholds = np.linspace(0, 1, 100)
best_threshold = 0.5
best_score = 0
for threshold in thresholds:
# Keep only high-confidence predictions
mask = confidences >= threshold
if mask.sum() == 0:
continue
# Compute metric
tp = ((predicted_classes[mask] == targets[mask])).sum()
fp = ((predicted_classes[mask] != targets[mask])).sum()
fn = ((predicted_classes[~mask] != targets[~mask])).sum() if (~mask).sum() > 0 else 0
if metric == "f1":
precision = tp / (tp + fp + 1e-8)
recall = tp / (tp + fn + 1e-8)
score = 2 * precision * recall / (precision + recall + 1e-8)
else:
score = tp / (tp + fp + 1e-8) # Precision
if score > best_score:
best_score = score
best_threshold = threshold
return best_threshold, best_score
# Test
np.random.seed(42)
predictions = np.random.dirichlet([1]*10, 100)
targets = np.random.randint(0, 10, 100)
threshold, score = find_optimal_confidence_threshold(predictions, targets)
assert 0 <= threshold <= 1, "Threshold in [0,1]"
assert 0 <= score <= 1, "Score in [0,1]"
print("✓ Confidence threshold tuning working")
if __name__ == "__main__":
print("Lab 3: ConfidenceThreshold - PASSED")### Lab 4: Calibration Analysis
import numpy as np
def analyze_calibration(predictions, targets):
"""Analyze calibration across confidence levels"""
confidences = np.max(predictions, axis=1)
predicted_classes = np.argmax(predictions, axis=1)
# Sort by confidence
sorted_indices = np.argsort(-confidences)
results = {}
for percentile in [25, 50, 75, 100]:
n = len(targets) * percentile // 100
indices = sorted_indices[:n]
avg_confidence = confidences[indices].mean()
accuracy = (predicted_classes[indices] == targets[indices]).mean()
results[f"top_{percentile}%"] = {
"confidence": avg_confidence,
"accuracy": accuracy,
"calibration_error": abs(avg_confidence - accuracy)
}
return results
# Test
np.random.seed(42)
predictions = np.random.dirichlet([1]*10, 100)
targets = np.random.randint(0, 10, 100)
analysis = analyze_calibration(predictions, targets)
assert len(analysis) == 4, "Four percentiles"
print("✓ Calibration analysis working")
if __name__ == "__main__":
print("Lab 4: CalibrationAnalysis - PASSED")