calibration
**Model Calibration** is the **property of a probabilistic classifier where predicted confidence scores accurately reflect empirical outcome probabilities** — a well-calibrated model that says "70% confidence" is correct approximately 70% of the time across all such predictions, making calibration essential for risk-sensitive applications where downstream decisions depend on the model's expressed uncertainty.
**What Is Model Calibration?**
- **Definition**: A model is perfectly calibrated when for all confidence levels p, among all predictions made with confidence p, exactly fraction p of those predictions are correct: P(Y=y | f(x)=p) = p for all p ∈ [0,1].
- **Calibration vs. Accuracy**: A model can be highly accurate but poorly calibrated (correct 95% of the time but expresses 99.9% confidence on every prediction) — or accurate and well-calibrated (correct 70% of the time when expressing 70% confidence).
- **Why It Matters**: In medical diagnosis, insurance pricing, weather forecasting, and financial risk — decisions are made based on predicted probabilities. If those probabilities are wrong, decisions are systematically miscalibrated.
**Why Calibration Matters**
- **Clinical Decision Support**: A radiology AI that outputs "99% probability of malignancy" on benign lesions causes unnecessary biopsies. Proper calibration ensures that a 90% confidence prediction leads to different clinical action than a 40% confidence prediction.
- **Weather Forecasting**: The gold standard of calibration — a forecast of 70% chance of rain should correspond to actual rain 70% of the days it is predicted. National Weather Service forecasts are among the best-calibrated probabilistic systems in existence.
- **Autonomous Vehicles**: Object detection confidence must be calibrated to trigger appropriate response — an over-confident pedestrian detector that expresses 99% confidence on false detections causes incorrect braking behavior.
- **LLM Alignment**: RLHF fine-tuning tends to make language models overconfident because human raters prefer assertive, direct answers — creating a systematic miscalibration toward false certainty.
- **Ensemble Systems**: Calibrated base models are required for proper ensemble combination — combining overconfident base models produces poorly calibrated ensembles.
**Measuring Calibration**
**Reliability Diagram (Calibration Plot)**:
- Bin predictions into ranges (0-10%, 10-20%, ..., 90-100%).
- Plot predicted confidence (x-axis) against empirical accuracy (y-axis).
- Perfect calibration = diagonal line; above diagonal = underconfident; below diagonal = overconfident.
**Expected Calibration Error (ECE)**:
ECE = Σ (|B_m| / n) × |acc(B_m) - conf(B_m)|
Where B_m = predictions in bin m, acc = accuracy, conf = mean confidence.
Lower ECE = better calibration.
**Maximum Calibration Error (MCE)**: Worst-case calibration error across all bins — more conservative than ECE.
**Negative Log-Likelihood (NLL)**: Proper scoring rule penalizing both accuracy and calibration — theoretically optimal measure.
**Why Modern Neural Networks Are Overconfident**
Guo et al. (ICML 2017) showed that modern deep neural networks trained with cross-entropy loss are significantly overconfident — they are more accurate than older networks but worse calibrated:
- **Early Stopping Effects**: Overfit models memorize training labels with near-zero loss, pushing output probabilities toward 0 or 1.
- **Batch Normalization**: Shifts internal representations in ways that increase output sharpness.
- **Skip Connections**: Allow gradient flow that sharpens predictions beyond calibrated levels.
- **Weight Decay Reduction**: Less regularization means less smoothing of output distributions.
- **RLHF**: Optimizing for human preference ratings rewards confident, assertive language — systematically increasing expressed certainty.
**Calibration Techniques**
| Technique | Method | When to Use | Complexity |
|-----------|--------|-------------|------------|
| Temperature Scaling | Single parameter T: softmax(logits/T) | Post-training, simple models | Very low |
| Platt Scaling | Sigmoid on output scores | Binary classification | Low |
| Isotonic Regression | Non-parametric monotonic mapping | When data abundant | Medium |
| Dirichlet Calibration | Multi-class generalization of Platt | Multi-class classification | Medium |
| Bayesian Deep Learning | Uncertainty in weights | Built-in calibration | High |
**Temperature Scaling in Practice**
The simplest and most effective post-hoc calibration method for neural networks:
1. Train the model normally (do not change weights).
2. On a held-out calibration set, find scalar T that minimizes NLL: T* = argmin_T NLL(softmax(logits/T)).
3. At inference: use softmax(logits/T*) as calibrated probability.
- T > 1: Softens distribution (reduces overconfidence).
- T < 1: Sharpens distribution (corrects underconfidence).
For LLMs, temperature scaling directly corresponds to the temperature parameter used during sampling — this is not coincidental; temperature was originally a calibration tool.
Model calibration is **the bridge between predicted confidence and trustworthy uncertainty communication** — in every domain where AI predictions inform real decisions, the gap between expressed confidence and empirical accuracy determines whether AI assistance improves or degrades human judgment.