expected calibration error (ece)
**Expected Calibration Error (ECE)** is the primary metric for evaluating the calibration quality of a probabilistic classifier, measuring the average absolute difference between predicted confidence and actual accuracy across binned prediction groups. A perfectly calibrated model has ECE = 0, meaning that among all predictions made with confidence p, exactly fraction p are correct (e.g., of all predictions made with 90% confidence, exactly 90% should be correct).
**Why ECE Matters in AI/ML:**
ECE provides a **single-number summary of how much a model's confidence estimates deviate from reality**, enabling direct comparison of calibration quality across models and guiding the selection and tuning of post-hoc calibration methods.
• **Binned computation** — ECE partitions predictions into M equal-width or equal-mass bins by predicted confidence, then computes: ECE = Σ(|B_m|/N) · |acc(B_m) - conf(B_m)| where acc(B_m) is the actual accuracy and conf(B_m) is the average confidence within bin m
• **Reliability diagrams** — ECE is visualized through reliability diagrams (calibration curves) plotting actual accuracy vs. predicted confidence for each bin; a perfectly calibrated model produces points along the diagonal; deviations above indicate underconfidence, below indicate overconfidence
• **Bin count sensitivity** — ECE values depend significantly on the number of bins M (typically 10-15): too few bins mask miscalibration patterns, too many bins create noisy estimates with high variance; this sensitivity is a known limitation
• **Variants** — Maximum Calibration Error (MCE) reports the worst-bin deviation; Adaptive ECE (AdaECE) uses equal-mass bins for more stable estimates; Classwise ECE evaluates calibration per class; Kernel Calibration Error (KCE) avoids binning entirely
• **Modern model miscalibration** — Despite high accuracy, modern deep networks are systematically overconfident with ECE of 5-15% before calibration; temperature scaling typically reduces ECE to 1-3%, and the remaining error guides further calibration efforts
| Metric | Formula | Sensitivity | Best For |
|--------|---------|-------------|----------|
| ECE | Weighted avg |acc - conf| | Bin count dependent | Overall calibration summary |
| MCE | Max |acc - conf| per bin | Worst-case analysis | Safety-critical applications |
| AdaECE | ECE with equal-mass bins | More stable | Small datasets |
| Classwise ECE | Per-class ECE averaged | Class-level calibration | Multi-class problems |
| Brier Score | Mean (p - y)² | Combines accuracy + calibration | Joint evaluation |
| KCE | Kernel-based (no bins) | Smooth, no binning | Rigorous evaluation |
**Expected Calibration Error is the standard metric for assessing whether a model's confidence scores are trustworthy, providing a quantitative measure of the gap between predicted probabilities and observed outcomes that directly guides calibration improvement and determines whether a model's uncertainty estimates are reliable enough for confidence-based decision making.**