Expected Calibration Error (ECE) is the primary metric for evaluating the calibration quality of a probabilistic classifier, measuring the average absolute difference between predicted confidence and actual accuracy across binned prediction groups. A perfectly calibrated model has ECE = 0, meaning that among all predictions made with confidence p, exactly fraction p are correct (e.g., of all predictions made with 90% confidence, exactly 90% should be correct).

Why ECE Matters in AI/ML: ECE provides a single-number summary of how much a model's confidence estimates deviate from reality, enabling direct comparison of calibration quality across models and guiding the selection and tuning of post-hoc calibration methods.

• Binned computation — ECE partitions predictions into M equal-width or equal-mass bins by predicted confidence, then computes: ECE = Σ(|B_m|/N) · |acc(B_m) - conf(B_m)| where acc(B_m) is the actual accuracy and conf(B_m) is the average confidence within bin m • Reliability diagrams — ECE is visualized through reliability diagrams (calibration curves) plotting actual accuracy vs. predicted confidence for each bin; a perfectly calibrated model produces points along the diagonal; deviations above indicate underconfidence, below indicate overconfidence • Bin count sensitivity — ECE values depend significantly on the number of bins M (typically 10-15): too few bins mask miscalibration patterns, too many bins create noisy estimates with high variance; this sensitivity is a known limitation • Variants — Maximum Calibration Error (MCE) reports the worst-bin deviation; Adaptive ECE (AdaECE) uses equal-mass bins for more stable estimates; Classwise ECE evaluates calibration per class; Kernel Calibration Error (KCE) avoids binning entirely • Modern model miscalibration — Despite high accuracy, modern deep networks are systematically overconfident with ECE of 5-15% before calibration; temperature scaling typically reduces ECE to 1-3%, and the remaining error guides further calibration efforts

Expected Calibration Error is the standard metric for assessing whether a model's confidence scores are trustworthy, providing a quantitative measure of the gap between predicted probabilities and observed outcomes that directly guides calibration improvement and determines whether a model's uncertainty estimates are reliable enough for confidence-based decision making.