Model Calibration Probability Estimation Confidence Adjustment

# Model Calibration: Probability Estimation & Confidence Adjustment

## Introduction & Motivation

Calibration ensures predicted probabilities match true frequency. Uncalibrated models (e.g., SVM, neural nets) output unreliable probabilities. Platt scaling, temperature scaling adjust post-hoc. Critical for risk-sensitive applications (medicine, finance).

Motivation: Many algorithms optimize accuracy not probability. Calibration bridges gap: predicted 80% should occur 80% of time. Essential for decision thresholds, cost-sensitive learning.

Applications: Medical diagnosis confidence, financial risk assessment, autonomous driving safety scores.

---

## Core Concepts & Theory

### Calibration Metrics

Expected Calibration Error (ECE): average difference between predicted and true probability per bin.

Brier Score: mean squared error between predicted and actual probability.

---

## Summary & Key Takeaways

Calibration aligns predicted probabilities with true frequency via post-hoc adjustment methods.

Principles:
1. Well-calibrated probabilities essential for decisions.
2. Post-hoc methods preserve ranking.
3. Temperature scaling simple, effective.
4. ECE measures calibration quality.
5. Cross-validation prevents overfitting in calibration.

---

---

## Appendix: Practical Labs

### Lab 1: Platt Scaling

from sklearn.calibration import CalibratedClassifierCV
from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

svm = SVC(kernel='rbf', probability=False)
calibrated = CalibratedClassifierCV(svm, method='sigmoid', cv=5)
calibrated.fit(X_train, y_train)

probs = calibrated.predict_proba(X_test)

print(f"Probability shape: {probs.shape}")
assert probs.shape == (len(X_test), 3), "Correct probability shape"
print("✓ Platt scaling working")

if __name__ == "__main__":
 print("Lab 1: Platt - PASSED")

### Lab 2: ECE Calculation

from sklearn.metrics import brier_score_loss
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import numpy as np

iris = load_iris()
X, y = iris.data, (iris.target == 0).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

lr = LogisticRegression(max_iter=200)
lr.fit(X_train, y_train)

probs = lr.predict_proba(X_test)[:, 1]
brier = brier_score_loss(y_test, probs)

print(f"Brier score: {brier:.4f}")
assert 0 <= brier <= 1, "Brier should be in [0,1]"
print("✓ ECE calculation working")

if __name__ == "__main__":
 print("Lab 2: ECE - PASSED")

### Lab 3: Temperature Scaling

from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import numpy as np

iris = load_iris()
X, y = iris.data, (iris.target == 0).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

lr = LogisticRegression(max_iter=200)
lr.fit(X_train, y_train)

logits = lr.decision_function(X_test)
temps = [0.5, 1.0, 2.0]

for T in temps:
 probs = 1 / (1 + np.exp(-logits / T))
 print(f"Temperature {T}: mean prob {probs.mean():.4f}")

print("✓ Temperature scaling working")

if __name__ == "__main__":
 print("Lab 3: Temperature - PASSED")

### Lab 4: Calibration Curve

from sklearn.calibration import calibration_curve
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

iris = load_iris()
X, y = iris.data, (iris.target == 0).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

lr = LogisticRegression(max_iter=200)
lr.fit(X_train, y_train)

probs = lr.predict_proba(X_test)[:, 1]
prob_true, prob_pred = calibration_curve(y_test, probs, n_bins=10)

print(f"Calibration curve bins: {len(prob_true)}")
assert len(prob_true) <= 10, "Should have <=10 bins"
print("✓ Calibration curve working")

if __name__ == "__main__":
 print("Lab 4: Calibration Curve - PASSED")