uncertainty quantification in machine learning

# Uncertainty Quantification in Machine Learning

## Introduction & Motivation

Quantifying prediction uncertainty is critical for decision-making in safety-critical applications. Bayesian and ensemble methods provide calibrated confidence intervals, enabling risk-aware decisions in materials discovery, process control, and autonomous systems.

Motivation: Predict uncertainty alongside point estimates.

Applications: Risk assessment, confidence intervals, model reliability, safety-critical systems.

---

## Core Concepts & Theory

### Epistemic Uncertainty

Model uncertainty.

### Aleatoric Uncertainty

Data noise and randomness.

### Calibration

Matching confidence to accuracy.

### Confidence Intervals

Quantile-based uncertainty.

---

## Mathematical Formulation

Predictive Variance:
$$\sigma^2_{pred} = \sigma^2_{aleatoric} + \sigma^2_{epistemic}$$

Calibration Error:
$$ ext{CE} = \frac{1}{N} \sum_i |p_i - \hat{p}_i|$$

Credible Interval:
$$P(\hat{y}_L < f(x) < \hat{y}_U) = 1 - \alpha$$

---

## Advanced Theory & Extensions

### Bayesian Deep Learning

Posterior approximation.

### Ensemble Uncertainty

Model disagreement.

### Density Prediction Networks

Full predictive distribution.

---

## Computational Considerations

Posterior: O(N·D²) approximation.

Ensemble: O(M·D) for M models.

Calibration: O(N·log N) sorting.

---

## Practical Implementation Strategies

### Monte Carlo Dropout

Efficient uncertainty.

### Ensemble Methods

Model disagreement.

### Temperature Scaling

Confidence calibration.

---

## Benchmark Datasets & Evaluation

UCI Repository: Standard datasets.

OOD Datasets: Distribution shift testing.

Uncertainty Benchmarks: Calibration studies.

---

## Key Challenges & Limitations

### Overconfidence

Miscalibrated predictions.

### Computational Cost

Uncertainty estimation overhead.

### Assumption Violations

Model limitations.

---

## Hyperparameter Tuning

Dropout rate: 0.1-0.5.

Ensemble size: 10-100.

Temperature: 1.0-5.0.

---

## Real-World Applications & Case Studies

Drug Discovery: Confidence in predictions.

Process Control: Risk quantification.

Autonomous Systems: Decision confidence.

---

## Integration with Other Methods

Uncertainty + neural networks; + Bayesian methods; + decision theory.

---

## Summary & Key Takeaways

Uncertainty quantification enables reliable decisions.

Principles:
1. Epistemic: Model uncertainty.
2. Aleatoric: Data noise.
3. Estimation: Bayesian or ensemble.
4. Calibration: Matching confidence.
5. Decision: Risk-aware choices.

---

## Appendix: Practical Labs

### Lab 1: Monte Carlo Dropout

import numpy as np

class MCDropoutUncertainty:
 def __init__(self, dropout_rate=0.5, model_dim=20):
 self.dropout_rate = dropout_rate
 self.W = np.random.randn(model_dim, 1) * 0.1
 
 def forward_with_dropout(self, X):
 """Forward pass with dropout"""
 mask = np.random.binomial(1, 1 - self.dropout_rate, X.shape)
 X_dropped = X * mask / (1 - self.dropout_rate)
 return X_dropped @ self.W
 
 def predict_with_uncertainty(self, X, n_mc_samples=100):
 """Uncertainty via MC sampling"""
 predictions = np.array([self.forward_with_dropout(X) for _ in range(n_mc_samples)])
 
 mean = np.mean(predictions, axis=0)
 std = np.std(predictions, axis=0)
 
 return mean, std

mc = MCDropoutUncertainty()
X = np.random.randn(10, 20)

mean, std = mc.predict_with_uncertainty(X, n_mc_samples=100)
print(f"✓ Predictions with uncertainty: mean shape {mean.shape}, std shape {std.shape}")

### Lab 2: Ensemble Uncertainty

import numpy as np

class EnsembleUncertainty:
 def __init__(self, n_models=10, model_dim=20):
 self.models = [np.random.randn(model_dim, 1) * 0.1 for _ in range(n_models)]
 
 def predict_ensemble(self, X):
 """Ensemble predictions"""
 predictions = np.array([X @ m for m in self.models])
 
 mean = np.mean(predictions, axis=0)
 std = np.std(predictions, axis=0)
 
 return mean, std

ensemble_unc = EnsembleUncertainty(n_models=10)
X = np.random.randn(5, 20)

mean, std = ensemble_unc.predict_ensemble(X)
print(f"✓ Ensemble uncertainty: mean {mean.shape}, std {std.shape}")

### Lab 3: Calibration

import numpy as np

def calibration_error(confidence, accuracy):
 """Compute calibration error"""
 return np.mean(np.abs(confidence - accuracy))

def temperature_scaling(logits, temperature=1.0):
 """Scale confidence via temperature"""
 return logits / temperature

logits = np.random.randn(100)
conf = 1 / (1 + np.exp(-logits))

# Calibrate
calib_conf = temperature_scaling(logits, temperature=2.0)
calib_conf = 1 / (1 + np.exp(-calib_conf))

print(f"✓ Temperature scaling applied")

### Lab 4: Uncertainty-Aware Decisions

import numpy as np

class RiskAwareDiagnosis:
 def __init__(self, confidence_threshold=0.9):
 self.threshold = confidence_threshold
 
 def predict_with_defer(self, predictions, uncertainties):
 """Predict or defer to expert"""
 decisions = []
 
 for pred, unc in zip(predictions, uncertainties):
 if unc < (1 - self.threshold):
 decisions.append(('predict', pred))
 else:
 decisions.append(('defer', None))
 
 return decisions

riskaware = RiskAwareDiagnosis(confidence_threshold=0.9)

predictions = np.random.rand(10)
uncertainties = np.random.rand(10) * 0.5

decisions = riskaware.predict_with_defer(predictions, uncertainties)
defer_count = sum(1 for d in decisions if d[0] == 'defer')

print(f"✓ Deferred {defer_count}/10 predictions")

---