model averaging
**Model Averaging** is an ensemble technique that combines predictions from multiple trained models by computing their weighted or unweighted average, producing a consensus prediction that is typically more accurate and better calibrated than any individual model. Model averaging encompasses both simple arithmetic averaging (equal weights) and sophisticated Bayesian Model Averaging (BMA, weights proportional to posterior model probabilities).
**Why Model Averaging Matters in AI/ML:**
Model averaging provides **consistent, low-effort accuracy improvements** over single models by exploiting the diversity of predictions across different model instances, reducing variance and improving calibration with minimal implementation complexity.
• **Simple averaging** — Averaging the predictions (probabilities, logits, or regression outputs) of N models trained with different random seeds consistently improves accuracy by 0.5-2% and reduces calibration error; this is the simplest and most robust ensemble technique
• **Bayesian Model Averaging** — BMA weights models by their posterior probability p(M_i|D) ∝ p(D|M_i)·p(M_i), giving higher weight to models that better explain the data; the averaged prediction p(y|x,D) = Σ p(y|x,M_i)·p(M_i|D) is the Bayesian-optimal combination
• **Stochastic Weight Averaging (SWA)** — Rather than averaging predictions, SWA averages model weights along the training trajectory, producing a single model approximating the average of an ensemble; this provides ensemble-like benefits with single-model inference cost
• **Uniform averaging robustness** — Surprisingly, simple uniform averaging (equal weights) often performs as well as or better than optimized weighting schemes because weight optimization can overfit to the validation set, especially with few models
• **Geometric averaging** — Averaging log-probabilities (equivalent to geometric mean of probabilities) and renormalizing provides an alternative that can outperform arithmetic averaging when models have different confidence scales
| Averaging Method | Weights | Inference Cost | Implementation Complexity |
|-----------------|---------|----------------|--------------------------|
| Simple Average | Uniform (1/N) | N× single model | Minimal |
| Bayesian Model Averaging | Posterior p(M|D) | N× + weight computation | Moderate |
| Weighted Average | Validation-optimized | N× + optimization | Moderate |
| Stochastic Weight Avg | Weight-space average | 1× (single model) | Low |
| Exponential Moving Avg | Decay-weighted | 1× (single model) | Low |
| Geometric Average | Uniform on log scale | N× | Minimal |
**Model averaging is the simplest and most reliable technique for improving prediction quality in machine learning, providing consistent accuracy and calibration improvements by combining multiple models through straightforward arithmetic averaging, with theoretical guarantees of variance reduction that make it the default first step in any production ensemble strategy.**