Deep Ensembles is the gold standard method for uncertainty quantification in deep learning, combining predictions from multiple independently trained neural networks to produce both improved accuracy and reliable uncertainty estimates — where prediction disagreement among ensemble members captures epistemic uncertainty (what the model doesn't know) while maintaining the simplicity of training M standard networks with different random initializations, consistently outperforming more sophisticated Bayesian approximations in empirical benchmarks.

What Are Deep Ensembles?

• Method: Train M neural networks (typically 3-10) independently with different random weight initializations and optionally different data shuffling.
• Prediction: Average the outputs for regression; average probabilities or use majority voting for classification.
• Uncertainty: Compute variance (disagreement) across ensemble members — high variance indicates the model is uncertain.
• Key Paper: Lakshminarayanan et al. (2017), "Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles."

Why Deep Ensembles Matter

• Uncertainty Quality: Empirically the best-calibrated uncertainty estimates among practical deep learning methods — consistently outperform MC Dropout, SWAG, and variational inference.
• OOD Detection: Ensemble disagreement naturally increases for out-of-distribution inputs — providing a built-in anomaly detector.
• Accuracy Boost: Averaging M networks reduces variance, typically improving accuracy by 1-3% over single models.
• Simplicity: No architectural changes, no special training procedures — just train M standard networks.
• Robustness: Each member sees slightly different loss landscapes due to random initialization, making the ensemble robust to local minima.

How Deep Ensembles Work

Training: For $m = 1, ldots, M$:

• Initialize network $f_m$ with random weights $ heta_m$.
• Train on the same dataset with standard procedure (optionally with different data augmentation or shuffling).

Inference:

• Mean Prediction: $ar{y} = frac{1}{M}sum_{m=1}^{M} f_m(x)$
• Epistemic Uncertainty: $ ext{Var}[y] = frac{1}{M}sum_{m=1}^{M}(f_m(x) - ar{y})^2$
• For classification: predictive entropy of averaged probabilities.

Comparison with Other Uncertainty Methods

Efficiency Improvements

• BatchEnsemble: Share most parameters, only learn per-member scaling factors — M × less memory.
• Snapshot Ensembles: Save checkpoints during cyclic learning rate schedule — single training run produces M models.
• Hyperensembles: Generate ensemble member weights from a hypernetwork.
• Multi-Head Ensembles: Shared backbone with M separate heads — reduced compute with similar uncertainty quality.
• Packed Ensembles: Efficient parameter sharing through structured subnetworks within a single model.

Deep Ensembles are the simple, powerful, and embarrassingly effective solution for knowing what your neural network doesn't know — proving that the most straightforward approach (just train multiple networks) remains the benchmark that more theoretically elegant methods struggle to surpass.