snapshot ensembles
**Snapshot Ensembles** are a computationally efficient ensemble technique that collects multiple diverse models along a single training run by using a cyclical learning rate schedule that periodically converges to different local minima, taking a "snapshot" (saved checkpoint) of the model at each convergence point. Instead of training N models independently (N× cost), snapshot ensembles produce N diverse models for approximately the cost of training a single model.
**Why Snapshot Ensembles Matter in AI/ML:**
Snapshot ensembles provide **ensemble benefits at near-single-model training cost** by exploiting the fact that cyclical learning rate schedules naturally visit diverse regions of the loss landscape, producing multiple functionally different models from one training trajectory.
• **Cyclical learning rate** — The learning rate follows a cosine annealing schedule that repeatedly warms up and decays: α(t) = α₀/2 · (cos(π·mod(t-1, T/M)/(T/M)) + 1), where T is total training iterations and M is the number of cycles; each cycle converges to a different local minimum
• **Snapshot collection** — At the end of each cosine cycle (when learning rate reaches its minimum and the model has converged to a local optimum), the model weights are saved as a snapshot; typically M=3-8 snapshots are collected per training run
• **Diversity through exploration** — Warming the learning rate back up after each snapshot escapes the current local minimum and explores new regions of the loss landscape; the subsequent cooldown converges to a different minimum, ensuring snapshot diversity
• **Ensemble at inference** — Predictions from all M snapshots are averaged (soft voting or probability averaging) to produce the final output; despite coming from a single training run, the diversity between snapshots provides meaningful variance reduction
• **Comparison to independent training** — While independent ensembles (training M separate models from scratch) typically produce slightly better diversity, snapshot ensembles achieve 70-90% of the full ensemble benefit at 1/M of the training cost
| Parameter | Typical Value | Notes |
|-----------|--------------|-------|
| Number of Cycles (M) | 3-8 | More cycles = more snapshots, less training per cycle |
| Initial Learning Rate | 0.1-0.3 | Warm restart maximum |
| Minimum Learning Rate | 10⁻⁴-10⁻⁶ | Convergence minimum |
| Schedule | Cosine annealing | Smooth decay per cycle |
| Training Cost | ~1× single model | Vs. M× for independent ensemble |
| Diversity | Moderate | Less than independent training |
| Accuracy Gain | 1-3% over single model | Task-dependent |
**Snapshot ensembles democratize ensemble learning by extracting multiple diverse models from a single training run through cyclical learning rate schedules, providing substantial accuracy and uncertainty estimation improvements at minimal additional training cost—making ensemble benefits accessible even when computational budgets prohibit training multiple independent models.**