stochastic weight averaging

**SWA** (Stochastic Weight Averaging) is an **optimization technique that averages multiple checkpoints collected during training with a high or cyclical learning rate** — the averaged weights converge to wider, flatter minima that generalize better than the final checkpoint alone. **How Does SWA Work?** - **Train**: Train normally until near convergence. - **SWA Phase**: Continue with a high or cyclic learning rate for additional epochs. - **Collect**: Save the model weights at the end of each SWA epoch. - **Average**: $ heta_{SWA} = frac{1}{T}sum_t heta_t$ (running average of collected checkpoints). - **Paper**: Izmailov et al. (2018). **Why It Matters** - **Flat Minima**: SWA finds wider minima that generalize better (loss landscape is flat around SWA solution). - **Free Improvement**: 0.5-1.5% accuracy improvement with minimal additional training cost. - **PyTorch Built-In**: Available as torch.optim.swa_utils.AveragedModel. **SWA** is **averaging your way to a better model** — collecting checkpoints along a high-learning-rate trajectory to find wide, flat minima.