exponential moving average

**EMA** (Exponential Moving Average) is an **optimization technique that maintains a shadow copy of model weights as an exponentially weighted moving average** — the EMA model is used for evaluation/inference while the original model is used for gradient-based training. **How Does EMA Work?** - **Update**: After each training step: $ heta_{EMA} = alpha cdot heta_{EMA} + (1-alpha) cdot heta_{train}$ (typically $alpha = 0.999$ or $0.9999$). - **Train**: The main model $ heta_{train}$ is updated by the optimizer normally. - **Evaluate**: Use $ heta_{EMA}$ for validation, testing, and deployment. - **Smooth**: EMA averages out the noise from individual gradient updates. **Why It Matters** - **Standard Practice**: EMA is used in virtually all modern training recipes (ViT, diffusion models, LLMs). - **Free Accuracy**: Typically 0.3-1.0% accuracy improvement at no additional training cost. - **Stability**: The EMA model is more stable and less susceptible to overfitting than the raw model. **EMA** is **the smooth shadow model** — maintaining a running average of weights that captures the model's best state throughout training.