polyak averaging

**Polyak Averaging** (Polyak-Ruppert Averaging) is a **convergence acceleration technique that averages all parameter iterates during optimization** — the average of all weights encountered during SGD converges faster than the final iterate for convex problems. **How Does Polyak Averaging Work?** - **Iterate**: Run SGD normally to get $ heta_1, heta_2, ..., heta_T$. - **Average**: $ar{ heta}_T = frac{1}{T}sum_{t=1}^T heta_t$ (or use a tail average for non-convex). - **Theory**: For convex problems, $ar{ heta}_T$ converges at the optimal $O(1/T)$ rate even with a constant learning rate. - **Papers**: Polyak (1990), Ruppert (1988). **Why It Matters** - **Theoretical Foundation**: Provides the theoretical justification for SWA and EMA techniques. - **Constant Learning Rate**: Enables using a larger, constant learning rate (the averaging cancels the noise). - **Practical**: EMA is the modern, practical version of Polyak averaging with exponential forgetting. **Polyak Averaging** is **the theoretical foundation for weight averaging** — the mathematically proven principle that averaging iterates accelerates convergence.