Polyak Averaging (Polyak-Ruppert Averaging) is a convergence acceleration technique that averages all parameter iterates during optimization — the average of all weights encountered during SGD converges faster than the final iterate for convex problems.

How Does Polyak Averaging Work?

• Iterate: Run SGD normally to get $ heta_1, heta_2, ..., heta_T$.
• Average: $ar{ heta}_T = frac{1}{T}sum_{t=1}^T heta_t$ (or use a tail average for non-convex).
• Theory: For convex problems, $ar{ heta}_T$ converges at the optimal $O(1/T)$ rate even with a constant learning rate.
• Papers: Polyak (1990), Ruppert (1988).

Why It Matters

• Theoretical Foundation: Provides the theoretical justification for SWA and EMA techniques.
• Constant Learning Rate: Enables using a larger, constant learning rate (the averaging cancels the noise).
• Practical: EMA is the modern, practical version of Polyak averaging with exponential forgetting.

Polyak Averaging is the theoretical foundation for weight averaging — the mathematically proven principle that averaging iterates accelerates convergence.