gradient noise

**Gradient Noise** is the **deliberate addition of noise to gradient updates during training** — typically Gaussian noise with decaying variance, which helps escape local minima, improves generalization, and can approximate Bayesian posterior sampling. **How Does Gradient Noise Work?** - **Injection**: $ ilde{g} = g + mathcal{N}(0, sigma_t^2)$ where $sigma_t$ decays over training. - **Schedule**: $sigma_t = sigma_0 / (1 + t)^gamma$ with $gamma approx 0.55$. - **Mini-Batch Noise**: SGD inherently has gradient noise from mini-batch sampling. Added noise amplifies this effect. - **Paper**: Neelakantan et al., "Adding Gradient Noise Improves Learning" (2015). **Why It Matters** - **Escape Local Minima**: Noise helps SGD escape sharp local minima and find flatter ones (better generalization). - **Bayesian Connection**: Gradient noise with appropriate scaling can approximate Langevin dynamics for Bayesian inference. - **Deep Networks**: Particularly helpful for very deep networks where deterministic gradients can get trapped. **Gradient Noise** is **controlled randomness in optimization** — deliberately shaking the optimizer to help it find better solutions in the loss landscape.