gradient normalization

**Gradient Normalization** is the **practice of normalizing gradient magnitudes during training** — either by clipping the gradient norm to a maximum value (gradient clipping) or by scaling gradients to have unit norm, preventing exploding gradients and stabilizing training. **Types of Gradient Normalization** - **Gradient Clipping by Norm**: $hat{g} = g cdot min(1, c/||g||)$. Clips when $||g|| > c$. - **Gradient Clipping by Value**: Clip each element independently: $hat{g}_i = ext{clip}(g_i, -c, c)$. - **Unit Norm**: Scale to unit norm: $hat{g} = g / ||g||$. - **Gradient Scaling**: Scale gradients by a constant factor (used in mixed-precision training). **Why It Matters** - **Stability**: Prevents exploding gradients in RNNs, transformers, and deep networks. - **Necessary for LLMs**: Gradient clipping (typically $c = 1.0$) is standard in all transformer pre-training. - **Mixed Precision**: Loss scaling + gradient unscaling is critical for FP16/BF16 training. **Gradient Normalization** is **the safety valve for deep learning** — preventing gradient explosions that would otherwise crash training.