Gradient Normalization is the practice of normalizing gradient magnitudes during training — either by clipping the gradient norm to a maximum value (gradient clipping) or by scaling gradients to have unit norm, preventing exploding gradients and stabilizing training.

Types of Gradient Normalization

• Gradient Clipping by Norm: $hat{g} = g cdot min(1, c/||g||)$. Clips when $||g|| > c$.
• Gradient Clipping by Value: Clip each element independently: $hat{g}_i = ext{clip}(g_i, -c, c)$.
• Unit Norm: Scale to unit norm: $hat{g} = g / ||g||$.
• Gradient Scaling: Scale gradients by a constant factor (used in mixed-precision training).

Why It Matters

• Stability: Prevents exploding gradients in RNNs, transformers, and deep networks.
• Necessary for LLMs: Gradient clipping (typically $c = 1.0$) is standard in all transformer pre-training.
• Mixed Precision: Loss scaling + gradient unscaling is critical for FP16/BF16 training.

Gradient Normalization is the safety valve for deep learning — preventing gradient explosions that would otherwise crash training.