gradient clipping
**Gradient Clipping** — a technique that limits the magnitude of gradients during backpropagation to prevent exploding gradients from destabilizing training.
**The Problem**
- In deep networks (especially RNNs/Transformers), gradients can grow exponentially during backpropagation
- One bad batch → huge gradient → catastrophic weight update → model diverges (loss goes to NaN)
**Methods**
- **Clip by Norm**: Scale the entire gradient vector if its norm exceeds a threshold
```python
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
```
If $||g|| > max\_norm$: $g \leftarrow g \times \frac{max\_norm}{||g||}$
Preserves gradient direction, just limits magnitude
- **Clip by Value**: Clamp each gradient element independently to [-value, +value]
```python
torch.nn.utils.clip_grad_value_(model.parameters(), clip_value=0.5)
```
Simpler but can change gradient direction
**Common Settings**
- Transformer training: `max_norm=1.0` (standard)
- RNN/LSTM training: `max_norm=5.0` (more aggressive needed)
- LLM training: `max_norm=1.0` (GPT, LLaMA, etc.)
**When to Use**
- Always for RNNs and Transformers
- When training with large learning rates
- When using mixed precision (FP16 gradients can overflow more easily)
**Gradient clipping** is a simple safety mechanism that virtually every modern deep learning training pipeline includes.