Gradient Clipping — a technique that limits the magnitude of gradients during backpropagation to prevent exploding gradients from destabilizing training.

The Problem

• In deep networks (especially RNNs/Transformers), gradients can grow exponentially during backpropagation
• One bad batch → huge gradient → catastrophic weight update → model diverges (loss goes to NaN)

Methods

• Clip by Norm: Scale the entire gradient vector if its norm exceeds a threshold

``python torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) `` If $||g|| > max\_norm$: $g \leftarrow g \times \frac{max\_norm}{||g||}$ Preserves gradient direction, just limits magnitude

• Clip by Value: Clamp each gradient element independently to [-value, +value]

``python torch.nn.utils.clip_grad_value_(model.parameters(), clip_value=0.5) `` Simpler but can change gradient direction

Common Settings

• Transformer training: max_norm=1.0 (standard)
• RNN/LSTM training: max_norm=5.0 (more aggressive needed)
• LLM training: max_norm=1.0 (GPT, LLaMA, etc.)

When to Use

• Always for RNNs and Transformers
• When training with large learning rates
• When using mixed precision (FP16 gradients can overflow more easily)

Gradient clipping is a simple safety mechanism that virtually every modern deep learning training pipeline includes.