Gradients and Backpropagation

What is Backpropagation? Backpropagation computes gradients of the loss with respect to each parameter, enabling gradient-based optimization.

The Chain Rule For a composition of functions $y = f(g(x))$: $$ \frac{dy}{dx} = \frac{dy}{dg} \cdot \frac{dg}{dx} $$

Backprop applies this recursively through the network.

Forward and Backward Pass

Forward Pass Compute outputs layer by layer, storing intermediate activations:

Input → Layer1 → (activations1) → Layer2 → (activations2) → ... → Loss

Backward Pass Compute gradients layer by layer, from loss to inputs:

dLoss → dLayer_n → dLayer_{n-1} → ... → dLayer_1

Gradient Flow in Transformers

Key Components

Residual Connections Are Critical

output = layer(x) + x  # Skip connection

# Gradient flows through both paths
d_output = d_layer + d_identity

Without residuals, gradients would vanish in deep networks.

Gradient Issues

Vanishing Gradients

• Gradients become too small in early layers
• Solutions: Residual connections, Layer Norm, careful initialization

Exploding Gradients

• Gradients become too large, causing instability
• Solutions: Gradient clipping, Layer Norm, lower learning rate

Gradient Clipping

# Clip gradient norm
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

Memory for Gradients Storing activations for backward pass is memory-intensive:

• Solution 1: Gradient checkpointing (recompute instead of store)
• Solution 2: Mixed precision (FP16/BF16 activations)
• Solution 3: Activation offloading to CPU

Monitoring Gradients

# Check gradient norms during training
for name, param in model.named_parameters():
    if param.grad is not None:
        print(f"{name}: {param.grad.norm():.4f}")