gradient
**Gradients and Backpropagation**
**What is Backpropagation?**
Backpropagation computes gradients of the loss with respect to each parameter, enabling gradient-based optimization.
**The Chain Rule**
For a composition of functions $y = f(g(x))$:
$$
\frac{dy}{dx} = \frac{dy}{dg} \cdot \frac{dg}{dx}
$$
Backprop applies this recursively through the network.
**Forward and Backward Pass**
**Forward Pass**
Compute outputs layer by layer, storing intermediate activations:
```
Input → Layer1 → (activations1) → Layer2 → (activations2) → ... → Loss
```
**Backward Pass**
Compute gradients layer by layer, from loss to inputs:
```
dLoss → dLayer_n → dLayer_{n-1} → ... → dLayer_1
```
**Gradient Flow in Transformers**
**Key Components**
| Component | Gradient Consideration |
|-----------|----------------------|
| Layer Norm | Stabilizes gradient magnitudes |
| Residual connections | Enable gradient flow to early layers |
| Attention | Gradients flow through softmax |
| FFN | Standard MLP gradients |
**Residual Connections Are Critical**
```
output = layer(x) + x # Skip connection
# Gradient flows through both paths
d_output = d_layer + d_identity
```
Without residuals, gradients would vanish in deep networks.
**Gradient Issues**
**Vanishing Gradients**
- Gradients become too small in early layers
- Solutions: Residual connections, Layer Norm, careful initialization
**Exploding Gradients**
- Gradients become too large, causing instability
- Solutions: Gradient clipping, Layer Norm, lower learning rate
**Gradient Clipping**
```python
# Clip gradient norm
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
```
**Memory for Gradients**
Storing activations for backward pass is memory-intensive:
- **Solution 1**: Gradient checkpointing (recompute instead of store)
- **Solution 2**: Mixed precision (FP16/BF16 activations)
- **Solution 3**: Activation offloading to CPU
**Monitoring Gradients**
```python
# Check gradient norms during training
for name, param in model.named_parameters():
if param.grad is not None:
print(f"{name}: {param.grad.norm():.4f}")
```