gradient checkpointing activation

**Gradient Checkpointing (Activation Recomputation)** is the **memory optimization technique for training deep neural networks that trades compute for memory by storing only a subset of intermediate activations during the forward pass and recomputing the discarded activations during the backward pass — reducing peak activation memory from O(N) to O(√N) for an N-layer network at the cost of one additional forward pass, enabling the training of models 3-10x larger on the same hardware**. **The Memory Problem** During training, the forward pass computes and stores activations at every layer because the backward pass needs them for gradient computation. For a transformer with 96 layers, batch size 32, sequence length 2048, and hidden dimension 12288, the stored activations consume ~150 GB — far exceeding any single GPU's memory. Without gradient checkpointing, training requires either smaller batch sizes, shorter sequences, or model parallelism. **How It Works** 1. **Forward Pass**: Divide the N layers into √N segments. Store only the activations at segment boundaries (√N checkpoints). Discard all intermediate activations within each segment. 2. **Backward Pass**: When gradients reach a segment boundary, re-execute the forward pass for that segment (recomputing the intermediate activations from the stored checkpoint) and immediately use them for gradient computation. 3. **Memory**: Only √N checkpoint activations + 1 segment's activations are stored simultaneously → O(√N) total activation memory. 4. **Compute**: Each layer's forward computation runs twice (once during forward, once during backward recomputation) → ~33% additional compute for a full recomputation strategy. **Selective Checkpointing** Not all layers consume equal memory. In transformers, the attention computation produces large intermediate tensors (batch × heads × seq × seq) while the linear layers produce smaller tensors. Selective checkpointing stores the cheap-to-store, expensive-to-recompute tensors and discards the expensive-to-store, cheap-to-recompute ones. **Implementation in Practice** - **PyTorch**: `torch.utils.checkpoint.checkpoint(function, *args)` wraps a module's forward pass. Activations within the checkpointed function are discarded and recomputed during backward. - **Megatron-LM / DeepSpeed**: Apply checkpointing at the transformer block level — each block's input activation is a checkpoint, and all internal activations (attention scores, intermediate FFN values) are recomputed. - **Full Recomputation**: Store nothing except the input. Recompute every activation during backward. Memory: O(1) activation memory. Compute: ~100% additional forward compute (2x total). Used only when memory is extremely constrained. **Combined with Other Techniques** Gradient checkpointing is typically combined with mixed-precision training (FP16/BF16 activations), ZeRO optimizer state sharding, and tensor parallelism to enable training of 100B+ parameter models on clusters of 80GB GPUs. Gradient Checkpointing is **the memory-compute exchange rate of deep learning training** — paying a 33% compute tax to reduce activation memory by 3-10x, enabling models far larger than GPU memory would otherwise permit.