Gradient Checkpointing (Activation Recomputation) is the memory optimization technique for training deep neural networks that trades compute for memory by storing only a subset of intermediate activations during the forward pass and recomputing the discarded activations during the backward pass — reducing peak activation memory from O(N) to O(√N) for an N-layer network at the cost of one additional forward pass, enabling the training of models 3-10x larger on the same hardware.

The Memory Problem

During training, the forward pass computes and stores activations at every layer because the backward pass needs them for gradient computation. For a transformer with 96 layers, batch size 32, sequence length 2048, and hidden dimension 12288, the stored activations consume ~150 GB — far exceeding any single GPU's memory. Without gradient checkpointing, training requires either smaller batch sizes, shorter sequences, or model parallelism.

How It Works

1. Forward Pass: Divide the N layers into √N segments. Store only the activations at segment boundaries (√N checkpoints). Discard all intermediate activations within each segment. 2. Backward Pass: When gradients reach a segment boundary, re-execute the forward pass for that segment (recomputing the intermediate activations from the stored checkpoint) and immediately use them for gradient computation. 3. Memory: Only √N checkpoint activations + 1 segment's activations are stored simultaneously → O(√N) total activation memory. 4. Compute: Each layer's forward computation runs twice (once during forward, once during backward recomputation) → ~33% additional compute for a full recomputation strategy.

Selective Checkpointing

Not all layers consume equal memory. In transformers, the attention computation produces large intermediate tensors (batch × heads × seq × seq) while the linear layers produce smaller tensors. Selective checkpointing stores the cheap-to-store, expensive-to-recompute tensors and discards the expensive-to-store, cheap-to-recompute ones.

Implementation in Practice

• PyTorch: torch.utils.checkpoint.checkpoint(function, *args) wraps a module's forward pass. Activations within the checkpointed function are discarded and recomputed during backward.
• Megatron-LM / DeepSpeed: Apply checkpointing at the transformer block level — each block's input activation is a checkpoint, and all internal activations (attention scores, intermediate FFN values) are recomputed.
• Full Recomputation: Store nothing except the input. Recompute every activation during backward. Memory: O(1) activation memory. Compute: ~100% additional forward compute (2x total). Used only when memory is extremely constrained.

Combined with Other Techniques

Gradient checkpointing is typically combined with mixed-precision training (FP16/BF16 activations), ZeRO optimizer state sharding, and tensor parallelism to enable training of 100B+ parameter models on clusters of 80GB GPUs.

Gradient Checkpointing is the memory-compute exchange rate of deep learning training — paying a 33% compute tax to reduce activation memory by 3-10x, enabling models far larger than GPU memory would otherwise permit.