selective recomputation
**Selective Activation Recomputation** is an **intelligent checkpointing strategy that analyzes the compute cost and memory footprint of each operation to decide which activations to save and which to recompute during the backward pass** — achieving a better speed-memory tradeoff than uniform checkpointing by always saving expensive activations (attention softmax outputs, large intermediate tensors) while recomputing cheap ones (linear projections, element-wise operations), standard practice in Megatron-LM and DeepSpeed for training large transformers.
**What Is Selective Recomputation?**
- **Definition**: A memory optimization technique for training large neural networks that selectively chooses which intermediate activations to keep in memory and which to discard and recompute during backpropagation — making targeted decisions based on each operation's compute cost versus memory footprint rather than applying a uniform checkpoint-every-N-layers strategy.
- **The Memory Problem**: Training a large transformer requires storing all intermediate activations from the forward pass for use in the backward pass — for a 175B parameter model, this can require hundreds of GB of GPU memory, far exceeding available VRAM.
- **Smart Selection Criteria**: Always save activations that are expensive to recompute (attention softmax outputs require the full QK^T computation) and always recompute activations that are cheap (element-wise ReLU, dropout masks, linear projections are fast to redo).
- **Compared to Uniform Checkpointing**: Uniform checkpointing saves every N-th layer's output regardless of cost — selective recomputation analyzes actual compute profiles and makes per-operation decisions, achieving ~50% memory reduction with less slowdown than uniform's ~70% memory at ~30% slowdown.
**How Selective Recomputation Works**
- **Profile Phase**: Analyze each operation in the transformer block — measure compute time (FLOPS) and memory footprint (bytes) to build a cost-benefit profile.
- **Classification**: Categorize operations as "save" (expensive to recompute, small memory) or "recompute" (cheap to recompute, large memory).
- **Always Save**: Attention softmax outputs (expensive QK^T matmul), normalization statistics (running mean/variance), dropout masks (must be identical in forward and backward).
- **Always Recompute**: Linear projections (fast matmul, large activation tensors), element-wise activations (GELU, ReLU — trivially cheap), residual additions.
**Memory Savings Comparison**
| Strategy | Memory Reduction | Speed Overhead | Complexity |
|----------|-----------------|---------------|-----------|
| No checkpointing | 0% (baseline) | 0% | None |
| Uniform (every layer) | ~70% | ~30% | Low |
| Uniform (every 2 layers) | ~50% | ~20% | Low |
| Selective recomputation | ~50-60% | ~10-15% | Medium |
| Full recomputation | ~90% | ~33% | Low |
**Implementation**
- **Megatron-LM**: Implements selective recomputation as the default checkpointing strategy — profiled for transformer architectures with attention-specific save decisions.
- **DeepSpeed**: Supports selective activation checkpointing through its ZeRO optimization stages — configurable per-layer save/recompute decisions.
- **PyTorch**: `torch.utils.checkpoint.checkpoint()` provides the building block — selective strategies wrap this with per-operation decision logic.
**Selective activation recomputation is the smart memory optimization that achieves the best speed-memory tradeoff for large model training** — by analyzing each operation's compute cost and making targeted save-or-recompute decisions rather than applying uniform checkpointing, it reduces memory by 50-60% with only 10-15% slowdown, enabling training of models that would otherwise exceed GPU memory limits.