gradient accumulation

**Gradient Accumulation** is a **critical memory optimization technique universally employed in large-scale Vision Transformer and LLM training that mathematically simulates the effect of enormous batch sizes — often 4,096 or higher — on consumer or mid-range GPUs by splitting a single logical optimization step across multiple sequential forward-backward passes, accumulating the gradient contributions before executing a single weight update.** **The Large Batch Requirement** - **The ViT Convergence Mandate**: Empirical research (DeiT, ViT-B/16) demonstrates that Vision Transformers require effective batch sizes of $1,024$ to $4,096$ to achieve reported accuracy. Smaller batch sizes produce noisy, high-variance gradient estimates that prevent the Self-Attention layers from learning stable, global feature representations. - **The Hardware Reality**: A ViT-B/16 model processing a batch of $4,096$ images at $224 imes 224$ resolution simultaneously requires approximately $64$ GB of GPU memory for activations alone. A single NVIDIA A100 (40GB) or consumer RTX 4090 (24GB) physically cannot fit this batch. **The Accumulation Protocol** Gradient Accumulation resolves this by fragmenting the logical batch across time: 1. **Micro-Batch Forward Pass**: Process a small micro-batch of $B_{micro} = 32$ images through the full forward pass. 2. **Backward Pass**: Compute the gradients for this micro-batch. Crucially, do NOT update the weights. 3. **Accumulate**: Add the computed gradients to a running gradient accumulator buffer. 4. **Repeat**: Execute steps 1-3 a total of $K = 128$ times (the accumulation steps). 5. **Update**: After all $K$ micro-batches, divide the accumulated gradients by $K$ to compute the average, then execute a single optimizer step (AdamW weight update). The effective batch size becomes $B_{effective} = B_{micro} imes K = 32 imes 128 = 4096$. **Mathematical Equivalence** Gradient accumulation produces mathematically identical gradients to true large-batch training under standard loss averaging. The gradient of the mean loss over $N$ samples is the mean of the per-sample gradients regardless of whether they are computed simultaneously or sequentially. The only difference is wall-clock time — accumulation processes the micro-batches serially rather than in parallel. **The Trade-Off** The technique trades approximately $30\%$ additional wall-clock training time (due to serial micro-batch processing) for a $50\%$ to $70\%$ reduction in peak GPU memory consumption, enabling the training of billion-parameter models on hardware that would otherwise be insufficient. **Gradient Accumulation** is **installment-plan optimization** — paying the computational cost of a massive batch size in small, affordable sequential installments while receiving the mathematically identical gradient signal that a single enormous parallel computation would produce.