gradient accumulation training

**Gradient Accumulation** is **the training technique that simulates large batch sizes by accumulating gradients over multiple forward-backward passes (micro-batches) before performing a single optimizer step — enabling training with effective batch sizes that exceed GPU memory capacity, achieving identical convergence to true large-batch training while using 4-16× less memory, making it essential for training large models on limited hardware and for hyperparameter tuning with consistent batch sizes across different GPU configurations**. **Gradient Accumulation Mechanism:** - **Micro-Batching**: divide logical batch (size B) into K micro-batches (size B/K each); perform forward and backward pass on each micro-batch; gradients accumulate (sum) across micro-batches; single optimizer step updates weights using accumulated gradients - **Memory Savings**: peak memory = model + optimizer state + activations for one micro-batch; without accumulation: peak memory = model + optimizer state + activations for full batch; 4-16× memory reduction enables training larger models or using larger effective batch sizes - **Computation**: K micro-batches require K forward passes and K backward passes; total compute identical to single large batch; but K optimizer steps replaced by 1 optimizer step; optimizer overhead reduced by K× - **Convergence**: gradient accumulation with K steps and batch size B/K is mathematically equivalent to batch size B; convergence curves identical (given proper learning rate scaling); no accuracy trade-off **Implementation Patterns:** - **PyTorch Manual**: for i, (data, target) in enumerate(dataloader): output = model(data); loss = criterion(output, target) / accumulation_steps; loss.backward(); if (i+1) % accumulation_steps == 0: optimizer.step(); optimizer.zero_grad() - **Gradient Scaling**: divide loss by accumulation_steps before backward(); ensures accumulated gradient has correct magnitude; equivalent to averaging gradients across micro-batches; critical for numerical correctness - **Zero Gradient Timing**: zero_grad() only after optimizer step; gradients accumulate across micro-batches; incorrect zero_grad() placement (every iteration) breaks accumulation - **Automatic Mixed Precision**: scaler.scale(loss).backward(); scaler.step(optimizer) only when (i+1) % accumulation_steps == 0; scaler.update() after step; AMP compatible with gradient accumulation **Effective Batch Size Calculation:** - **Single GPU**: effective_batch_size = micro_batch_size × accumulation_steps; micro_batch_size=32, accumulation_steps=4 → effective_batch_size=128 - **Multi-GPU Data Parallel**: effective_batch_size = micro_batch_size × accumulation_steps × num_gpus; 8 GPUs, micro_batch_size=16, accumulation_steps=8 → effective_batch_size=1024 - **Learning Rate Scaling**: when increasing effective batch size, scale learning rate proportionally; linear scaling rule: lr_new = lr_base × (batch_new / batch_base); maintains convergence speed - **Warmup Adjustment**: scale warmup steps proportionally to batch size; larger batches require longer warmup; warmup_steps_new = warmup_steps_base × (batch_new / batch_base) **Batch Normalization Considerations:** - **BatchNorm Statistics**: BatchNorm computes mean/variance over micro-batch, not effective batch; micro-batch statistics are noisier; may hurt convergence for very small micro-batches (<8) - **SyncBatchNorm**: synchronizes statistics across GPUs; computes mean/variance over micro_batch_size × num_gpus; improves stability but adds communication overhead; use when micro-batch size <16 - **GroupNorm/LayerNorm**: normalization independent of batch size; unaffected by gradient accumulation; preferred for small micro-batches; GroupNorm widely used in vision transformers - **Running Statistics**: BatchNorm running mean/variance updated every micro-batch; K× more updates than without accumulation; may cause slight divergence; typically negligible impact **Memory-Compute Trade-offs:** - **Accumulation Steps**: more steps → less memory, more time; 2× accumulation steps → 1.5× training time (due to reduced optimizer overhead); 4× steps → 1.8× time; 8× steps → 2× time - **Optimal Micro-Batch Size**: too small → poor GPU utilization, excessive overhead; too large → insufficient memory savings; optimal typically 8-32 samples per GPU; measure GPU utilization with profiler - **Activation Checkpointing**: combine with gradient accumulation for maximum memory savings; checkpointing saves 50-70% activation memory; accumulation saves 75-90% activation memory; together enable 10-20× larger models - **Gradient Checkpointing + Accumulation**: checkpoint every N layers; accumulate over K micro-batches; enables training 100B+ parameter models on 8×40GB GPUs **Distributed Training Integration:** - **Data Parallel**: each GPU accumulates gradients independently; all-reduce after accumulation completes; reduces communication frequency by K×; improves scaling efficiency - **Pipeline Parallel**: micro-batches naturally fit pipeline parallelism; each stage processes different micro-batch; gradient accumulation across pipeline flushes; enables efficient pipeline utilization - **ZeRO Optimizer**: gradient accumulation compatible with ZeRO stages 1-3; reduces optimizer state memory; combined with accumulation enables training 100B+ models on consumer GPUs - **FSDP (Fully Sharded Data Parallel)**: accumulation reduces all-gather frequency; sharded parameters gathered once per accumulation cycle; reduces communication overhead by K× **Hyperparameter Tuning:** - **Consistent Batch Size**: use gradient accumulation to maintain constant effective batch size across different GPU counts; 1 GPU: micro=128, accum=1; 4 GPUs: micro=32, accum=1; 8 GPUs: micro=16, accum=1 — all achieve effective batch size 128 - **Memory-Constrained Tuning**: when GPU memory limits batch size, use accumulation to explore larger batch sizes; compare batch sizes 256, 512, 1024 without changing hardware - **Throughput Optimization**: measure samples/second for different micro-batch and accumulation combinations; larger micro-batches improve GPU utilization; more accumulation reduces optimizer overhead; find optimal balance **Profiling and Optimization:** - **GPU Utilization**: nsight systems shows GPU active time; low utilization (<70%) indicates micro-batch too small; increase micro-batch size, reduce accumulation steps - **Memory Usage**: nvidia-smi shows memory consumption; if memory usage <<90%, increase micro-batch size; if memory usage >95%, increase accumulation steps - **Throughput Measurement**: measure samples/second = (micro_batch_size × accumulation_steps × num_gpus) / time_per_step; optimize for maximum throughput while maintaining convergence - **Communication Overhead**: with data parallel, measure all-reduce time; accumulation reduces all-reduce frequency; K× accumulation → K× less communication; improves scaling efficiency **Common Pitfalls:** - **Forgetting Loss Scaling**: loss.backward() without dividing by accumulation_steps causes K× larger gradients; leads to divergence or numerical instability; always scale loss or gradients - **Incorrect Zero Grad**: calling zero_grad() every iteration clears accumulated gradients; breaks accumulation; only zero after optimizer step - **BatchNorm with Small Micro-Batches**: micro-batch size <8 causes noisy BatchNorm statistics; use GroupNorm, LayerNorm, or SyncBatchNorm instead - **Learning Rate Not Scaled**: increasing effective batch size without scaling learning rate causes slow convergence; use linear scaling rule or learning rate finder **Use Cases:** - **Large Model Training**: train 70B parameter model on 8×40GB GPUs; micro-batch=1, accumulation=64, effective batch=512; without accumulation, model doesn't fit - **High-Resolution Images**: train on 1024×1024 images with batch size 64; micro-batch=4, accumulation=16; without accumulation, OOM error - **Consistent Hyperparameters**: maintain batch size 256 across 1, 2, 4, 8 GPU configurations; adjust accumulation steps to keep effective batch constant; simplifies hyperparameter transfer - **Memory-Bandwidth Trade-off**: when memory-bound, use accumulation to reduce memory; when compute-bound, reduce accumulation to improve throughput; balance based on bottleneck Gradient accumulation is **the essential technique for training large models on limited hardware — by decoupling effective batch size from GPU memory constraints, it enables training with optimal batch sizes regardless of hardware limitations, achieving 4-16× memory savings with minimal computational overhead and making large-scale model training accessible on consumer and mid-range professional GPUs**.