batch size

Batch size impacts training throughput, convergence dynamics, and generalization, with larger batches enabling better hardware utilization but potentially requiring learning rate adjustments and reaching diminishing returns beyond a critical batch size. Throughput: larger batches use GPU parallelism more efficiently; more samples per second; reduced training wall-clock time. Gradient noise: small batches have noisy gradients (high variance); large batches have smoother gradients; noise can help generalization. Learning rate scaling: when increasing batch size, often increase LR proportionally (linear scaling rule) to maintain similar gradient step magnitude. Warmup: large batch training often needs LR warmup; start small, ramp up to target LR. Critical batch size: beyond this point, increasing batch size doesn't improve training speed proportionally; communication overhead dominates. Generalization: research suggests small batch training may find flatter minima, potentially better generalization; debated topic. Memory constraints: batch size limited by GPU memory; gradient accumulation simulates larger batches without memory increase. Effective batch size: with gradient accumulation over k steps and N GPUs, effective batch = batch_per_GPU × N × k. Domain dependence: optimal batch size varies by task; NLP often uses larger batches than vision. Hyperparameter tuning: treat batch size as hyperparameter; don't assume largest possible is best. Batch size choice significantly impacts training dynamics and efficiency.