batch size

Batch size is the number of examples processed together in one forward-backward pass before weight update. **Trade-offs**: **Large batches**: More stable gradients, GPU utilization, faster wall-clock (with parallelism), but may generalize worse. **Small batches**: Noisier gradients (regularization effect), less memory, possibly better generalization. **Memory impact**: Larger batch = more activation memory. Often the limiting factor for batch size. **Learning rate scaling**: Large batches often need higher learning rate. Linear scaling rule: double batch, double LR (with warmup). **Gradient accumulation**: Simulate large batches on limited memory by accumulating across steps. **Effective batch size**: Per-device batch x devices x accumulation steps. What matters for training dynamics. **LLM training**: Large batches (millions of tokens) for efficiency. Requires careful LR tuning. **Critical batch size**: Beyond some size, more compute without proportional improvement. Diminishing returns. **Recommendations**: Maximize batch size within memory, scale LR appropriately, use accumulation if needed. **Hyperparameter**: Often tuned alongside learning rate. Larger models may benefit from larger batches.