global batch

**Global batch** is the **total number of samples contributing to one optimizer update across all devices and accumulation passes** - it is the optimizer-facing batch size that determines gradient statistics and learning-rate scaling behavior. **What Is Global batch?** - **Definition**: Global batch aggregates local micro-batches from all parallel workers over accumulation steps. - **Optimization Link**: Many hyperparameters, especially learning rate and warmup, depend on global batch. - **System Decoupling**: Hardware topology may change while preserving the same global batch target. - **Measurement**: Should be logged explicitly for every run to ensure comparable experiment interpretation. **Why Global batch Matters** - **Convergence Consistency**: Matching global batch helps maintain similar optimization dynamics across cluster sizes. - **Scaling Decisions**: Global batch is the key anchor for linear scaling and large-batch experiments. - **Benchmark Fairness**: Performance comparisons are misleading if global batch differs silently. - **Reproducibility**: Exact batch semantics are required to recreate prior model quality outcomes. - **Cost Analysis**: Batch size affects step count and runtime, directly influencing training economics. **How It Is Used in Practice** - **Formula Tracking**: Compute and log global batch from micro-batch, world size, and accumulation settings. - **Policy Coupling**: Tie LR, momentum, and scheduler parameters to explicit global batch checkpoints. - **Scale Migration**: When adding GPUs, rebalance micro-batch and accumulation to preserve intended global batch. Global batch is **the central quantity that connects distributed systems configuration to optimizer behavior** - controlling it explicitly is required for reliable scaling and reproducibility.