communication overhead

**Communication overhead** is the **portion of distributed training time spent moving and synchronizing data instead of performing model computation** - it is the primary scaling tax that grows as cluster size increases and compute per rank decreases. **What Is Communication overhead?** - **Definition**: Aggregate latency and bandwidth cost of collectives, point-to-point transfers, and synchronization barriers. - **Dominant Sources**: Gradient all-reduce, parameter exchange, and pipeline stage boundary transfers. - **Scaling Effect**: Relative overhead rises when per-device compute workload becomes smaller. - **Measurement**: Computed from step-time breakdown comparing communication phases against compute phases. **Why Communication overhead Matters** - **Scaling Limit**: High communication tax prevents near-linear acceleration with added GPUs. - **Cost Impact**: Idle compute during communication increases price per useful training step. - **Architecture Choice**: Overhead profile guides choice of parallelism and topology strategy. - **Performance Debugging**: Communication-heavy traces reveal network or collective bottlenecks. - **Optimization Prioritization**: Reducing overhead often yields larger gains than pure kernel tuning at scale. **How It Is Used in Practice** - **Ratio Tracking**: Monitor compute-to-communication ratio across model sizes and cluster configurations. - **Collective Tuning**: Optimize bucket sizes, algorithm selection, and rank placement for fabric locality. - **Overlap Adoption**: Hide communication behind backprop compute where framework supports asynchronous collectives. Communication overhead is **the scaling tax that governs distributed training efficiency** - understanding and reducing this tax is essential for cost-effective multi-GPU expansion.