computation-communication overlap

**Computation-communication overlap** is the **optimization technique that schedules data exchange concurrently with ongoing model computation** - it reduces visible communication cost by filling network time under useful compute work. **What Is Computation-communication overlap?** - **Definition**: Launch communication for ready gradient buckets while later layers continue backward computation. - **Mechanism**: Asynchronous collectives and stream scheduling allow concurrent kernel and network activity. - **Dependency Constraint**: Only gradients whose dependencies are complete can be communicated early. - **Implementation Complexity**: Requires careful bucketization, stream control, and synchronization correctness. **Why Computation-communication overlap Matters** - **Step-Time Reduction**: Hidden communication lowers apparent synchronization overhead. - **Scaling Improvement**: Overlap becomes increasingly valuable as cluster size and communication volume grow. - **Resource Utilization**: Keeps both compute engines and network links active simultaneously. - **Cost Efficiency**: Faster effective steps reduce total runtime and infrastructure spend. - **Performance Stability**: Overlap can smooth communication spikes that otherwise stall all workers. **How It Is Used in Practice** - **Bucket Ordering**: Arrange gradients so early-ready layers trigger communication promptly. - **Stream Architecture**: Use separate CUDA streams for compute and communication with explicit event dependencies. - **Profiler Verification**: Confirm real overlap in timeline traces rather than relying on theoretical configuration. Computation-communication overlap is **a critical optimization for high-scale distributed training** - effective overlap converts network wait time into productive parallel progress.