Communication-Computation Overlap is the technique of executing data transfers concurrently with useful computation — hiding the latency of inter-GPU, inter-node, or device-host communication behind productive work, which is the single most important optimization for scaling distributed training and HPC applications efficiently across multiple devices.

Why Overlap Matters

• Without overlap: Total time = Compute + Communication (serial).
• With overlap: Total time = max(Compute, Communication).
• At scale (hundreds of GPUs): Communication can be 30-50% of total time → overlap recovers most of this.

Overlap Techniques in Distributed Training

1. Gradient AllReduce Overlap (DDP standard)

• Backward pass computes gradients layer by layer.
• As soon as a layer's gradient is ready → start AllReduce for that layer.
• While AllReduce runs → backward pass continues computing next layer's gradients.
• Result: AllReduce mostly hidden behind backward computation.

2. Prefetch Parameters (FSDP/ZeRO-3)

• FSDP must all-gather parameters before each layer's forward pass.
• Prefetch: Start all-gathering layer N+1 while computing layer N.
• Result: Communication for next layer overlaps with current layer's computation.

3. Pipeline Parallelism Overlap

• While microbatch K is in forward on stage N → microbatch K-1 is in backward on stage N.
• Different stages process different microbatches simultaneously.
• Pipeline fill/drain bubbles remain but steady-state achieves full overlap.

Implementation on GPUs

CUDA Stream Overlap Pattern

• Stream 1: Compute kernel.
• Stream 2: Communication (NCCL AllReduce or memcpy).
• Both streams execute concurrently on different GPU hardware units.
• GPU has dedicated copy engines separate from compute SMs → true overlap.

Measuring Overlap Efficiency

• Overlap ratio: $\frac{T_{serial} - T_{overlapped}}{T_{comm}}$
• 100% = perfect overlap (all communication hidden).
• 0% = no overlap (fully serial).
• Profile with NVIDIA Nsight Systems: Visual timeline shows concurrent stream execution.

Challenges

• Data dependencies: Cannot prefetch too far ahead — limited by data flow order.
• Memory pressure: Prefetching requires buffering data → increases memory usage.
• Synchronization: Must ensure communication completes before result is needed.

Communication-computation overlap is the fundamental technique that makes distributed computing practical — without it, the communication overhead of multi-GPU and multi-node training would make scaling beyond a few devices economically infeasible.