communication-efficient training

**Communication-Efficient Training** encompasses the **set of techniques to reduce the communication overhead in distributed deep learning** — addressing the key bottleneck where gradient synchronization between workers dominates training time. **Communication Reduction Strategies** - **Gradient Compression**: Sparsification (top-K, random) and quantization (1-bit, ternary) reduce message size. - **Local SGD**: Workers perform multiple local gradient steps before synchronizing — reduce communication frequency. - **Gradient Accumulation**: Accumulate gradients over multiple mini-batches before communicating. - **Decentralized**: Replace the central parameter server with peer-to-peer gossip communication. **Why It Matters** - **Scalability**: Communication cost grows with number of workers — communication efficiency enables scaling to more GPUs. - **Network Bottleneck**: In datacenter training, network bandwidth is 100-1000× slower than compute — communication dominates. - **Edge/Federated**: In federated learning, communication is extremely expensive (slow WAN links) — efficiency is critical. **Communication-Efficient Training** is **maximizing compute-per-byte** — reducing the communication needed to synchronize distributed training without sacrificing model quality.