gradient compression

**Gradient Compression** is a **distributed training optimization technique that reduces the communication volume of gradients** — sending only the most important gradient information between workers, cutting communication overhead by 100-1000x at the cost of a small approximation. **The Communication Bottleneck** - AllReduce of gradients: Must communicate all parameters each step. - GPT-3 (175B params): 175B × 4 bytes = 700GB per AllReduce step. - Inter-node bandwidth: 100Gbps = 12.5 GB/s → 56 seconds per step. - Solution: Reduce what's communicated without hurting convergence. **Top-K Sparsification** - Gradient vector: Most values are small, few are large. - Top-K: Communicate only the K largest (by magnitude) gradient elements. - K = 0.1%: 1000x compression — only 0.1% of gradients transmitted. - **Error feedback**: Accumulate skipped gradients locally → include in next step. - Without error feedback: Top-K diverges. With it: Convergence preserved. **PowerSGD (2019)** - Low-rank approximation: $G \approx PQ^T$ where P, Q are low-rank factors. - Compress gradient matrix G (m×n) to P (m×r) + Q (n×r), $r << \min(m,n)$. - Rank-4 PowerSGD: 16x compression with minimal accuracy loss. - Default optimizer option in PyTorch DDP. **1-bit SGD / SignSGD** - Extreme compression: Communicate only sign of gradient (1 bit per element). - 32x compression vs. FP32. - QSGD: Stochastic quantization to k bits — adjustable compression ratio. **Communication Overlap** - Combine compression with overlap: Compute layer N+1 while communicating layer N gradients. - Bucket allreduce: Group small layers into buckets — amortize communication overhead. **Convergence Guarantees** - With error feedback: Top-K and PowerSGD converge to same quality as uncompressed SGD. - Trade-off: Compression ratio vs. wall-clock speedup vs. accuracy degradation. Gradient compression is **a key technique for scaling distributed training beyond NVLink speed** — when training across multiple nodes connected by slower Ethernet or InfiniBand, compression can save $50-200K in compute costs for large model training runs.