gradient sparsification

**Gradient Sparsification** is a **communication reduction technique for distributed training that transmits only a subset of gradient components** — sending only the most important (largest) gradients and accumulating the rest locally, reducing communication by 100-1000× with minimal accuracy loss. **Gradient Sparsification Methods** - **Top-K**: Send only the K largest gradient components by magnitude — deterministic selection. - **Random-K**: Randomly sample K gradient components — stochastic, unbiased estimator. - **Threshold**: Send only gradients exceeding a magnitude threshold — adaptive sparsity. - **Error Feedback**: Accumulate unsent gradients locally and add them to the next round — prevents information loss. **Why It Matters** - **Communication Bottleneck**: In distributed training, gradient communication is often the bottleneck — sparsification eliminates it. - **99%+ Sparsity**: Deep learning gradients are often very sparse — sending only 0.1-1% of gradients suffices. - **Error Feedback**: The error feedback mechanism ensures convergence despite extreme sparsification. **Gradient Sparsification** is **sending only the important gradients** — reducing communication by orders of magnitude while maintaining training quality.