top-k gradient sparsification

**Top-K Gradient Sparsification** is the **most common gradient sparsification strategy** — selecting only the K gradient components with the largest magnitude for communication, where K is typically 0.1-1% of the total gradient dimension. **Top-K Algorithm** - **Compute**: Compute the full gradient locally. - **Select**: Find the top-K components by absolute magnitude. - **Communicate**: Send only these K (index, value) pairs — ~99% compression. - **Error Feedback**: Accumulate the unsent components and add to the next gradient: $e_{t+1} = g_t - TopK(g_t + e_t)$. **Why It Matters** - **Convergence Guarantee**: With error feedback, top-K sparsification converges to the same solution as full gradient communication. - **All-Reduce**: Sparse all-reduce is more complex than dense all-reduce — specialized communication primitives needed. - **Hardware**: Modern accelerators (GPUs, TPUs) have high compute-to-communication ratios — sparsification exploits this. **Top-K Sparsification** is **selecting the most impactful gradients** — sending only the largest updates for massive communication savings.