error feedback in compressed communication

**Error Feedback** (Memory) is a **mechanism that compensates for gradient compression losses by accumulating unsent gradient components locally** — the accumulated error is added to the next round's gradient before compression, ensuring that all gradient information is eventually communicated. **How Error Feedback Works** - **Compress**: Apply compression $C(g_t + e_t)$ to the gradient plus accumulated error. - **Communicate**: Send the compressed gradient $C(g_t + e_t)$. - **Accumulate**: Store the compression error: $e_{t+1} = (g_t + e_t) - C(g_t + e_t)$. - **Next Round**: Add accumulated error to next gradient: $g_{t+1} + e_{t+1}$. **Why It Matters** - **Convergence Fix**: Without error feedback, aggressive compression prevents convergence. With error feedback, convergence is guaranteed. - **No Information Loss**: Every gradient component is eventually communicated — just delayed, not lost. - **Universal**: Error feedback works with any compression method (top-K, random, quantization). **Error Feedback** is **remembering what you didn't send** — accumulating compression residuals to ensure no gradient information is permanently lost.