powersgd

**PowerSGD** is a **low-rank gradient compression method that approximates gradient matrices with their top-$k$ singular vectors** — using power iteration to efficiently compute a low-rank approximation, achieving high compression with better accuracy than sparsification or quantization. **How PowerSGD Works** - **Low-Rank**: Approximate gradient matrix $G approx P Q^T$ where $P$ and $Q$ are tall, thin matrices (rank $k$). - **Power Iteration**: Use 1-2 steps of power iteration starting from the previous $Q$ to quickly approximate top singular vectors. - **Communication**: Communicate $P$ and $Q$ (total size = $k(m+n)$) instead of $G$ (size = $m imes n$) — compression ratio = $mn / k(m+n)$. - **Error Feedback**: Accumulate the compression residual for next iteration. **Why It Matters** - **Better Trade-Off**: PowerSGD achieves better accuracy-compression trade-offs than sparsification or quantization. - **Warm Start**: Reusing the previous iteration's $Q$ makes power iteration converge in just 1-2 steps. - **Practical**: Integrated into PyTorch's distributed data parallel (DDP) as a built-in communication hook. **PowerSGD** is **low-rank gradient communication** — transmitting compact matrix factorizations instead of full gradients for efficient, high-quality compression.