ternary gradients

**Ternary Gradients** is a **gradient quantization scheme that compresses each gradient component to one of three values: {-1, 0, +1}** — achieving very high compression while preserving sparsity, as zero gradients are explicitly represented. **Ternary Quantization Methods** - **TernGrad**: Stochastic ternary quantization — $hat{g}_i in {-s, 0, +s}$ where $s$ is a scaling factor. - **Threshold-Based**: Components with magnitude below a threshold are set to 0, others to $pm s$. - **Stochastic Rounding**: $P(hat{g}_i = s cdot ext{sign}(g_i)) = |g_i|/s$ — unbiased with controlled variance. - **Encoding**: {-1, 0, +1} requires ~1.585 bits per component — encode efficiently with run-length encoding. **Why It Matters** - **Sparsity Aware**: Unlike 1-bit SGD, ternary gradients preserve gradient sparsity — zero gradients stay zero. - **Unbiased**: Stochastic ternary quantization is an unbiased estimator — convergence is theoretically guaranteed. - **Hardware Friendly**: Ternary operations can be implemented efficiently on specialized hardware. **Ternary Gradients** are **the three-symbol gradient alphabet** — compressing gradients to {-1, 0, +1} for efficient communication with sparsity awareness.