adafactor

**Adafactor** is a **memory-efficient adaptive optimizer designed for training large models** — replacing Adam's per-parameter second moment buffer with a factored approximation, reducing optimizer memory from $O(mn)$ to $O(m + n)$ for each matrix parameter. **How Does Adafactor Work?** - **Factored Second Moments**: For a weight matrix $W in mathbb{R}^{m imes n}$, instead of storing the full $m imes n$ second moment, store row factors ($m$) and column factors ($n$). - **Reconstruction**: $v_{ij} approx r_i cdot c_j / ar{r}$ (outer product approximation). - **No Momentum**: Optionally omits first moment (momentum) to save more memory. - **Paper**: Shazeer & Stern (2018). **Why It Matters** - **Memory Savings**: For large transformer models (billions of parameters), Adafactor saves 30-50% optimizer memory vs. Adam. - **T5**: Used to train Google's T5 model family (11B parameters). - **Large Models**: Essential when model size pushes against GPU memory limits. **Adafactor** is **Adam on a memory diet** — achieving comparable optimization quality with dramatically less memory through smart factorization.