novograd

**NovoGrad** is an **adaptive optimizer that uses layer-wise second moments instead of per-parameter moments** — dramatically reducing optimizer memory while maintaining competitive training performance, especially for NLP and speech models. **How Does NovoGrad Work?** - **Layer-Wise Second Moment**: $v_l = eta_2 v_l + (1-eta_2) ||g_l||^2$ (one scalar per layer, not per parameter). - **Normalized Gradient**: $hat{g}_l = g_l / sqrt{v_l}$ (normalize by layer-wise second moment). - **Momentum**: Standard first-moment EMA on the normalized gradient. - **Paper**: Ginsburg et al. (2019). **Why It Matters** - **Memory Savings**: One scalar per layer vs. one value per parameter -> massive memory reduction for the second moment buffer. - **Speech/NLP**: Designed for and effective on Jasper (speech) and BERT (NLP) training. - **Large Models**: Memory savings enable larger models or batch sizes within the same GPU memory. **NovoGrad** is **the frugal adaptive optimizer** — achieving Adam-like adaptation with a fraction of the memory by thinking in layers instead of parameters.