sophia

**Sophia** (Second-order Clipped Stochastic Optimization) is a **lightweight second-order optimizer designed for large language model pre-training** — using a diagonal Hessian estimate to provide per-parameter curvature information, achieving 2x faster training than Adam with minimal overhead. **How Does Sophia Work?** - **Hessian Estimate**: Uses a diagonal Gauss-Newton approximation or Hutchinson estimator to estimate per-parameter curvature $h_t$. - **Update**: $ heta_{t+1} = heta_t - eta cdot ext{clip}(m_t / max(h_t, gamma), 1)$ (gradient divided by curvature, clipped). - **Clipping**: The clipping prevents excessively large steps in flat directions (where $h_t approx 0$). - **Cost**: ~15% overhead over Adam (Hessian estimated every 10-20 steps). **Why It Matters** - **2x Speedup**: Reaches the same loss as Adam in half the training steps for LLMs (GPT-2, LLaMA). - **Curvature-Aware**: Steps more aggressively in flat directions and conservatively in sharp ones — optimal for the heterogeneous loss landscape of LLMs. - **Scalable**: Designed for billion-parameter models. **Sophia** is **the curvature-aware optimizer for LLMs** — using lightweight second-order information to navigate the loss landscape twice as fast as Adam.