radam

**RAdam** (Rectified Adam) is an **adaptive optimizer that addresses the high variance of Adam's adaptive learning rate in early training** — by computing the variance of the adaptive rate and automatically switching from SGD-like behavior (when variance is high) to full Adam (when variance is low). **How Does RAdam Work?** - **Problem with Adam**: In early training, the exponential moving average of squared gradients ($v_t$) is estimated from few samples -> high variance -> erratic learning rate. - **Solution**: RAdam computes the variance of the adaptive rate. If variance exceeds a threshold, it falls back to momentum SGD. - **Automatic Warmup**: Effectively provides automatic learning rate warmup without a warmup schedule. - **Paper**: Liu et al. (2020). **Why It Matters** - **No Warmup Needed**: Eliminates the need for manual learning rate warmup schedules. - **Robust**: More stable than Adam in the critical early training phase. - **Drop-In**: Same API as Adam with no additional hyperparameters. **RAdam** is **Adam with self-correcting early training** — automatically providing the warmup behavior that Adam needs but doesn't have.