adabelief

**AdaBelief** is an **adaptive optimizer that adapts the learning rate based on the "belief" in the current gradient direction** — using the deviation of the gradient from the expected gradient (EMA), rather than the gradient magnitude itself, as the adaptive scaling factor. **How Does AdaBelief Work?** - **Key Change**: Instead of $v_t = eta_2 v_{t-1} + (1-eta_2) g_t^2$ (Adam), use $v_t = eta_2 v_{t-1} + (1-eta_2)(g_t - m_t)^2$ (AdaBelief). - **Interpretation**: If the gradient $g_t$ matches the momentum $m_t$ (strong belief), take a large step. If they diverge (weak belief), take a small step. - **Effect**: Adapts to gradient predictability, not just magnitude. - **Paper**: Zhuang et al. (2020). **Why It Matters** - **Fast Convergence**: Combines the fast convergence of Adam with the generalization of SGD. - **Better Generalization**: Outperforms Adam on test accuracy while maintaining fast training. - **Stability**: Less likely to diverge on noisy gradients compared to Adam. **AdaBelief** is **the confidence-weighted optimizer** — stepping boldly when gradients are predictable and cautiously when they're erratic.