Neural Network Optimizers are the algorithms that update model parameters based on computed gradients to minimize the training loss function — with the choice of optimizer (SGD, Adam, AdamW, LAMB) and its hyperparameters (learning rate, momentum, weight decay) directly determining convergence speed, final accuracy, and generalization quality of the trained model.

Stochastic Gradient Descent (SGD):

• Vanilla SGD: θ_{t+1} = θ_t - η∇L(θ_t) — learning rate η scales gradient; noisy gradient estimates from mini-batches provide implicit regularization but cause slow convergence
• Momentum: accumulate exponentially decayed gradient history — v_t = βv_{t-1} + ∇L(θ_t), θ_{t+1} = θ_t - ηv_t; β=0.9 typical; accelerates convergence in consistent gradient directions while dampening oscillations
• Nesterov Momentum: evaluate gradient at the "look-ahead" position — computes gradient at θ_t - ηβv_{t-1} instead of θ_t; provides better convergence for convex objectives; slightly better in practice than standard momentum
• SGD + Momentum: still achieves best generalization for many vision tasks — requires careful learning rate tuning and schedule but often produces models that generalize better than adaptive methods

Adaptive Learning Rate Methods:

• Adam: maintains per-parameter first moment (mean) and second moment (uncentered variance) of gradients — m_t = β₁m_{t-1} + (1-β₁)g_t, v_t = β₂v_{t-1} + (1-β₂)g_t²; update = η × m̂_t/(√v̂_t + ε) where m̂, v̂ are bias-corrected; default β₁=0.9, β₂=0.999, ε=1e-8
• AdamW: fixes weight decay implementation in Adam — standard Adam applies L2 regularization to gradient before adaptive scaling (incorrect), AdamW applies weight decay directly to weights after Adam step (correct); consistently outperforms Adam with L2 regularization
• AdaGrad: accumulates squared gradients from all past steps — effective for sparse gradients (NLP embeddings) but learning rate monotonically decreases, eventually becoming too small to learn
• RMSProp: AdaGrad with exponential moving average of squared gradients — prevents learning rate from shrinking to zero; predecessor to Adam; still used for RNN training in some settings

Large Batch Optimization:

• LARS (Layer-wise Adaptive Rate Scaling): adjusts learning rate per layer based on weight-to-gradient norm ratio — enables training with batch sizes up to 32K without accuracy loss; used for large-batch ImageNet training
• LAMB (Layer-wise Adaptive Moments for Batch training): combines LARS-style layer adaptation with Adam — enables BERT pre-training with batch size 64K in 76 minutes; critical for distributed training efficiency
• Gradient Accumulation: simulate large batch by accumulating gradients over multiple forward-backward passes — equivalent to large batch training without additional GPU memory; division by accumulation steps normalizes gradient scale

Optimizer selection is a foundational decision in deep learning training — AdamW has become the default for Transformer-based models (NLP, ViT), while SGD with momentum remains competitive for CNNs; understanding the tradeoffs between convergence speed, memory overhead, and generalization quality enables practitioners to choose the optimal optimizer for each architecture and dataset.