neural network optimization

**Neural Network Optimizers** are the **algorithms that update model parameters to minimize the loss function during training — where the choice of optimizer (SGD, Adam, AdamW, LAMB) and its hyperparameters (learning rate, momentum, weight decay) directly determines training speed, final model quality, and generalization performance, making optimizer selection one of the most impactful decisions in deep learning practice**. **Stochastic Gradient Descent (SGD) Foundation** The simplest optimizer: θ_{t+1} = θ_t - η × ∇L(θ_t), where η is the learning rate and ∇L is the gradient computed on a mini-batch. SGD with momentum adds a velocity term: v_t = β × v_{t-1} + ∇L(θ_t); θ_{t+1} = θ_t - η × v_t. Momentum smooths gradient noise and accelerates convergence along consistent gradient directions. SGD+momentum remains the strongest optimizer for computer vision (ResNet, ConvNeXt) when properly tuned. **Adaptive Learning Rate Optimizers** - **Adam (Adaptive Moment Estimation)**: Maintains per-parameter running averages of the first moment (mean, m_t) and second moment (variance, v_t) of gradients. The learning rate for each parameter is scaled by 1/√v_t — parameters with large gradients get smaller updates, parameters with small gradients get larger updates. Less sensitive to learning rate choice than SGD; faster initial convergence. - **AdamW**: Decouples weight decay from gradient-based updates. Standard L2 regularization in Adam interacts poorly with adaptive learning rates (different parameters with different effective learning rates should have different regularization strengths). AdamW applies weight decay directly to parameters: θ_{t+1} = (1-λ) × θ_t - η × m_t/√v_t. The default optimizer for Transformer training. - **LAMB (Layer-wise Adaptive Moments)**: Extends Adam with per-layer learning rate scaling based on the ratio of parameter norm to update norm. Enables large-batch training (batch size 32K-64K) without accuracy loss. Used for BERT pre-training at scale. - **Lion (EvoLved Sign Momentum)**: Discovered through program search (Google, 2023). Uses only the sign of the momentum (not magnitude), reducing memory by 50% compared to Adam (no second moment). Competitive with AdamW while using less memory. **Learning Rate Schedules** - **Warmup**: Start with a very small learning rate and linearly increase to the target over the first 1-10% of training. Essential for Transformers where early large updates destabilize attention weights. - **Cosine Decay**: After warmup, decrease the learning rate following a cosine curve to near-zero. Smooth schedule that avoids the abrupt drops of step decay. The standard for most modern training. - **Cosine with Restarts**: Periodically reset the learning rate to the maximum, creating multiple cosine cycles. Can escape local minima and improve final performance. - **One-Cycle Policy**: Single cosine cycle from low → high → low learning rate. Super-convergence: achieves the same accuracy in 10x fewer iterations with 10x higher peak learning rate. **Practical Guidelines** - **Vision (CNNs)**: SGD+momentum (0.9) with cosine decay. Learning rate 0.1 for batch size 256, scale linearly with batch size. - **Transformers/LLMs**: AdamW with β1=0.9, β2=0.95-0.999, weight decay 0.01-0.1, warmup 1-5% of training, cosine decay. - **Fine-tuning**: Lower learning rate (1e-5 to 5e-5) than pretraining. Layer-wise learning rate decay (lower layers get smaller rates). Neural Network Optimizers are **the engines that drive learning** — converting loss gradients into parameter updates through algorithms whose subtle mathematical differences translate into significant real-world differences in training cost, final accuracy, and model robustness.