Gradient Descent Optimizers
# Gradient Descent Optimizers
## Introduction & Motivation
Optimization: update parameters to minimize loss. SGD, Adam, AdamW. Applications: training convergence, hyperparameter tuning.
Motivation: Efficiently solve non-convex optimization.
Applications: Model training, parameter updates.
---
## Core Concepts & Theory
### Stochastic Gradient Descent
Mini-batch updates.
### Momentum Methods
Accelerate convergence.
### Adaptive Learning Rates
Per-parameter adjustments.
### Gradient Clipping
Stability control.
---
## Mathematical Formulation
SGD with Momentum:
$$v_t = \beta v_{t-1} + g_t, \quad heta_t = heta_{t-1} - \alpha v_t$$
Adam:
$$m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t, \quad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2$$
$$ heta_t = heta_{t-1} - \alpha \frac{m_t}{\sqrt{v_t} + \epsilon}$$
AdamW:
$$ heta_t = heta_{t-1} - \alpha m_t / (\sqrt{v_t} + \epsilon) - \lambda heta_{t-1}$$
---
## Advanced Theory & Extensions
### LARS Optimizer
Layer-wise adaptive rates.
### LAMB Optimizer
Large batch training.
### Lookahead
Acceleration wrapper.
---
## Computational Considerations
SGD: O(params).
Adam: O(params) with 2x memory.
Sparse updates: O(nnz).
---
## Practical Implementation Strategies
### Learning Rate Scheduling
Decay schedules.
### Warmup Phases
Stability initialization.
### Optimizer Tuning
Hyperparameter selection.
---
## Benchmark Datasets & Evaluation
ImageNet: Convergence speed.
CIFAR-10: Small-scale comparison.
NLP: GLUE benchmark.
---
## Key Challenges & Limitations
### Hyperparameter Sensitivity
Learning rate tuning.
### Generalization Gap
Adam vs. SGD tradeoff.
### Computational Cost
Memory overhead.
---
## Hyperparameter Tuning
Learning rate: 1e-5 to 1e-1.
Beta1 (momentum): 0.9.
Beta2 (variance): 0.999.
---
## Real-World Applications & Case Studies
Vision: SGD with momentum standard.
NLP: Adam popular.
Large-scale: LARS/LAMB for distributed.
---
## Integration with Other Methods
Optimizers + learning rate schedules for convergence; + gradient clipping for stability.
---
## Summary & Key Takeaways
Optimization algorithms enable efficient training.
Principles:
1. SGD: Simple updates.
2. Momentum: Acceleration.
3. Adaptive rates: Per-parameter tuning.
4. Scheduling: Dynamic adjustment.
5. Stability: Gradient control.
---
## Appendix: Practical Labs
### Lab 1: SGD with Momentum
import numpy as np
def sgd_momentum(params, grads, learning_rate=0.01, momentum=0.9, velocity=None):
"""SGD with momentum update"""
if velocity is None:
velocity = [np.zeros_like(p) for p in params]
updated_params = []
for p, g, v in zip(params, grads, velocity):
v = momentum * v + g
p = p - learning_rate * v
updated_params.append(p)
return updated_params, velocity
np.random.seed(42)
params = [np.random.randn(100, 100)]
grads = [np.random.randn(100, 100) * 0.1]
updated, vel = sgd_momentum(params, grads)
assert updated[0].shape == params[0].shape, "Correct update shape"
print("✓ SGD momentum working")### Lab 2: Adam Optimizer
import numpy as np
def adam_step(params, grads, learning_rate=0.001, beta1=0.9, beta2=0.999,
epsilon=1e-8, m=None, v=None, t=1):
"""Adam optimization step"""
if m is None:
m = [np.zeros_like(p) for p in params]
if v is None:
v = [np.zeros_like(p) for p in params]
updated_params = []
for p, g, m_t, v_t in zip(params, grads, m, v):
m_t = beta1 * m_t + (1 - beta1) * g
v_t = beta2 * v_t + (1 - beta2) * (g ** 2)
m_hat = m_t / (1 - beta1 ** t)
v_hat = v_t / (1 - beta2 ** t)
p = p - learning_rate * m_hat / (np.sqrt(v_hat) + epsilon)
updated_params.append(p)
return updated_params, m, v
np.random.seed(42)
params = [np.random.randn(100, 100)]
grads = [np.random.randn(100, 100) * 0.1]
updated, m, v = adam_step(params, grads)
assert updated[0].shape == params[0].shape, "Correct update shape"
print("✓ Adam optimizer working")### Lab 3: Learning Rate Scheduling
import numpy as np
def exponential_decay(step, initial_lr=0.1, decay_rate=0.96, decay_steps=1000):
"""Exponential decay schedule"""
lr = initial_lr * (decay_rate ** (step / decay_steps))
return lr
def cosine_annealing(step, total_steps, initial_lr=0.1):
"""Cosine annealing schedule"""
lr = initial_lr * (1 + np.cos(np.pi * step / total_steps)) / 2
return lr
lrs_exp = [exponential_decay(s) for s in range(5000)]
lrs_cos = [cosine_annealing(s, 5000) for s in range(5000)]
assert lrs_exp[-1] < lrs_exp[0], "Exponential decay"
print("✓ Learning rate scheduling working")### Lab 4: Gradient Clipping
import numpy as np
def clip_gradients_by_norm(grads, clip_norm=1.0):
"""Clip gradients by global norm"""
global_norm = np.sqrt(sum(np.sum(g**2) for g in grads))
scale = clip_norm / (global_norm + 1e-8)
scale = min(1.0, scale)
clipped = [g * scale for g in grads]
return clipped
np.random.seed(42)
grads = [np.random.randn(100, 100) for _ in range(3)]
clipped = clip_gradients_by_norm(grads, clip_norm=1.0)
total_norm = np.sqrt(sum(np.sum(g**2) for g in clipped))
assert total_norm <= 1.01, "Gradients clipped"
print("✓ Gradient clipping working")---