Learning Rate Schedules Warmup Cosine Decay Step Decay
# Learning Rate Schedules: Warmup, Cosine Decay & Step Decay
## Introduction & Motivation
Learning rate schedule: dynamic adjustment during training. Warmup: ramp from low to target; stabilize early training. Cosine decay: smooth decrease; avoid premature convergence. Step decay: discrete drops; simple baseline. Exponential decay: smooth exponential reduction. Applications: all deep learning; critical for reproducible training.
Motivation: Fixed learning rate: instability early, suboptimal late. Scheduling balances speed and stability.
Applications: Neural networks, transformers, fine-tuning.
---
## Core Concepts & Theory
### Warmup Phase
Linear ramp from 0 to target LR; stabilize gradients.
### Cosine Annealing
Smooth decrease via cosine curve; prevents oscillation.
### Step Decay
Multiply by factor at fixed epochs; classical schedule.
---
## Mathematical Formulation
Linear Warmup (first t_warmup steps):
$$\alpha_t = \alpha_{ ext{max}} \cdot \frac{t}{t_{ ext{warmup}}}$$
Cosine Annealing (after warmup):
$$\alpha_t = \alpha_{ ext{min}} + \frac{\alpha_{ ext{max}} - \alpha_{ ext{min}}}{2} \left[1 + \cos\left(\pi \frac{t - t_{ ext{warmup}}}{T - t_{ ext{warmup}}}
ight)
ight]$$
Step Decay:
$$\alpha_t = \alpha_0 \cdot \gamma^{\lfloor t / s
floor}$$
where γ = decay factor, s = step size.
---
## Advanced Theory & Extensions
### Warm Restarts
Cosine annealing with periodic resets; escape local minima.
### Exponential Decay
$$\alpha_t = \alpha_0 e^{-\lambda t}$$
### Polynomial Decay
$$\alpha_t = \alpha_0 \left(\frac{1 - t/T}{1} ight)^p$$
---
## Computational Considerations
Warmup: O(1) per step; negligible overhead.
Cosine/Step: O(1) per step; simple arithmetic.
Warm restarts: O(1) with reset logic.
---
## Practical Implementation Strategies
### Warmup Duration
Typically 5-10% of total steps; longer for transformer.
### Cosine T_max
One epoch or full training; depends on dataset size.
### Decay Factor γ
0.1 common; drop LR 10× at each milestone.
---
## Benchmark Datasets & Evaluation
ImageNet: Linear warmup + cosine annealing standard.
BERT: Warmup 10k steps; cosine to 0 over training.
ResNets: Step decay at epochs 30, 60, 90 (standard).
---
## Key Challenges & Limitations
### Warmup Hyperparameter
Too short: unstable early. Too long: slows convergence.
### Schedule-Optimizer Coupling
LR decay interacts with momentum; careful tuning needed.
### No Universal Schedule
Task and architecture dependent; empirical tuning required.
---
## Hyperparameter Tuning
Warmup ratio: 5-10% of total steps.
Cosine T_max: One full training run or per epoch.
Decay factor γ: 0.1 standard; 0.2 more aggressive.
---
## Real-World Applications & Case Studies
Vision: Linear warmup + cosine standard for ImageNet.
NLP: BERT uses linear warmup + polynomial decay.
Fine-tuning: Lower warmup; shorter total schedule.
---
## Integration with Other Methods
Schedule + Gradient Clipping → stabilize transformer training.
Schedule + Weight Decay → coupled dynamics.
---
## Summary & Key Takeaways
Learning rate scheduling via warmup, cosine annealing, and decay enables stable training across diverse optimization landscapes.
Principles:
1. Warmup: linear ramp stabilizes early training.
2. Cosine: smooth decay without abrupt jumps.
3. Step decay: simple, effective baseline.
4. Warmup ∝ data size; cosine ∝ total steps.
5. Empirical tuning essential; no universal schedule.
---
---
## Appendix: Practical Labs
### Lab 1: Linear Warmup
import torch
import numpy as np
def linear_warmup(step, warmup_steps, base_lr):
"""Linear warmup schedule"""
if step < warmup_steps:
return base_lr * (step / warmup_steps)
else:
return base_lr
# Test
np.random.seed(42)
base_lr = 0.001
warmup_steps = 1000
lrs = [linear_warmup(t, warmup_steps, base_lr) for t in range(2000)]
assert lrs[0] == 0, "Should start at 0"
assert lrs[warmup_steps - 1] < base_lr, "Should be below base at warmup end"
assert lrs[warmup_steps] == base_lr, "Should reach base at warmup"
assert all(lrs[i] <= lrs[i+1] for i in range(warmup_steps - 1)), "Should monotonically increase"
print("✓ Linear warmup working")
if __name__ == "__main__":
print("Lab 1: Warmup - PASSED")### Lab 2: Cosine Annealing
import torch
import numpy as np
def cosine_annealing(step, total_steps, base_lr, min_lr=1e-6):
"""Cosine annealing schedule"""
return min_lr + (base_lr - min_lr) * 0.5 * (1 + np.cos(np.pi * step / total_steps))
# Test
np.random.seed(42)
base_lr = 0.001
total_steps = 10000
min_lr = 1e-6
lrs = [cosine_annealing(t, total_steps, base_lr, min_lr) for t in range(total_steps)]
assert lrs[0] == base_lr, "Should start at base_lr"
assert lrs[-1] == min_lr, "Should end at min_lr"
assert all(np.isfinite(l) for l in lrs), "All LRs should be finite"
assert all(l >= min_lr for l in lrs), "LR should not go below min"
print("✓ Cosine annealing working")
if __name__ == "__main__":
print("Lab 2: Cosine - PASSED")### Lab 3: Step Decay
import torch
import numpy as np
def step_decay(step, base_lr, decay_factor=0.1, milestone_steps=[3000, 6000]):
"""Step decay schedule"""
decay_count = sum(1 for m in milestone_steps if step >= m)
return base_lr * (decay_factor ** decay_count)
# Test
np.random.seed(42)
base_lr = 0.001
milestones = [3000, 6000, 9000]
lrs = [step_decay(t, base_lr, 0.1, milestones) for t in range(10000)]
assert lrs[0] == base_lr, "Should start at base_lr"
assert lrs[3000] == base_lr * 0.1, "Should decay at first milestone"
assert lrs[6000] == base_lr * 0.01, "Should decay at second milestone"
assert lrs[9000] == base_lr * 0.001, "Should decay at third milestone"
print("✓ Step decay working")
if __name__ == "__main__":
print("Lab 3: Step Decay - PASSED")### Lab 4: Combined Schedule (Warmup + Cosine)
import torch
import numpy as np
def warmup_cosine_schedule(step, total_steps, base_lr, warmup_steps, min_lr=1e-6):
"""Warmup then cosine annealing"""
if step < warmup_steps:
return base_lr * (step / warmup_steps)
else:
progress = (step - warmup_steps) / (total_steps - warmup_steps)
return min_lr + (base_lr - min_lr) * 0.5 * (1 + np.cos(np.pi * progress))
# Test
np.random.seed(42)
base_lr = 0.001
total_steps = 10000
warmup_steps = 1000
lrs = [warmup_cosine_schedule(t, total_steps, base_lr, warmup_steps) for t in range(total_steps)]
assert lrs[0] == 0, "Should start at 0"
assert lrs[warmup_steps - 1] < base_lr, "Warmup should be below base"
assert abs(lrs[warmup_steps] - base_lr) < 1e-6, "Should reach base at warmup end"
assert lrs[-1] < base_lr, "Should decay by end"
assert all(np.isfinite(l) for l in lrs), "All finite"
print("✓ Warmup + cosine working")
if __name__ == "__main__":
print("Lab 4: Combined - PASSED")