Learning Rate Scheduling Strategies

# Learning Rate Scheduling Strategies

## Introduction & Motivation

Learning rate scheduling: vary learning rate during training. Improve convergence and final accuracy. Applications: effective training, convergence control.

Motivation: Adapt learning rate for different training phases.

Applications: Optimizing convergence, improving final performance.

---

## Core Concepts & Theory

### Decay Schedules

Reduce learning rate over time.

### Step Decay

Drop at milestones.

### Cosine Annealing

Smooth periodic decay.

### Exponential Decay

Exponential reduction.

---

## Mathematical Formulation

Step Decay:
$$\eta_t = \eta_0 \cdot \gamma^{\lfloor t / T floor}$$

Cosine Annealing:
$$\eta_t = \eta_{\min} + \frac{\eta_0 - \eta_{\min}}{2}(1 + \cos(\pi t / T))$$

Exponential:
$$\eta_t = \eta_0 \cdot e^{-\lambda t}$$

---

## Advanced Theory & Extensions

### Cyclical Learning Rates

Periodic variation.

### Warm Restarts

Periodic resets.

### Adaptive Schedules

Task-specific adaptation.

---

## Computational Considerations

Schedule computation: O(1) per step.

Impact: Significant on convergence.

Memory: No additional memory.

---

## Practical Implementation Strategies

### Milestone Selection

Choose decay points.

### Initial Learning Rate

Set carefully.

### Final Learning Rate

Set minimum value.

---

## Benchmark Datasets & Evaluation

ImageNet: Vision tasks.

CIFAR-10: Small-scale classification.

Language Models: NLP tasks.

---

## Key Challenges & Limitations

### Hyperparameter Sensitivity

Schedule affects results.

### Task Dependency

Different schedules per task.

### Tuning Effort

Requires experimentation.

---

## Hyperparameter Tuning

Initial LR: 1e-2 to 1e-3.

Decay factor: 0.1-0.5.

Milestone epochs: Task-dependent.

---

## Real-World Applications & Case Studies

Training Stability: Improve convergence.

Final Accuracy: Often improves.

Convergence Speed: Faster learning.

---

## Integration with Other Methods

Learning rate scheduling + warm-up; + adaptive optimizers.

---

## Summary & Key Takeaways

Learning rate scheduling improves training dynamics.

Principles:
1. Decay: Reduce over time.
2. Step: Discrete milestones.
3. Cosine: Smooth annealing.
4. Cyclical: Periodic variation.
5. Adaptation: Task-specific tuning.

---

## Appendix: Practical Labs

### Lab 1: Step Decay Schedule

import numpy as np

def step_decay_schedule(epoch, initial_lr=0.1, decay_factor=0.1, decay_epochs=[10, 20]):
 """Step decay learning rate schedule"""
 lr = initial_lr
 for milestone in decay_epochs:
 if epoch >= milestone:
 lr *= decay_factor
 return lr

lrs = [step_decay_schedule(e) for e in range(30)]
assert lrs[0] > lrs[15] > lrs[25]
print(f"✓ Step decay: LR progression correct")

### Lab 2: Cosine Annealing

import numpy as np

def cosine_annealing_schedule(epoch, total_epochs=100, initial_lr=0.1, min_lr=0.0):
 """Cosine annealing learning rate schedule"""
 lr = min_lr + (initial_lr - min_lr) * (1 + np.cos(np.pi * epoch / total_epochs)) / 2
 return lr

lrs = [cosine_annealing_schedule(e, 100) for e in range(100)]
assert lrs[0] == 0.1
assert abs(lrs[50] - 0.05) < 0.01
print(f"✓ Cosine annealing: Smooth decay")

### Lab 3: Exponential Decay

import numpy as np

def exponential_decay_schedule(step, initial_lr=0.1, decay_rate=0.96, decay_steps=1000):
 """Exponential decay learning rate schedule"""
 lr = initial_lr * (decay_rate ** (step / decay_steps))
 return lr

lrs = [exponential_decay_schedule(s) for s in range(0, 5000, 1000)]
assert lrs[0] > lrs[1] > lrs[2]
print(f"✓ Exponential decay: {lrs}")

### Lab 4: Cyclical Learning Rates

import numpy as np

def cyclical_learning_rate(epoch, base_lr=0.001, max_lr=0.1, cycle_length=20):
 """Cyclical learning rate schedule"""
 cycle = epoch % cycle_length
 lr = base_lr + (max_lr - base_lr) * abs(1 - 2 * cycle / cycle_length)
 return lr

lrs = [cyclical_learning_rate(e, cycle_length=10) for e in range(30)]
assert len(lrs) == 30
print(f"✓ Cyclical LR: {len(lrs)} values")

---