Warm-Up Cool-Down Schedules

# Warm-up & Cool-down Schedules

## Introduction & Motivation

Warm-up: gradually increase learning rate. Cool-down: gradually decrease. Improve stability and convergence. Applications: stable training, better final models.

Motivation: Stabilize training in early phases and refinement.

Applications: Robust convergence, improved performance.

---

## Core Concepts & Theory

### Linear Warm-up

Gradual increase from zero.

### Cool-down Phase

Final refinement period.

### Two-Stage Training

Combine warm-up and decay.

### Stability Enhancement

Prevent early instability.

---

## Mathematical Formulation

Linear Warm-up:
$$\eta_t = \eta_0 \cdot \frac{t}{T_{ ext{warmup}}} ext{ for } t \leq T_{ ext{warmup}}$$

Combined Schedule:
$$\eta_t = \begin{cases} \eta_0 \frac{t}{T_w} & t \leq T_w \\ ext{decay}(t) & t > T_w \end{cases}$$

---

## Advanced Theory & Extensions

### Polynomial Warm-up

Non-linear increase.

### LAMB Warm-up

Layer-adaptive warm-up.

### Cool-down Tuning

Optimize refinement phase.

---

## Computational Considerations

Overhead: Negligible.

Impact: Significant on convergence.

Memory: No additional memory.

---

## Practical Implementation Strategies

### Warm-up Duration

Typically 5-10% of training.

### Cool-down Duration

1-5% of total epochs.

### Temperature Control

Gradual adjustment.

---

## Benchmark Datasets & Evaluation

ImageNet: Large-scale vision.

BERT: Language model pretraining.

Large LLMs: Billion-parameter models.

---

## Key Challenges & Limitations

### Duration Tuning

Task-specific duration.

### Interaction

Effects with optimizer.

### Instability

May still occur without careful tuning.

---

## Hyperparameter Tuning

Warm-up epochs: 5-20% of total.

Cool-down epochs: 1-10% of total.

Initial LR: 1e-3 to 1e-2.

---

## Real-World Applications & Case Studies

Large Models: BERT, GPT pretraining.

Mixed Precision: Critical for stability.

Distributed Training: Synchronization.

---

## Integration with Other Methods

Warm-up + mixed precision; + gradient clipping.

---

## Summary & Key Takeaways

Warm-up and cool-down improve training stability.

Principles:
1. Warm-up: Gradual increase.
2. Stability: Prevent early divergence.
3. Cool-down: Refinement phase.
4. Timing: Critical for convergence.
5. Effectiveness: Often necessary for large models.

---

## Appendix: Practical Labs

### Lab 1: Linear Warm-up

import numpy as np

def linear_warmup(step, total_warmup_steps, initial_lr=0.0, target_lr=0.1):
 """Linear learning rate warm-up"""
 if step < total_warmup_steps:
 lr = initial_lr + (target_lr - initial_lr) * (step / total_warmup_steps)
 else:
 lr = target_lr
 return lr

lrs = [linear_warmup(s, 1000, 0.0, 0.1) for s in range(0, 2000, 200)]
assert lrs[0] < lrs[2] < lrs[4]
print(f"✓ Linear warmup: {lrs}")

### Lab 2: Cool-down Phase

import numpy as np

def cooldown_schedule(epoch, total_epochs=100, peak_lr=0.1, cooldown_start=80, min_lr=0.0):
 """Cool-down phase schedule"""
 if epoch < cooldown_start:
 lr = peak_lr
 else:
 fraction = (epoch - cooldown_start) / (total_epochs - cooldown_start)
 lr = peak_lr - (peak_lr - min_lr) * fraction
 return lr

lrs = [cooldown_schedule(e, 100) for e in range(100)]
assert lrs[50] > lrs[90]
print(f"✓ Cool-down schedule working")

### Lab 3: Combined Warm-up + Decay

import numpy as np

def warmup_then_decay(step, total_steps, warmup_fraction=0.1, initial_lr=0.1):
 """Warm-up then decay schedule"""
 warmup_steps = int(total_steps * warmup_fraction)
 
 if step < warmup_steps:
 lr = initial_lr * (step / warmup_steps)
 else:
 progress = (step - warmup_steps) / (total_steps - warmup_steps)
 lr = initial_lr * 0.5 * (1 + np.cos(np.pi * progress))
 
 return lr

lrs = [warmup_then_decay(s, 1000) for s in range(0, 1000, 100)]
assert lrs[0] < lrs[1] # Warming up
print(f"✓ Combined schedule working")

### Lab 4: Effective Learning Rate Analysis

import numpy as np

def analyze_schedule(schedule_fn, total_steps):
 """Analyze learning rate schedule"""
 lrs = [schedule_fn(s) for s in range(total_steps)]
 
 return {
 'initial': lrs[0],
 'peak': max(lrs),
 'final': lrs[-1],
 'mean': np.mean(lrs)
 }

schedule = lambda s: 0.1 * (s / 100) if s < 100 else 0.1 * (1 - (s-100)/900)
stats = analyze_schedule(schedule, 1000)
print(f"✓ Schedule stats: peak={stats['peak']:.3f}, final={stats['final']:.3f}")

---