warmup epochs in vit

**Warmup epochs in ViT** are the **initial training phase where learning rate increases gradually from a small value to target value to avoid early optimization shocks** - this controlled ramp is critical because random initialization plus large step sizes can destabilize deep transformer training. **What Is Learning Rate Warmup?** - **Definition**: A schedule that linearly or smoothly raises learning rate during first few epochs. - **Purpose**: Prevents large destructive updates before normalization and gradients stabilize. - **Typical Range**: Commonly 5 to 20 warmup epochs depending on dataset size and batch scale. - **Compatibility**: Usually followed by cosine decay or polynomial decay schedule. **Why Warmup Matters** - **Stability**: Reduces early divergence and gradient explosions. - **Convergence Quality**: Helps model reach better basins by avoiding chaotic start. - **Scale Support**: Necessary when using large batch sizes and aggressive base learning rates. - **Reproducibility**: Makes training less sensitive to random seed and hardware variation. - **Optimization Synergy**: Works well with AdamW and pre-norm transformers. **Warmup Strategies** **Linear Warmup**: - Increase learning rate by constant increment each step. - Simple and widely adopted baseline. **Cosine Warmup**: - Smooth ramp to target with curved profile. - Can reduce abrupt transition at warmup end. **Layerwise Warmup**: - Use different warmup scales for backbone and head during fine-tuning. - Helpful when head is randomly initialized. **How It Works** **Step 1**: Start with very low learning rate near zero and increase it each iteration until reaching configured base rate. **Step 2**: Switch to main decay schedule after warmup while monitoring loss spikes and gradient norms. **Tools & Platforms** - **timm schedulers**: Built in warmup plus cosine decay options. - **PyTorch optim wrappers**: Easy to chain warmup and main schedule. - **Training dashboards**: Visualize learning rate curve against loss behavior. Warmup epochs are **the controlled launch sequence that keeps ViT optimization from collapsing in the first minutes of training** - they convert unstable starts into smooth convergence trajectories.