layer decay
**Layer Decay (Layer-wise Learning Rate Decay)** is a **highly effective, carefully calibrated fine-tuning hyperparameter strategy that assigns progressively smaller learning rates to the earlier (deeper) layers of a pre-trained Vision Transformer while allowing the later (shallower, task-specific) layers to train with aggressively higher learning rates — mathematically preserving the universal low-level features learned during massive pre-training while rapidly adapting the high-level classification head to the new downstream task.**
**The Fine-Tuning Dilemma**
- **The Catastrophe**: When a pre-trained ViT (trained on millions of images to recognize universal edges, textures, and shapes) is fine-tuned on a small downstream dataset with a single, uniform learning rate, the aggressive gradient updates violently overwrite the carefully learned universal features in the early layers. The model catastrophically "forgets" how to see basic visual primitives.
- **The Opposite Extreme**: If the learning rate is set too conservatively (to protect the early layers), the later task-specific layers barely update at all, and the model fails to adapt to the new domain.
**The Layer Decay Formula**
Layer Decay introduces a multiplicative decay factor ($alpha$, typically $0.65$ to $0.85$) applied layer-by-layer from the top of the network downward:
$$LR_i = LR_{base} imes alpha^{(N - i)}$$
Where $N$ is the total number of layers and $i$ is the current layer index (starting from 1 at the bottom). The result is a smooth exponential gradient: the final classification head trains at the full $LR_{base}$, while the first patch embedding layer trains at a learning rate that may be $100 imes$ smaller.
**Why Layer Decay is Critical for ViTs**
- **The CNN Contrast**: Standard CNNs (ResNets) are relatively robust to uniform fine-tuning because their convolutional filters are small and localized. ViT Self-Attention layers, however, encode massive, global, interrelated feature dependencies across the entire image. A single aggressive gradient update to an early attention layer can cascade catastrophic representation damage throughout the entire network.
- **The Empirical Rule**: BEiT, MAE, and DINOv2 all demonstrated that Layer Decay is essentially mandatory for achieving state-of-the-art fine-tuning results with large Vision Transformers. Without it, performance drops by $1\%$ to $3\%$ on standard benchmarks.
**Layer Decay** is **the principle of frozen roots** — training a transplanted neural network by aggressively renovating the penthouse while barely touching the foundation, mathematically guaranteeing that the universal knowledge embedded in the deepest layers survives the transfer intact.