Layer Decay (Layer-wise Learning Rate Decay) is a highly effective, carefully calibrated fine-tuning hyperparameter strategy that assigns progressively smaller learning rates to the earlier (deeper) layers of a pre-trained Vision Transformer while allowing the later (shallower, task-specific) layers to train with aggressively higher learning rates — mathematically preserving the universal low-level features learned during massive pre-training while rapidly adapting the high-level classification head to the new downstream task.

The Fine-Tuning Dilemma

• The Catastrophe: When a pre-trained ViT (trained on millions of images to recognize universal edges, textures, and shapes) is fine-tuned on a small downstream dataset with a single, uniform learning rate, the aggressive gradient updates violently overwrite the carefully learned universal features in the early layers. The model catastrophically "forgets" how to see basic visual primitives.
• The Opposite Extreme: If the learning rate is set too conservatively (to protect the early layers), the later task-specific layers barely update at all, and the model fails to adapt to the new domain.

The Layer Decay Formula

Layer Decay introduces a multiplicative decay factor ($alpha$, typically $0.65$ to $0.85$) applied layer-by-layer from the top of the network downward:

Where $N$ is the total number of layers and $i$ is the current layer index (starting from 1 at the bottom). The result is a smooth exponential gradient: the final classification head trains at the full $LR_{base}$, while the first patch embedding layer trains at a learning rate that may be $100 imes$ smaller.

Why Layer Decay is Critical for ViTs

• The CNN Contrast: Standard CNNs (ResNets) are relatively robust to uniform fine-tuning because their convolutional filters are small and localized. ViT Self-Attention layers, however, encode massive, global, interrelated feature dependencies across the entire image. A single aggressive gradient update to an early attention layer can cascade catastrophic representation damage throughout the entire network.
• The Empirical Rule: BEiT, MAE, and DINOv2 all demonstrated that Layer Decay is essentially mandatory for achieving state-of-the-art fine-tuning results with large Vision Transformers. Without it, performance drops by $1\%$ to $3\%$ on standard benchmarks.

Layer Decay is the principle of frozen roots — training a transplanted neural network by aggressively renovating the penthouse while barely touching the foundation, mathematically guaranteeing that the universal knowledge embedded in the deepest layers survives the transfer intact.