Weight Decay in Vision Transformers is a critically important, aggressively calibrated regularization hyperparameter that applies continuous, mathematically enforced shrinkage pressure on the model's weight matrices during every optimization step — and is empirically proven to be far more essential for training ViTs than for traditional CNNs, typically requiring values 10 to 50 times larger than standard ResNet configurations.

The Overfitting Vulnerability

• The CNN Advantage: Convolutional Neural Networks possess a powerful built-in inductive bias — their small, spatially local filters inherently constrain the hypothesis space. A $3 imes 3$ convolution kernel can only see 9 pixels at a time, providing a natural regularization effect that limits overfitting.
• The ViT Catastrophe: A Vision Transformer has no such built-in constraint. Every Self-Attention head can attend to every patch in the image from the very first layer. This enormous flexibility grants the model the mathematical capacity to trivially memorize the entire training set by constructing unique, complex attention patterns for each individual training image.

The Aggressive Weight Decay Strategy

• The CNN Baseline: Standard ResNet training uses weight decay values around $10^{-4}$ (0.0001). Any higher and the convolutional filters are over-constrained.
• The ViT Requirement: DeiT, BEiT, MAE, and virtually all competitive ViT training recipes mandate weight decay values of $0.05$ to $0.1$ — up to $1000 imes$ larger than CNN baselines.
• The Mechanism: At every optimization step, the algorithm multiplies every weight by $(1 - lambda)$ where $lambda$ is the weight decay factor. This relentless mathematical pressure continuously shrinks the weight magnitudes toward zero, forcing the attention weights to become sparse and preventing the model from memorizing individual training examples.

The Critical Exclusion Rule

Not all parameters tolerate aggressive weight decay:

• Bias Terms: These are small scalar offsets. Applying heavy decay to them forces them toward zero, effectively removing the learned offset and damaging the model.
• LayerNorm Parameters ($gamma$, $eta$): These affine scale and shift parameters are exquisitely sensitive. Decaying $gamma$ toward zero collapses the normalization, while decaying $eta$ eliminates the shift. Both actions severely destabilize training.

All competitive ViT training recipes explicitly exclude Bias and LayerNorm parameters from the weight decay parameter group.

Weight Decay in ViT is the mathematical pruning shears — relentlessly trimming the explosive, unconstrained attention weights of a Vision Transformer to prevent the model from lazily memorizing the training data instead of learning genuinely transferable visual representations.