weight decay
**Weight Decay** is the **regularization technique that penalizes large weight values by adding a fraction of the weight magnitude to the loss function or directly shrinking weights toward zero at each update step** — preventing overfitting by discouraging the model from relying on any single feature too heavily, and representing one of the most universally applied and effective regularization methods across all deep learning architectures.
**L2 Regularization vs. Decoupled Weight Decay**
- **L2 Regularization**: Add λ||w||² to loss → gradient becomes: ∇L + 2λw.
- **Decoupled Weight Decay**: Directly multiply weights: w ← w × (1 - λ·lr) after gradient step.
- With vanilla SGD: Both are equivalent.
- With Adam/AdaGrad: They are NOT equivalent!
- L2 regularization interacts with adaptive learning rates → effective decay varies per parameter.
- Decoupled weight decay (AdamW) applies uniform decay regardless of adaptive rate.
**AdamW vs. Adam + L2**
```
Adam + L2 regularization:
g_t = ∇L(w) + 2λw (L2 added to gradient)
m_t = β₁m_{t-1} + (1-β₁)g_t
v_t = β₂v_{t-1} + (1-β₂)g_t²
w = w - lr × m_t / (√v_t + ε)
# Problem: Decay is scaled by 1/√v_t → uneven
AdamW (decoupled weight decay):
g_t = ∇L(w) (gradient without L2)
m_t = β₁m_{t-1} + (1-β₁)g_t
v_t = β₂v_{t-1} + (1-β₂)g_t²
w = w - lr × m_t / (√v_t + ε) - lr × λ × w
# Decay is uniform → better regularization
```
**Typical Weight Decay Values**
| Model Type | Weight Decay (λ) | Notes |
|-----------|-----------------|-------|
| CNN (ResNet, etc.) | 1e-4 to 5e-4 | Standard for ImageNet training |
| Transformer (NLP) | 0.01 to 0.1 | Higher values common in LLMs |
| LLM pre-training | 0.1 | GPT-3, LLaMA use λ=0.1 |
| Fine-tuning | 0.0 to 0.01 | Lower to preserve pre-trained features |
| ViT | 0.05 to 0.3 | Vision transformers need stronger regularization |
**What NOT to Decay**
- **Bias terms**: Typically excluded (don't contribute to model complexity).
- **LayerNorm/BatchNorm parameters**: Scale (γ) and shift (β) excluded.
- **Embedding layers**: Sometimes excluded.
- Implementation: Parameter groups with different weight decay values.
**Effect of Weight Decay**
- Too little (λ → 0): Model overfits — weights grow large, memorizes training data.
- Too much (λ → ∞): Model underfits — weights forced too small, can't learn.
- Sweet spot: Depends on model size, dataset size, and other regularization.
- Weight decay + dropout + data augmentation: Complementary regularization effects.
Weight decay is **the most fundamental regularization technique in deep learning** — its simplicity and universal effectiveness across architectures make it one of the few hyperparameters that is always present in modern training configurations, with the AdamW formulation establishing decoupled weight decay as the standard for transformer-based models.