weight decay
**Weight decay** is a **regularization technique that penalizes large weight values during training** — adding a term proportional to the squared weights to the loss function, weight decay prevents overfitting by encouraging simpler models with smaller parameter magnitudes.
**What Is Weight Decay?**
- **Definition**: Penalty term on weight magnitude during optimization.
- **Mechanism**: Adds λ||w||² to loss function.
- **Effect**: Shrinks weights toward zero each update.
- **Goal**: Prevent overfitting, improve generalization.
**Why Weight Decay Works**
- **Simpler Models**: Smaller weights = simpler decision boundaries.
- **Reduced Variance**: Less sensitive to training noise.
- **Implicit Prior**: Assumes smaller weights are better.
- **Prevents Memorization**: Limits model capacity.
- **Stable Training**: Bounds weight magnitudes.
**Mathematical Formulation**
**L2 Regularization**:
```
Total Loss = Original Loss + λ × Σ(w²)
L_total = L_task + λ ||w||²
Where:
- L_task: Task loss (cross-entropy, MSE, etc.)
- λ: Weight decay coefficient (e.g., 0.01)
- ||w||²: Sum of squared weights
```
**Gradient Impact**:
```
Without decay: w = w - η × ∇L
With decay: w = w - η × (∇L + 2λw)
= w × (1 - 2ηλ) - η × ∇L
Each step shrinks weights by factor (1 - 2ηλ)
```
**L2 vs. AdamW Weight Decay**
**Key Difference**:
```
L2 Regularization (Adam):
- Adds λw to gradient before adaptive scaling
- Effect varies with learning rate adaptation
- Not true regularization with Adam
AdamW (Decoupled):
- Applies decay directly: w = w - ηλw
- Independent of gradient adaptation
- Proper regularization behavior
```
**Comparison**:
```
Method | Implementation | Adam Behavior
----------------|---------------------|---------------
L2 (traditional)| Loss += λ||w||² | Entangled
AdamW | w -= ηλw (separate) | Proper decay
```
**Typical Values**
**By Application**:
```
Application | Weight Decay
---------------------|-------------
LLM pre-training | 0.1
LLM fine-tuning | 0.01-0.1
Vision models | 0.0001-0.01
Small datasets | 0.1-0.5
Large datasets | 0.0001-0.01
```
**Implementation**
**PyTorch**:
```python
# AdamW with weight decay
optimizer = torch.optim.AdamW(
model.parameters(),
lr=1e-4,
weight_decay=0.01, # λ value
)
# Different decay for different layers
param_groups = [
{"params": model.embeddings.parameters(), "weight_decay": 0.0},
{"params": model.layers.parameters(), "weight_decay": 0.01},
]
optimizer = torch.optim.AdamW(param_groups, lr=1e-4)
```
**What to Exclude**:
```python
# Common practice: no decay on bias and layer norm
no_decay = ["bias", "LayerNorm.weight", "layer_norm.weight"]
param_groups = [
{
"params": [p for n, p in model.named_parameters()
if not any(nd in n for nd in no_decay)],
"weight_decay": 0.01,
},
{
"params": [p for n, p in model.named_parameters()
if any(nd in n for nd in no_decay)],
"weight_decay": 0.0,
},
]
```
**Tuning Weight Decay**
**Grid Search**:
```
Values to try: [0.0, 0.001, 0.01, 0.1]
Signs of too much: Underfitting, low training accuracy
Signs of too little: Overfitting, high train/val gap
Optimal: Best validation performance
```
**Relationship with Learning Rate**:
```
Higher LR often pairs with higher weight decay
Lower LR may need lower weight decay
Some papers report optimal λ/η ratio
```
Weight decay is **foundational to modern deep learning regularization** — by continuously shrinking weights toward zero, it prevents the large parameter values that lead to overfitting and ensures models generalize beyond their training data.