Weight Decay L2 Regularization

# Weight Decay & L2 Regularization

## Introduction & Motivation

Weight decay: penalize large weights during training. Regularize model complexity. Applications: improve generalization, prevent overfitting.

Motivation: Control model capacity and improve generalization.

Applications: Reduce overfitting, better test performance.

---

## Core Concepts & Theory

### L2 Regularization

Penalty on squared weights.

### Weight Decay

Multiplicative weight reduction.

### Regularization Parameter

Control strength.

### Generalization Effect

Improve test accuracy.

---

## Mathematical Formulation

L2 Regularization Loss:
$$\mathcal{L}_{ ext{total}} = \mathcal{L} + \lambda \sum_i w_i^2$$

Weight Decay Update:
$$w_t = w_{t-1} (1 - \lambda \eta) - \eta abla L$$

Difference:
$$ ext{Weight decay} eq ext{L2 regularization in adaptive optimizers}$$

---

## Advanced Theory & Extensions

### Decoupled Weight Decay

AdamW style.

### Per-Layer Decay

Different rates per layer.

### Adaptive Regularization

Task-dependent strength.

---

## Computational Considerations

Overhead: O(1) per parameter.

Memory: No additional.

Impact: Significant on generalization.

---

## Practical Implementation Strategies

### Lambda Selection

Tune regularization strength.

### Optimizer Choice

AdamW recommended.

### Layer Differentiation

Different decay rates.

---

## Benchmark Datasets & Evaluation

ImageNet: Classification.

CIFAR-10: Generalization.

Language Models: Perplexity.

---

## Key Challenges & Limitations

### Interaction with Optimizer

Different behavior in Adam vs SGD.

### Hyperparameter Sensitivity

Lambda requires tuning.

### Task Dependency

Varies by problem.

---

## Hyperparameter Tuning

Lambda: 1e-4 to 1e-2.

Per-layer factors: 0.1 to 10.

Optimizer: AdamW typically better.

---

## Real-World Applications & Case Studies

Overfitting Prevention: Improves generalization.

Model Robustness: Better test performance.

Regularization: Complementary to dropout.

---

## Integration with Other Methods

Weight decay + dropout; + batch normalization.

---

## Summary & Key Takeaways

Weight decay improves generalization effectively.

Principles:
1. L2 penalty: Squared weight sum.
2. Generalization: Reduce overfitting.
3. Decoupling: Separate from optimizer.
4. Tuning: Lambda is critical.
5. Optimizer: AdamW recommended.

---

## Appendix: Practical Labs

### Lab 1: L2 Regularization Loss

import numpy as np

def l2_regularization_loss(weights, lambda_reg=0.01):
 """Compute L2 regularization penalty"""
 l2_loss = lambda_reg * np.sum(weights ** 2)
 return l2_loss

np.random.seed(42)
weights = np.random.randn(1000)
l2_loss = l2_regularization_loss(weights, 0.01)
assert l2_loss > 0
print(f"✓ L2 loss: {l2_loss:.3f}")

### Lab 2: Weight Decay Update

import numpy as np

def weight_decay_update(weights, gradients, learning_rate=0.01, decay=0.01):
 """Apply weight decay update"""
 # Weight decay (multiplicative)
 weights = weights * (1 - decay * learning_rate)
 # Gradient update
 weights = weights - learning_rate * gradients
 return weights

np.random.seed(42)
w = np.random.randn(100)
g = np.random.randn(100)
w_new = weight_decay_update(w, g)
assert np.linalg.norm(w_new) < np.linalg.norm(w)
print("✓ Weight decay update working")

### Lab 3: AdamW vs Adam

import numpy as np

def adam_l2(weights, gradients, lr=0.001, lambda_l2=0.01):
 """Adam with L2 regularization"""
 l2_grad = 2 * lambda_l2 * weights
 gradients_with_l2 = gradients + l2_grad
 # Update (simplified)
 weights = weights - lr * gradients_with_l2
 return weights

def adamw(weights, gradients, lr=0.001, weight_decay=0.01):
 """AdamW with weight decay"""
 # Decoupled weight decay
 weights = weights * (1 - weight_decay * lr)
 # Update (simplified)
 weights = weights - lr * gradients
 return weights

np.random.seed(42)
w = np.random.randn(10)
g = np.random.randn(10)
w_adam_l2 = adam_l2(w.copy(), g)
w_adamw = adamw(w.copy(), g)
print("✓ Adam vs AdamW comparison working")

### Lab 4: Lambda Sensitivity

import numpy as np

def test_lambda_effect(weights, lambda_values=[0.0, 0.01, 0.1, 1.0]):
 """Analyze effect of regularization strength"""
 losses = []
 for lam in lambda_values:
 l2_loss = lam * np.sum(weights ** 2)
 losses.append(l2_loss)
 return losses

np.random.seed(42)
weights = np.random.randn(1000)
losses = test_lambda_effect(weights)
assert losses[0] < losses[1] < losses[3]
print(f"✓ Lambda effect: {losses}")

---