Regularization Techniques Dropout Weight Decay Early Stopping
# Regularization Techniques: Dropout, Weight Decay & Early Stopping
## Introduction & Motivation
Regularization: prevent overfitting by constraining model complexity. Dropout: randomly zero activations; ensemble effect. Weight decay (L2): penalize large weights; smoother functions. Early stopping: halt training when validation performance plateaus. Applications: all deep learning tasks; critical for generalization.
Motivation: Training error ↓ but test error ↑ indicates overfitting. Regularization trades accuracy for robustness.
Applications: Deep neural networks, all supervised learning, transfer learning fine-tuning.
---
## Core Concepts & Theory
### Dropout
Binary mask: probability p zeroed. Scaling: divide by (1-p) at test time (or use inverted dropout).
### Weight Decay
L2 penalty: λ·||w||². Equivalent to Gaussian prior on weights.
### Early Stopping
Monitor validation loss; stop if no improvement for k epochs.
---
## Mathematical Formulation
Dropout in forward pass:
$$h' = h \odot m, \quad m_i \sim ext{Bernoulli}(1-p)$$
With inverted dropout:
$$h' = \frac{h \odot m}{1-p}$$
Weight decay loss:
$$L = L_{ ext{task}} + \lambda ||w||_2^2$$
---
## Advanced Theory & Extensions
### DropConnect
Drop weights (not activations); similar effect.
### Batch Normalization
Implicit regularization; shifts distribution.
### Mixup
Interpolate samples; data-level regularization.
---
## Computational Considerations
Dropout: O(1) per activation; negligible overhead.
Weight decay: O(1) per parameter; regularization cost.
Early stopping: Reduces training time; no inference cost.
---
## Practical Implementation Strategies
### Dropout Rate
Typically 0.2-0.5; higher in larger networks.
### Weight Decay (λ)
1e-4 to 1e-2; dataset/architecture dependent.
### Early Stopping Patience
10-20 epochs; balance convergence and data efficiency.
---
## Benchmark Datasets & Evaluation
CIFAR-10: Dropout ~0.3 standard.
ImageNet: Weight decay ~1e-4 common.
Time Series: Early stopping critical.
---
## Key Challenges & Limitations
### Hyperparameter Tuning
Dropout, weight decay interact; requires tuning.
### Computational Cost
Early stopping reduces training; but requires validation.
### Task Dependency
Optimal regularization strength varies by task.
---
## Hyperparameter Tuning
Dropout p: 0.2-0.5; higher for larger models.
Weight decay λ: 1e-5 to 1e-2; log scale search.
Early stopping patience: 5-20 epochs.
---
## Real-World Applications & Case Studies
Vision: Dropout in ResNets; weight decay standard.
NLP: Dropout in BERT fine-tuning.
Time Series: Early stopping prevents overfitting.
---
## Integration with Other Methods
Dropout + Batch Norm → implicit + explicit regularization.
Weight Decay + Learning Rate Schedule → coupled dynamics.
---
## Summary & Key Takeaways
Regularization via dropout, weight decay, and early stopping prevents overfitting by constraining model complexity and monitoring generalization performance.
Principles:
1. Dropout: stochastic zeroing; ensemble effect.
2. Weight decay: L2 penalty; Gaussian prior.
3. Early stopping: validation monitoring; halt condition.
4. Interaction effects: combined regularization stronger.
5. Task-dependent: tuning essential.
---
---
## Appendix: Practical Labs
### Lab 1: Dropout Implementation
import torch
import torch.nn as nn
class DropoutLayer(nn.Module):
def __init__(self, p=0.5):
super().__init__()
self.p = p
def forward(self, x):
if self.training:
mask = torch.bernoulli(torch.ones_like(x) * (1 - self.p))
return x * mask / (1 - self.p)
else:
return x
# Test
dropout = DropoutLayer(p=0.3)
x = torch.randn(32, 100)
dropout.train()
x_train = dropout(x)
assert (x_train == 0).any(), "Should drop some activations"
dropout.eval()
x_test = dropout(x)
assert torch.allclose(x_test, x), "Should be identity at test time"
print("✓ Dropout working")
if __name__ == "__main__":
print("Lab 1: Dropout - PASSED")### Lab 2: Weight Decay
import torch
import torch.nn as nn
import torch.optim as optim
def compute_weight_decay_loss(model, lambda_wd):
"""Compute L2 regularization term"""
wd_loss = 0
for param in model.parameters():
wd_loss += torch.norm(param) ** 2
return lambda_wd * wd_loss / 2
# Test
model = nn.Linear(10, 5)
lambda_wd = 0.001
wd_loss = compute_weight_decay_loss(model, lambda_wd)
assert wd_loss > 0, "Weight decay should be positive"
print(f"Weight decay loss: {wd_loss:.4f}")
print("✓ Weight decay working")
if __name__ == "__main__":
print("Lab 2: Weight Decay - PASSED")### Lab 3: Early Stopping
import numpy as np
class EarlyStopping:
def __init__(self, patience=10):
self.patience = patience
self.best_loss = float('inf')
self.counter = 0
self.should_stop = False
def __call__(self, val_loss):
if val_loss < self.best_loss:
self.best_loss = val_loss
self.counter = 0
else:
self.counter += 1
if self.counter >= self.patience:
self.should_stop = True
return self.should_stop
# Test
es = EarlyStopping(patience=3)
val_losses = [0.5, 0.4, 0.35, 0.36, 0.37, 0.38, 0.39]
for i, loss in enumerate(val_losses):
stop = es(loss)
if stop:
print(f"Early stopping at epoch {i}")
break
assert es.should_stop, "Should stop after patience exceeded"
print("✓ Early stopping working")
if __name__ == "__main__":
print("Lab 3: Early Stopping - PASSED")### Lab 4: Regularization Comparison
import torch
import torch.nn as nn
import numpy as np
def compare_regularization(X_train, y_train, X_test, y_test, regularization_types):
"""Compare different regularization strengths"""
results = []
for reg_type in regularization_types:
model = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 1))
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=reg_type if reg_type else 0)
model.train()
for _ in range(10):
logits = model(X_train)
loss = nn.functional.mse_loss(logits.squeeze(), y_train)
optimizer.zero_grad()
loss.backward()
optimizer.step()
# Test loss
model.eval()
with torch.no_grad():
test_logits = model(X_test)
test_loss = nn.functional.mse_loss(test_logits.squeeze(), y_test)
results.append({'reg': reg_type, 'test_loss': test_loss.item()})
return results
# Test
X_train = torch.randn(100, 10)
y_train = torch.randn(100)
X_test = torch.randn(50, 10)
y_test = torch.randn(50)
results = compare_regularization(X_train, y_train, X_test, y_test, [0, 0.0001, 0.001])
print(f"Regularization comparison:")
for r in results:
print(f" λ={r['reg']}: test_loss={r['test_loss']:.4f}")
assert len(results) == 3, "Should have 3 results"
print("✓ Regularization comparison working")
if __name__ == "__main__":
print("Lab 4: Comparison - PASSED")