Data Augmentation Mixup Cutmix Consistency Regularization
# Data Augmentation: Mixup, CutMix & Consistency Regularization
## Introduction & Motivation
Data augmentation artificially increases training data via transformations. Mixup: interpolate samples + labels. CutMix: cut-paste patches. Augmentation improves generalization, reduces overfitting, robustness to perturbations. Critical when data limited.
Motivation: More data improves learning. Augmentation cheaper than collecting new data. Encourages models to learn invariances.
Applications: Computer vision (images), NLP (text), speech, medical imaging.
---
## Core Concepts & Theory
### Mixup
λ ∈ [0,1] sampled from Beta distribution. x̃ = λx_i + (1-λ)x_j; ỹ = λy_i + (1-λ)y_j.
### CutMix
Random patch cut from one image, pasted into another; labels mixed by patch ratio.
### Consistency Regularization
Augment twice; penalize prediction differences.
---
## Mathematical Formulation
Mixup:
$$ ilde{x} = \lambda x_i + (1-\lambda) x_j$$
$$ ilde{y} = \lambda y_i + (1-\lambda) y_j$$
CutMix:
$$ ilde{x} = x_i \odot M + x_j \odot (1-M)$$
$$ ilde{y} = \lambda y_i + (1-\lambda) y_j, \quad \lambda = 1 - |M|$$
Consistency loss:
$$\mathcal{L}_{ ext{cons}} = \mathbb{E}_{x,\xi_1,\xi_2}[D(f(g_{\xi_1}(x)), f(g_{\xi_2}(x)))]$$
---
## Advanced Theory & Extensions
### MixUp Variants
CutMix (spatial), Mixup (feature space), Mosaic (multi-sample).
### Augmentation Policies
AutoAugment: search for optimal augmentation policies.
### Test-Time Augmentation
Average predictions over augmented versions; improves robustness.
---
## Computational Considerations
Mixup: O(batch_size) interpolation; no overhead.
CutMix: O(batch_size) patching; minimal overhead.
Consistency: 2× forward passes; doubles compute.
---
## Practical Implementation Strategies
### Beta Distribution
λ ~ Beta(α, α) with α ∈ [0.1, 1.0]; controls interpolation strength.
### Patch Selection
Random location, size. CutMix uses image-level patch ratio.
### Augmentation Scheduling
Start weak, increase strength; curriculum for stability.
---
## Benchmark Datasets & Evaluation
ImageNet: Standard vision benchmark; 1M images.
CIFAR-10: 50K images; small, high-aug domain.
Metrics: Top-1/5 accuracy, robustness under perturbations.
---
## Key Challenges & Limitations
### Label Smoothing Interaction
Mixup + label smoothing redundant; may hurt.
### Distribution Shift
Augmented samples off-distribution; can mislead.
### Task-Dependency
Optimal augmentation depends on task; requires tuning.
---
## Hyperparameter Tuning
Alpha (Beta param): 0.1-1.0; lower = stronger mixup.
Augmentation strength: 0.1-1.0.
Consistency weight λ: 0.1-1.0.
---
## Real-World Applications & Case Studies
ImageNet Models: Mixup standard in modern training; +1-2% accuracy.
Medical Imaging: CutMix reduces overfitting on limited data.
Semi-Supervised: Consistency regularization + pseudo-labeling.
---
## Integration with Other Methods
Augmentation + Transfer Learning → pretrain with aug, finetune.
Augmentation + Uncertainty → perturb for UQ.
---
## Summary & Key Takeaways
Data augmentation via Mixup, CutMix, and consistency regularization improves generalization and robustness without additional labeled data.
Principles:
1. Mixup: linear interpolation in input and label space.
2. CutMix: spatial cutout-paste with label mixing.
3. Consistency: penalize prediction changes under augmentation.
4. Augmentation strength and policy task-dependent.
5. Test-time augmentation improves robustness.
---
---
## Appendix: Practical Labs
### Lab 1: Mixup Implementation
import torch
import torch.nn as nn
import numpy as np
def mixup(x, y, alpha=1.0):
"""Mixup augmentation"""
batch_size = x.size(0)
lam = np.random.beta(alpha, alpha)
index = torch.randperm(batch_size)
x_mixed = lam * x + (1 - lam) * x[index]
y_mixed_a, y_mixed_b = y, y[index]
return x_mixed, y_mixed_a, y_mixed_b, lam
def mixup_criterion(criterion, pred, y_a, y_b, lam):
"""Loss with mixed labels"""
return lam * criterion(pred, y_a) + (1 - lam) * criterion(pred, y_b)
# Data
x = torch.randn(32, 10)
y = torch.randint(0, 5, (32,))
x_mixed, y_a, y_b, lam = mixup(x, y, alpha=0.2)
print(f"Mixup coefficient: {lam:.3f}")
print(f"Mixed sample shape: {x_mixed.shape}")
assert x_mixed.shape == x.shape, "Should preserve shape"
assert 0 <= lam <= 1, "Lambda should be in [0,1]"
print("✓ Mixup working")
if __name__ == "__main__":
print("Lab 1: Mixup - PASSED")### Lab 2: CutMix Implementation
import torch
import numpy as np
def cutmix(x, y, alpha=1.0):
"""CutMix augmentation"""
batch_size = x.size(0)
lam = np.random.beta(alpha, alpha)
# Random box
w, h = x.size(3), x.size(2)
cut_ratio = np.sqrt(1 - lam)
cut_h = int(h * cut_ratio)
cut_w = int(w * cut_ratio)
cx = np.random.randint(0, w)
cy = np.random.randint(0, h)
bbx1 = np.clip(cx - cut_w // 2, 0, w)
bby1 = np.clip(cy - cut_h // 2, 0, h)
bbx2 = np.clip(cx + cut_w // 2, 0, w)
bby2 = np.clip(cy + cut_h // 2, 0, h)
index = torch.randperm(batch_size)
x_cutmix = x.clone()
x_cutmix[:, :, bby1:bby2, bbx1:bbx2] = x[index, :, bby1:bby2, bbx1:bbx2]
lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (h * w))
return x_cutmix, y, y[index], lam
# Data (4D: batch, channels, height, width)
x = torch.randn(8, 3, 32, 32)
y = torch.randint(0, 10, (8,))
x_cut, y_a, y_b, lam = cutmix(x, y, alpha=0.2)
print(f"CutMix lambda: {lam:.3f}")
print(f"Cut image shape: {x_cut.shape}")
assert x_cut.shape == x.shape, "Should preserve shape"
print("✓ CutMix working")
if __name__ == "__main__":
print("Lab 2: CutMix - PASSED")### Lab 3: Consistency Regularization
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
def consistency_loss(logits1, logits2, loss_fn=nn.KLDivLoss(reduction='mean')):
"""KL divergence between two augmentations"""
probs1 = torch.softmax(logits1, dim=1)
probs2 = torch.softmax(logits2, dim=1)
return loss_fn(probs2.log(), probs1.detach())
class SimpleModel(nn.Module):
def __init__(self):
super().__init__()
self.net = nn.Sequential(nn.Linear(20, 32), nn.ReLU(), nn.Linear(32, 5))
def forward(self, x):
return self.net(x)
def weak_augment(x):
return x + 0.05 * torch.randn_like(x)
def strong_augment(x):
return x + 0.2 * torch.randn_like(x)
model = SimpleModel()
optimizer = optim.Adam(model.parameters(), lr=0.01)
x = torch.randn(16, 20)
y = torch.randint(0, 5, (16,))
# Training step with consistency
optimizer.zero_grad()
logits = model(x)
supervised_loss = nn.functional.cross_entropy(logits, y)
x_weak = weak_augment(x)
x_strong = strong_augment(x)
logits_weak = model(x_weak)
logits_strong = model(x_strong)
cons_loss = consistency_loss(logits_weak, logits_strong)
total_loss = supervised_loss + 0.1 * cons_loss
total_loss.backward()
optimizer.step()
print(f"Supervised loss: {supervised_loss:.4f}, Consistency loss: {cons_loss:.4f}")
assert supervised_loss > 0, "Supervised loss should be positive"
assert cons_loss > 0, "Consistency loss should be positive"
print("✓ Consistency regularization working")
if __name__ == "__main__":
print("Lab 3: Consistency - PASSED")### Lab 4: Augmentation Evaluation
import numpy as np
import torch
def evaluate_augmentation_robustness(model, x_clean, y, augment_fn, n_augs=10):
"""Test accuracy with augmented samples"""
accuracies = []
for _ in range(n_augs):
x_aug = augment_fn(x_clean)
with torch.no_grad():
logits = model(x_aug)
acc = (logits.argmax(dim=1) == y).float().mean()
accuracies.append(acc.item())
mean_acc = np.mean(accuracies)
std_acc = np.std(accuracies)
return mean_acc, std_acc
def dummy_augment(x):
return x + 0.1 * torch.randn_like(x)
model = torch.nn.Sequential(torch.nn.Linear(20, 32), torch.nn.ReLU(), torch.nn.Linear(32, 5))
x_test = torch.randn(50, 20)
y_test = torch.randint(0, 5, (50,))
mean_acc, std_acc = evaluate_augmentation_robustness(model, x_test, y_test, dummy_augment, n_augs=5)
print(f"Augmentation robustness: {mean_acc:.2%} ± {std_acc:.2%}")
assert 0 <= mean_acc <= 1, "Accuracy should be in [0,1]"
print("✓ Augmentation evaluation working")
if __name__ == "__main__":
print("Lab 4: Evaluation - PASSED")