Data Augmentation Mixup Cutmix Augmentation Strategies
# Data Augmentation: Mixup, CutMix & Augmentation Strategies
## Introduction & Motivation
Data Augmentation: increase training data variety. Mixup: linear interpolation in input space. CutMix: regional dropout. Applications: improve generalization, reduce overfitting, limited data scenarios.
Motivation: Limited data; regularization; robustness.
Applications: Image classification, detection, segmentation.
---
## Core Concepts & Theory
### Mixup
Linear combination of samples and labels.
### CutMix
Random cropping and mixing regions.
### Regional Dropout
Spatial regularization strategy.
---
## Mathematical Formulation
Mixup:
$$ ilde{x} = \lambda x_i + (1-\lambda) x_j$$
$$ ilde{y} = \lambda y_i + (1-\lambda) y_j$$
where \lambda \sim ext{Beta}(\alpha, \alpha).
CutMix:
$$ ilde{x} = x_i \odot M + x_j \odot (1-M)$$
where M is random binary mask.
---
## Advanced Theory & Extensions
### MixUp Variants
CutMix, MixBlock, FMix, SmoothMix.
### AutoAugment
Learned augmentation policies.
### RandAugment
Random augmentation chains.
---
## Computational Considerations
Mixup: O(1) per sample; no extra computation.
CutMix: O(1) masking operation.
AutoAugment: O(search_cost) one-time.
---
## Practical Implementation Strategies
### Beta Distribution
Alpha parameter; controls interpolation.
### Label Smoothing
Regularization with soft labels.
### Probability Tuning
When to apply augmentation.
---
## Benchmark Datasets & Evaluation
CIFAR-10/100: Standard augmentation benchmark.
ImageNet: Large-scale verification.
Detection/Segmentation: Task-specific evaluation.
---
## Key Challenges & Limitations
### Label Interpretation
Mixed labels semantic meaning.
### Hyperparameter Tuning
Alpha, cutmix probability sensitive.
### Task Dependency
Works better for some tasks.
---
## Hyperparameter Tuning
Alpha (Mixup): 0.1-1.0; controls blend.
CutMix probability: 0.5; when to apply.
Cut probability: 0.5; region size.
---
## Real-World Applications & Case Studies
Image Classification: Improved accuracy.
Object Detection: Better localization.
Semantic Segmentation: Robustness.
---
## Integration with Other Methods
Mixup + Ensemble → diversity.
CutMix + Regularization → robustness.
---
## Summary & Key Takeaways
Data Augmentation via Mixup and CutMix enables improved generalization through sample interpolation and spatial regularization.
Principles:
1. Mixup: linear interpolation.
2. CutMix: spatial mixing.
3. Beta distribution: blending control.
4. Label smoothing: soft targets.
5. Regularization: overfitting prevention.
---
---
## Appendix: Practical Labs
### Lab 1: Mixup
import numpy as np
def mixup_batch(X, y, alpha=1.0):
"""Apply Mixup augmentation"""
batch_size = len(X)
# Sample lambda from Beta distribution
lam = np.random.beta(alpha, alpha)
# Random permutation
index = np.random.permutation(batch_size)
# Mix inputs and labels
X_mixed = lam * X + (1 - lam) * X[index]
y_mixed = lam * y + (1 - lam) * y[index]
return X_mixed, y_mixed
# Test
np.random.seed(42)
X = np.random.randn(32, 3, 224, 224)
y = np.eye(10)[np.random.randint(0, 10, 32)]
X_mix, y_mix = mixup_batch(X, y, alpha=1.0)
assert X_mix.shape == X.shape, "Mixed input shape"
assert y_mix.shape == y.shape, "Mixed label shape"
print("✓ Mixup working")
if __name__ == "__main__":
print("Lab 1: Mixup - PASSED")### Lab 2: CutMix
import numpy as np
def cutmix_batch(X, y, alpha=1.0, cutmix_prob=0.5):
"""Apply CutMix augmentation"""
if np.random.rand() > cutmix_prob:
return X, y
batch_size = len(X)
# Sample lambda
lam = np.random.beta(alpha, alpha)
# Random indices
index = np.random.permutation(batch_size)
# Get spatial dimensions
_, _, H, W = X.shape
# Random box
cut_ratio = np.sqrt(1 - lam)
cut_h = int(H * cut_ratio)
cut_w = int(W * cut_ratio)
cx = np.random.randint(0, W)
cy = np.random.randint(0, H)
x1 = max(0, cx - cut_w // 2)
y1 = max(0, cy - cut_h // 2)
x2 = min(W, cx + cut_w // 2)
y2 = min(H, cy + cut_h // 2)
# Apply CutMix
X_mixed = X.copy()
X_mixed[:, :, y1:y2, x1:x2] = X[index, :, y1:y2, x1:x2]
# Recompute lambda based on actual cut
lam = 1 - ((x2 - x1) * (y2 - y1)) / (H * W)
y_mixed = lam * y + (1 - lam) * y[index]
return X_mixed, y_mixed
# Test
np.random.seed(42)
X = np.random.randn(32, 3, 224, 224)
y = np.eye(10)[np.random.randint(0, 10, 32)]
X_cut, y_cut = cutmix_batch(X, y)
assert X_cut.shape == X.shape, "Cut input shape"
assert y_cut.shape == y.shape, "Cut label shape"
print("✓ CutMix working")
if __name__ == "__main__":
print("Lab 2: CutMix - PASSED")### Lab 3: Beta Distribution Sampling
import numpy as np
def sample_lambda_distribution(alpha, num_samples=1000):
"""Sample from Beta distribution for lambda"""
lambdas = np.random.beta(alpha, alpha, num_samples)
return lambdas
# Test
alphas = [0.1, 0.5, 1.0, 2.0]
for alpha in alphas:
lambdas = sample_lambda_distribution(alpha, 1000)
assert np.all((lambdas >= 0) & (lambdas <= 1)), f"Valid lambdas for alpha={alpha}"
print("✓ Beta distribution sampling working")
if __name__ == "__main__":
print("Lab 3: BetaDistribution - PASSED")### Lab 4: Label Smoothing with Augmentation
import numpy as np
def label_smoothing_augmented(y, epsilon=0.1):
"""Apply label smoothing"""
num_classes = y.shape[1] if len(y.shape) > 1 else y.max() + 1
# Convert to one-hot if needed
if len(y.shape) == 1:
y_one_hot = np.eye(num_classes)[y]
else:
y_one_hot = y
# Smooth labels
y_smoothed = (1 - epsilon) * y_one_hot + epsilon / num_classes
return y_smoothed
# Test
np.random.seed(42)
y = np.eye(10)[np.random.randint(0, 10, 32)]
y_smoothed = label_smoothing_augmented(y, epsilon=0.1)
assert y_smoothed.shape == y.shape, "Smoothed shape"
assert np.allclose(y_smoothed.sum(axis=1), 1.0), "Normalized"
print("✓ Label smoothing working")
if __name__ == "__main__":
print("Lab 4: LabelSmoothing - PASSED")