Curriculum Learning Sample Ordering Progressive Difficulty
# Curriculum Learning: Sample Ordering & Progressive Difficulty
## Introduction & Motivation
Curriculum learning: train on easy examples first, gradually increase difficulty. Self-paced learning: learner selects sample difficulty. Hard example mining: focus on hard samples. Sample weighting: weight by difficulty. Applications: accelerate training, improve generalization, cold-start learning.
Motivation: Random order: early training noisy. Curriculum: stable early learning, refine on hard samples.
Applications: Image classification, NLP, object detection.
---
## Core Concepts & Theory
### Curriculum Strategy
Easy-to-hard ordering; can be predefined or learned.
### Self-Paced Learning
Learner selects samples via loss; avoid early mislabeling.
### Hard Example Mining
Focus on misclassified; balance sample importance.
---
## Mathematical Formulation
Curriculum weighting:
$$L = \sum_i w_i(t) L(y_i, \hat{y}_i)$$
where w_i(t) = difficulty weight, increasing with time.
Self-paced weighting:
$$w_i = \begin{cases} 1 & L_i \leq \lambda \\ 0 & L_i > \lambda \end{cases}$$
keep only samples with loss < threshold λ.
---
## Advanced Theory & Extensions
### Teacher-Student Curriculum
Teacher model defines curriculum for student.
### Adversarial Curriculum
Adversarial examples generated; increasing hardness.
### Noisy Label Learning
Curriculum helps with noisy labels; learn easy samples first.
---
## Computational Considerations
Sorting: O(N log N) at each epoch; negligible.
Weighting: O(1) per sample; no overhead.
Loss-based selection: O(N) per epoch.
---
## Practical Implementation Strategies
### Difficulty Metric
Loss (classification loss) or loss-based ranking.
### Scheduling
Linear or exponential transition from easy to hard.
### Threshold
Control how much hard data; empirical tuning.
---
## Benchmark Datasets & Evaluation
CIFAR-10: Curriculum reduces training time ~20%.
ImageNet: Helps with noisy labels; stable training.
MNIST: Limited benefit; already easy.
---
## Key Challenges & Limitations
### Difficulty Estimation
How to define difficulty a priori.
### Hyperparameter Tuning
Curriculum schedule, thresholds; many parameters.
### Computation
Loss-based curriculum requires periodic re-sorting.
---
## Hyperparameter Tuning
Schedule: Linear or exponential progression.
Starting difficulty: Top 10-50% easiest samples.
Transition time: Gradual over epochs.
---
## Real-World Applications & Case Studies
Object Detection: Hard negatives via curriculum; improves mAP.
Noise Robust: Curriculum learns clean examples first.
Domain Adaptation: Easy target samples first.
---
## Integration with Other Methods
Curriculum + Mixup → progressive sample mixing.
Curriculum + Hard Negatives → balanced sampling.
---
## Summary & Key Takeaways
Curriculum learning via progressive difficulty improves training stability and generalization by presenting samples in order of increasing complexity.
Principles:
1. Easy-to-hard: stable early learning.
2. Self-paced: learner selects difficulty.
3. Hard mining: focus on difficult samples.
4. Weighting: gradual transition from easy to hard.
5. Robust to noise: learn clean first.
---
---
## Appendix: Practical Labs
### Lab 1: Difficulty Estimation
import numpy as np
def compute_sample_difficulty(X, y, model, metric='loss'):
"""Estimate sample difficulty via loss"""
losses = []
for i in range(len(X)):
output = model(X[i:i+1])
loss = ((output - y[i:i+1]) ** 2).mean()
losses.append(loss.item())
return np.array(losses)
# Test
import torch
np.random.seed(42)
model = torch.nn.Linear(10, 1)
X = torch.randn(20, 10)
y = torch.randn(20, 1)
difficulty = compute_sample_difficulty(X, y, model)
assert len(difficulty) == 20, "Should have difficulty per sample"
assert all(d >= 0 for d in difficulty), "Difficulty non-negative"
print("✓ Difficulty estimation working")
if __name__ == "__main__":
print("Lab 1: Difficulty - PASSED")### Lab 2: Easy-to-Hard Curriculum
import numpy as np
def curriculum_sort(X, y, difficulties, start_easy_ratio=0.5):
"""Sort by difficulty; start with easy samples"""
sorted_idx = np.argsort(difficulties)
X_sorted = X[sorted_idx]
y_sorted = y[sorted_idx]
return X_sorted, y_sorted, sorted_idx
# Test
np.random.seed(42)
X = np.random.randn(100, 10)
y = np.random.randn(100)
difficulties = np.random.uniform(0, 1, 100)
X_sorted, y_sorted, idx = curriculum_sort(X, y, difficulties)
assert len(idx) == 100, "Should have all indices"
assert all(difficulties[idx[i]] <= difficulties[idx[i+1]] for i in range(99)), "Should be sorted"
print("✓ Easy-to-hard curriculum working")
if __name__ == "__main__":
print("Lab 2: Curriculum - PASSED")### Lab 3: Self-Paced Learning
import numpy as np
def self_paced_weights(losses, lambda_param):
"""Compute self-paced weights: keep easy, discard hard"""
weights = (losses <= lambda_param).astype(float)
return weights
def update_lambda(epoch, max_epochs, lambda_start=1.0, lambda_end=0.1):
"""Update threshold lambda over epochs"""
# Exponential decay
lambda_t = lambda_start * (lambda_end / lambda_start) ** (epoch / max_epochs)
return lambda_t
# Test
np.random.seed(42)
losses = np.random.uniform(0, 1, 20)
weights = self_paced_weights(losses, 0.5)
assert len(weights) == 20, "Should have weights per sample"
assert all(w in [0, 1] for w in weights), "Weights binary"
print("✓ Self-paced learning working")
if __name__ == "__main__":
print("Lab 3: SelfPaced - PASSED")### Lab 4: Curriculum Training Simulation
import numpy as np
def train_with_curriculum(X, y, difficulties, epochs=10):
"""Simulate training with curriculum"""
sorted_idx = np.argsort(difficulties)
losses_per_epoch = []
for epoch in range(epochs):
# Gradually include harder samples
ratio = (epoch + 1) / epochs
n_samples = int(len(X) * ratio)
selected_idx = sorted_idx[:n_samples]
# Simulate loss (inversely proportional to difficulty)
avg_loss = np.mean(difficulties[selected_idx])
losses_per_epoch.append(avg_loss)
return losses_per_epoch
# Test
np.random.seed(42)
X = np.random.randn(100, 10)
y = np.random.randn(100)
difficulties = np.random.uniform(0, 1, 100)
losses = train_with_curriculum(X, y, difficulties, epochs=10)
assert len(losses) == 10, "Should have 10 epochs"
assert all(np.isfinite(l) for l in losses), "All finite"
print("✓ Curriculum training working")
if __name__ == "__main__":
print("Lab 4: Training - PASSED")