Curriculum Learning Progressive Training Strategy
# Curriculum Learning: Progressive Training Strategy
## Introduction & Motivation
Curriculum Learning: train on easy samples first, progressively harder. Self-paced learning: model selects samples. Applications: convergence speedup, better generalization, learning stability.
Motivation: Mimic human learning; easy-to-hard progression.
Applications: Improvement in convergence and accuracy.
---
## Core Concepts & Theory
### Difficulty Metric
Measure sample complexity.
### Sample Selection
Progressive curriculum design.
### Self-Paced Learning
Model-selected curriculum.
---
## Mathematical Formulation
Curriculum-weighted loss:
$$L = \sum_i w_i(t) L(f(x_i), y_i)$$
Self-paced weight:
$$w_i = \mathbb{1}[ ext{loss}_i \leq \lambda(t)]$$
where \lambda(t) is dynamic threshold.
---
## Advanced Theory & Extensions
### Self-Paced Curriculum
Automatic difficulty selection.
### Teacher-Student Curriculum
One model guides another.
### Adversarial Curriculum
Hard negative samples.
---
## Computational Considerations
Difficulty estimation: O(N) per epoch.
Sample selection: O(N) sorting.
Training: No significant overhead.
---
## Practical Implementation Strategies
### Difficulty Estimation
Loss-based, model uncertainty, novelty.
### Curriculum Design
Exponential, linear, or custom schedule.
### Warmup Phase
Initialize with full curriculum.
---
## Benchmark Datasets & Evaluation
CIFAR-10/100: Curriculum benchmark.
ImageNet: Large-scale validation.
Noisy Labels: Robust training.
---
## Key Challenges & Limitations
### Curriculum Design
Problem-dependent; manual effort.
### Computational Overhead
Difficulty estimation adds cost.
### Hyperparameter Tuning
Curriculum rate; initial difficulty.
---
## Hyperparameter Tuning
Curriculum rate: Slow to fast.
Initial difficulty: Start with easy samples.
Hardness metric: Loss, confidence, or other.
---
## Real-World Applications & Case Studies
Noisy Datasets: Robust to label noise.
Domain Adaptation: Easy-to-hard transfer.
Object Detection: Sample difficulty ordering.
---
## Integration with Other Methods
Curriculum + Regularization → stable training.
Curriculum + Ensemble → diversity.
---
## Summary & Key Takeaways
Curriculum Learning via progressive training enables improved convergence and generalization through difficulty-ordered sample selection.
Principles:
1. Easy-to-hard: sample difficulty.
2. Self-paced: dynamic selection.
3. Curriculum rate: progression speed.
4. Loss threshold: sample filtering.
5. Robustness: noise handling.
---
---
## Appendix: Practical Labs
### Lab 1: Sample Difficulty Ranking
import numpy as np
def rank_sample_difficulty(losses, method='loss'):
"""Rank samples by difficulty"""
if method == 'loss':
difficulty = losses
elif method == 'entropy':
# Assuming losses are logits
probs = 1 / (1 + np.exp(-losses))
entropy = -probs * np.log(probs + 1e-8) - (1-probs) * np.log(1-probs + 1e-8)
difficulty = entropy
else:
raise ValueError(f"Unknown method: {method}")
# Rank: 0=easiest, N-1=hardest
ranking = np.argsort(difficulty)
return ranking, difficulty
# Test
np.random.seed(42)
losses = np.random.rand(100)
ranking, diff = rank_sample_difficulty(losses)
assert len(ranking) == 100, "Ranking length"
assert np.all(np.argsort(diff) == ranking), "Correct ranking"
print("✓ Sample ranking working")
if __name__ == "__main__":
print("Lab 1: SampleRanking - PASSED")### Lab 2: Curriculum Scheduling
import numpy as np
def curriculum_schedule(epoch, total_epochs, strategy='linear', hardness_start=0.2):
"""Compute curriculum threshold by epoch"""
progress = epoch / total_epochs
if strategy == 'linear':
hardness = hardness_start + (1 - hardness_start) * progress
elif strategy == 'exponential':
hardness = hardness_start * np.exp(np.log(1 / hardness_start) * progress)
elif strategy == 'sigmoid':
hardness = hardness_start + (1 - hardness_start) / (1 + np.exp(-10 * (progress - 0.5)))
else:
raise ValueError(f"Unknown strategy: {strategy}")
return hardness
# Test
for strategy in ['linear', 'exponential', 'sigmoid']:
hardness_vals = [curriculum_schedule(e, 100, strategy) for e in range(100)]
assert hardness_vals[0] >= hardness_vals[0], "Non-decreasing"
print("✓ Curriculum scheduling working")
if __name__ == "__main__":
print("Lab 2: CurriculumScheduling - PASSED")### Lab 3: Self-Paced Selection
import numpy as np
def self_paced_selection(losses, lambda_param):
"""Select samples below difficulty threshold"""
weights = (losses <= lambda_param).astype(float)
return weights
# Test
np.random.seed(42)
losses = np.random.rand(100)
lambda_val = 0.5
weights = self_paced_selection(losses, lambda_val)
assert np.sum(weights) > 0, "At least some samples selected"
assert np.all((weights == 0) | (weights == 1)), "Binary weights"
print("✓ Self-paced selection working")
if __name__ == "__main__":
print("Lab 3: SelfPacedSelection - PASSED")### Lab 4: Weighted Loss
import numpy as np
def curriculum_weighted_loss(logits, targets, weights):
"""Compute weighted loss with curriculum weights"""
# Cross-entropy
exp_logits = np.exp(logits - np.max(logits, axis=1, keepdims=True))
probs = exp_logits / exp_logits.sum(axis=1, keepdims=True)
batch_size = len(logits)
base_loss = -np.log(probs[np.arange(batch_size), targets] + 1e-8)
# Weighted loss
weighted_loss = (base_loss * weights).sum() / (weights.sum() + 1e-8)
return weighted_loss
# Test
np.random.seed(42)
logits = np.random.randn(32, 10)
targets = np.random.randint(0, 10, 32)
weights = np.random.rand(32)
loss = curriculum_weighted_loss(logits, targets, weights)
assert np.isfinite(loss), "Loss finite"
print("✓ Weighted loss working")
if __name__ == "__main__":
print("Lab 4: WeightedLoss - PASSED")