curriculum learning
**Curriculum Learning**
**What is Curriculum Learning?**
Training models on examples ordered by difficulty, starting with easy examples and progressing to harder ones, mimicking human learning.
**Curriculum Types**
**Predefined Curriculum**
Order by known difficulty:
```python
def difficulty_score(example):
return len(example["text"]) # Simple: shorter is easier
# Sort by difficulty
curriculum = sorted(data, key=difficulty_score)
# Train in batches of increasing difficulty
for epoch in range(epochs):
current_data = curriculum[:epoch_fraction * len(curriculum)]
train(model, current_data)
```
**Self-Paced Learning**
Model determines what is easy:
```python
def self_paced_weights(losses, threshold):
# Easy examples have low loss
weights = (losses < threshold).float()
return weights
# Increase threshold over training
for epoch in range(epochs):
threshold = initial + epoch * increment
losses = model.get_losses(data)
weights = self_paced_weights(losses, threshold)
train(model, data, weights)
```
**Difficulty Metrics**
| Metric | Description |
|--------|-------------|
| Length | Shorter sequences are easier |
| Vocabulary | Common words are easier |
| Syntax complexity | Simple grammar is easier |
| Model loss | Low loss = easy for current model |
| Human annotation | Expert-labeled difficulty |
**Curriculum Strategies**
| Strategy | Description |
|----------|-------------|
| Baby Steps | Very gradual difficulty increase |
| One-pass | Single sweep from easy to hard |
| Interleaved | Mix difficulties, weighted toward easy |
| Anti-curriculum | Hard first (sometimes works) |
**Benefits**
- Faster convergence
- Better generalization
- More stable training
- Can help with difficult examples
**Implementation Example**
```python
class CurriculumDataLoader:
def __init__(self, data, difficulty_fn, pacing_fn):
self.data = sorted(data, key=difficulty_fn)
self.pacing_fn = pacing_fn
def get_epoch_data(self, epoch):
fraction = self.pacing_fn(epoch)
cutoff = int(fraction * len(self.data))
return self.data[:cutoff]
```
**Use Cases**
- Training LLMs (simple to complex examples)
- Computer vision (clear to ambiguous images)
- Reinforcement learning (easy to hard tasks)
- Low-resource scenarios (maximize data efficiency)