Curriculum Learning

What is Curriculum Learning? Training models on examples ordered by difficulty, starting with easy examples and progressing to harder ones, mimicking human learning.

Curriculum Types

Predefined Curriculum Order by known difficulty:

def difficulty_score(example):
    return len(example["text"])  # Simple: shorter is easier

# Sort by difficulty
curriculum = sorted(data, key=difficulty_score)

# Train in batches of increasing difficulty
for epoch in range(epochs):
    current_data = curriculum[:epoch_fraction * len(curriculum)]
    train(model, current_data)

Self-Paced Learning Model determines what is easy:

def self_paced_weights(losses, threshold):
    # Easy examples have low loss
    weights = (losses < threshold).float()
    return weights

# Increase threshold over training
for epoch in range(epochs):
    threshold = initial + epoch * increment
    losses = model.get_losses(data)
    weights = self_paced_weights(losses, threshold)
    train(model, data, weights)

Difficulty Metrics

Curriculum Strategies

Benefits

• Faster convergence
• Better generalization
• More stable training
• Can help with difficult examples

Implementation Example

class CurriculumDataLoader:
    def __init__(self, data, difficulty_fn, pacing_fn):
        self.data = sorted(data, key=difficulty_fn)
        self.pacing_fn = pacing_fn

    def get_epoch_data(self, epoch):
        fraction = self.pacing_fn(epoch)
        cutoff = int(fraction * len(self.data))
        return self.data[:cutoff]

Use Cases

• Training LLMs (simple to complex examples)
• Computer vision (clear to ambiguous images)
• Reinforcement learning (easy to hard tasks)
• Low-resource scenarios (maximize data efficiency)