Validation in Machine Learning

Train-Validation-Test Split

Purpose

Why Separate Sets?

• Training: Model sees this data during learning
• Validation: Check generalization, tune settings
• Test: Unbiased final performance estimate

Creating Validation Sets

Random Split

from sklearn.model_selection import train_test_split

train, temp = train_test_split(data, test_size=0.2, random_state=42)
val, test = train_test_split(temp, test_size=0.5, random_state=42)
# 80% train, 10% val, 10% test

Stratified Split (for classification)

train, val = train_test_split(
    data, test_size=0.1,
    stratify=data["label"],  # Preserve class distribution
    random_state=42
)

Time-Based Split (for temporal data)

# Sort by date, use recent data for validation
data = data.sort_values("date")
train = data[:int(len(data)*0.8)]
val = data[int(len(data)*0.8):]

Validation During Training

Standard Loop

for epoch in range(num_epochs):
    # Training
    model.train()
    for batch in train_loader:
        loss = train_step(model, batch)

    # Validation
    model.eval()
    val_losses = []
    with torch.no_grad():
        for batch in val_loader:
            val_loss = model(batch)
            val_losses.append(val_loss)

    avg_val_loss = sum(val_losses) / len(val_losses)
    print(f"Epoch {epoch}: val_loss={avg_val_loss:.4f}")

Early Stopping

patience = 3
best_val_loss = float("inf")
patience_counter = 0

for epoch in range(num_epochs):
    val_loss = evaluate(model, val_loader)

    if val_loss < best_val_loss:
        best_val_loss = val_loss
        patience_counter = 0
        save_checkpoint(model, "best_model.pt")
    else:
        patience_counter += 1
        if patience_counter >= patience:
            print("Early stopping triggered")
            break

LLM Validation Considerations

For Fine-Tuning

• Use held-out examples from same distribution
• Evaluate on task-specific metrics (not just loss)
• Consider multiple evaluation tasks

For Pretraining

• Use separate validation text corpus
• Evaluate perplexity on diverse domains
• Check downstream task performance periodically