validation
**Validation in Machine Learning**
**Train-Validation-Test Split**
**Purpose**
| Set | Purpose | Typical Size |
|-----|---------|--------------|
| Training | Learn model parameters | 70-80% |
| Validation | Tune hyperparameters, early stopping | 10-15% |
| Test | Final evaluation (touch once!) | 10-15% |
**Why Separate Sets?**
- **Training**: Model sees this data during learning
- **Validation**: Check generalization, tune settings
- **Test**: Unbiased final performance estimate
**Creating Validation Sets**
**Random Split**
```python
from sklearn.model_selection import train_test_split
train, temp = train_test_split(data, test_size=0.2, random_state=42)
val, test = train_test_split(temp, test_size=0.5, random_state=42)
# 80% train, 10% val, 10% test
```
**Stratified Split (for classification)**
```python
train, val = train_test_split(
data, test_size=0.1,
stratify=data["label"], # Preserve class distribution
random_state=42
)
```
**Time-Based Split (for temporal data)**
```python
# Sort by date, use recent data for validation
data = data.sort_values("date")
train = data[:int(len(data)*0.8)]
val = data[int(len(data)*0.8):]
```
**Validation During Training**
**Standard Loop**
```python
for epoch in range(num_epochs):
# Training
model.train()
for batch in train_loader:
loss = train_step(model, batch)
# Validation
model.eval()
val_losses = []
with torch.no_grad():
for batch in val_loader:
val_loss = model(batch)
val_losses.append(val_loss)
avg_val_loss = sum(val_losses) / len(val_losses)
print(f"Epoch {epoch}: val_loss={avg_val_loss:.4f}")
```
**Early Stopping**
```python
patience = 3
best_val_loss = float("inf")
patience_counter = 0
for epoch in range(num_epochs):
val_loss = evaluate(model, val_loader)
if val_loss < best_val_loss:
best_val_loss = val_loss
patience_counter = 0
save_checkpoint(model, "best_model.pt")
else:
patience_counter += 1
if patience_counter >= patience:
print("Early stopping triggered")
break
```
**LLM Validation Considerations**
**For Fine-Tuning**
- Use held-out examples from same distribution
- Evaluate on task-specific metrics (not just loss)
- Consider multiple evaluation tasks
**For Pretraining**
- Use separate validation text corpus
- Evaluate perplexity on diverse domains
- Check downstream task performance periodically