Cross-Validation Model Evaluation Reliable Performance Estimation
# Cross-Validation & Model Evaluation: Reliable Performance Estimation
## Introduction & Motivation
Cross-validation (CV): partition data into folds; train/test on different folds. Reduces variance in performance estimates. K-fold: standard (k=5,10). Stratified: preserve class distribution. Time series: respect temporal order. Applications: model selection, hyperparameter tuning, generalization assessment.
Motivation: Single train-test split: high variance estimate. CV averages over multiple splits; stable estimate.
Applications: Model comparison, hyperparameter selection, final evaluation.
---
## Core Concepts & Theory
### K-Fold Cross-Validation
Partition into k folds; train on k-1, test on 1; repeat k times.
### Stratified K-Fold
Balance class distribution across folds; important for imbalanced data.
### Time Series Split
Respect temporal order; no information leakage from future.
---
## Mathematical Formulation
K-fold CV error:
$$ ext{CV}(\lambda) = \frac{1}{k} \sum_{i=1}^k ext{error}( ext{test}_i, f_\lambda( ext{train}_{-i}))$$
Variance reduction:
$$ ext{Var}( ext{CV}) \approx \frac{1}{k} ext{Var}( ext{test})$$
---
## Advanced Theory & Extensions
### Leave-One-Out Cross-Validation (LOOCV)
Extreme CV (k=n); expensive but unbiased.
### Nested Cross-Validation
Outer CV for evaluation; inner CV for hyperparameter selection.
### Bootstrap
Sampling with replacement; alternative to CV.
---
## Computational Considerations
Time: O(k × training_time).
Memory: Single fold in memory; efficient.
Parallelization: k folds independent; fully parallelizable.
---
## Practical Implementation Strategies
### Random State
Set seed; reproducible folds.
### Stratification
Always stratify for classification; maintain class balance.
### Shuffle
Shuffle before splitting; removes order bias.
---
## Benchmark Datasets & Evaluation
MNIST: Simple; small variance with 5-fold CV.
Imbalanced Data: Stratified essential; prevents skewed evaluation.
Time Series: Temporal splits for realistic assessment.
---
## Key Challenges & Limitations
### Data Leakage
Preprocessing before CV → biased estimates.
### Time Series Leakage
Information from future → invalid evaluation.
### Computational Cost
k× overhead; limits k for expensive models.
---
## Hyperparameter Tuning
k (folds): 5-10; balance variance-bias.
Shuffle: True (after sorting).
Stratify: True for classification.
---
## Real-World Applications & Case Studies
Model Selection: Compare models via nested CV.
Hyperparameter Tuning: Outer CV final eval; inner CV tuning.
Final Evaluation: Report CV scores + std dev.
---
## Integration with Other Methods
CV + Grid Search → nested CV for reliable tuning.
CV + Ensemble → average CV predictions.
---
## Summary & Key Takeaways
Cross-validation via k-fold partitioning provides stable performance estimates by averaging over multiple train-test splits, essential for reliable model evaluation.
Principles:
1. K-fold: k independent evaluations; low variance.
2. Stratified: maintain class distribution across folds.
3. Time series: respect temporal order.
4. Nested CV: separate hyperparameter selection from evaluation.
5. Nested CV: avoid overfitting to test set.
---
---
## Appendix: Practical Labs
### Lab 1: K-Fold Cross-Validation
import numpy as np
from sklearn.model_selection import KFold
def k_fold_cv(X, y, k=5):
"""Manual k-fold cross-validation"""
kf = KFold(n_splits=k, shuffle=True, random_state=42)
fold_scores = []
for train_idx, test_idx in kf.split(X):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
# Simple: predict mean (baseline)
y_pred = np.full_like(y_test, y_train.mean())
mse = np.mean((y_test - y_pred) ** 2)
fold_scores.append(mse)
return np.array(fold_scores)
# Test
X = np.random.randn(100, 5)
y = np.random.randn(100)
scores = k_fold_cv(X, y, k=5)
print(f"CV scores: {scores}, Mean: {scores.mean():.4f}, Std: {scores.std():.4f}")
assert len(scores) == 5, "Should have 5 fold scores"
assert all(s > 0 for s in scores), "MSE should be positive"
print("✓ K-fold CV working")
if __name__ == "__main__":
print("Lab 1: K-Fold - PASSED")### Lab 2: Stratified K-Fold
import numpy as np
from sklearn.model_selection import StratifiedKFold
def stratified_cv(X, y, k=5):
"""Stratified k-fold for classification"""
skf = StratifiedKFold(n_splits=k, shuffle=True, random_state=42)
fold_ratios = []
for train_idx, test_idx in skf.split(X, y):
y_train = y[train_idx]
y_test = y[test_idx]
# Check class balance
train_pos_ratio = (y_train == 1).sum() / len(y_train)
test_pos_ratio = (y_test == 1).sum() / len(y_test)
fold_ratios.append({'train': train_pos_ratio, 'test': test_pos_ratio})
return fold_ratios
# Test
X = np.random.randn(100, 5)
y = np.concatenate([np.ones(30), np.zeros(70)])
ratios = stratified_cv(X, y, k=5)
print(f"Class balance in folds:")
for i, r in enumerate(ratios):
print(f" Fold {i}: train={r['train']:.2f}, test={r['test']:.2f}")
assert all(abs(r['train'] - 0.3) < 0.05 for r in ratios), "Train ratio should be ~0.3"
print("✓ Stratified CV working")
if __name__ == "__main__":
print("Lab 2: Stratified - PASSED")### Lab 3: Nested Cross-Validation
import numpy as np
from sklearn.model_selection import cross_val_score, GridSearchCV, KFold
def nested_cv(X, y, param_grid):
"""Nested CV: inner for tuning, outer for evaluation"""
from sklearn.linear_model import Ridge
outer_cv = KFold(n_splits=3, shuffle=True, random_state=42)
scores = []
for train_idx, test_idx in outer_cv.split(X):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
# Inner CV for hyperparameter tuning
model = Ridge()
grid_search = GridSearchCV(model, param_grid, cv=3, scoring='neg_mean_squared_error')
grid_search.fit(X_train, y_train)
# Evaluate on test fold
test_score = grid_search.score(X_test, y_test)
scores.append(test_score)
return np.array(scores)
# Test
X = np.random.randn(100, 5)
y = np.random.randn(100)
param_grid = {'alpha': [0.1, 1.0, 10.0]}
scores = nested_cv(X, y, param_grid)
print(f"Nested CV scores: {scores}, Mean: {scores.mean():.4f}")
assert len(scores) == 3, "Should have 3 outer folds"
print("✓ Nested CV working")
if __name__ == "__main__":
print("Lab 3: Nested - PASSED")### Lab 4: Time Series Cross-Validation
import numpy as np
def time_series_cv(X, y, n_splits=3):
"""Time series CV: respect temporal order"""
n_samples = len(X)
train_size = n_samples // (n_splits + 1)
fold_metrics = []
for i in range(n_splits):
train_end = (i + 1) * train_size
test_end = train_end + train_size
X_train, y_train = X[:train_end], y[:train_end]
X_test, y_test = X[train_end:test_end], y[train_end:test_end]
# Simple: predict last value
y_pred = np.full_like(y_test, y_train[-1])
mse = np.mean((y_test - y_pred) ** 2)
fold_metrics.append(mse)
return np.array(fold_metrics)
# Test
X = np.arange(100).reshape(-1, 1)
y = np.sin(np.arange(100) / 10)
scores = time_series_cv(X, y, n_splits=3)
print(f"Time series CV scores: {scores}")
assert len(scores) == 3, "Should have 3 folds"
assert all(s > 0 for s in scores), "MSE should be positive"
print("✓ Time series CV working")
if __name__ == "__main__":
print("Lab 4: Time Series - PASSED")