Cross-Validation Model Evaluation Reliable Performance Estimation

# Cross-Validation & Model Evaluation: Reliable Performance Estimation

## Introduction & Motivation

Cross-validation (CV): partition data into folds; train/test on different folds. Reduces variance in performance estimates. K-fold: standard (k=5,10). Stratified: preserve class distribution. Time series: respect temporal order. Applications: model selection, hyperparameter tuning, generalization assessment.

Motivation: Single train-test split: high variance estimate. CV averages over multiple splits; stable estimate.

Applications: Model comparison, hyperparameter selection, final evaluation.

---

## Core Concepts & Theory

### K-Fold Cross-Validation

Partition into k folds; train on k-1, test on 1; repeat k times.

### Stratified K-Fold

Balance class distribution across folds; important for imbalanced data.

### Time Series Split

Respect temporal order; no information leakage from future.

---

## Mathematical Formulation

K-fold CV error:
$$ ext{CV}(\lambda) = \frac{1}{k} \sum_{i=1}^k ext{error}( ext{test}_i, f_\lambda( ext{train}_{-i}))$$

Variance reduction:
$$ ext{Var}( ext{CV}) \approx \frac{1}{k} ext{Var}( ext{test})$$

---

## Advanced Theory & Extensions

### Leave-One-Out Cross-Validation (LOOCV)

Extreme CV (k=n); expensive but unbiased.

### Nested Cross-Validation

Outer CV for evaluation; inner CV for hyperparameter selection.

### Bootstrap

Sampling with replacement; alternative to CV.

---

## Computational Considerations

Time: O(k × training_time).

Memory: Single fold in memory; efficient.

Parallelization: k folds independent; fully parallelizable.

---

## Practical Implementation Strategies

### Random State

Set seed; reproducible folds.

### Stratification

Always stratify for classification; maintain class balance.

### Shuffle

Shuffle before splitting; removes order bias.

---

## Benchmark Datasets & Evaluation

MNIST: Simple; small variance with 5-fold CV.

Imbalanced Data: Stratified essential; prevents skewed evaluation.

Time Series: Temporal splits for realistic assessment.

---

## Key Challenges & Limitations

### Data Leakage

Preprocessing before CV → biased estimates.

### Time Series Leakage

Information from future → invalid evaluation.

### Computational Cost

k× overhead; limits k for expensive models.

---

## Hyperparameter Tuning

k (folds): 5-10; balance variance-bias.

Shuffle: True (after sorting).

Stratify: True for classification.

---

## Real-World Applications & Case Studies

Model Selection: Compare models via nested CV.

Hyperparameter Tuning: Outer CV final eval; inner CV tuning.

Final Evaluation: Report CV scores + std dev.

---

## Integration with Other Methods

CV + Grid Search → nested CV for reliable tuning.

CV + Ensemble → average CV predictions.

---

## Summary & Key Takeaways

Cross-validation via k-fold partitioning provides stable performance estimates by averaging over multiple train-test splits, essential for reliable model evaluation.

Principles:
1. K-fold: k independent evaluations; low variance.
2. Stratified: maintain class distribution across folds.
3. Time series: respect temporal order.
4. Nested CV: separate hyperparameter selection from evaluation.
5. Nested CV: avoid overfitting to test set.

---

---

## Appendix: Practical Labs

### Lab 1: K-Fold Cross-Validation

import numpy as np
from sklearn.model_selection import KFold

def k_fold_cv(X, y, k=5):
 """Manual k-fold cross-validation"""
 kf = KFold(n_splits=k, shuffle=True, random_state=42)
 
 fold_scores = []
 for train_idx, test_idx in kf.split(X):
 X_train, X_test = X[train_idx], X[test_idx]
 y_train, y_test = y[train_idx], y[test_idx]
 
 # Simple: predict mean (baseline)
 y_pred = np.full_like(y_test, y_train.mean())
 mse = np.mean((y_test - y_pred) ** 2)
 fold_scores.append(mse)
 
 return np.array(fold_scores)

# Test
X = np.random.randn(100, 5)
y = np.random.randn(100)

scores = k_fold_cv(X, y, k=5)

print(f"CV scores: {scores}, Mean: {scores.mean():.4f}, Std: {scores.std():.4f}")
assert len(scores) == 5, "Should have 5 fold scores"
assert all(s > 0 for s in scores), "MSE should be positive"
print("✓ K-fold CV working")

if __name__ == "__main__":
 print("Lab 1: K-Fold - PASSED")

### Lab 2: Stratified K-Fold

import numpy as np
from sklearn.model_selection import StratifiedKFold

def stratified_cv(X, y, k=5):
 """Stratified k-fold for classification"""
 skf = StratifiedKFold(n_splits=k, shuffle=True, random_state=42)
 
 fold_ratios = []
 for train_idx, test_idx in skf.split(X, y):
 y_train = y[train_idx]
 y_test = y[test_idx]
 
 # Check class balance
 train_pos_ratio = (y_train == 1).sum() / len(y_train)
 test_pos_ratio = (y_test == 1).sum() / len(y_test)
 
 fold_ratios.append({'train': train_pos_ratio, 'test': test_pos_ratio})
 
 return fold_ratios

# Test
X = np.random.randn(100, 5)
y = np.concatenate([np.ones(30), np.zeros(70)])

ratios = stratified_cv(X, y, k=5)

print(f"Class balance in folds:")
for i, r in enumerate(ratios):
 print(f" Fold {i}: train={r['train']:.2f}, test={r['test']:.2f}")
assert all(abs(r['train'] - 0.3) < 0.05 for r in ratios), "Train ratio should be ~0.3"
print("✓ Stratified CV working")

if __name__ == "__main__":
 print("Lab 2: Stratified - PASSED")

### Lab 3: Nested Cross-Validation

import numpy as np
from sklearn.model_selection import cross_val_score, GridSearchCV, KFold

def nested_cv(X, y, param_grid):
 """Nested CV: inner for tuning, outer for evaluation"""
 
 from sklearn.linear_model import Ridge
 
 outer_cv = KFold(n_splits=3, shuffle=True, random_state=42)
 scores = []
 
 for train_idx, test_idx in outer_cv.split(X):
 X_train, X_test = X[train_idx], X[test_idx]
 y_train, y_test = y[train_idx], y[test_idx]
 
 # Inner CV for hyperparameter tuning
 model = Ridge()
 grid_search = GridSearchCV(model, param_grid, cv=3, scoring='neg_mean_squared_error')
 grid_search.fit(X_train, y_train)
 
 # Evaluate on test fold
 test_score = grid_search.score(X_test, y_test)
 scores.append(test_score)
 
 return np.array(scores)

# Test
X = np.random.randn(100, 5)
y = np.random.randn(100)
param_grid = {'alpha': [0.1, 1.0, 10.0]}

scores = nested_cv(X, y, param_grid)

print(f"Nested CV scores: {scores}, Mean: {scores.mean():.4f}")
assert len(scores) == 3, "Should have 3 outer folds"
print("✓ Nested CV working")

if __name__ == "__main__":
 print("Lab 3: Nested - PASSED")

### Lab 4: Time Series Cross-Validation

import numpy as np

def time_series_cv(X, y, n_splits=3):
 """Time series CV: respect temporal order"""
 
 n_samples = len(X)
 train_size = n_samples // (n_splits + 1)
 
 fold_metrics = []
 for i in range(n_splits):
 train_end = (i + 1) * train_size
 test_end = train_end + train_size
 
 X_train, y_train = X[:train_end], y[:train_end]
 X_test, y_test = X[train_end:test_end], y[train_end:test_end]
 
 # Simple: predict last value
 y_pred = np.full_like(y_test, y_train[-1])
 mse = np.mean((y_test - y_pred) ** 2)
 
 fold_metrics.append(mse)
 
 return np.array(fold_metrics)

# Test
X = np.arange(100).reshape(-1, 1)
y = np.sin(np.arange(100) / 10)

scores = time_series_cv(X, y, n_splits=3)

print(f"Time series CV scores: {scores}")
assert len(scores) == 3, "Should have 3 folds"
assert all(s > 0 for s in scores), "MSE should be positive"
print("✓ Time series CV working")

if __name__ == "__main__":
 print("Lab 4: Time Series - PASSED")