cross-validation k-fold stratified temporal
# Cross-Validation: K-Fold, Stratified & Temporal
## Introduction & Motivation
Cross-validation: estimate generalization performance. K-Fold: partition into K folds; average metrics. Stratified: preserve class distribution. Temporal: respect time ordering. Applications: model evaluation, hyperparameter tuning, performance estimation.
Motivation: Single train-test split unreliable; high variance. Cross-validation provides robust estimates.
Applications: Model evaluation, hyperparameter optimization.
---
## Core Concepts & Theory
### K-Fold CV
Partition into K disjoint folds; train K models.
### Stratified K-Fold
Preserve class proportions in each fold.
### Time Series CV
Respect temporal ordering; forward chaining.
---
## Mathematical Formulation
K-Fold CV estimate:
$$ ext{Score} = \frac{1}{K} \sum_{k=1}^K ext{Score}_k$$
where Score_k evaluated on test fold k.
Stratified splits: |class_train| / |all_train| ≈ |class_test| / |all_test| per fold.
---
## Advanced Theory & Extensions
### Leave-One-Out CV
K = N; maximum variance reduction; expensive.
### Repeated K-Fold
Multiple random K-fold splits; ensemble of estimates.
### Nested CV
Outer for evaluation; inner for tuning.
---
## Computational Considerations
K-Fold: O(K · train_time); K=5-10 typical.
Stratified: O(sort) preprocessing cost.
Time series: O(N) forward chaining; sequential.
---
## Practical Implementation Strategies
### K Selection
K=5, 10 standard; tradeoff variance-computation.
### Stratification
Always use for imbalanced classification.
### Reproducibility
Set random seed; deterministic splits.
---
## Benchmark Datasets & Evaluation
CIFAR-10: 5-Fold or 10-Fold standard.
Imbalanced Data: Stratified K-Fold essential.
Time Series: Forward chaining mandatory.
---
## Key Challenges & Limitations
### Computational Cost
K models trained; expensive for large datasets.
### Dependence
Folds not independent; biased variance estimates.
### Temporal Data
Standard CV violates temporal ordering.
---
## Hyperparameter Tuning
K: 5-10 typical; balance variance-bias.
Shuffle: False for time series; True otherwise.
Random state: Set for reproducibility.
---
## Real-World Applications & Case Studies
Kaggle: 5-Fold CV standard; validation strategy.
Medical: Stratified CV for balanced evaluation.
Finance: Walk-forward CV for time series.
---
## Integration with Other Methods
CV + Hyperparameter Optimization → nested CV.
CV + Ensemble → cross-validated ensemble.
---
## Summary & Key Takeaways
Cross-validation via k-fold, stratified, and temporal variants provides robust performance estimation with appropriate data structure respect.
Principles:
1. K-Fold: partition-based averaging.
2. Stratified: class balance preservation.
3. Temporal: time-ordering respect.
4. Nested: tuning + evaluation separation.
5. Reproducibility: set random seeds.
---
---
## Appendix: Practical Labs
### Lab 1: K-Fold Cross-Validation
import numpy as np
class KFold:
def __init__(self, n_splits=5, shuffle=False, random_state=None):
self.n_splits = n_splits
self.shuffle = shuffle
self.random_state = random_state
def split(self, X, y=None):
"""Generate train-test indices"""
n_samples = len(X)
indices = np.arange(n_samples)
if self.shuffle:
np.random.RandomState(self.random_state).shuffle(indices)
fold_size = n_samples // self.n_splits
for fold in range(self.n_splits):
start = fold * fold_size
end = start + fold_size if fold < self.n_splits - 1 else n_samples
test_indices = indices[start:end]
train_indices = np.concatenate([indices[:start], indices[end:]])
yield train_indices, test_indices
# Test
np.random.seed(42)
kf = KFold(n_splits=5)
X = np.random.randn(100, 10)
y = np.random.randint(0, 2, 100)
fold_count = 0
for train_idx, test_idx in kf.split(X, y):
assert len(train_idx) + len(test_idx) == len(X), "Complete coverage"
assert len(set(train_idx) & set(test_idx)) == 0, "No overlap"
fold_count += 1
assert fold_count == 5, "5 folds"
print("✓ K-Fold CV working")
if __name__ == "__main__":
print("Lab 1: KFold - PASSED")### Lab 2: Stratified K-Fold
import numpy as np
class StratifiedKFold:
def __init__(self, n_splits=5):
self.n_splits = n_splits
def split(self, X, y):
"""Generate stratified folds"""
n_samples = len(X)
# Get class indices
classes = np.unique(y)
class_indices = {c: np.where(y == c)[0] for c in classes}
train_indices = []
test_indices = []
for fold in range(self.n_splits):
fold_train = []
fold_test = []
# Stratify by class
for c, indices in class_indices.items():
fold_size = len(indices) // self.n_splits
start = fold * fold_size
end = start + fold_size if fold < self.n_splits - 1 else len(indices)
fold_test.extend(indices[start:end])
fold_train.extend(np.concatenate([indices[:start], indices[end:]]))
yield np.array(fold_train), np.array(fold_test)
# Test
np.random.seed(42)
skf = StratifiedKFold(n_splits=5)
X = np.random.randn(100, 10)
y = np.array([0] * 30 + [1] * 70)
for train_idx, test_idx in skf.split(X, y):
# Check stratification
train_ratio = (y[train_idx] == 0).sum() / len(y[train_idx])
test_ratio = (y[test_idx] == 0).sum() / len(y[test_idx])
assert 0.2 < train_ratio < 0.4, "Stratified training"
assert 0.2 < test_ratio < 0.4, "Stratified test"
print("✓ Stratified K-Fold working")
if __name__ == "__main__":
print("Lab 2: StratifiedKFold - PASSED")### Lab 3: Time Series Cross-Validation
import numpy as np
class TimeSeriesSplit:
def __init__(self, n_splits=5):
self.n_splits = n_splits
def split(self, X):
"""Generate time series folds (forward chaining)"""
n_samples = len(X)
fold_size = n_samples // (self.n_splits + 1)
for fold in range(self.n_splits):
train_end = (fold + 1) * fold_size
test_end = train_end + fold_size
train_indices = np.arange(train_end)
test_indices = np.arange(train_end, test_end)
yield train_indices, test_indices
# Test
np.random.seed(42)
ts_split = TimeSeriesSplit(n_splits=5)
X = np.random.randn(100, 10)
for train_idx, test_idx in ts_split.split(X):
# Check temporal ordering
assert train_idx[-1] < test_idx[0], "Train before test"
assert len(set(train_idx) & set(test_idx)) == 0, "No overlap"
print("✓ Time Series CV working")
if __name__ == "__main__":
print("Lab 3: TimeSeriesSplit - PASSED")### Lab 4: Nested Cross-Validation
import numpy as np
def nested_cross_validation(X, y, inner_cv=5, outer_cv=5):
"""Nested CV: inner for HPO, outer for evaluation"""
n_samples = len(X)
# Outer fold loop
outer_scores = []
for out_fold in range(outer_cv):
# Create outer train-test split
out_test_size = n_samples // outer_cv
out_test_start = out_fold * out_test_size
out_test_end = out_test_start + out_test_size if out_fold < outer_cv - 1 else n_samples
out_test_idx = np.arange(out_test_start, out_test_end)
out_train_idx = np.concatenate([np.arange(out_test_start), np.arange(out_test_end, n_samples)])
# Inner loop: hyperparameter tuning
inner_scores = []
for in_fold in range(inner_cv):
in_test_size = len(out_train_idx) // inner_cv
in_test_start = in_fold * in_test_size
in_test_end = in_test_start + in_test_size if in_fold < inner_cv - 1 else len(out_train_idx)
in_test_idx = out_train_idx[in_test_start:in_test_end]
in_train_idx = np.concatenate([out_train_idx[:in_test_start], out_train_idx[in_test_end:]])
# Train and evaluate (simplified)
score = np.random.random()
inner_scores.append(score)
# Use best inner hyperparameters for outer evaluation
outer_score = np.mean(inner_scores)
outer_scores.append(outer_score)
return np.mean(outer_scores), np.std(outer_scores)
# Test
np.random.seed(42)
X = np.random.randn(100, 10)
y = np.random.randint(0, 2, 100)
mean_score, std_score = nested_cross_validation(X, y, inner_cv=3, outer_cv=3)
assert 0 <= mean_score <= 1, "Score in [0,1]"
assert std_score >= 0, "Std non-negative"
print("✓ Nested CV working")
if __name__ == "__main__":
print("Lab 4: NestedCV - PASSED")