Hyperparameter Optimization
# Hyperparameter Optimization
## Introduction & Motivation
Hyperparameter Optimization: automatically tune model hyperparameters. Bayesian optimization, grid search, random search. Applications: AutoML, model tuning, efficient search.
Motivation: Find optimal hyperparameters automatically; reduce manual tuning.
Applications: AutoML platforms, model selection.
---
## Core Concepts & Theory
### Grid Search
Exhaustive search over parameter grid.
### Random Search
Random sampling from parameter space.
### Bayesian Optimization
Probabilistic model-based search.
### Hyperband
Successive halving for efficient search.
---
## Mathematical Formulation
Objective Function:
$$\min_h ext{val\_loss}(h) + \lambda \|h\|_2^2$$
Bayesian Optimization (Expected Improvement):
$$EI(h) = \mathbb{E}[\max(f(h) - f(h^*), 0)]$$
Hyperband Allocation:
$$n_i = n_0 \cdot r^{-i}, \quad s_i = s_0 \cdot i$$
---
## Advanced Theory & Extensions
### Population-Based Training (PBT)
Evolutionary hyperparameter search.
### Successive Halving
Early stopping for efficiency.
### Multi-Fidelity Optimization
Use low-cost approximations.
---
## Computational Considerations
Grid search: O(grid_size·train_time).
Random search: O(budget·train_time).
Bayesian optimization: O(iterations·(GP_fit + acquisition))\.
---
## Practical Implementation Strategies
### Early Stopping
Stop poor performers early.
### Logarithmic Sampling
Search in log scale.
### Warm-Start
Initialize from good configurations.
---
## Benchmark Datasets & Evaluation
CIFAR-10: Quick benchmarking.
ImageNet: Large-scale evaluation.
NAS-Bench: Pre-computed search space.
---
## Key Challenges & Limitations
### High Dimensionality
Large hyperparameter space.
### Noisy Evaluations
Stochastic performance.
### Long Training Time
Expensive function evaluations.
---
## Hyperparameter Tuning
Search iterations: 10-1000.
Parallel workers: 1-100.
Early stopping patience: 5-50 epochs.
---
## Real-World Applications & Case Studies
AutoML: Automated model selection.
Kaggle: Competition parameter tuning.
Industry Models: Production model optimization.
---
## Integration with Other Methods
Hyperparameter optimization + neural architecture search for full automation; + early stopping for efficiency.
---
## Summary & Key Takeaways
Hyperparameter Optimization via Bayesian optimization and successive halving enables efficient parameter search.
Principles:
1. Grid search: Exhaustive baseline.
2. Random search: Better than grid in high dimensions.
3. Bayesian optimization: Probabilistic guidance.
4. Hyperband: Efficient budget allocation.
5. Early stopping: Cost reduction.
---
---
## Appendix: Practical Labs
### Lab 1: Grid Search
import numpy as np
def grid_search(param_grid, train_function):
"""Exhaustive grid search"""
best_score = -np.inf
best_params = None
# Generate all combinations
param_names = list(param_grid.keys())
param_values = [param_grid[name] for name in param_names]
# Iterate through all combinations
from itertools import product
for values in product(*param_values):
params = dict(zip(param_names, values))
score = train_function(params)
if score > best_score:
best_score = score
best_params = params
return best_params, best_score
# Test
def mock_train(params):
return np.sum(np.array(list(params.values())))
grid = {'lr': [0.01, 0.1], 'batch_size': [32, 64]}
best, score = grid_search(grid, mock_train)
assert best is not None, "Best params found"
print("✓ Grid search working")
if __name__ == "__main__":
print("Lab 1: GridSearch - PASSED")### Lab 2: Random Search
import numpy as np
def random_search(param_distributions, train_function, n_iter=10):
"""Random hyperparameter search"""
best_score = -np.inf
best_params = None
for _ in range(n_iter):
# Sample random parameters
params = {}
for param_name, param_dist in param_distributions.items():
if isinstance(param_dist, list):
params[param_name] = np.random.choice(param_dist)
else:
# Assume distribution object with sample method
params[param_name] = param_dist
score = train_function(params)
if score > best_score:
best_score = score
best_params = params
return best_params, best_score
# Test
def mock_train(params):
return np.random.rand()
dists = {'lr': [0.01, 0.1, 1.0], 'batch_size': [32, 64, 128]}
best, score = random_search(dists, mock_train, n_iter=5)
assert best is not None, "Best params found"
print("✓ Random search working")
if __name__ == "__main__":
print("Lab 2: RandomSearch - PASSED")### Lab 3: Early Stopping
import numpy as np
def should_stop_early(val_losses, patience=3, min_delta=0.001):
"""Check if should stop training early"""
if len(val_losses) < patience + 1:
return False
# Check if best loss hasn't improved
best_loss = np.min(val_losses[:-patience])
current_loss = val_losses[-1]
if current_loss > best_loss - min_delta:
# Check if all recent losses are worse
recent = val_losses[-patience:]
if all(loss >= best_loss - min_delta for loss in recent):
return True
return False
# Test
losses = [0.5, 0.4, 0.35, 0.36, 0.37, 0.38]
should_stop = should_stop_early(losses, patience=3)
assert isinstance(should_stop, bool), "Boolean result"
print("✓ Early stopping working")
if __name__ == "__main__":
print("Lab 3: EarlyStopping - PASSED")### Lab 4: Hyperparameter Range
import numpy as np
def log_sample_range(min_val, max_val, num_samples=10):
"""Sample hyperparameters in log scale"""
log_min = np.log10(min_val)
log_max = np.log10(max_val)
log_samples = np.linspace(log_min, log_max, num_samples)
samples = 10 ** log_samples
return samples
# Test
samples = log_sample_range(0.00001, 1.0, num_samples=5)
assert len(samples) == 5, "Correct number of samples"
assert samples[0] < samples[-1], "Monotonic increase"
print("✓ Log-scale sampling working")
if __name__ == "__main__":
print("Lab 4: LogScaleSampling - PASSED")