Hyperparameter Optimization Bayesian Random Grid Search
# Hyperparameter Optimization: Bayesian, Random & Grid Search
## Introduction & Motivation
Hyperparameter optimization: find best model configuration. Grid search: exhaustive parameter combinations. Random search: random sampling; better scaling. Bayesian optimization: probabilistic surrogate; adaptive. Applications: model tuning, AutoML, architecture search.
Motivation: Hyperparameters impact performance. Manual tuning expensive; automated search needed.
Applications: Model selection, AutoML, architecture optimization.
---
## Core Concepts & Theory
### Grid Search
Exhaustive; guaranteed coverage; combinatorial explosion.
### Random Search
Random sampling; better high-dimensional scaling.
### Bayesian Optimization
Gaussian process surrogate; acquisition function.
---
## Mathematical Formulation
Grid search: Evaluate all combinations in grid.
Random search:
$$ heta_i \sim ext{Uniform}( ext{domain})$$
Bayesian optimization:
$$ ext{max}_{x} ext{EI}(x) = \mathbb{E}[\max(f(x) - f(x^*), 0)]$$
where f = surrogate model, EI = expected improvement.
---
## Advanced Theory & Extensions
### Bandit-Based Methods
Multi-armed bandit framework; Successive Halving.
### Hyperband
Combination of random search and early stopping.
### Population-Based Training
Evolving hyperparameters during training.
---
## Computational Considerations
Grid search: O(G^D) where G = grid size, D = dimensions.
Random search: O(R) where R = number samples; more efficient.
Bayesian: O(R · M) where M = surrogate training cost.
---
## Practical Implementation Strategies
### Search Space Definition
Bounded ranges; log scale for some parameters.
### Cross-Validation
Nested CV for hyperparameter tuning.
### Early Stopping
Stop unpromising trials; save computation.
---
## Benchmark Datasets & Evaluation
CIFAR-10: Hyperparameter tuning significant gains.
ImageNet: Extensive tuning for state-of-the-art.
AutoML Benchmarks: HPO standard evaluation.
---
## Key Challenges & Limitations
### Computational Cost
Grid/Bayesian expensive; many evaluations.
### Search Space Curse
Dimensionality explosion; exponential combinations.
### Transferability
Optimal hyperparameters task-specific.
---
## Hyperparameter Tuning Meta
Learning rate: log scale [1e-5, 1e-1].
Batch size: {32, 64, 128, 256}.
Regularization: log scale [1e-5, 1e-1].
---
## Real-World Applications & Case Studies
ImageNet Training: Extensive learning rate schedules.
BERT: HPO for downstream tasks.
AutoML: Automatic configuration search.
---
## Integration with Other Methods
HPO + Cross-Validation → robust tuning.
HPO + Ensemble → diverse models.
---
## Summary & Key Takeaways
Hyperparameter optimization via grid, random, and Bayesian methods systematically search configuration spaces for optimal performance.
Principles:
1. Grid: exhaustive; exponential cost.
2. Random: efficient scaling; better high-D.
3. Bayesian: surrogate-guided; sample efficient.
4. Bandit: multi-armed framework.
5. Early stopping: computational efficiency.
---
---
## Appendix: Practical Labs
### Lab 1: Grid Search
import numpy as np
import itertools
def grid_search(param_grid, evaluate_fn):
"""Grid search over parameter combinations"""
# Generate all combinations
keys = param_grid.keys()
values = param_grid.values()
combinations = list(itertools.product(*values))
best_params = None
best_score = -np.inf
for combo in combinations:
params = dict(zip(keys, combo))
score = evaluate_fn(params)
if score > best_score:
best_score = score
best_params = params
return best_params, best_score
# Test
np.random.seed(42)
param_grid = {
"learning_rate": [0.001, 0.01, 0.1],
"batch_size": [32, 64, 128]
}
def dummy_evaluate(params):
# Simulate evaluation (higher is better)
return -np.log(params["learning_rate"] + 0.01) * (params["batch_size"] / 100)
best_params, best_score = grid_search(param_grid, dummy_evaluate)
assert best_params is not None, "Best params found"
assert np.isfinite(best_score), "Best score finite"
print("✓ Grid search working")
if __name__ == "__main__":
print("Lab 1: GridSearch - PASSED")### Lab 2: Random Search
import numpy as np
def random_search(param_ranges, evaluate_fn, num_trials=10):
"""Random search"""
best_params = None
best_score = -np.inf
for _ in range(num_trials):
# Sample random parameters
params = {}
for param, (low, high) in param_ranges.items():
params[param] = np.random.uniform(low, high)
score = evaluate_fn(params)
if score > best_score:
best_score = score
best_params = params
return best_params, best_score
# Test
np.random.seed(42)
param_ranges = {
"learning_rate": (0.001, 0.1),
"dropout": (0.0, 0.5)
}
def dummy_evaluate(params):
return -params["learning_rate"] ** 2 - params["dropout"]
best_params, best_score = random_search(param_ranges, dummy_evaluate, num_trials=20)
assert best_params is not None, "Best params found"
print("✓ Random search working")
if __name__ == "__main__":
print("Lab 2: RandomSearch - PASSED")### Lab 3: Bayesian Optimization (Gaussian Process)
import numpy as np
def bayesian_optimization(param_range, evaluate_fn, num_iterations=10):
"""Simplified Bayesian optimization"""
history_x = []
history_y = []
# Random initial sample
for _ in range(3):
x = np.random.uniform(param_range[0], param_range[1])
y = evaluate_fn(x)
history_x.append(x)
history_y.append(y)
# Bayesian iterations
for _ in range(num_iterations - 3):
# Simple acquisition: max mean + uncertainty
x_test = np.linspace(param_range[0], param_range[1], 100)
# Compute expected improvement (simplified)
best_y = np.max(history_y)
# Simple heuristic: prefer unexplored regions
# (real BO uses GP variance)
acq_values = []
for xt in x_test:
dist_to_observed = np.min(np.abs(np.array(history_x) - xt))
acq = dist_to_observed + 0.1 * (best_y + np.random.randn())
acq_values.append(acq)
# Select next point
best_idx = np.argmax(acq_values)
x_next = x_test[best_idx]
y_next = evaluate_fn(x_next)
history_x.append(x_next)
history_y.append(y_next)
best_idx = np.argmax(history_y)
return history_x[best_idx], history_y[best_idx]
# Test
np.random.seed(42)
def f(x):
return -(x - 0.5) ** 2 + 1
x_opt, y_opt = bayesian_optimization((0, 1), f, num_iterations=20)
assert 0 <= x_opt <= 1, "Solution in range"
print("✓ Bayesian optimization working")
if __name__ == "__main__":
print("Lab 3: BayesianOptimization - PASSED")### Lab 4: Hyperparameter Importance
import numpy as np
def compute_param_importance(history_params, history_scores, param_names):
"""Simple parameter importance via variance"""
importance = {}
for i, param_name in enumerate(param_names):
param_values = np.array([p[i] for p in history_params])
# Correlation between param and score
correlation = np.corrcoef(param_values, history_scores)[0, 1]
# Importance: absolute correlation
importance[param_name] = abs(correlation) if not np.isnan(correlation) else 0
return importance
# Test
np.random.seed(42)
history_params = [
[0.001, 32, 0.2],
[0.01, 64, 0.3],
[0.1, 128, 0.1],
[0.005, 32, 0.25]
]
history_scores = [0.85, 0.90, 0.88, 0.87]
param_names = ["lr", "batch_size", "dropout"]
importance = compute_param_importance(history_params, history_scores, param_names)
assert len(importance) == 3, "Importance for all params"
assert all(0 <= v <= 1 for v in importance.values()), "Importance in [0,1]"
print("✓ Parameter importance working")
if __name__ == "__main__":
print("Lab 4: ParamImportance - PASSED")