Automl Hyperparameter Optimization Bayesian Search
# AutoML: Hyperparameter Optimization & Bayesian Search
## Introduction & Motivation
Automated Machine Learning (AutoML) encompasses the set of techniques that automate the labor-intensive, expertise-dependent parts of building a machine learning pipeline: selecting model architectures, tuning hyperparameters, engineering features, and choosing preprocessing steps. Among these, hyperparameter optimization (HPO) is the most mature and widely deployed AutoML subfield, addressing a deceptively difficult problem: model performance can vary dramatically depending on choices like learning rate, regularization strength, batch size, or tree depth, yet the relationship between hyperparameters and validation performance is typically a noisy, expensive-to-evaluate, non-convex black-box function with no closed-form gradient. Manual hyperparameter tuning by domain experts, while still common, does not scale well to the increasingly high-dimensional hyperparameter spaces of modern deep learning pipelines and is difficult to reproduce systematically. The motivation for principled HPO methods is to make efficient use of a limited evaluation budget, since each hyperparameter configuration typically requires training an entire model to assess its quality, which can cost anywhere from seconds (small models) to weeks of compute (large-scale deep learning), making naive exhaustive search prohibitively expensive. Beyond pure performance gains, AutoML and HPO also serve a democratization role, lowering the expertise barrier required to obtain a well-tuned model and enabling more reproducible, less anecdotal model development workflows across research and industry.
## Core Concepts & Theory
The hyperparameter optimization problem is formalized as finding \lambda^* = \arg\min_{\lambda \in \Lambda} f(\lambda), where \lambda is a hyperparameter configuration drawn from a search space \Lambda (which may mix continuous, discrete, and categorical dimensions), and f(\lambda) is the validation loss or error obtained by training a model with configuration \lambda and evaluating it on held-out data. Because f has no analytical form and no gradient with respect to \lambda is directly available, HPO is a black-box optimization problem, and methods are distinguished by how they use the limited number of f(\lambda) evaluations they can afford. Grid search exhaustively evaluates every combination of a discretized set of values per hyperparameter, guaranteeing coverage but scaling exponentially with the number of tuned hyperparameters. Random search, shown by Bergstra and Bengio (2012) to often outperform grid search under a fixed budget, samples configurations independently at random, which is more efficient when only a few hyperparameters actually matter for a given problem (since random search allocates more distinct values to each dimension than an equivalently-sized grid). Bayesian optimization builds a probabilistic surrogate model of f (commonly a Gaussian process) from previously observed configuration-performance pairs, and uses this surrogate together with an acquisition function to decide which configuration to evaluate next, explicitly balancing exploration (trying regions of high uncertainty) against exploitation (trying regions the surrogate predicts will perform well). Multi-fidelity methods such as Successive Halving and Hyperband exploit the observation that cheap, low-fidelity evaluations (e.g., training for a few epochs, or on a data subsample) are often predictive enough of final performance to allow early termination of poor configurations, dramatically improving sample efficiency over full-fidelity-only search.
## Mathematical Formulation
In Bayesian optimization, a Gaussian process (GP) surrogate models f(\lambda) as a random function with a prior mean \mu_0 and covariance kernel k(\lambda, \lambda') (commonly the Matérn or RBF kernel), such that for any finite set of points, the function values are jointly Gaussian:
$$ f(\lambda_{1:n}) \sim \mathcal{N}(\mu_0(\lambda_{1:n}), K) $$
where K_{ij} = k(\lambda_i, \lambda_j). Given n observed pairs \{(\lambda_i, y_i)\}_{i=1}^n, the GP posterior at a new point \lambda has closed-form mean and variance:
$$ \mu_n(\lambda) = k(\lambda)^ op K^{-1} y, \qquad \sigma_n^2(\lambda) = k(\lambda, \lambda) - k(\lambda)^ op K^{-1} k(\lambda) $$
where k(\lambda) = [k(\lambda, \lambda_1), \ldots, k(\lambda, \lambda_n)]^ op. The Expected Improvement (EI) acquisition function, one of the most widely used, quantifies the expected amount by which a new evaluation at \lambda would improve upon the current best observed value y^* = \min_i y_i:
$$ ext{EI}(\lambda) = \mathbb{E}\left[\max(y^* - f(\lambda), 0) ight] = (y^* - \mu_n(\lambda)) \, \Phi(Z) + \sigma_n(\lambda) \, \phi(Z) $$
where Z = \frac{y^* - \mu_n(\lambda)}{\sigma_n(\lambda)}, and \Phi, \phi are the standard normal CDF and PDF respectively. The next configuration to evaluate is chosen by maximizing this acquisition function: \lambda_{n+1} = \arg\max_\lambda ext{EI}(\lambda). For Hyperband, given a maximum resource budget R and a downsampling rate \eta, the number of configurations n evaluated at each successive halving rung and the resource r allocated per configuration follow a geometric schedule designed to allocate exponentially more resources to configurations that survive earlier, cheaper rounds:
$$ n_i = \left\lceil \frac{n \cdot \eta^{-i}}{ ight ceil}, \qquad r_i = r \cdot \eta^{i} $$
## Advanced Theory & Extensions
Tree-structured Parzen Estimator (TPE), used by the popular Optuna and Hyperopt libraries, takes a different modeling approach than GP-based Bayesian optimization: rather than modeling p(y \mid \lambda) directly, TPE models p(\lambda \mid y) via two separate density estimators — one fit to the configurations that produced "good" outcomes (below some quantile threshold) and one fit to the rest — and selects new configurations that maximize the ratio of these two densities, which naturally handles high-dimensional, conditional, and mixed continuous-categorical search spaces more gracefully than a standard GP kernel. BOHB (Bayesian Optimization and Hyperband) combines the sample efficiency of Bayesian optimization with the multi-fidelity resource allocation of Hyperband, using TPE-style modeling to select promising configurations to feed into a Hyperband-style successive halving schedule, generally outperforming either technique in isolation. Neural Architecture Search (NAS) can be viewed as an extreme case of hyperparameter optimization where the search space includes discrete architectural choices (number of layers, operation types, connectivity patterns) rather than only continuous training hyperparameters; NAS methods span the same spectrum from black-box search (evolutionary algorithms, reinforcement learning controllers) to more efficient gradient-based and weight-sharing approaches such as DARTS and ENAS. Population-based training (PBT), developed by DeepMind, interleaves hyperparameter search with the training process itself: a population of models trains in parallel, and periodically the worst-performing members copy the weights and hyperparameters of better-performing members with small perturbations, allowing hyperparameters to be scheduled and adapted dynamically over the course of a single training run rather than fixed for its entire duration. Multi-objective HPO extends the single-objective formulation to jointly optimize competing goals such as accuracy versus inference latency or model size, typically using Pareto-front-based acquisition functions to identify a set of non-dominated trade-off configurations rather than a single optimum.
## Computational Considerations
The dominant computational cost in HPO is almost always the cost of training and evaluating candidate configurations, not the overhead of the search algorithm itself, which means the primary lever for improving HPO efficiency is reducing the number and/or cost of full-fidelity evaluations required. Gaussian process surrogates scale cubically, O(n^3), in the number of observations n due to the matrix inversion required for posterior inference, which limits standard GP-based Bayesian optimization to a few hundred to a few thousand evaluations before approximate or sparse GP methods become necessary. Parallelizing Bayesian optimization requires care because the sequential exploitation-exploration logic assumes each new observation informs the next query; batch acquisition strategies (e.g., local penalization, or Thompson sampling across an ensemble of surrogate posterior draws) allow multiple configurations to be evaluated concurrently on distributed compute without simply reverting to random search. Multi-fidelity methods introduce their own computational trade-off: the correlation between low-fidelity (e.g., 5-epoch) and high-fidelity (e.g., 100-epoch) performance rankings is not perfect, so overly aggressive early stopping in Hyperband-style methods risks discarding configurations that would have performed well with more training, requiring careful tuning of the fidelity schedule itself. For very large-scale deep learning where a single training run costs substantial GPU-time, warm-starting search with transfer learning from HPO results on related, smaller-scale tasks or datasets can substantially reduce the number of expensive full-scale evaluations needed to reach a good configuration.
## Practical Implementation Strategies
A practical default recommendation for most teams starting HPO is to begin with random search over a well-chosen, appropriately-scaled search space (using log-scale sampling for learning rates, regularization coefficients, and similar multiplicative hyperparameters) as a strong, simple baseline before investing in more sophisticated Bayesian or multi-fidelity methods, since random search is trivially parallelizable and surprisingly competitive when the evaluation budget is modest relative to the dimensionality of the search space. Search space design deserves as much attention as the search algorithm itself: overly wide or poorly parameterized ranges waste evaluation budget on clearly nonviable regions, while overly narrow ranges risk excluding the true optimum, and log-uniform versus uniform sampling should be chosen based on whether the hyperparameter's effect on performance is expected to be roughly multiplicative or additive. Using multi-fidelity methods like Hyperband or ASHA (Asynchronous Successive Halving) is generally the most impactful single change teams can make to their HPO workflow when training is expensive, since aggressive early-stopping of clearly underperforming configurations frees up budget for exploring more configurations overall. Popular open-source frameworks — Optuna (TPE-based with pruning support), Ray Tune (supporting a wide range of search algorithms and native distributed execution), and scikit-optimize (GP-based Bayesian optimization) — abstract away much of the implementation complexity and should generally be preferred over hand-rolled search loops except for research into new HPO algorithms themselves. Logging and tracking every evaluated configuration and its outcome (e.g., via MLflow, Weights & Biases, or Optuna's built-in storage) is essential not only for reproducibility but because post-hoc analysis of the search trajectory (e.g., hyperparameter importance analysis) often surfaces actionable insights beyond just the single best configuration found.
## Benchmark Datasets & Evaluation
HPOBench and HPO-B provide large collections of precomputed or surrogate-modeled hyperparameter response surfaces across many datasets and model families (including XGBoost, neural networks, and support vector machines), enabling fast, reproducible benchmarking of new HPO algorithms without requiring the actual expensive model training for every evaluated configuration. NAS-Bench-101/201/301 serve the analogous role for neural architecture search, providing precomputed accuracy tables (or accurate surrogate predictors) for large but enumerable architecture search spaces, allowing NAS algorithms to be compared on equal footing without the confound of differing compute budgets across papers. The AutoML Benchmark (AMLB) evaluates full end-to-end AutoML systems (Auto-sklearn, AutoGluon, H2O AutoML, TPOT) across a standardized suite of tabular classification and regression datasets, measuring both predictive performance and wall-clock time budget adherence. Evaluation of HPO methods typically reports either the best validation performance found within a fixed evaluation budget (anytime performance curves plotting best-so-far performance against number of evaluations or wall-clock time), or the number of evaluations required to reach a target performance threshold, since different algorithms have different strengths at different points along the budget-performance trade-off curve. Because HPO algorithms themselves have stochastic components (random initialization, sampling), rigorous evaluation requires averaging results over multiple independent search runs with different random seeds, and single-run comparisons in published work should be treated with appropriate skepticism.
## Key Challenges & Limitations
High-dimensional search spaces remain challenging for standard Bayesian optimization, since GP surrogates and many acquisition function optimization routines degrade in effectiveness beyond roughly 20-50 dimensions, motivating specialized high-dimensional BO methods (e.g., random embeddings, additive GP structure) that are less mature and less widely adopted than standard low-dimensional BO. Conditional and hierarchical search spaces — where some hyperparameters are only relevant given specific values of others (e.g., a "number of attention heads" hyperparameter that only applies if "architecture type" is set to Transformer) — require specialized search-space representations and are handled more naturally by tree-based methods like TPE than by standard GP kernels, which assume a fixed-dimensional continuous space. The correlation between validation performance and true generalization performance can break down under distribution shift, meaning an HPO process that aggressively optimizes a single validation split risks a form of overfitting to that specific split, which nested cross-validation or repeated validation splits can mitigate at additional computational cost. Multi-fidelity methods introduce a genuine risk of discarding eventually-strong configurations that simply start slowly (a common pattern with some learning rate schedules or regularization strategies that show their benefit only late in training), and the reliability of early-stopping decisions is problem-dependent and not always easy to verify in advance. Finally, AutoML systems in general, including full pipeline search beyond just HPO, can produce brittle solutions that are difficult to interpret, debug, or trust in high-stakes applications, and can also silently encode dataset-specific quirks (data leakage, spurious correlations) into an automatically selected pipeline that a domain expert would have caught through manual inspection.
## Hyperparameter Tuning
Yes, HPO itself has "hyperparameters" that require tuning, or at least sensible default choices: the acquisition function type (Expected Improvement, Upper Confidence Bound, Probability of Improvement) affects the exploration-exploitation trade-off, with UCB's explicit exploration coefficient offering more direct control than EI's implicit balance. The GP kernel choice and its own hyperparameters (length scales, signal variance, noise variance) are typically fit via maximum marginal likelihood after each new observation, but the kernel family itself (RBF versus Matérn, with Matérn generally preferred for its less restrictive smoothness assumptions matching real optimization landscapes better) is a modeling choice that affects surrogate quality. For Hyperband and successive-halving methods, the downsampling rate \eta (commonly 3 or 4) and the maximum resource budget R jointly determine the trade-off between how many configurations are explored versus how much resource each surviving configuration ultimately receives, with more aggressive (higher) \eta values screening more configurations more cheaply at the risk of higher variance in early-stage rankings. The number of random initial "exploration" evaluations performed before switching to model-based Bayesian optimization proposals (commonly 10-20% of total budget) affects surrogate model quality early in the search, with too few initial points risking a poorly calibrated surrogate that misguides early exploitation decisions. For TPE specifically, the quantile threshold \gamma separating "good" from "bad" observed configurations (commonly around 0.15-0.25) controls how selectively the algorithm defines success, with lower values producing a more exploitative, narrowly-focused search.
## Real-World Applications & Case Studies
Google's Vizier system provides HPO-as-a-service internally, powering hyperparameter tuning across a vast range of internal machine learning products and demonstrating that a centralized, algorithm-agnostic Bayesian optimization service can serve heterogeneous workloads at massive scale more efficiently than per-team ad hoc tuning scripts. AWS SageMaker Autopilot and Google Cloud AutoML productize end-to-end AutoML (including HPO, feature engineering, and model selection) as managed cloud services aimed at practitioners without deep machine learning expertise, trading some degree of customizability for substantially reduced time-to-first-model. In industrial recommender systems and ad-ranking pipelines, where model retraining occurs frequently on shifting data distributions, automated HPO pipelines (often using multi-fidelity methods to control the cost of frequent retuning) are standard practice to prevent gradual performance degradation as data distributions drift over time without requiring constant manual intervention. In scientific machine learning and drug discovery, AutoML pipelines including HPO have been used to systematically tune graph neural network architectures for molecular property prediction, where the enormous space of possible architectural and training choices makes manual tuning impractical for teams without dedicated ML infrastructure expertise. Kaggle competition winners frequently rely on systematic HPO (commonly Optuna) as a standard, almost obligatory step in their pipelines, particularly for gradient-boosted tree models (XGBoost, LightGBM, CatBoost) where a handful of well-tuned hyperparameters (learning rate, tree depth, regularization terms, subsampling rates) can produce meaningfully better leaderboard performance than default settings.
## Integration with Other Methods
HPO is tightly integrated with neural architecture search, since modern NAS methods frequently treat architectural choices as an extension of the same hyperparameter search space handled by general-purpose HPO tools, and frameworks like Optuna and Ray Tune support both use cases within a shared API. Meta-learning approaches to HPO (sometimes called "learning to optimize" or warm-starting) use historical HPO results across many prior tasks or datasets to build a meta-model that predicts good starting configurations or promising search directions for a new task, substantially reducing the number of evaluations needed compared to starting a fresh search with no prior knowledge, connecting directly to the broader meta-learning and few-shot adaptation literature. Federated and distributed training pipelines increasingly incorporate HPO as an integrated stage rather than a separate offline step, requiring HPO algorithms that are robust to the higher variance and partial-participation dynamics characteristic of federated learning environments. HPO is also used as an internal component within larger automated machine learning pipelines that jointly search over feature engineering, feature selection, model family, and hyperparameters (e.g., Auto-sklearn's combined algorithm selection and hyperparameter optimization, or "CASH" problem formulation), rather than being applied only after a model family has already been manually fixed. Differentiable and gradient-based HPO methods (e.g., hypergradient descent, which computes gradients of validation loss with respect to hyperparameters via implicit differentiation or unrolled optimization) bridge classical black-box HPO with the gradient-based optimization machinery of standard deep learning training, though they remain less general-purpose than black-box methods since they require the hyperparameter's effect on the loss to be differentiable.
## Future Research Directions
Improving sample efficiency further, particularly for the increasingly common setting where a single training run costs many thousands of GPU-hours (as in large language model pretraining), is a pressing research direction, since even highly sample-efficient Bayesian optimization requires dozens of evaluations that may simply be unaffordable at frontier model scale, motivating research into scaling-law-based extrapolation and small-to-large transfer of tuned hyperparameters. Automated, principled search space design — automatically determining sensible ranges and parameterizations for hyperparameters rather than relying on manually specified search spaces — remains a relatively underexplored area with potential to remove one of the last major manual, expertise-dependent steps in the AutoML pipeline. Robustness and reliability of AutoML-produced pipelines under distribution shift and adversarial conditions is an increasingly important direction as AutoML systems are deployed in higher-stakes settings, requiring HPO and pipeline search objectives that account for robustness metrics rather than optimizing purely for in-distribution validation performance. Green and cost-aware AutoML, explicitly incorporating energy consumption, carbon footprint, or dollar cost as objectives or constraints alongside predictive performance, is gaining attention as the compute cost of both model training and the HPO search process itself have become material environmental and financial concerns. Finally, integrating large language models into the HPO and AutoML loop itself — using LLMs to propose promising configurations, interpret search results, or even write custom feature engineering and model code informed by natural-language problem descriptions — is an emerging direction blurring the line between classical black-box HPO and LLM-driven agentic machine learning pipeline construction.
## Summary & Key Takeaways
AutoML and hyperparameter optimization automate the search over model configurations, architectures, and training settings that would otherwise require expensive, expertise-dependent manual tuning, formalized as black-box optimization of an expensive-to-evaluate validation performance function. Random search provides a strong, simple, embarrassingly parallel baseline, while Bayesian optimization (using Gaussian process or Tree-structured Parzen Estimator surrogates and an acquisition function like Expected Improvement) makes more sample-efficient use of a limited evaluation budget by explicitly modeling uncertainty and balancing exploration against exploitation. Multi-fidelity methods such as Hyperband and BOHB combine early-stopping of clearly underperforming configurations with Bayesian modeling, and represent the most impactful practical technique for expensive-to-train deep learning models. Neural architecture search, population-based training, and multi-objective HPO extend the core black-box optimization framework to discrete architectural search spaces, dynamic hyperparameter schedules, and competing objectives like accuracy versus latency, respectively. As training costs continue to grow, particularly for large-scale deep learning, future HPO research is increasingly focused on extreme sample efficiency, cost-and-carbon-aware search objectives, and the integration of transfer learning and LLM-driven guidance into the search process itself.
---
## Appendix: Practical Labs
### Lab 1: Random Search vs. Grid Search Under a Fixed Budget
import numpy as np
def black_box_objective(lr, reg, seed=0):
"""A synthetic validation-loss surface with a single optimum, standing
in for an expensive real model-training-and-evaluation call."""
rng = np.random.RandomState(seed)
log_lr, log_reg = np.log10(lr), np.log10(reg)
true_loss = (log_lr - (-3.0)) ** 2 + (log_reg - (-2.0)) ** 2
noise = rng.normal(0, 0.05)
return true_loss + noise
def grid_search(lr_range, reg_range, n_per_dim, seed=0):
lrs = np.logspace(np.log10(lr_range[0]), np.log10(lr_range[1]), n_per_dim)
regs = np.logspace(np.log10(reg_range[0]), np.log10(reg_range[1]), n_per_dim)
best_loss, best_config = np.inf, None
for lr in lrs:
for reg in regs:
loss = black_box_objective(lr, reg, seed)
if loss < best_loss:
best_loss, best_config = loss, (lr, reg)
return best_config, best_loss
def random_search(lr_range, reg_range, n_trials, seed=0):
rng = np.random.RandomState(seed)
best_loss, best_config = np.inf, None
for _ in range(n_trials):
lr = 10 ** rng.uniform(np.log10(lr_range[0]), np.log10(lr_range[1]))
reg = 10 ** rng.uniform(np.log10(reg_range[0]), np.log10(reg_range[1]))
loss = black_box_objective(lr, reg, seed)
if loss < best_loss:
best_loss, best_config = loss, (lr, reg)
return best_config, best_loss
def test_random_vs_grid_search():
lr_range, reg_range = (1e-5, 1e-1), (1e-4, 1e0)
budget = 25 # 5x5 grid, or 25 random trials
grid_config, grid_loss = grid_search(lr_range, reg_range, n_per_dim=5)
random_config, random_loss = random_search(lr_range, reg_range, n_trials=budget, seed=1)
print(f"Grid search: best_loss={grid_loss:.4f}, config={grid_config}")
print(f"Random search: best_loss={random_loss:.4f}, config={random_config}")
assert grid_loss < 5.0 and random_loss < 5.0, "Both methods should find a reasonable region of the space"
print("Random vs. grid search test passed.")
if __name__ == "__main__":
test_random_vs_grid_search()### Lab 2: Bayesian Optimization with a Gaussian Process Surrogate
import numpy as np
from scipy.stats import norm
def rbf_kernel(X1, X2, length_scale=1.0, signal_var=1.0):
sq_dists = np.sum(X1**2, axis=1).reshape(-1, 1) + np.sum(X2**2, axis=1) - 2 * X1 @ X2.T
return signal_var * np.exp(-0.5 * sq_dists / length_scale**2)
def gp_posterior(X_train, y_train, X_test, length_scale=1.0, noise=1e-3):
K = rbf_kernel(X_train, X_train, length_scale) + noise * np.eye(len(X_train))
K_s = rbf_kernel(X_train, X_test, length_scale)
K_ss = rbf_kernel(X_test, X_test, length_scale) + 1e-8 * np.eye(len(X_test))
K_inv = np.linalg.inv(K)
mu = K_s.T @ K_inv @ y_train
cov = K_ss - K_s.T @ K_inv @ K_s
return mu, np.sqrt(np.maximum(np.diag(cov), 1e-12))
def expected_improvement(mu, sigma, y_best):
with np.errstate(divide="ignore"):
Z = (y_best - mu) / sigma
ei = (y_best - mu) * norm.cdf(Z) + sigma * norm.pdf(Z)
ei[sigma < 1e-9] = 0.0
return ei
def objective(x):
"""1D toy objective with a clear global minimum near x=0.7."""
return (x - 0.7) ** 2 + 0.05 * np.sin(20 * x)
def bayesian_optimize(n_init=3, n_iter=15, seed=0):
rng = np.random.RandomState(seed)
X = rng.uniform(0, 1, size=(n_init, 1))
y = np.array([objective(x[0]) for x in X])
candidates = np.linspace(0, 1, 200).reshape(-1, 1)
for _ in range(n_iter):
mu, sigma = gp_posterior(X, y, candidates, length_scale=0.15)
ei = expected_improvement(mu, sigma, y.min())
next_x = candidates[np.argmax(ei)]
next_y = objective(next_x[0])
X = np.vstack([X, next_x])
y = np.append(y, next_y)
best_idx = np.argmin(y)
return X[best_idx][0], y[best_idx], X, y
def test_bayesian_optimization():
best_x, best_y, X, y = bayesian_optimize()
print(f"Best x found: {best_x:.4f} (true optimum near 0.7)")
print(f"Best objective value: {best_y:.4f}")
print(f"Total evaluations: {len(y)}")
assert abs(best_x - 0.7) < 0.15, "Bayesian optimization should converge near the true optimum"
print("Bayesian optimization test passed.")
if __name__ == "__main__":
test_bayesian_optimization()### Lab 3: Successive Halving (Hyperband Core Subroutine)
import numpy as np
def noisy_learning_curve(config_quality, budget, seed):
"""Simulates a learning curve: higher config_quality and higher budget
(e.g., epochs trained) produce lower loss, with realistic noise and
diminishing returns as budget increases."""
rng = np.random.RandomState(seed)
base_loss = 1.0 / (1.0 + config_quality * np.log1p(budget))
noise = rng.normal(0, 0.02)
return base_loss + noise
def successive_halving(n_configs=27, min_budget=1, eta=3, seed=0):
rng = np.random.RandomState(seed)
# Each configuration has an unknown "true quality" the search must discover.
config_qualities = rng.uniform(0.1, 2.0, size=n_configs)
config_ids = list(range(n_configs))
budget = min_budget
round_num = 0
history = []
while len(config_ids) > 1:
losses = {
cid: noisy_learning_curve(config_qualities[cid], budget, seed=cid * 100 + round_num)
for cid in config_ids
}
n_survivors = max(1, len(config_ids) // eta)
survivors = sorted(config_ids, key=lambda cid: losses[cid])[:n_survivors]
history.append({
"round": round_num, "budget": budget,
"n_configs": len(config_ids), "n_survivors": n_survivors,
})
config_ids = survivors
budget *= eta
round_num += 1
final_config = config_ids[0]
return final_config, config_qualities[final_config], history
def test_successive_halving():
best_config, best_quality, history = successive_halving(n_configs=27, eta=3, seed=42)
print("Successive halving schedule:")
for round_info in history:
print(f" round={round_info['round']}: budget={round_info['budget']}, "
f"{round_info['n_configs']} -> {round_info['n_survivors']} configs")
print(f"Final selected config quality: {best_quality:.4f}")
# With eta=3 and 27 initial configs, only one should remain after 3 rounds.
assert history[-1]["n_survivors"] == 1, "Successive halving should converge to a single configuration"
# The surviving configuration should generally be among the better ones,
# though multi-fidelity noise means it need not be the single best.
all_qualities_rank = np.argsort(-np.array([best_quality]))
print("Successive halving test passed.")
if __name__ == "__main__":
test_successive_halving()### Lab 4: Tree-Structured Parzen Estimator (TPE) Core Logic
import numpy as np
from scipy.stats import gaussian_kde
def tpe_propose(observed_configs, observed_losses, bounds, gamma=0.2, n_candidates=100, seed=0):
"""Simplified TPE: split observed (config, loss) pairs into 'good' (below
the gamma-quantile of losses) and 'bad' groups, fit a density estimator
to each, and propose the candidate maximizing the good/bad density ratio."""
rng = np.random.RandomState(seed)
observed_configs = np.array(observed_configs)
observed_losses = np.array(observed_losses)
threshold = np.quantile(observed_losses, gamma)
good_mask = observed_losses <= threshold
good_configs = observed_configs[good_mask]
bad_configs = observed_configs[~good_mask]
candidates = rng.uniform(bounds[0], bounds[1], size=(n_candidates, observed_configs.shape[1]))
# Fall back to random proposal if either group is too small for density estimation.
if len(good_configs) < 2 or len(bad_configs) < 2:
return candidates[rng.randint(n_candidates)]
good_kde = gaussian_kde(good_configs.T)
bad_kde = gaussian_kde(bad_configs.T)
good_density = good_kde(candidates.T)
bad_density = bad_kde(candidates.T) + 1e-8
ratio = good_density / bad_density
best_candidate = candidates[np.argmax(ratio)]
return best_candidate
def objective(config):
"""2D toy objective, minimized near (0.3, 0.8)."""
x, y = config
return (x - 0.3) ** 2 + (y - 0.8) ** 2
def run_tpe_search(n_iters=40, seed=0):
rng = np.random.RandomState(seed)
bounds = (np.array([0.0, 0.0]), np.array([1.0, 1.0]))
configs = [rng.uniform(0, 1, size=2) for _ in range(5)]
losses = [objective(c) for c in configs]
for i in range(n_iters):
next_config = tpe_propose(configs, losses, bounds, seed=seed + i)
next_loss = objective(next_config)
configs.append(next_config)
losses.append(next_loss)
best_idx = int(np.argmin(losses))
return configs[best_idx], losses[best_idx]
def test_tpe_search():
best_config, best_loss = run_tpe_search()
print(f"Best config found: ({best_config[0]:.3f}, {best_config[1]:.3f})")
print(f"Best loss: {best_loss:.4f} (true optimum at (0.3, 0.8), loss=0)")
assert best_loss < 0.1, "TPE search should find a configuration reasonably close to the true optimum"
print("TPE search test passed.")
if __name__ == "__main__":
test_tpe_search()