hyperparameter optimization bayesian

**Hyperparameter Optimization (HPO)** is **the systematic process of selecting the best configuration of training hyperparameters — learning rate, batch size, architecture choices, regularization strength, and optimizer settings — using principled search strategies that maximize model performance while minimizing computational cost** — replacing manual trial-and-error tuning with automated methods ranging from Bayesian optimization to population-based training. **Search Strategy Taxonomy:** - **Grid Search**: Evaluate all combinations of discretized hyperparameter values; exhaustive but exponentially expensive in the number of hyperparameters (curse of dimensionality) - **Random Search**: Sample hyperparameter configurations uniformly at random; provably more efficient than grid search when only a few hyperparameters matter (Bergstra & Bengio, 2012) - **Bayesian Optimization**: Build a probabilistic surrogate model of the objective function and use an acquisition function to select the most promising configuration to evaluate next - **Tree-Structured Parzen Estimator (TPE)**: Model the density of good and bad configurations separately using kernel density estimators, selecting points with high probability under the good distribution (used in Optuna and Hyperopt) - **Gaussian Process (GP)**: Fit a Gaussian process to observed (configuration, performance) pairs, using Expected Improvement or Upper Confidence Bound acquisition functions - **Successive Halving / Hyperband**: Allocate a small budget to many configurations, then progressively eliminate the worst performers and allocate more resources to survivors - **Population-Based Training (PBT)**: Maintain a population of models training in parallel, periodically replacing poor performers with perturbed copies of good performers — enabling hyperparameter schedules to evolve during training **Key Frameworks and Tools:** - **Optuna**: Python framework with TPE-based sampler, pruning via median/percentile stopping, multi-objective optimization, and rich visualization (contour plots, parameter importance, optimization history) - **Ray Tune**: Distributed HPO library integrated with Ray, supporting multiple search algorithms (Bayesian, Hyperband, PBT, BOHB), fault-tolerant distributed execution, and seamless scaling from laptop to cluster - **Weights & Biases Sweeps**: Cloud-integrated HPO with Bayesian and random search, real-time experiment tracking, and collaborative visualization - **KerasTuner**: Keras-native HPO with built-in Hyperband, random search, and Bayesian optimization for Keras/TensorFlow models - **SMAC3**: Sequential Model-Based Algorithm Configuration using random forests as surrogate models, excelling on conditional and high-dimensional search spaces - **Ax/BoTorch**: Meta's adaptive experimentation platform built on BoTorch (Bayesian optimization in PyTorch), supporting multi-objective and constrained optimization **Early Stopping and Pruning:** - **Median Pruner**: Stop a trial if its intermediate performance falls below the median of completed trials at the same step - **Percentile Pruner**: Generalize median pruning to any percentile threshold, trading aggressiveness for risk of pruning eventually-good trials - **ASHA (Asynchronous Successive Halving)**: Asynchronously promote or stop trials based on their performance at predefined rungs, enabling efficient utilization of distributed resources - **Learning Curve Extrapolation**: Fit parametric curves to partial training histories to predict final performance and prune unlikely candidates early **Multi-Objective and Constrained HPO:** - **Pareto Optimization**: Simultaneously optimize accuracy, latency, and model size, returning a Pareto front of non-dominated solutions - **Constrained Optimization**: Enforce hard constraints (e.g., model must be under 50MB, inference under 10ms) while maximizing accuracy - **Cost-Aware Search**: Weight the acquisition function by the computational cost of each configuration, preferring cheap evaluations when uncertainty is high **Practical Recommendations:** - **Start with Random Search**: Establish baselines and understand the hyperparameter landscape before deploying more sophisticated methods - **Use Log-Uniform Sampling**: For learning rates, weight decay, and other scale-sensitive parameters, sample uniformly in log space - **Budget Allocation**: Allocate 20–50% of total compute budget to HPO; use Hyperband-style early stopping to maximize configurations evaluated - **Warm-Starting**: Initialize Bayesian optimization with previously observed configurations from related tasks or model architectures - **Feature Importance Analysis**: Use fANOVA (functional ANOVA) to quantify which hyperparameters most impact performance, focusing future search on the most influential ones Hyperparameter optimization has **evolved from a manual art into a rigorous engineering discipline — with modern frameworks enabling practitioners to efficiently navigate vast configuration spaces, discover non-obvious hyperparameter interactions, and systematically extract maximum performance from deep learning models within fixed computational budgets**.