hyperparameter optimization
**Hyperparameter Optimization (HPO)** is the **systematic search for the best configuration of training settings (learning rate, batch size, architecture choices, regularization) that maximizes model performance** — automating what was traditionally a manual trial-and-error process, with methods ranging from simple grid search to sophisticated Bayesian optimization that can efficiently explore high-dimensional configuration spaces.
**Common Hyperparameters**
| Category | Parameters | Typical Range |
|----------|-----------|---------------|
| Optimization | Learning rate, weight decay, momentum | LR: 1e-5 to 1e-1 |
| Architecture | Hidden size, num layers, num heads | Problem-dependent |
| Regularization | Dropout, label smoothing, data augmentation | 0.0 to 0.5 |
| Training | Batch size, epochs, warmup steps | 16 to 4096 |
| LR Schedule | Cosine, linear, step decay | Schedule type + params |
**Search Strategies**
**Grid Search**
- Evaluate all combinations of pre-specified values.
- Cost: Exponential in number of hyperparameters — $O(V^D)$ for V values per D dimensions.
- Effective only for 1-3 hyperparameters.
**Random Search (Bergstra & Bengio 2012)**
- Sample configurations randomly from distributions.
- Provably more efficient than grid search: Better at finding narrow optima.
- Widely used as a strong baseline.
**Bayesian Optimization**
- Build a **surrogate model** (Gaussian Process, Tree-structured Parzen Estimator) of the objective function.
- **Acquisition function** (Expected Improvement, UCB) selects next configuration to try.
- After each trial: Update surrogate model with new result.
- Efficient: Finds good configurations in 20-100 trials — 10-50x fewer than random search.
**Multi-Fidelity Methods**
- **Hyperband / ASHA**: Train many configurations for a few epochs → prune bad ones → train survivors longer.
- Successive halving: Start 81 configs for 1 epoch → keep top 27 for 3 epochs → top 9 for 9 epochs → top 3 for 27 epochs → best 1 for 81 epochs.
- Dramatically reduces total compute compared to full training of each configuration.
**HPO Frameworks**
| Framework | Backend | Highlights |
|-----------|---------|------------|
| Optuna | TPE, CMA-ES | Pythonic, pruning, visualization |
| Ray Tune | Any (Optuna, BO, PBT) | Distributed, multi-GPU support |
| Weights & Biases Sweeps | Bayes, Random, Grid | Integrated experiment tracking |
| Ax (Meta) | Bayesian (BoTorch) | Multi-objective, neural BO |
**Population-Based Training (PBT)**
- Run multiple training runs in parallel.
- Periodically: Poorly performing runs copy weights and hyperparameters from top performers, with random perturbation.
- Hyperparameters evolve during training — adapts LR schedule automatically.
Hyperparameter optimization is **a critical but often undervalued component of ML development** — a well-tuned baseline model frequently outperforms a poorly-tuned novel architecture, making systematic HPO one of the highest-ROI investments in any machine learning project.