hyperparameter optimization neural

**Hyperparameter Optimization (HPO)** is the **automated search for the optimal configuration of neural network training hyperparameters (learning rate, batch size, weight decay, architecture choices, augmentation policies) — using principled methods (Bayesian optimization, bandit-based early stopping, evolutionary search) that explore the hyperparameter space more efficiently than manual tuning or grid search, finding configurations that improve model accuracy by 1-5% while reducing the human effort and compute cost of the tuning process**. **Why HPO Matters** Neural network performance is highly sensitive to hyperparameters: learning rate wrong by 2× can reduce accuracy by 5%+. Manual tuning requires deep expertise and many trial-and-error runs. Production scale: a team training hundreds of models per week needs automated HPO to achieve consistent quality. **Search Methods** **Grid Search**: Evaluate all combinations of discrete hyperparameter values. Curse of dimensionality: 5 hyperparameters with 10 values each = 100,000 configurations. Impractical for more than 2-3 hyperparameters. **Random Search (Bergstra & Bengio, 2012)**: Sample hyperparameter configurations randomly from defined distributions. Surprisingly effective — in high-dimensional spaces, random search covers important dimensions better than grid search (which wastes evaluations on unimportant dimensions). 60 random trials often match or exceed exhaustive grid search. **Bayesian Optimization (BO)**: - Build a probabilistic surrogate model (Gaussian Process or Tree-Parzen Estimator) of the objective function (validation accuracy as a function of hyperparameters). - Surrogate predicts both the expected performance and uncertainty for untested configurations. - Acquisition function (Expected Improvement, Upper Confidence Bound) selects the next configuration to evaluate — balancing exploitation (high predicted performance) and exploration (high uncertainty). - Each evaluation enriches the surrogate model → subsequent selections are better informed. - 2-10× more efficient than random search for expensive evaluations (each trial = full training run). **Early Stopping Methods** **Successive Halving / Hyperband (Li et al., 2017)**: - Start many configurations (e.g., 81) with a small budget (e.g., 1 epoch each). - Evaluate and keep only the top 1/3. Give them 3× more budget (3 epochs). - Repeat: keep top 1/3 with 3× budget, until 1 configuration trained to full budget. - Total compute: N × B_max instead of N × B_max configurations — dramatic savings. - Hyperband runs multiple instances of successive halving with different starting budgets to balance exploration breadth and individual trial depth. **HPO Frameworks** - **Optuna**: Python HPO framework. Supports BO (TPE), grid, random. Pruning (early stopping of poor trials via successive halving). Integration with PyTorch Lightning, Hugging Face. - **Ray Tune**: Distributed HPO on Ray clusters. ASHA (Asynchronous Successive Halving), PBT (Population-Based Training), BO. - **Weights & Biases Sweeps**: HPO integrated with experiment tracking. Bayesian and random search with visualization. **Population-Based Training (PBT)** Evolutionary approach: run N training jobs in parallel. Periodically, poor-performing jobs clone the weights and hyperparameters of better-performing jobs (exploit), then mutate hyperparameters slightly (explore). Hyperparameters evolve during training — schedules emerge naturally. 1.5-2× faster than fixed-schedule HPO. Hyperparameter Optimization is **the automation layer that removes the most unreliable component from the ML training pipeline — human intuition about hyperparameter settings** — replacing guesswork with principled search that consistently finds better configurations in fewer trials.