hyperparameter tuning

**Hyperparameter Tuning** — finding the optimal settings for values not learned during training (learning rate, batch size, architecture choices, regularization strength). **Methods** - **Manual**: Intuition + trial and error. Common in practice but not systematic - **Grid Search**: Try all combinations of predefined values. Exhaustive but exponentially expensive - **Random Search**: Sample random combinations. Often better than grid search — more efficient exploration (Bergstra & Bengio, 2012) - **Bayesian Optimization**: Build probabilistic model of objective function, sample promising points. Tools: Optuna, Weights & Biases Sweeps - **Population-Based Training (PBT)**: Evolve hyperparameters during training. Used by DeepMind **Key Hyperparameters** - Learning rate (most important) - Batch size, weight decay, dropout rate - Architecture: depth, width, number of heads - Schedule: warmup steps, decay type **Best Practices** - Start with published defaults for your architecture - Tune learning rate first (log scale: 1e-5 to 1e-1) - Use validation set, never test set, for selection