hyperparameter tuning

Hyperparameter tuning searches for optimal training settings like learning rate, batch size, and architecture choices. **What are hyperparameters**: Settings not learned by training - learning rate, batch size, layer count, regularization strength, optimizer choice. **Search methods**: **Grid search**: Try all combinations. Exhaustive but exponentially expensive. **Random search**: Random combinations. Often more efficient than grid (Bergstra and Bengio). **Bayesian optimization**: Model performance surface, sample promising regions. Efficient for expensive evaluations. **Population-based training**: Evolutionary approach, mutate and select best configurations during training. **Key hyperparameters for LLMs**: Learning rate (most important), warmup steps, batch size, weight decay, dropout. **Practical approach**: Start with known good defaults, tune learning rate first, then batch size, then minor parameters. **Tools**: Optuna, Ray Tune, Weights and Biases sweeps, Keras Tuner. **Compute considerations**: Each trial is a training run. Budget limits thorough search. Use early stopping, parallel trials. **Best practices**: Log all hyperparameters, use validation set (not test), consider reproducibility.