Hyperparameter tuning searches for optimal training settings like learning rate, batch size, and architecture choices. What are hyperparameters: Settings not learned by training - learning rate, batch size, layer count, regularization strength, optimizer choice. Search methods: Grid search: Try all combinations. Exhaustive but exponentially expensive. Random search: Random combinations. Often more efficient than grid (Bergstra and Bengio). Bayesian optimization: Model performance surface, sample promising regions. Efficient for expensive evaluations. Population-based training: Evolutionary approach, mutate and select best configurations during training. Key hyperparameters for LLMs: Learning rate (most important), warmup steps, batch size, weight decay, dropout. Practical approach: Start with known good defaults, tune learning rate first, then batch size, then minor parameters. Tools: Optuna, Ray Tune, Weights and Biases sweeps, Keras Tuner. Compute considerations: Each trial is a training run. Budget limits thorough search. Use early stopping, parallel trials. Best practices: Log all hyperparameters, use validation set (not test), consider reproducibility.