Hyperparameter Tuning

Key Hyperparameters for LLMs

Learning Rate

Training

Tuning Strategies

Grid Search Try all combinations:

learning_rates = [1e-5, 5e-5, 1e-4]
batch_sizes = [8, 16, 32]

for lr in learning_rates:
    for bs in batch_sizes:
        result = train_and_eval(lr=lr, batch_size=bs)

Exhaustive but expensive.

Random Search Sample randomly from distributions:

import random

lr = 10 ** random.uniform(-5, -3)  # Log-uniform
bs = random.choice([8, 16, 32, 64])

More efficient than grid search for most problems.

Bayesian Optimization Use past results to guide search:

from optuna import create_study

def objective(trial):
    lr = trial.suggest_float("lr", 1e-5, 1e-3, log=True)
    bs = trial.suggest_int("batch_size", 8, 64, step=8)
    return train_and_eval(lr=lr, batch_size=bs)

study = create_study(direction="minimize")
study.optimize(objective, n_trials=20)

Tools for HP Sweeps

Weights & Biases Sweep

# sweep.yaml
method: bayes
metric:
  name: val_loss
  goal: minimize
parameters:
  learning_rate:
    min: 0.00001
    max: 0.001
  batch_size:
    values: [8, 16, 32]

wandb sweep sweep.yaml
wandb agent <sweep_id>

Best Practices

Start Simple 1. Use published hyperparameters as baseline 2. Tune one hyperparameter at a time 3. Focus on learning rate first

Resource Allocation

• Use smaller model/dataset for initial sweeps
• Verify best settings transfer to full scale
• Budget compute for tuning (10-20% of total)

Common Mistakes

• Tuning on test set (data leakage!)
• Not setting random seeds
• Comparing runs with different # of steps
• Ignoring variability across runs