hyperparameter
**Hyperparameter Tuning**
**Key Hyperparameters for LLMs**
**Learning Rate**
| Setting | Typical Range |
|---------|---------------|
| Pretraining | 1e-4 to 3e-4 |
| Full fine-tuning | 1e-5 to 5e-5 |
| LoRA | 1e-4 to 3e-4 |
| LoRA rank | 8, 16, 32, 64 |
**Training**
| Hyperparameter | Considerations |
|----------------|----------------|
| Batch size | Larger = more stable, memory permitting |
| Warmup steps | 1-5% of total steps |
| Weight decay | 0.01 to 0.1 |
| Max sequence length | Task-dependent |
| Epochs | 1-5 for fine-tuning |
**Tuning Strategies**
**Grid Search**
Try all combinations:
```python
learning_rates = [1e-5, 5e-5, 1e-4]
batch_sizes = [8, 16, 32]
for lr in learning_rates:
for bs in batch_sizes:
result = train_and_eval(lr=lr, batch_size=bs)
```
Exhaustive but expensive.
**Random Search**
Sample randomly from distributions:
```python
import random
lr = 10 ** random.uniform(-5, -3) # Log-uniform
bs = random.choice([8, 16, 32, 64])
```
More efficient than grid search for most problems.
**Bayesian Optimization**
Use past results to guide search:
```python
from optuna import create_study
def objective(trial):
lr = trial.suggest_float("lr", 1e-5, 1e-3, log=True)
bs = trial.suggest_int("batch_size", 8, 64, step=8)
return train_and_eval(lr=lr, batch_size=bs)
study = create_study(direction="minimize")
study.optimize(objective, n_trials=20)
```
**Tools for HP Sweeps**
| Tool | Type | Features |
|------|------|----------|
| Optuna | Python library | Bayesian optimization |
| Ray Tune | Distributed | Scales to clusters |
| W&B Sweeps | Commercial | Great visualization |
| Hydra | Config | Config management |
**Weights & Biases Sweep**
```yaml
# sweep.yaml
method: bayes
metric:
name: val_loss
goal: minimize
parameters:
learning_rate:
min: 0.00001
max: 0.001
batch_size:
values: [8, 16, 32]
```
```bash
wandb sweep sweep.yaml
wandb agent
```
**Best Practices**
**Start Simple**
1. Use published hyperparameters as baseline
2. Tune one hyperparameter at a time
3. Focus on learning rate first
**Resource Allocation**
- Use smaller model/dataset for initial sweeps
- Verify best settings transfer to full scale
- Budget compute for tuning (10-20% of total)
**Common Mistakes**
- Tuning on test set (data leakage!)
- Not setting random seeds
- Comparing runs with different # of steps
- Ignoring variability across runs