fine-tune
**Fine-tuning** is the **process of adapting a pretrained language model to specific tasks, domains, or behaviors** — taking a foundation model trained on general data and updating its weights using smaller, curated datasets, enabling specialized performance that outperforms generic models while requiring far less compute than training from scratch.
**What Is Fine-Tuning?**
- **Definition**: Continued training of a pretrained model on task-specific data.
- **Input**: Pretrained base model + domain-specific dataset.
- **Output**: Specialized model adapted to target task/domain.
- **Purpose**: Customize behavior without pretraining costs.
**Why Fine-Tuning Matters**
- **Specialization**: Adapt general models to specific domains (medical, legal, code).
- **Efficiency**: 1000× cheaper than pretraining from scratch.
- **Quality**: Often outperforms in-context learning for specialized tasks.
- **Consistency**: Reliable output format and style.
- **Proprietary Data**: Incorporate private or specialized knowledge.
- **Reduced Prompt Length**: Bake instructions into weights.
**Fine-Tuning Methods**
**Supervised Fine-Tuning (SFT)**:
- Train on (instruction, response) pairs.
- Direct demonstration of desired behavior.
- Most common and straightforward approach.
**Reinforcement Learning from Human Feedback (RLHF)**:
- Train reward model on human preference comparisons.
- Optimize policy via PPO to maximize reward.
- More complex but enables nuanced alignment.
**Direct Preference Optimization (DPO)**:
- Directly optimize on preference data without reward model.
- Simpler than RLHF, similar results.
- Increasingly popular for alignment.
**Constitutional AI (CAI)**:
- Self-critique using principles.
- Model evaluates and improves its own responses.
- Reduces need for human labeling.
**Parameter-Efficient Fine-Tuning (PEFT)**
**LoRA (Low-Rank Adaptation)**:
```
Original: W (d × d matrix, frozen)
LoRA: W + BA (B is d × r, A is r × d)
r << d (e.g., r=16, d=4096)
Train only A and B: 0.1-1% of parameters
Merge at inference: W' = W + BA
```
**QLoRA**:
- Load base model in 4-bit quantization.
- Train LoRA adapters in FP16.
- Fine-tune 70B models on single 24-48GB GPU.
**Other PEFT Methods**:
- **Prefix Tuning**: Learn continuous prompt embeddings.
- **Adapters**: Insert small trainable modules between layers.
- **IA³**: Scale activations with learned vectors.
**When to Fine-Tune vs. Prompt**
```
Approach | Best For
-----------------|------------------------------------------
Prompting/RAG | Variable tasks, fast iteration, small data
Fine-Tuning | Consistent format, domain expertise, scale
Full FT | New capabilities, architecture changes
PEFT (LoRA) | Limited compute, multiple adapters
```
**Fine-Tuning Pipeline**
```svg
```
**Tools & Frameworks**
- **Hugging Face**: transformers, peft, trl libraries.
- **Axolotl**: Streamlined fine-tuning configuration.
- **LLaMA-Factory**: GUI and CLI for fine-tuning.
- **Unsloth**: Memory-efficient fine-tuning.
- **Together AI, Modal, Lambda**: Cloud fine-tuning services.
Fine-tuning is **the bridge between general AI and domain-specific solutions** — it enables organizations to create customized models that understand their specific terminology, formats, and requirements while building on the massive investment in foundation model pretraining.