direct preference optimization
**Direct Preference Optimization (DPO)** is a **stable, reward-model-free alternative to RLHF that directly optimizes LLM policy on preference data** — achieving comparable alignment results without the complexity and instability of PPO training.
**The Problem DPO Solves**
- RLHF requires training a separate reward model, then running PPO (complex, unstable).
- PPO has many failure modes: reward hacking, KL explosion, mode collapse.
- DPO (Rafailov et al., 2023) shows: the optimal policy under RLHF has a closed-form relationship to the reward — no need to train the reward model explicitly.
**DPO Objective**
$$L_{DPO} = -E_{(x,y_w,y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)} \right) \right]$$
- $y_w$: preferred (winning) response, $y_l$: rejected (losing) response.
- $\pi_{ref}$: SFT reference model (frozen).
- Increase probability of preferred response relative to reference; decrease rejected.
**DPO Variants**
- **IPO (Identity Preference Optimization)**: Addresses overfitting in DPO.
- **KTO (Kahneman-Tversky Optimization)**: Uses non-paired preference data.
- **SimPO**: Reference-model-free, uses sequence length normalization.
- **ORPO**: Combines SFT and preference learning in one stage.
**DPO vs. RLHF Comparison**
| Aspect | RLHF (PPO) | DPO |
|--------|------------|-----|
| Reward model | Required | Not needed |
| Training stability | Lower | Higher |
| Hyperparameter sensitivity | High | Low |
| Performance | Slightly higher | Close/comparable |
| Implementation complexity | High | Low |
**Adoption**: DPO is now widely used — Llama 2, Mistral, Gemma, and most open-source aligned models use DPO or variants.
DPO is **the standard preference alignment method for open-source LLMs** — its simplicity and stability democratized alignment beyond large labs with PPO infrastructure.