direct preference optimization dpo

**Direct Preference Optimization (DPO)** is the **alignment training algorithm that optimizes language models directly on human preference data without requiring a separate reward model or reinforcement learning loop — reformulating the RLHF objective into a simple classification loss on preferred vs. rejected response pairs, achieving comparable alignment quality to PPO-based RLHF with dramatically simpler implementation and more stable training**. **The RLHF Complexity Problem** Standard RLHF has three stages: (1) supervised fine-tuning (SFT), (2) reward model training on preference data, (3) PPO optimization of the policy against the reward model with KL constraint. Stage 3 is notoriously unstable — PPO requires careful tuning of learning rate, KL coefficient, advantage estimation, value function warmup, and reward normalization. DPO eliminates stages 2 and 3 entirely. **The DPO Insight** Rafailov et al. (2023) showed that the optimal policy under the KL-constrained RLHF objective has a closed-form relationship to the reward function: r(x, y) = β · log(π(y|x) / π_ref(y|x)) + f(x) where π is the policy, π_ref is the reference (SFT) model, and β is the KL constraint strength. This means the reward is implicitly defined by the policy — no separate reward model is needed. **DPO Loss** Substituting the implicit reward into the Bradley-Terry preference model: L_DPO = −E[log σ(β · (log π(y_w|x)/π_ref(y_w|x) − log π(y_l|x)/π_ref(y_l|x)))] where y_w is the preferred response and y_l is the rejected response. This is simply a binary cross-entropy loss on the log-probability ratios. The policy is trained to increase the probability of preferred responses and decrease the probability of rejected responses, relative to the reference model. **Advantages Over RLHF** - **Simplicity**: No reward model training, no PPO, no value function, no advantage estimation. DPO is a straightforward supervised loss on preference pairs. - **Stability**: No RL instability (reward hacking, KL divergence explosion, reward model exploitation). Training curves are smooth and predictable. - **Efficiency**: Single stage of training after SFT. No need to maintain four models in memory simultaneously (policy, reference, reward, value — required by PPO). **Practical Considerations** - **On-Policy vs. Off-Policy**: DPO trains on a fixed dataset of preference pairs (off-policy). If the SFT model distribution has shifted significantly, the preference data may be out-of-distribution. Iterative DPO (regenerating responses with the current policy) partially addresses this. - **Reference Model**: The π_ref model (typically the SFT checkpoint) must be kept in memory during training for computing log-probability ratios. This doubles the memory requirement compared to standard fine-tuning. - **β Sensitivity**: The temperature β controls how much the policy can deviate from the reference. Too low: little alignment effect. Too high: policy collapses to always choosing safe but uninformative responses. Direct Preference Optimization is **the simplification that made RLHF practical for everyone** — proving that the complex RL machinery of PPO was solving a problem that had a much simpler direct solution, opening alignment training to any team that can fine-tune a language model.