preference learning

**Preference learning** is the machine learning paradigm of learning **what humans prefer** from comparative judgments rather than absolute scores. In the context of LLMs, it refers to training models to understand and generate responses that humans would choose over alternatives. **Why Comparisons Over Ratings** - **Easier for Humans**: Asking "Which response is better, A or B?" is more natural and reliable than "Rate this response from 1 to 10." People are much better at **relative** judgments than **absolute** ones. - **More Consistent**: Different annotators may use rating scales differently (one person's 7 is another's 5), but pairwise preferences are more stable across annotators. - **Naturally Ordered**: Preferences directly encode the ordering relationship needed for training — response A is better than B. **Preference Learning Methods** - **RLHF**: Train a reward model on preferences, then use RL to optimize the LLM against that reward. The classic approach used by **ChatGPT** and **Claude**. - **DPO (Direct Preference Optimization)**: Directly optimize the LLM on preference data without training a separate reward model. Simpler and more stable than RLHF. - **KTO (Kahneman-Tversky Optimization)**: Learns from **binary feedback** (good/bad) rather than pairwise comparisons. - **IPO (Identity Preference Optimization)**: A variant of DPO with better theoretical properties. - **ORPO (Odds Ratio Preference Optimization)**: Combines preference learning with supervised fine-tuning in a single objective. **Data Collection** - **Human Annotators**: Trained evaluators compare model outputs and select the better one. Gold standard but expensive. - **AI Feedback**: Use a strong LLM (like GPT-4) to generate preference labels. Cheaper but may introduce systematic biases. - **Implicit Feedback**: Derive preferences from user behavior — which responses users accept, regenerate, or edit. **Challenges** - **Intransitive Preferences**: Humans may prefer A over B, B over C, but C over A — not a consistent ranking. - **Subjectivity**: Different users have different preferences, both factual and stylistic. - **Annotation Cost**: High-quality human preferences remain expensive at scale. Preference learning is now the dominant approach for **post-training alignment** of large language models.