Preference learning is the machine learning paradigm of learning what humans prefer from comparative judgments rather than absolute scores. In the context of LLMs, it refers to training models to understand and generate responses that humans would choose over alternatives.

Why Comparisons Over Ratings

• Easier for Humans: Asking "Which response is better, A or B?" is more natural and reliable than "Rate this response from 1 to 10." People are much better at relative judgments than absolute ones.
• More Consistent: Different annotators may use rating scales differently (one person's 7 is another's 5), but pairwise preferences are more stable across annotators.
• Naturally Ordered: Preferences directly encode the ordering relationship needed for training — response A is better than B.

Preference Learning Methods

• RLHF: Train a reward model on preferences, then use RL to optimize the LLM against that reward. The classic approach used by ChatGPT and Claude.
• DPO (Direct Preference Optimization): Directly optimize the LLM on preference data without training a separate reward model. Simpler and more stable than RLHF.
• KTO (Kahneman-Tversky Optimization): Learns from binary feedback (good/bad) rather than pairwise comparisons.
• IPO (Identity Preference Optimization): A variant of DPO with better theoretical properties.
• ORPO (Odds Ratio Preference Optimization): Combines preference learning with supervised fine-tuning in a single objective.

Data Collection

• Human Annotators: Trained evaluators compare model outputs and select the better one. Gold standard but expensive.
• AI Feedback: Use a strong LLM (like GPT-4) to generate preference labels. Cheaper but may introduce systematic biases.
• Implicit Feedback: Derive preferences from user behavior — which responses users accept, regenerate, or edit.

Challenges

• Intransitive Preferences: Humans may prefer A over B, B over C, but C over A — not a consistent ranking.
• Subjectivity: Different users have different preferences, both factual and stylistic.
• Annotation Cost: High-quality human preferences remain expensive at scale.

Preference learning is now the dominant approach for post-training alignment of large language models.