reward modeling
**Reward Modeling and Preference Learning** — Reward modeling trains neural networks to predict human preferences over model outputs, providing the optimization signal that aligns language models with human values and intentions through reinforcement learning from human feedback.
**Reward Model Architecture** — Reward models typically share the same architecture as the language model being aligned, with the final unembedding layer replaced by a scalar value head. Given an input prompt and a completion, the reward model outputs a single score representing quality. Training uses comparison data where human annotators rank multiple completions for the same prompt, and the model learns to assign higher scores to preferred outputs through pairwise ranking losses.
**Bradley-Terry Preference Framework** — The standard approach models human preferences using the Bradley-Terry model, where the probability of preferring response A over B is a sigmoid function of their reward difference. This formulation enables training from pairwise comparisons without requiring absolute quality scores. The loss function maximizes the log-likelihood of observed preferences, naturally calibrating reward differences to reflect preference strength.
**Data Collection and Quality** — High-quality preference data requires careful annotator selection, clear guidelines, and calibration procedures. Inter-annotator agreement metrics identify ambiguous examples and unreliable annotators. Diverse prompt distributions ensure the reward model generalizes across topics and styles. Active learning strategies prioritize labeling examples where the current reward model is most uncertain, maximizing information gain per annotation dollar spent.
**Direct Preference Optimization** — DPO eliminates the need for explicit reward model training by directly optimizing the language model policy using preference data. The key insight reformulates the reward modeling objective as a classification loss on the policy itself, treating the log-ratio of policy probabilities as an implicit reward. Variants like IPO, KTO, and ORPO further simplify preference learning with different theoretical foundations and practical trade-offs.
**Reward modeling serves as the critical translation layer between subjective human judgment and mathematical optimization, and its fidelity fundamentally determines whether aligned models truly capture human preferences or merely exploit superficial patterns in annotation data.**