Preference Data
Preference data is a collection of human or machine judgments about which of several model outputs is better, most commonly stored as a prompt together with a chosen response and a rejected response. It is the raw material of reinforcement learning from human feedback and of direct preference optimization: instead of showing a model the single correct answer, it records relative judgments of quality, which capture things that are easy to recognize but hard to write down, such as helpfulness, tone, honesty and safety. A person who cannot easily compose a perfect summary can usually say which of two summaries is better, and that asymmetry is the whole reason this kind of data exists.
Pairwise comparison is the standard format because comparing is more reliable and cheaper for people than assigning absolute scores. Ratings on a one-to-ten scale drift between annotators and between days, while a choice between two responses is more consistent. The statistical backbone is the Bradley-Terry model, from 1952, which treats the probability that one item beats another as a function of the difference in their hidden quality scores; a reward model trained on preference pairs is essentially fitting those scores. Collecting several responses per prompt is efficient, because ranking four candidates yields six pairs from a single reading. The Christiano et al. 2017 paper on deep reinforcement learning from human preferences showed that comparisons of short video clips were enough to teach agents behaviors no one could specify as a reward.
The InstructGPT paper (Ouyang et al., OpenAI, 2022) established the template that most later systems follow. Contracted labelers wrote prompts and ranked several model outputs per prompt; those rankings trained a reward model, and a reinforcement learning stage then tuned the language model against it. The striking finding was that a much smaller model tuned this way was preferred by human raters over the far larger base GPT-3. Anthropic's 2022 work on helpful and harmless assistants released a large public preference dataset, and Stiennon and colleagues' 2020 summarization work had applied the same loop to text. Later datasets such as UltraFeedback and the Chatbot Arena votes collected by LMSYS in 2023 broadened sources, and some pipelines replace humans with a model judge, an approach called reinforcement learning from AI feedback (RLAIF), as in Anthropic's Constitutional AI paper of 2022.
Annotation is where most of the hidden problems enter, because a preference label is an opinion and models faithfully learn the opinions' biases. Annotators tend to favor longer answers, confident tone and polished formatting, so a reward model absorbs a length bias and the tuned model grows wordy. They may reward agreeable answers over correct ones, which feeds sycophancy. Position bias, where the first or second option is favored regardless of content, shows up in both humans and model judges. Annotators also disagree genuinely, since values differ across cultures and individuals, and a single chosen label hides that spread. Fact-checking is hard under time pressure, so plausible falsehoods can be marked as better than honest hedges.
Quality control therefore matters more than raw volume, and good practice mixes several safeguards. Teams write detailed guidelines, measure inter-annotator agreement, keep gold examples to test labelers, and review a sample of labels personally. Balancing the pairs so that chosen responses are not systematically longer, randomizing display order, and collecting multiple labels for contested pairs all reduce artefacts. Preference pairs also go stale: once a policy improves, its outputs look different from those the data was collected on, so on-policy data collected from the current model is generally more informative than a frozen offline set. Privacy and consent for the prompts used, and the working conditions of the annotators, are part of the data's real cost.
One persistent misconception is that preference data teaches a model what is true or what humanity values. It teaches what a particular set of raters, following particular instructions, preferred on a particular sample of prompts. A model tuned on such data is steered toward those judgments, which is useful for style and safety but not a guarantee of factual accuracy, and it can be pushed beyond the range where the labels were reliable, the classic setting for reward hacking.
Read preference data as a record of human taste captured one comparison at a time: it is the cheapest way yet found to say what good looks like to a model, and its limits are exactly the limits of the judges who produced it.