Pairwise comparison is an evaluation method where two model outputs are placed side by side and a judge (human or LLM) determines which response is better. It is the most common format for evaluating large language models because it produces more reliable and consistent judgments than absolute scoring.

Why Pairwise Over Absolute Rating

• Easier Judgment: Humans find it much easier to say "A is better than B" than to assign a precise score like "This is a 7 out of 10."
• More Consistent: Different annotators calibrate absolute scales differently, but pairwise preferences show higher inter-annotator agreement.
• Directly Useful: Pairwise preferences are exactly the data format needed for reward model training (RLHF) and ranking algorithms (Bradley-Terry, Elo).

How It Works

• Input: A prompt plus two candidate responses (A and B).
• Judge: A human evaluator or strong LLM compares the responses on criteria like helpfulness, accuracy, safety, clarity, and completeness.
• Output: One of: A wins, B wins, or Tie.

Key Considerations

• Position Bias: Judges may prefer whichever response is shown first (or second). Mitigation: Run each comparison twice with positions swapped.
• Length Bias: Longer responses often appear more thorough. Mitigation: Use length-controlled evaluation protocols.
• Criteria Specification: Clear evaluation criteria improve consistency. Without them, judges weigh factors differently.

Applications

• LMSYS Chatbot Arena: Blind pairwise comparisons by real users to rank LLMs.
• AlpacaEval: GPT-4 as judge performing pairwise comparisons against a reference model.
• RLHF Data Collection: Human annotators provide pairwise preferences for reward model training.
• A/B Testing: Compare model versions during development using pairwise evaluation.

Pairwise comparison is the gold standard evaluation format for LLMs — it provides the most reliable signal about relative model quality.