pairwise comparison
**Pairwise comparison** is an evaluation method where two model outputs are placed **side by side** and a judge (human or LLM) determines which response is **better**. It is the most common format for evaluating large language models because it produces more reliable and consistent judgments than absolute scoring.
**Why Pairwise Over Absolute Rating**
- **Easier Judgment**: Humans find it much easier to say "A is better than B" than to assign a precise score like "This is a 7 out of 10."
- **More Consistent**: Different annotators calibrate absolute scales differently, but pairwise preferences show higher **inter-annotator agreement**.
- **Directly Useful**: Pairwise preferences are exactly the data format needed for **reward model training** (RLHF) and **ranking algorithms** (Bradley-Terry, Elo).
**How It Works**
- **Input**: A prompt plus two candidate responses (A and B).
- **Judge**: A human evaluator or strong LLM compares the responses on criteria like helpfulness, accuracy, safety, clarity, and completeness.
- **Output**: One of: A wins, B wins, or Tie.
**Key Considerations**
- **Position Bias**: Judges may prefer whichever response is shown first (or second). **Mitigation**: Run each comparison twice with positions swapped.
- **Length Bias**: Longer responses often appear more thorough. **Mitigation**: Use length-controlled evaluation protocols.
- **Criteria Specification**: Clear evaluation criteria improve consistency. Without them, judges weigh factors differently.
**Applications**
- **LMSYS Chatbot Arena**: Blind pairwise comparisons by real users to rank LLMs.
- **AlpacaEval**: GPT-4 as judge performing pairwise comparisons against a reference model.
- **RLHF Data Collection**: Human annotators provide pairwise preferences for reward model training.
- **A/B Testing**: Compare model versions during development using pairwise evaluation.
Pairwise comparison is the **gold standard evaluation format** for LLMs — it provides the most reliable signal about relative model quality.