bradley-terry model
**The Bradley-Terry model** is a probabilistic framework for modeling **pairwise comparison** outcomes — given two options, it predicts the probability of each one being preferred. It is the mathematical foundation underlying **reward model training** in RLHF.
**The Model**
Each option i has a latent strength parameter $\beta_i$. The probability that option i is preferred over option j is:
$$P(i \succ j) = \frac{e^{\beta_i}}{e^{\beta_i} + e^{\beta_j}} = \sigma(\beta_i - \beta_j)$$
Where $\sigma$ is the **sigmoid function**. The preference probability depends only on the **difference in strengths**, not their absolute values.
**Connection to RLHF**
- In RLHF reward modeling, the reward model assigns scores $r(x, y)$ to each response y given prompt x.
- The Bradley-Terry model assumes the probability of preferring response $y_w$ over $y_l$ is:
$$P(y_w \succ y_l | x) = \sigma(r(x, y_w) - r(x, y_l))$$
- The reward model is trained by **maximizing the log-likelihood** of the observed human preferences under this model.
**Key Properties**
- **Transitivity**: The model assumes consistent preferences — if A is strongly preferred over B and B over C, then A will be strongly preferred over C.
- **Scale Invariance**: Adding a constant to all strengths doesn't change preferences — only differences matter.
- **Maximum Likelihood**: Parameters are estimated by maximizing the likelihood of observed comparison outcomes.
**Extensions**
- **Thurstone Model**: Alternative where strengths are sampled from Normal distributions rather than Gumbel distributions.
- **Plackett-Luce Model**: Extends Bradley-Terry to **rankings** of more than two items.
- **Ties**: Extensions exist for handling "equally good" outcomes.
**Practical Usage**
Beyond RLHF, the Bradley-Terry model is used in **chess/Elo ratings**, **sports ranking**, **A/B testing**, and any domain involving pairwise comparisons. The **LMSYS Chatbot Arena leaderboard** uses it to rank LLMs based on human votes.