The Bradley-Terry model is a probabilistic framework for modeling pairwise comparison outcomes — given two options, it predicts the probability of each one being preferred. It is the mathematical foundation underlying reward model training in RLHF.

The Model

Each option i has a latent strength parameter $\beta_i$. The probability that option i is preferred over option j is:

$$P(i \succ j) = \frac{e^{\beta_i}}{e^{\beta_i} + e^{\beta_j}} = \sigma(\beta_i - \beta_j)$$

Where $\sigma$ is the sigmoid function. The preference probability depends only on the difference in strengths, not their absolute values.

Connection to RLHF

• In RLHF reward modeling, the reward model assigns scores $r(x, y)$ to each response y given prompt x.
• The Bradley-Terry model assumes the probability of preferring response $y_w$ over $y_l$ is:

$$P(y_w \succ y_l | x) = \sigma(r(x, y_w) - r(x, y_l))$$

• The reward model is trained by maximizing the log-likelihood of the observed human preferences under this model.

Key Properties

• Transitivity: The model assumes consistent preferences — if A is strongly preferred over B and B over C, then A will be strongly preferred over C.
• Scale Invariance: Adding a constant to all strengths doesn't change preferences — only differences matter.
• Maximum Likelihood: Parameters are estimated by maximizing the likelihood of observed comparison outcomes.

Extensions

• Thurstone Model: Alternative where strengths are sampled from Normal distributions rather than Gumbel distributions.
• Plackett-Luce Model: Extends Bradley-Terry to rankings of more than two items.
• Ties: Extensions exist for handling "equally good" outcomes.

Practical Usage

Beyond RLHF, the Bradley-Terry model is used in chess/Elo ratings, sports ranking, A/B testing, and any domain involving pairwise comparisons. The LMSYS Chatbot Arena leaderboard uses it to rank LLMs based on human votes.