Preference-Based RL is a reinforcement learning paradigm where the reward signal comes from human preferences over trajectory pairs — instead of numeric rewards, a human evaluator compares two behaviors and indicates which is preferred, and a reward model is learned from these comparisons.

Preference Learning Pipeline

• Query: Present the human with two trajectory segments $(sigma_1, sigma_2)$.
• Label: Human indicates preference: $sigma_1 succ sigma_2$, $sigma_2 succ sigma_1$, or roughly equal.
• Reward Model: Train $R(s,a)$ such that $P(sigma_1 succ sigma_2) = frac{exp(sum R(sigma_1))}{exp(sum R(sigma_1)) + exp(sum R(sigma_2))}$ (Bradley-Terry model).
• RL: Optimize policy using the learned reward model via standard RL (PPO, SAC, etc.).

Why It Matters

• Reward-Free: No need to hand-craft reward functions — preferences define the objective implicitly.
• Scalable: Preferences are faster to provide than demonstrations — binary comparison is cognitively easy.
• Active Queries: Active learning selects the most informative trajectory pairs to query — minimizes human effort.

Preference-Based RL is learning rewards from comparisons — using human preferences over behaviors to automatically derive reward functions.