preference-based rl

**Preference-Based RL** is a **reinforcement learning paradigm where the reward signal comes from human preferences over trajectory pairs** — instead of numeric rewards, a human evaluator compares two behaviors and indicates which is preferred, and a reward model is learned from these comparisons. **Preference Learning Pipeline** - **Query**: Present the human with two trajectory segments $(sigma_1, sigma_2)$. - **Label**: Human indicates preference: $sigma_1 succ sigma_2$, $sigma_2 succ sigma_1$, or roughly equal. - **Reward Model**: Train $R(s,a)$ such that $P(sigma_1 succ sigma_2) = frac{exp(sum R(sigma_1))}{exp(sum R(sigma_1)) + exp(sum R(sigma_2))}$ (Bradley-Terry model). - **RL**: Optimize policy using the learned reward model via standard RL (PPO, SAC, etc.). **Why It Matters** - **Reward-Free**: No need to hand-craft reward functions — preferences define the objective implicitly. - **Scalable**: Preferences are faster to provide than demonstrations — binary comparison is cognitively easy. - **Active Queries**: Active learning selects the most informative trajectory pairs to query — minimizes human effort. **Preference-Based RL** is **learning rewards from comparisons** — using human preferences over behaviors to automatically derive reward functions.