reward model rlhf

**Reinforcement Learning from Human Feedback (RLHF)** is the **training methodology that aligns large language models with human preferences and values — using a reward model trained on human comparison data to score model outputs, then optimizing the language model to maximize this reward via reinforcement learning (PPO) or direct preference optimization (DPO), transforming raw pretrained models that predict the next token into helpful, harmless, and honest assistants that follow instructions and refuse harmful requests**. **The Alignment Problem** A pretrained LLM maximizes P(next token | context) — it models human text, including helpful answers, toxic rants, misinformation, and everything else. RLHF steers the model toward producing specifically helpful and safe outputs, not just likely text. **Three-Stage Pipeline** **Stage 1 — Supervised Fine-Tuning (SFT)**: - Fine-tune the pretrained LLM on a dataset of (instruction, high-quality response) pairs. Typically 10K-100K examples, often written by human annotators. - Produces a model that follows instructions but may still generate harmful, verbose, or unhelpful content. **Stage 2 — Reward Model Training**: - Collect comparison data: for each prompt, generate K responses (K=4-8), have human annotators rank them from best to worst. - Train a reward model (initialized from the SFT model, with a scalar output head) to predict the human preference: R(prompt, response) → scalar score. - Loss: Bradley-Terry model — for preferred response y_w and dispreferred y_l: L = -log(σ(R(x, y_w) - R(x, y_l))). Trains the reward model to score preferred responses higher. - Scale: InstructGPP used 33K comparison data points. ChatGPT's RLHF used significantly more. **Stage 3 — RL Optimization (PPO)**: - The language model is the RL policy. For each prompt, generate a response, score it with the reward model, update the policy to increase reward. - PPO (Proximal Policy Optimization): clips the policy gradient to prevent large updates. KL penalty: distance between the RL policy and the original SFT model is penalized — prevents reward hacking (exploiting reward model weaknesses at the expense of coherent language). - Objective: maximize E[R(x, y)] - β × KL(π || π_ref), where π is the RL policy and π_ref is the SFT reference model. **Direct Preference Optimization (DPO)** Bypasses the reward model entirely: - Derives a closed-form relationship between the optimal policy and the human preferences. - Loss: L = -log(σ(β × (log π(y_w|x)/π_ref(y_w|x) - log π(y_l|x)/π_ref(y_l|x)))). Directly optimizes the policy on preference data. - Simpler pipeline (no separate reward model training, no RL loop), more stable training, comparable performance to PPO-based RLHF. - Used by LLaMA 2, Zephyr, Mistral, and many open-source aligned models. **Challenges** - **Reward Hacking**: RL policy discovers outputs that score high with the reward model but are meaninglessly repetitive, excessively verbose, or otherwise low-quality. Mitigated by KL constraint and reward model iteration. - **Annotation Quality**: Human preferences are noisy, inconsistent, and influenced by biases. Inter-annotator agreement is typically 70-80%. Constitutional AI (Anthropic) uses AI feedback instead of human feedback for scaling. - **Alignment Tax**: RLHF slightly reduces raw capability (helpfulness-harmlessness trade-off). The model becomes more cautious, occasionally refusing valid requests. RLHF is **the alignment technology that transformed language models from text completion engines into controllable AI assistants** — providing the mechanism to steer model behavior toward human values, safety, and helpfulness at scale.