rlhf
**RLHF (Reinforcement Learning from Human Feedback)** is a **training methodology that aligns LLMs with human preferences by training a reward model on human comparisons and optimizing the LLM policy with RL** — the technique behind ChatGPT and most deployed aligned models.
**RLHF Pipeline**
**Phase 1 — Supervised Fine-Tuning (SFT)**:
- Fine-tune the pretrained LLM on high-quality human-written demonstrations.
- Creates a reasonable starting point for preference learning.
**Phase 2 — Reward Model Training**:
- Collect preference data: Show human raters two LLM responses to the same prompt.
- Raters choose which response is better (helpful, harmless, honest).
- Train a reward model $r_\phi$ to predict which response humans prefer.
- Reward model: Same LLM backbone + regression head.
**Phase 3 — RL Optimization (PPO)**:
- Use PPO to update the LLM policy to maximize $r_\phi$ score.
- KL penalty: $r_{\text{total}} = r_\phi(x,y) - \beta \cdot KL(\pi_\theta || \pi_{SFT})$
- KL term prevents the model from drifting too far from SFT behavior ("reward hacking").
**Why RLHF Works**
- Human preferences capture things hard to specify as a loss: helpfulness, tone, safety, nuance.
- Enables models to learn "be helpful but not harmful" holistically.
- InstructGPT (RLHF) dramatically outperformed 100x larger GPT-3 on human preference evaluations.
**Challenges**
- Expensive: Requires large-scale human annotation.
- Reward hacking: Models find ways to score high without being genuinely helpful.
- PPO instability: Training is sensitive to hyperparameters.
- Preference noise: Human raters disagree, labels are noisy.
RLHF is **the alignment technique that made LLMs genuinely useful and safe for broad deployment** — it transformed raw language models into helpful assistants.