learning from human feedback

**RLHF** (Reinforcement Learning from Human Feedback) is the **technique of training AI models using human preferences as the reward signal** — instead of a hand-crafted reward function, humans compare model outputs, these preferences train a reward model, and the reward model guides RL-based policy optimization. **RLHF Pipeline** - **SFT**: Supervised Fine-Tuning on curated demonstrations — baseline model. - **Reward Model**: Train a reward model $R(x, y)$ on human preference comparisons: "output A is better than output B." - **RL Fine-Tuning**: Optimize the SFT model with PPO to maximize the learned reward $R$, with a KL penalty to stay near SFT. - **Iteration**: Collect more preferences on the RL-tuned model, retrain reward model, re-optimize. **Why It Matters** - **Alignment**: RLHF aligns AI behavior with human values and preferences — the key technique behind ChatGPT, Claude. - **Beyond Demonstrations**: Preferences are easier to provide than demonstrations — comparing is easier than generating. - **LLMs**: RLHF transformed language models from next-word predictors into helpful, harmless assistants. **RLHF** is **aligning AI with human preferences** — using human comparisons to create a reward signal for training helpful, safe AI systems.