reinforcement

**Reinforcement Learning from Human Feedback RLHF** is **a technique aligning language models with human preferences by training a reward model from human comparisons, then optimizing model behavior via reinforcement learning** — enables superior instruction-following and alignment without explicit reward specification. RLHF combines human judgment and RL optimization. **Reward Model Training** human raters compare model outputs (e.g., model A vs. B for same prompt). Preference pairs (output_a, output_b, winner) are labeled. Reward model (neural network) trained to predict preference: P(output_a preferred) = σ(r(output_a) - r(output_b)) where r is reward function. Bradley-Terry model for preference prediction. **Quality of Annotation** human annotators' consistency critical—disagreement increases label noise. Rater training, guidelines clarification, inter-rater agreement metrics (Cohen's kappa) ensure quality. Single rater vs. multiple raters (consensus labeling). **Preference Signal Definition** what signal to optimize for? Helpfulness, harmlessness, honesty (HHH). Multidimensional preferences: trade-offs between factors. Difficulty: specifying objectives precisely. **Data Collection and Scaling** collecting preferences expensive—requires human evaluation. Options: crowd workers, domain experts (higher cost, better quality), model-based preference prediction (proxy). **Reinforcement Learning from Reward Model** finetune pretrained LLM using policy gradient with reward signal from reward model. Prevents divergence from original pretraining via KL divergence penalty: loss = -E[reward] + β * KL(policy || pretrained). **Algorithmic Approaches** PPO (Proximal Policy Optimization): popular RL algorithm for language generation. GRPO (Generalized Reward optimization): simpler alternative. DPO (direct preference optimization): newer approach avoiding explicit reward model. **Training Stability and Reward Hacking** reward model imperfect—model learns to exploit shortcomings. Reward increase doesn't guarantee true preference improvement. KL penalty prevents excessive divergence. **Distribution Shift** as model improves, generating outputs outside reward model's training distribution. Reward model loses accuracy. Online learning updates reward model on new model outputs. **Evaluation and Metrics** human evaluation gold standard but expensive. Automatic metrics via reference models, proxy tasks. A/B testing with users. **Preference Diversity** different users prefer different styles. Single reward model suboptimal. Conditional reward models: condition on user preferences. Multi-objective RLHF: Pareto frontier of objectives. **Practical Considerations** computational cost of RLHF significant: reward modeling, RL training, human evaluation. Approximate methods (e.g., DPO) reduce cost. **Constitutional AI Alternative** instead of human preferences, specify principles (constitution), LLM self-critique via principle. Less human-intensive but potentially less aligned. **Applications** in commercial LLMs (ChatGPT, Claude, Gemini). Critical for safe, aligned deployment. **RLHF enables LLMs to optimize for human preferences** making models more useful and safer.