Reinforcement Learning from Human Feedback RLHF is a technique aligning language models with human preferences by training a reward model from human comparisons, then optimizing model behavior via reinforcement learning — enables superior instruction-following and alignment without explicit reward specification. RLHF combines human judgment and RL optimization. Reward Model Training human raters compare model outputs (e.g., model A vs. B for same prompt). Preference pairs (output_a, output_b, winner) are labeled. Reward model (neural network) trained to predict preference: P(output_a preferred) = σ(r(output_a) - r(output_b)) where r is reward function. Bradley-Terry model for preference prediction. Quality of Annotation human annotators' consistency critical—disagreement increases label noise. Rater training, guidelines clarification, inter-rater agreement metrics (Cohen's kappa) ensure quality. Single rater vs. multiple raters (consensus labeling). Preference Signal Definition what signal to optimize for? Helpfulness, harmlessness, honesty (HHH). Multidimensional preferences: trade-offs between factors. Difficulty: specifying objectives precisely. Data Collection and Scaling collecting preferences expensive—requires human evaluation. Options: crowd workers, domain experts (higher cost, better quality), model-based preference prediction (proxy). Reinforcement Learning from Reward Model finetune pretrained LLM using policy gradient with reward signal from reward model. Prevents divergence from original pretraining via KL divergence penalty: loss = -E[reward] + β * KL(policy || pretrained). Algorithmic Approaches PPO (Proximal Policy Optimization): popular RL algorithm for language generation. GRPO (Generalized Reward optimization): simpler alternative. DPO (direct preference optimization): newer approach avoiding explicit reward model. Training Stability and Reward Hacking reward model imperfect—model learns to exploit shortcomings. Reward increase doesn't guarantee true preference improvement. KL penalty prevents excessive divergence. Distribution Shift as model improves, generating outputs outside reward model's training distribution. Reward model loses accuracy. Online learning updates reward model on new model outputs. Evaluation and Metrics human evaluation gold standard but expensive. Automatic metrics via reference models, proxy tasks. A/B testing with users. Preference Diversity different users prefer different styles. Single reward model suboptimal. Conditional reward models: condition on user preferences. Multi-objective RLHF: Pareto frontier of objectives. Practical Considerations computational cost of RLHF significant: reward modeling, RL training, human evaluation. Approximate methods (e.g., DPO) reduce cost. Constitutional AI Alternative instead of human preferences, specify principles (constitution), LLM self-critique via principle. Less human-intensive but potentially less aligned. Applications in commercial LLMs (ChatGPT, Claude, Gemini). Critical for safe, aligned deployment. RLHF enables LLMs to optimize for human preferences making models more useful and safer.