<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">RLHF — Aligning LLMs with Human Preferences</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">pretrain → supervised fine-tune → train reward model on human comparisons → optimize policy with PPO</text>

  <!-- === TOP: Three-stage pipeline === -->
  <rect x="25" y="62" width="710" height="120" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="380" y="80" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">The RLHF Pipeline — Three Training Stages</text>

  <!-- Stage 1: SFT -->
  <rect x="40" y="92" width="145" height="55" rx="5" fill="#0f1a2a" stroke="#60a5fa" stroke-width="1"/>
  <text x="112" y="110" fill="#93c5fd" font-size="9" text-anchor="middle" font-weight="600">Stage 1: SFT</text>
  <text x="112" y="124" fill="#8b98a5" font-size="8" text-anchor="middle">fine-tune base model on</text>
  <text x="112" y="136" fill="#8b98a5" font-size="8" text-anchor="middle">human demonstrations</text>
  <text x="112" y="150" fill="#6b7684" font-size="7.5" text-anchor="middle">→ π_SFT</text>

  <!-- Arrow -->
  <path d="M188,120 L212,120" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="210,117 216,120 210,123" fill="#8b98a5"/>

  <!-- Stage 2: Reward Model -->
  <rect x="218" y="92" width="180" height="55" rx="5" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1"/>
  <text x="308" y="110" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="600">Stage 2: Reward Model</text>
  <text x="308" y="124" fill="#8b98a5" font-size="8" text-anchor="middle">train on human comparisons (A ≻ B)</text>
  <text x="308" y="136" fill="#8b98a5" font-size="8" text-anchor="middle">outputs scalar reward r(x, y)</text>
  <text x="308" y="150" fill="#6b7684" font-size="7.5" text-anchor="middle">Bradley-Terry model: P(A≻B) = σ(r_A - r_B)</text>

  <!-- Arrow -->
  <path d="M401,120 L425,120" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="423,117 429,120 423,123" fill="#8b98a5"/>

  <!-- Stage 3: PPO -->
  <rect x="431" y="92" width="175" height="55" rx="5" fill="#14261f" stroke="#34d399" stroke-width="1"/>
  <text x="518" y="110" fill="#6ee7b7" font-size="9" text-anchor="middle" font-weight="600">Stage 3: RL (PPO)</text>
  <text x="518" y="124" fill="#8b98a5" font-size="8" text-anchor="middle">maximize E[r(x,y)] with KL penalty</text>
  <text x="518" y="136" fill="#8b98a5" font-size="8" text-anchor="middle">π* = argmax r(x,y) - β·KL(π‖π_SFT)</text>
  <text x="518" y="150" fill="#6b7684" font-size="7.5" text-anchor="middle">keep policy near SFT to avoid reward hacking</text>

  <!-- Final output -->
  <path d="M609,120 L633,120" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="631,117 637,120 631,123" fill="#8b98a5"/>
  <rect x="639" y="104" width="55" height="30" rx="4" fill="#1a1520" stroke="#a78bfa" stroke-width="0.9"/>
  <text x="666" y="122" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="600">π_RLHF</text>

  <!-- === MIDDLE LEFT: PPO optimization loop === -->
  <rect x="25" y="192" width="345" height="130" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="197" y="210" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">PPO Training Loop (one step)</text>

  <text x="45" y="230" fill="#60a5fa" font-size="8.5">1. Sample prompt x from dataset</text>
  <text x="45" y="246" fill="#60a5fa" font-size="8.5">2. Generate response y ~ π(·|x)</text>
  <text x="45" y="262" fill="#fbbf24" font-size="8.5">3. Score: R = r(x,y) - β·KL(π‖π_ref)</text>
  <text x="45" y="278" fill="#34d399" font-size="8.5">4. Compute advantage A_t (GAE-λ)</text>
  <text x="45" y="294" fill="#c4b5fd" font-size="8.5">5. PPO clip update: clip(ratio, 1±ε) · A_t</text>
  <text x="45" y="310" fill="#f87171" font-size="8.5">6. Value head update: L_V = (V - R_target)²</text>

  <!-- === MIDDLE RIGHT: Alternatives === -->
  <rect x="385" y="192" width="350" height="130" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="560" y="210" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">RLHF vs Alternatives</text>

  <text x="405" y="232" fill="#34d399" font-size="9" font-weight="600">DPO (Direct Preference)</text>
  <text x="405" y="246" fill="#8b98a5" font-size="8.5">no reward model — optimize preferences directly</text>
  <text x="405" y="258" fill="#6b7684" font-size="8">L = -log σ(β(log π/π_ref for chosen - rejected))</text>

  <text x="405" y="278" fill="#fbbf24" font-size="9" font-weight="600">RLAIF (AI Feedback)</text>
  <text x="405" y="292" fill="#8b98a5" font-size="8.5">replace human annotators with LLM judge</text>

  <text x="405" y="312" fill="#c4b5fd" font-size="9" font-weight="600">Constitutional AI</text>
  <text x="405" y="324" fill="#8b98a5" font-size="8.5">self-critique against written principles</text>

  <!-- === BOTTOM: Challenges and models === -->
  <rect x="25" y="332" width="710" height="72" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="350" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Challenges and Failure Modes</text>

  <text x="130" y="372" fill="#f87171" font-size="9" text-anchor="middle" font-weight="600">Reward Hacking</text>
  <text x="130" y="386" fill="#8b98a5" font-size="8" text-anchor="middle">policy exploits reward proxy</text>
  <text x="130" y="398" fill="#6b7684" font-size="7.5" text-anchor="middle">verbose but empty responses</text>

  <text x="310" y="372" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="600">Mode Collapse</text>
  <text x="310" y="386" fill="#8b98a5" font-size="8" text-anchor="middle">diversity decreases</text>
  <text x="310" y="398" fill="#6b7684" font-size="7.5" text-anchor="middle">KL penalty mitigates</text>

  <text x="490" y="372" fill="#60a5fa" font-size="9" text-anchor="middle" font-weight="600">Annotation Cost</text>
  <text x="490" y="386" fill="#8b98a5" font-size="8" text-anchor="middle">human labels expensive</text>
  <text x="490" y="398" fill="#6b7684" font-size="7.5" text-anchor="middle">→ DPO, RLAIF reduce cost</text>

  <text x="650" y="372" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="600">Instability</text>
  <text x="650" y="386" fill="#8b98a5" font-size="8" text-anchor="middle">PPO requires 4 models in RAM</text>
  <text x="650" y="398" fill="#6b7684" font-size="7.5" text-anchor="middle">policy, ref, reward, value</text>

  <!-- Models that use RLHF -->
  <rect x="25" y="412" width="710" height="26" rx="4" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="429" fill="#fbbf24" font-size="9.5" text-anchor="middle">Used by: ChatGPT (PPO) · Claude (RLHF+Constitutional) · Llama-3 (DPO+RLHF) · Gemini (RLHF) · DeepSeek (GRPO)</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">RLHF closes the gap between "next-token prediction" and "helpful, harmless, honest" — alignment is the final training stage.</text>
</svg>
```inforcement Learning from Human Feedback (RLHF)** is the alignment technique that turned raw language models into usable assistants. A pretrained model is fluent but aimless — it predicts plausible next tokens without any sense of which responses are helpful, honest, or safe. RLHF fixes that by learning a model of human preference and then optimizing the language model against it. It is the method behind the "instruct" and "chat" versions of most frontier models, and the reason they follow instructions and refuse harmful requests instead of merely autocompleting.\n\n```svg\n<svg viewBox="0 0 820 560" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">\n  <rect x="0" y="0" width="820" height="560" fill="#0d1117"/>\n  <text x="410" y="36" fill="#e6edf3" font-size="22" font-weight="700" text-anchor="middle">RLHF — Turning Human Preference into a Training Signal</text>\n  <text x="410" y="60" fill="#8b98a5" font-size="13" text-anchor="middle">a base model knows how to predict text; RLHF teaches it which answers people actually want</text>\n  <rect x="40" y="88" width="150" height="58" rx="8" fill="#0f1c17" stroke="#34d399" stroke-width="1.6"/>\n  <text x="115" y="112" fill="#6ee7b7" font-size="13.5" font-weight="700" text-anchor="middle">Pretrained</text>\n  <text x="115" y="132" fill="#8b98a5" font-size="13" text-anchor="middle">base LLM</text>\n  <rect x="232" y="88" width="150" height="58" rx="8" fill="#0f1c17" stroke="#60a5fa" stroke-width="1.6"/>\n  <text x="307" y="112" fill="#93c5fd" font-size="13.5" font-weight="700" text-anchor="middle">1. SFT</text>\n  <text x="307" y="132" fill="#8b98a5" font-size="13" text-anchor="middle">demo answers</text>\n  <rect x="424" y="88" width="176" height="58" rx="8" fill="#1a1226" stroke="#c084fc" stroke-width="1.6"/>\n  <text x="512" y="112" fill="#d8b4fe" font-size="13.5" font-weight="700" text-anchor="middle">2. Reward Model</text>\n  <text x="512" y="132" fill="#8b98a5" font-size="13" text-anchor="middle">learns human taste</text>\n  <rect x="632" y="88" width="150" height="58" rx="8" fill="#2a1420" stroke="#fbbf24" stroke-width="1.6"/>\n  <text x="707" y="112" fill="#fcd34d" font-size="13.5" font-weight="700" text-anchor="middle">3. RL / PPO</text>\n  <text x="707" y="132" fill="#8b98a5" font-size="13" text-anchor="middle">optimize reward</text>\n  <line x1="190" y1="117.0" x2="232" y2="117.0" stroke="#3f5169" stroke-width="2.0"/>\n  <path d="M224 121 L232 117 L224 113" stroke="#3f5169" stroke-width="2.0" fill="none"/>\n  <line x1="382" y1="117.0" x2="424" y2="117.0" stroke="#3f5169" stroke-width="2.0"/>\n  <path d="M416 121 L424 117 L416 113" stroke="#3f5169" stroke-width="2.0" fill="none"/>\n  <line x1="600" y1="117.0" x2="632" y2="117.0" stroke="#3f5169" stroke-width="2.0"/>\n  <path d="M624 121 L632 117 L624 113" stroke="#3f5169" stroke-width="2.0" fill="none"/>\n  <line x1="707" y1="146" x2="707" y2="180" stroke="#3f5169" stroke-width="2.0"/>\n  <path d="M703 172 L707 180 L711 172" stroke="#3f5169" stroke-width="2.0" fill="none"/>\n  <rect x="617" y="180" width="180" height="54" rx="8" fill="#0f1c17" stroke="#34d399" stroke-width="1.6"/>\n  <text x="707" y="204" fill="#6ee7b7" font-size="13.5" font-weight="700" text-anchor="middle">Aligned model</text>\n  <text x="707" y="224" fill="#8b98a5" font-size="13" text-anchor="middle">helpful + harmless</text>\n  <rect x="40" y="262" width="372" height="206" rx="8" fill="#0b1220" stroke="#233043" stroke-width="1.2"/>\n  <text x="58" y="288" fill="#cbd5e1" font-size="13" font-weight="700">How the reward model learns</text>\n  <text x="58" y="308" fill="#8b98a5" font-size="12">Same prompt, two answers — a human picks the better one.</text>\n  <rect x="58" y="322" width="132" height="34" rx="5" fill="#151b23" stroke="#3f5169" stroke-width="1"/>\n  <text x="124" y="344" fill="#c9d4e0" font-size="13" text-anchor="middle">prompt</text>\n  <rect x="222" y="316" width="168" height="30" rx="5" fill="#0f1c17" stroke="#34d399" stroke-width="1.3"/>\n  <text x="238" y="336" fill="#6ee7b7" font-size="13" text-anchor="start">answer A   ✓ chosen</text>\n  <rect x="222" y="356" width="168" height="30" rx="5" fill="#1a0f14" stroke="#6b7684" stroke-width="1"/>\n  <text x="238" y="376" fill="#8b98a5" font-size="13" text-anchor="start">answer B   ✕ rejected</text>\n  <line x1="190" y1="339" x2="222" y2="331" stroke="#3f5169" stroke-width="1.4"/>\n  <path d="M215 336 L222 331 L213 330" stroke="#3f5169" stroke-width="1.4" fill="none"/>\n  <line x1="190" y1="339" x2="222" y2="371" stroke="#3f5169" stroke-width="1.4"/>\n  <path d="M214 368 L222 371 L219 363" stroke="#3f5169" stroke-width="1.4" fill="none"/>\n  <text x="58" y="418" fill="#93c5fd" font-size="13" font-family="ui-monospace,monospace">loss = -log σ( r(A) − r(B) )</text>\n  <text x="58" y="442" fill="#6b7684" font-size="12">score the chosen answer above the rejected one</text>\n  <rect x="432" y="262" width="348" height="206" rx="8" fill="#0b1220" stroke="#233043" stroke-width="1.2"/>\n  <text x="450" y="288" fill="#cbd5e1" font-size="13" font-weight="700">The RL loop, on a leash</text>\n  <rect x="452" y="314" width="124" height="34" rx="5" fill="#2a1420" stroke="#fbbf24" stroke-width="1.2"/>\n  <text x="514" y="336" fill="#fcd34d" font-size="13" text-anchor="middle">policy (LLM)</text>\n  <rect x="636" y="314" width="130" height="34" rx="5" fill="#1a1226" stroke="#c084fc" stroke-width="1.2"/>\n  <text x="701" y="336" fill="#d8b4fe" font-size="13" text-anchor="middle">reward model</text>\n  <line x1="576" y1="331" x2="636" y2="331" stroke="#3f5169" stroke-width="1.4"/>\n  <path d="M628 335 L636 331 L628 327" stroke="#3f5169" stroke-width="1.4" fill="none"/>\n  <text x="606" y="308" fill="#8b98a5" font-size="12" text-anchor="middle">answer</text>\n  <path d="M701 348 L701 366 L514 366 L514 348" stroke="#3f5169" stroke-width="1.4" fill="none"/>\n  <line x1="516" y1="362" x2="514" y2="350" stroke="#3f5169" stroke-width="1.4"/>\n  <path d="M519 358 L514 350 L512 359" stroke="#3f5169" stroke-width="1.4" fill="none"/>\n  <text x="607" y="382" fill="#8b98a5" font-size="12" text-anchor="middle">reward signal → update policy</text>\n  <text x="764" y="392" fill="#6b7684" font-size="13" text-anchor="end">anti-drift leash</text>\n  <rect x="452" y="398" width="314" height="36" rx="5" fill="#151b23" stroke="#f87171" stroke-width="1.2"/>\n  <text x="609" y="421" fill="#fca5a5" font-size="13" font-family="ui-monospace,monospace" text-anchor="middle">− β · KL( policy ‖ frozen reference )</text>\n  <rect x="40" y="490" width="740" height="52" rx="6" fill="#101826" stroke="#38bdf8" stroke-width="1.2"/>\n  <text x="58" y="521" fill="#7dd3fc" font-size="12.5" font-weight="700">DPO shortcut:</text>\n  <text x="164" y="512" fill="#9fb4c6" font-size="13">skip the separate reward model and RL loop — train the language model</text>\n  <text x="164" y="531" fill="#9fb4c6" font-size="13">directly on the chosen/rejected pairs with one classification-style loss.</text>\n</svg>\n```\n\n**Stage one is supervised fine-tuning (SFT).** Human contractors write high-quality example answers to a range of prompts, and the base model is fine-tuned to imitate them. This alone gets the model into the neighborhood of helpful behavior — it now answers questions rather than continuing them — but imitation has a ceiling: humans cannot demonstrate the best possible answer to every prompt, and writing demonstrations is slow and expensive.\n\n**Stage two trains a reward model from comparisons, not demonstrations.** Instead of writing ideal answers, humans are shown two model outputs for the same prompt and simply pick the better one. Preference judgments are far cheaper and more reliable than authored answers. A separate reward model is trained on these pairs to output a scalar score, using a loss that pushes the chosen answer's score above the rejected one. The reward model becomes a learned, automatable stand-in for human taste.\n\n**Stage three optimizes the policy with reinforcement learning, usually PPO.** The language model (now the "policy") generates answers, the reward model scores them, and the score is used as a reward signal to update the policy toward higher-scoring outputs. Crucially, a KL-divergence penalty tethers the policy to the original reference model so it cannot drift into degenerate text that games the reward. This leash is what keeps RLHF stable.\n\n**Reward hacking is the central failure mode.** Because the policy optimizes the reward model rather than true human preference, it will exploit any gap between them — becoming sycophantic, verbose, or confidently wrong in ways the reward model happens to score highly. Managing this trade-off, sometimes called the alignment tax (aligned models can lose a little raw capability), is much of the practical craft of RLHF.\n\n**DPO and its relatives simplify the pipeline.** Direct Preference Optimization skips the separate reward model and RL loop entirely, deriving a loss that trains the language model directly on the chosen/rejected pairs. It is far simpler and cheaper to run and has become a popular default, though PPO-style RLHF still tends to reach the highest quality at the frontier. RLAIF replaces human labels with AI-generated preferences to scale the data further.\n\n| Stage | Data it needs | What it produces | Main risk |\n|---|---|---|---|\n| SFT | human-written answers | a model that follows instructions | limited by demonstration quality |\n| Reward model | human A-vs-B preferences | a scalar "human taste" scorer | mislabeled or noisy preferences |\n| PPO / RL | prompts + reward model | a preference-optimized policy | reward hacking, drift |\n| DPO (alt.) | the preference pairs directly | aligned model, no RM or RL loop | slightly lower ceiling than PPO |\n\nRead RLHF through a *preference-signal* lens rather than a *teach-it-the-answer* lens: the breakthrough is not that humans show the model what to say, but that humans only have to say which of two answers is better, and that cheap comparative signal is amplified — first into a reward model, then into a full optimization objective — until it reshapes a fluent-but-aimless predictor into an assistant that reliably does what people want.\n