<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">RLHF — Aligning LLMs with Human Preferences</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">3 stages: SFT → Reward Model → PPO — turns a base model into a helpful, harmless assistant</text>

  <!-- === THREE STAGES (left to right) === -->
  <text x="380" y="74" fill="#e6edf3" font-size="12" font-weight="600" text-anchor="middle">The RLHF Pipeline</text>

  <!-- Stage 1: SFT -->
  <rect x="30" y="86" width="210" height="130" rx="6" fill="#080d14" stroke="#34d399" stroke-width="1.4"/>
  <text x="135" y="104" fill="#6ee7b7" font-size="11" text-anchor="middle" font-weight="600">Stage 1: SFT</text>
  <text x="135" y="118" fill="#8b98a5" font-size="8.5" text-anchor="middle">supervised fine-tuning</text>

  <!-- Human demonstrations -->
  <rect x="45" y="128" width="85" height="30" rx="3" fill="#14261f" stroke="#2f6d55" stroke-width="0.8"/>
  <text x="87" y="142" fill="#6ee7b7" font-size="8" text-anchor="middle">Human demos</text>
  <text x="87" y="153" fill="#6b7684" font-size="7" text-anchor="middle">(prompt, ideal answer)</text>

  <!-- Arrow -->
  <path d="M132,143 L148,143" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="146,140 152,143 146,146" fill="#8b98a5"/>

  <!-- Fine-tuned model -->
  <rect x="154" y="128" width="72" height="30" rx="3" fill="#1c1633" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="190" y="142" fill="#c4b5fd" font-size="8" text-anchor="middle">Base LLM</text>
  <text x="190" y="153" fill="#8b98a5" font-size="7" text-anchor="middle">→ SFT model</text>

  <text x="135" y="178" fill="#8b98a5" font-size="8" text-anchor="middle">train on ~100K examples</text>
  <text x="135" y="192" fill="#6b7684" font-size="7.5" text-anchor="middle">learn format + following instructions</text>
  <text x="135" y="206" fill="#6b7684" font-size="7.5" text-anchor="middle">output: π_SFT</text>

  <!-- Stage 2: Reward Model -->
  <rect x="255" y="86" width="220" height="130" rx="6" fill="#080d14" stroke="#f59e0b" stroke-width="1.4"/>
  <text x="365" y="104" fill="#fbbf24" font-size="11" text-anchor="middle" font-weight="600">Stage 2: Reward Model</text>
  <text x="365" y="118" fill="#8b98a5" font-size="8.5" text-anchor="middle">learn human preferences</text>

  <!-- Comparison pairs -->
  <rect x="270" y="128" width="60" height="40" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.8"/>
  <text x="300" y="143" fill="#fbbf24" font-size="7.5" text-anchor="middle">Response A</text>
  <text x="300" y="155" fill="#34d399" font-size="7" text-anchor="middle">✓ preferred</text>
  <text x="300" y="166" fill="#6b7684" font-size="6.5" text-anchor="middle">(human picks)</text>

  <rect x="340" y="128" width="60" height="40" rx="3" fill="#2a1a0a" stroke="#6b7684" stroke-width="0.8"/>
  <text x="370" y="143" fill="#8b98a5" font-size="7.5" text-anchor="middle">Response B</text>
  <text x="370" y="155" fill="#f87171" font-size="7" text-anchor="middle">✗ rejected</text>

  <!-- Arrow to RM -->
  <path d="M402,148 L418,148" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="416,145 422,148 416,151" fill="#8b98a5"/>

  <!-- RM output -->
  <rect x="424" y="132" width="40" height="32" rx="3" fill="#2a1a0a" stroke="#fbbf24" stroke-width="1"/>
  <text x="444" y="147" fill="#fbbf24" font-size="8" text-anchor="middle">RM</text>
  <text x="444" y="160" fill="#8b98a5" font-size="7" text-anchor="middle">score</text>

  <text x="365" y="188" fill="#8b98a5" font-size="8" text-anchor="middle">train on ~300K human comparisons</text>
  <text x="365" y="202" fill="#6b7684" font-size="7.5" text-anchor="middle">Bradley-Terry model: P(A≻B) = σ(r_A - r_B)</text>
  <text x="365" y="216" fill="#6b7684" font-size="7.5" text-anchor="middle">output: reward function r(prompt, response)</text>

  <!-- Stage 3: PPO / RL -->
  <rect x="490" y="86" width="240" height="130" rx="6" fill="#080d14" stroke="#a78bfa" stroke-width="1.4"/>
  <text x="610" y="104" fill="#c4b5fd" font-size="11" text-anchor="middle" font-weight="600">Stage 3: RL (PPO)</text>
  <text x="610" y="118" fill="#8b98a5" font-size="8.5" text-anchor="middle">optimize policy against reward</text>

  <!-- RL loop diagram -->
  <rect x="510" y="130" width="55" height="28" rx="3" fill="#1c1633" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="537" y="148" fill="#c4b5fd" font-size="8" text-anchor="middle">Policy π</text>

  <path d="M567,144 L588,144" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <text x="578" y="138" fill="#8b98a5" font-size="6.5" text-anchor="middle">gen</text>

  <rect x="590" y="130" width="55" height="28" rx="3" fill="#14202c" stroke="#8b98a5" stroke-width="0.8"/>
  <text x="617" y="148" fill="#8b98a5" font-size="8" text-anchor="middle">response</text>

  <path d="M647,144 L668,144" fill="none" stroke="#8b98a5" stroke-width="0.8"/>

  <rect x="670" y="130" width="45" height="28" rx="3" fill="#2a1a0a" stroke="#fbbf24" stroke-width="0.8"/>
  <text x="692" y="148" fill="#fbbf24" font-size="8" text-anchor="middle">RM</text>

  <!-- Reward signal back -->
  <path d="M692,158 C692,178 537,178 537,158" fill="none" stroke="#f87171" stroke-width="1.2" stroke-dasharray="3,2"/>
  <polygon points="539,160 535,155 541,155" fill="#f87171"/>
  <text x="615" y="175" fill="#f87171" font-size="7.5" text-anchor="middle">reward signal → update π</text>

  <text x="610" y="196" fill="#8b98a5" font-size="8" text-anchor="middle">KL penalty: stay near π_SFT</text>
  <text x="610" y="210" fill="#6b7684" font-size="7.5" text-anchor="middle">maximize E[r(x,y)] - β·KL(π||π_SFT)</text>

  <!-- === BOTTOM LEFT: DPO alternative === -->
  <rect x="30" y="228" width="340" height="95" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="246" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">DPO — The Simpler Alternative (skip RM+PPO)</text>

  <text x="50" y="268" fill="#34d399" font-size="9">Direct Preference Optimization:</text>
  <text x="50" y="284" fill="#8b98a5" font-size="8.5">Skip reward model entirely — train policy directly on preferences</text>
  <text x="50" y="300" fill="#8b98a5" font-size="8.5">loss = -log σ(β · (log π(y_w) - log π(y_l) - log π_ref(y_w) + log π_ref(y_l)))</text>
  <text x="50" y="318" fill="#6b7684" font-size="8">Used by: Llama-3, Zephyr, Qwen-2.5, Gemma | Simpler, more stable, similar quality</text>

  <!-- === BOTTOM RIGHT: What it produces === -->
  <rect x="385" y="228" width="345" height="95" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="557" y="246" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">What Alignment Achieves</text>

  <text x="405" y="266" fill="#34d399" font-size="9">✓ Follow instructions precisely</text>
  <text x="405" y="282" fill="#34d399" font-size="9">✓ Refuse harmful requests</text>
  <text x="405" y="298" fill="#34d399" font-size="9">✓ Admit uncertainty ("I don't know")</text>
  <text x="405" y="314" fill="#34d399" font-size="9">✓ Format answers helpfully</text>
  <text x="620" y="266" fill="#f87171" font-size="9">Risks:</text>
  <text x="620" y="282" fill="#8b98a5" font-size="8.5">sycophancy</text>
  <text x="620" y="296" fill="#8b98a5" font-size="8.5">reward hacking</text>
  <text x="620" y="310" fill="#8b98a5" font-size="8.5">over-refusal</text>

  <!-- === BOTTOM: Frontier methods === -->
  <rect x="30" y="335" width="700" height="80" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="353" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Beyond RLHF (2024-25)</text>

  <text x="50" y="373" fill="#c4b5fd" font-size="9">RLAIF: AI-generated preferences (Constitutional AI)</text>
  <text x="50" y="389" fill="#60a5fa" font-size="9">Process Reward Models (PRM): reward each reasoning step</text>
  <text x="50" y="405" fill="#fbbf24" font-size="9">RLHF + RL on reasoning (o1): reward correct chains-of-thought</text>
  <text x="440" y="373" fill="#34d399" font-size="9">Online DPO: iterative preference collection</text>
  <text x="440" y="389" fill="#8b98a5" font-size="9">SPIN: self-play (model generates its own prefs)</text>
  <text x="440" y="405" fill="#6b7684" font-size="8.5">Trend: RLHF cost → 5-15% of pretraining compute</text>

  <!-- Scale -->
  <text x="380" y="430" fill="#e6edf3" font-size="9.5" text-anchor="middle" font-weight="600">InstructGPT (2022): first RLHF product. ChatGPT: RLHF made GPT-3.5 conversational overnight.</text>
  <text x="380" y="446" fill="#8b98a5" font-size="9" text-anchor="middle">Without alignment: brilliant but uncontrollable. With alignment: the difference between a model and a product.</text>

  <text x="380" y="464" fill="#6b7684" font-size="11" text-anchor="middle">RLHF bridges the gap between "can do" and "should do" — it's how raw intelligence becomes a useful tool.</text>
</svg>
```inforcement Learning from Human Feedback (RLHF)** is the alignment technique that turned raw language models into usable assistants. A pretrained model is fluent but aimless — it predicts plausible next tokens without any sense of which responses are helpful, honest, or safe. RLHF fixes that by learning a model of human preference and then optimizing the language model against it. It is the method behind the "instruct" and "chat" versions of most frontier models, and the reason they follow instructions and refuse harmful requests instead of merely autocompleting.\n\n```svg\n<svg viewBox="0 0 820 560" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">\n  <rect x="0" y="0" width="820" height="560" fill="#0d1117"/>\n  <text x="410" y="36" fill="#e6edf3" font-size="22" font-weight="700" text-anchor="middle">RLHF — Turning Human Preference into a Training Signal</text>\n  <text x="410" y="60" fill="#8b98a5" font-size="13" text-anchor="middle">a base model knows how to predict text; RLHF teaches it which answers people actually want</text>\n  <rect x="40" y="88" width="150" height="58" rx="8" fill="#0f1c17" stroke="#34d399" stroke-width="1.6"/>\n  <text x="115" y="112" fill="#6ee7b7" font-size="13.5" font-weight="700" text-anchor="middle">Pretrained</text>\n  <text x="115" y="132" fill="#8b98a5" font-size="13" text-anchor="middle">base LLM</text>\n  <rect x="232" y="88" width="150" height="58" rx="8" fill="#0f1c17" stroke="#60a5fa" stroke-width="1.6"/>\n  <text x="307" y="112" fill="#93c5fd" font-size="13.5" font-weight="700" text-anchor="middle">1. SFT</text>\n  <text x="307" y="132" fill="#8b98a5" font-size="13" text-anchor="middle">demo answers</text>\n  <rect x="424" y="88" width="176" height="58" rx="8" fill="#1a1226" stroke="#c084fc" stroke-width="1.6"/>\n  <text x="512" y="112" fill="#d8b4fe" font-size="13.5" font-weight="700" text-anchor="middle">2. Reward Model</text>\n  <text x="512" y="132" fill="#8b98a5" font-size="13" text-anchor="middle">learns human taste</text>\n  <rect x="632" y="88" width="150" height="58" rx="8" fill="#2a1420" stroke="#fbbf24" stroke-width="1.6"/>\n  <text x="707" y="112" fill="#fcd34d" font-size="13.5" font-weight="700" text-anchor="middle">3. RL / PPO</text>\n  <text x="707" y="132" fill="#8b98a5" font-size="13" text-anchor="middle">optimize reward</text>\n  <line x1="190" y1="117.0" x2="232" y2="117.0" stroke="#3f5169" stroke-width="2.0"/>\n  <path d="M224 121 L232 117 L224 113" stroke="#3f5169" stroke-width="2.0" fill="none"/>\n  <line x1="382" y1="117.0" x2="424" y2="117.0" stroke="#3f5169" stroke-width="2.0"/>\n  <path d="M416 121 L424 117 L416 113" stroke="#3f5169" stroke-width="2.0" fill="none"/>\n  <line x1="600" y1="117.0" x2="632" y2="117.0" stroke="#3f5169" stroke-width="2.0"/>\n  <path d="M624 121 L632 117 L624 113" stroke="#3f5169" stroke-width="2.0" fill="none"/>\n  <line x1="707" y1="146" x2="707" y2="180" stroke="#3f5169" stroke-width="2.0"/>\n  <path d="M703 172 L707 180 L711 172" stroke="#3f5169" stroke-width="2.0" fill="none"/>\n  <rect x="617" y="180" width="180" height="54" rx="8" fill="#0f1c17" stroke="#34d399" stroke-width="1.6"/>\n  <text x="707" y="204" fill="#6ee7b7" font-size="13.5" font-weight="700" text-anchor="middle">Aligned model</text>\n  <text x="707" y="224" fill="#8b98a5" font-size="13" text-anchor="middle">helpful + harmless</text>\n  <rect x="40" y="262" width="372" height="206" rx="8" fill="#0b1220" stroke="#233043" stroke-width="1.2"/>\n  <text x="58" y="288" fill="#cbd5e1" font-size="13" font-weight="700">How the reward model learns</text>\n  <text x="58" y="308" fill="#8b98a5" font-size="12">Same prompt, two answers — a human picks the better one.</text>\n  <rect x="58" y="322" width="132" height="34" rx="5" fill="#151b23" stroke="#3f5169" stroke-width="1"/>\n  <text x="124" y="344" fill="#c9d4e0" font-size="13" text-anchor="middle">prompt</text>\n  <rect x="222" y="316" width="168" height="30" rx="5" fill="#0f1c17" stroke="#34d399" stroke-width="1.3"/>\n  <text x="238" y="336" fill="#6ee7b7" font-size="13" text-anchor="start">answer A   ✓ chosen</text>\n  <rect x="222" y="356" width="168" height="30" rx="5" fill="#1a0f14" stroke="#6b7684" stroke-width="1"/>\n  <text x="238" y="376" fill="#8b98a5" font-size="13" text-anchor="start">answer B   ✕ rejected</text>\n  <line x1="190" y1="339" x2="222" y2="331" stroke="#3f5169" stroke-width="1.4"/>\n  <path d="M215 336 L222 331 L213 330" stroke="#3f5169" stroke-width="1.4" fill="none"/>\n  <line x1="190" y1="339" x2="222" y2="371" stroke="#3f5169" stroke-width="1.4"/>\n  <path d="M214 368 L222 371 L219 363" stroke="#3f5169" stroke-width="1.4" fill="none"/>\n  <text x="58" y="418" fill="#93c5fd" font-size="13" font-family="ui-monospace,monospace">loss = -log σ( r(A) − r(B) )</text>\n  <text x="58" y="442" fill="#6b7684" font-size="12">score the chosen answer above the rejected one</text>\n  <rect x="432" y="262" width="348" height="206" rx="8" fill="#0b1220" stroke="#233043" stroke-width="1.2"/>\n  <text x="450" y="288" fill="#cbd5e1" font-size="13" font-weight="700">The RL loop, on a leash</text>\n  <rect x="452" y="314" width="124" height="34" rx="5" fill="#2a1420" stroke="#fbbf24" stroke-width="1.2"/>\n  <text x="514" y="336" fill="#fcd34d" font-size="13" text-anchor="middle">policy (LLM)</text>\n  <rect x="636" y="314" width="130" height="34" rx="5" fill="#1a1226" stroke="#c084fc" stroke-width="1.2"/>\n  <text x="701" y="336" fill="#d8b4fe" font-size="13" text-anchor="middle">reward model</text>\n  <line x1="576" y1="331" x2="636" y2="331" stroke="#3f5169" stroke-width="1.4"/>\n  <path d="M628 335 L636 331 L628 327" stroke="#3f5169" stroke-width="1.4" fill="none"/>\n  <text x="606" y="308" fill="#8b98a5" font-size="12" text-anchor="middle">answer</text>\n  <path d="M701 348 L701 366 L514 366 L514 348" stroke="#3f5169" stroke-width="1.4" fill="none"/>\n  <line x1="516" y1="362" x2="514" y2="350" stroke="#3f5169" stroke-width="1.4"/>\n  <path d="M519 358 L514 350 L512 359" stroke="#3f5169" stroke-width="1.4" fill="none"/>\n  <text x="607" y="382" fill="#8b98a5" font-size="12" text-anchor="middle">reward signal → update policy</text>\n  <text x="764" y="392" fill="#6b7684" font-size="13" text-anchor="end">anti-drift leash</text>\n  <rect x="452" y="398" width="314" height="36" rx="5" fill="#151b23" stroke="#f87171" stroke-width="1.2"/>\n  <text x="609" y="421" fill="#fca5a5" font-size="13" font-family="ui-monospace,monospace" text-anchor="middle">− β · KL( policy ‖ frozen reference )</text>\n  <rect x="40" y="490" width="740" height="52" rx="6" fill="#101826" stroke="#38bdf8" stroke-width="1.2"/>\n  <text x="58" y="521" fill="#7dd3fc" font-size="12.5" font-weight="700">DPO shortcut:</text>\n  <text x="164" y="512" fill="#9fb4c6" font-size="13">skip the separate reward model and RL loop — train the language model</text>\n  <text x="164" y="531" fill="#9fb4c6" font-size="13">directly on the chosen/rejected pairs with one classification-style loss.</text>\n</svg>\n```\n\n**Stage one is supervised fine-tuning (SFT).** Human contractors write high-quality example answers to a range of prompts, and the base model is fine-tuned to imitate them. This alone gets the model into the neighborhood of helpful behavior — it now answers questions rather than continuing them — but imitation has a ceiling: humans cannot demonstrate the best possible answer to every prompt, and writing demonstrations is slow and expensive.\n\n**Stage two trains a reward model from comparisons, not demonstrations.** Instead of writing ideal answers, humans are shown two model outputs for the same prompt and simply pick the better one. Preference judgments are far cheaper and more reliable than authored answers. A separate reward model is trained on these pairs to output a scalar score, using a loss that pushes the chosen answer's score above the rejected one. The reward model becomes a learned, automatable stand-in for human taste.\n\n**Stage three optimizes the policy with reinforcement learning, usually PPO.** The language model (now the "policy") generates answers, the reward model scores them, and the score is used as a reward signal to update the policy toward higher-scoring outputs. Crucially, a KL-divergence penalty tethers the policy to the original reference model so it cannot drift into degenerate text that games the reward. This leash is what keeps RLHF stable.\n\n**Reward hacking is the central failure mode.** Because the policy optimizes the reward model rather than true human preference, it will exploit any gap between them — becoming sycophantic, verbose, or confidently wrong in ways the reward model happens to score highly. Managing this trade-off, sometimes called the alignment tax (aligned models can lose a little raw capability), is much of the practical craft of RLHF.\n\n**DPO and its relatives simplify the pipeline.** Direct Preference Optimization skips the separate reward model and RL loop entirely, deriving a loss that trains the language model directly on the chosen/rejected pairs. It is far simpler and cheaper to run and has become a popular default, though PPO-style RLHF still tends to reach the highest quality at the frontier. RLAIF replaces human labels with AI-generated preferences to scale the data further.\n\n| Stage | Data it needs | What it produces | Main risk |\n|---|---|---|---|\n| SFT | human-written answers | a model that follows instructions | limited by demonstration quality |\n| Reward model | human A-vs-B preferences | a scalar "human taste" scorer | mislabeled or noisy preferences |\n| PPO / RL | prompts + reward model | a preference-optimized policy | reward hacking, drift |\n| DPO (alt.) | the preference pairs directly | aligned model, no RM or RL loop | slightly lower ceiling than PPO |\n\nRead RLHF through a *preference-signal* lens rather than a *teach-it-the-answer* lens: the breakthrough is not that humans show the model what to say, but that humans only have to say which of two answers is better, and that cheap comparative signal is amplified — first into a reward model, then into a full optimization objective — until it reshapes a fluent-but-aimless predictor into an assistant that reliably does what people want.\n