<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect width="760" height="470" fill="#0d1117"/>
  <defs>
    <marker id="ppBlueArrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#60a5fa"/></marker>
    <marker id="ppGreenArrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#34d399"/></marker>
    <marker id="ppGoldArrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="6" markerHeight="6" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#fbbf24"/></marker>
    <filter id="ppGlow" x="-100%" y="-100%" width="300%" height="300%"><feGaussianBlur stdDeviation="4" result="b"/><feMerge><feMergeNode in="b"/><feMergeNode in="SourceGraphic"/></feMerge></filter>
  </defs>

  <text x="380" y="30" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Perplexity — Average Next-Token Surprise</text>
  <text x="380" y="52" fill="#8b98a5" font-size="12.5" text-anchor="middle">score each observed token with the probability assigned before seeing it, average the log loss, then exponentiate</text>

  <!-- Token-by-token scoring -->
  <g transform="translate(24 84)">
    <text x="356" y="-9" fill="#e6edf3" font-size="10.5" font-weight="700" text-anchor="middle">TEACHER FORCING · THE TRUE PREFIX IS GIVEN AT EVERY POSITION</text>
    <rect width="712" height="224" rx="13" fill="#10151b" stroke="#3a4453" stroke-width="1.5"/>

    <text x="22" y="28" fill="#8b98a5" font-size="9">evaluation sequence</text>
    <g transform="translate(21 39)">
      <rect width="84" height="34" rx="7" fill="#172238" stroke="#60a5fa"/><text x="42" y="22" fill="#93c5fd" font-size="11" font-weight="700" text-anchor="middle">The</text>
      <path d="M84 17h22" stroke="#60a5fa" stroke-width="2" marker-end="url(#ppBlueArrow)"/>
      <g transform="translate(110)"><rect width="84" height="34" rx="7" fill="#172238" stroke="#60a5fa"/><text x="42" y="22" fill="#93c5fd" font-size="11" font-weight="700" text-anchor="middle">chip</text></g>
      <path d="M194 17h22" stroke="#60a5fa" stroke-width="2" marker-end="url(#ppBlueArrow)"/>
      <g transform="translate(220)"><rect width="84" height="34" rx="7" fill="#172238" stroke="#60a5fa"/><text x="42" y="22" fill="#93c5fd" font-size="11" font-weight="700" text-anchor="middle">passes</text></g>
      <path d="M304 17h22" stroke="#60a5fa" stroke-width="2" marker-end="url(#ppBlueArrow)"/>
      <g transform="translate(330)"><rect width="84" height="34" rx="7" fill="#172238" stroke="#60a5fa"/><text x="42" y="22" fill="#93c5fd" font-size="11" font-weight="700" text-anchor="middle">final</text></g>
      <path d="M414 17h22" stroke="#60a5fa" stroke-width="2" marker-end="url(#ppBlueArrow)"/>
      <g transform="translate(440)"><rect width="84" height="34" rx="7" fill="#172238" stroke="#60a5fa"/><text x="42" y="22" fill="#93c5fd" font-size="11" font-weight="700" text-anchor="middle">test</text></g>
    </g>

    <!-- predicted distributions for actual tokens -->
    <g transform="translate(21 93)">
      <g>
        <text x="42" y="0" fill="#8b98a5" font-size="7.5" text-anchor="middle">p(The | BOS)</text>
        <rect x="5" y="10" width="74" height="13" rx="4" fill="#1f2937"/><rect x="5" y="10" width="61" height="13" rx="4" fill="#34d399"/>
        <text x="42" y="20" fill="#e6edf3" font-size="8" font-weight="700" text-anchor="middle">0.82</text>
        <text x="42" y="39" fill="#6ee7b7" font-size="8" text-anchor="middle">surprise 0.20</text>
      </g>
      <g transform="translate(110)">
        <text x="42" y="0" fill="#8b98a5" font-size="7.5" text-anchor="middle">p(chip | The)</text>
        <rect x="5" y="10" width="74" height="13" rx="4" fill="#1f2937"/><rect x="5" y="10" width="31" height="13" rx="4" fill="#fbbf24"/>
        <text x="42" y="20" fill="#e6edf3" font-size="8" font-weight="700" text-anchor="middle">0.42</text>
        <text x="42" y="39" fill="#fde68a" font-size="8" text-anchor="middle">surprise 0.87</text>
      </g>
      <g transform="translate(220)">
        <text x="42" y="0" fill="#8b98a5" font-size="7.5" text-anchor="middle">p(passes | …)</text>
        <rect x="5" y="10" width="74" height="13" rx="4" fill="#1f2937"/><rect x="5" y="10" width="47" height="13" rx="4" fill="#60a5fa"/>
        <text x="42" y="20" fill="#e6edf3" font-size="8" font-weight="700" text-anchor="middle">0.64</text>
        <text x="42" y="39" fill="#93c5fd" font-size="8" text-anchor="middle">surprise 0.45</text>
      </g>
      <g transform="translate(330)">
        <text x="42" y="0" fill="#8b98a5" font-size="7.5" text-anchor="middle">p(final | …)</text>
        <rect x="5" y="10" width="74" height="13" rx="4" fill="#1f2937"/><rect x="5" y="10" width="17" height="13" rx="4" fill="#f87171"/>
        <text x="42" y="20" fill="#e6edf3" font-size="8" font-weight="700" text-anchor="middle">0.23</text>
        <text x="42" y="39" fill="#fca5a5" font-size="8" text-anchor="middle">surprise 1.47</text>
      </g>
      <g transform="translate(440)">
        <text x="42" y="0" fill="#8b98a5" font-size="7.5" text-anchor="middle">p(test | …)</text>
        <rect x="5" y="10" width="74" height="13" rx="4" fill="#1f2937"/><rect x="5" y="10" width="51" height="13" rx="4" fill="#60a5fa"/>
        <text x="42" y="20" fill="#e6edf3" font-size="8" font-weight="700" text-anchor="middle">0.69</text>
        <text x="42" y="39" fill="#93c5fd" font-size="8" text-anchor="middle">surprise 0.37</text>
      </g>
    </g>

    <!-- Calculation -->
    <path d="M550 61H579" stroke="#fbbf24" stroke-width="2.5" marker-end="url(#ppGoldArrow)"/>
    <g transform="translate(583 31)">
      <rect width="114" height="168" rx="10" fill="#17150f" stroke="#fbbf24" stroke-width="1.6"/>
      <text x="57" y="18" fill="#fde68a" font-size="9.5" font-weight="700" text-anchor="middle">AGGREGATE</text>
      <text x="57" y="42" fill="#8b98a5" font-size="8" text-anchor="middle">token loss</text>
      <text x="57" y="57" fill="#c4b5fd" font-size="10" font-weight="700" text-anchor="middle">−ln p(actual)</text>
      <path d="M18 69h78" stroke="#745d1d"/>
      <text x="57" y="88" fill="#8b98a5" font-size="8" text-anchor="middle">mean NLL</text>
      <text x="57" y="105" fill="#fbbf24" font-size="13" font-weight="700" text-anchor="middle">0.67</text>
      <path d="M18 116h78" stroke="#745d1d"/>
      <text x="57" y="133" fill="#8b98a5" font-size="8" text-anchor="middle">PPL = exp(NLL)</text>
      <circle cx="57" cy="151" r="17" fill="#fbbf24" fill-opacity=".15" stroke="#fbbf24"/>
      <text x="57" y="156" fill="#fde68a" font-size="13" font-weight="700" text-anchor="middle">1.95</text>
    </g>
    <text x="281" y="205" fill="#8b98a5" font-size="8.5" text-anchor="middle">a low assigned probability creates a large log penalty; averaging in log space prevents products from underflowing</text>
  </g>

  <!-- Interpretation -->
  <g transform="translate(24 337)">
    <text x="250" y="-10" fill="#6ee7b7" font-size="10.5" font-weight="700" text-anchor="middle">LOWER PERPLEXITY MEANS THE OBSERVED TOKENS WERE LESS SURPRISING</text>
    <rect width="500" height="98" rx="12" fill="#101d1a" stroke="#33705e" stroke-width="1.5"/>
    <g transform="translate(14 14)">
      <text x="103" y="10" fill="#6ee7b7" font-size="8.5" font-weight="700" text-anchor="middle">CONCENTRATED DISTRIBUTION · PPL ≈ 2</text>
      <path d="M9 65H198" stroke="#46525f"/>
      <rect x="22" y="18" width="25" height="47" rx="3" fill="#34d399"/>
      <rect x="63" y="42" width="25" height="23" rx="3" fill="#60a5fa"/>
      <rect x="104" y="51" width="25" height="14" rx="3" fill="#a78bfa"/>
      <rect x="145" y="58" width="25" height="7" rx="3" fill="#52647b"/>
      <circle cx="34" cy="18" r="4" fill="#6ee7b7" filter="url(#ppGlow)"/>
      <text x="103" y="82" fill="#8b98a5" font-size="7.8" text-anchor="middle">model narrows uncertainty to a few plausible choices</text>
    </g>
    <path d="M221 49h35" stroke="#34d399" stroke-width="2" marker-end="url(#ppGreenArrow)"/>
    <g transform="translate(264 14)">
      <text x="103" y="10" fill="#fca5a5" font-size="8.5" font-weight="700" text-anchor="middle">DIFFUSE DISTRIBUTION · PPL ≈ 8</text>
      <path d="M9 65H198" stroke="#46525f"/>
      <rect x="16" y="45" width="20" height="20" rx="3" fill="#f87171"/><rect x="43" y="47" width="20" height="18" rx="3" fill="#f87171"/>
      <rect x="70" y="48" width="20" height="17" rx="3" fill="#f87171"/><rect x="97" y="49" width="20" height="16" rx="3" fill="#f87171"/>
      <rect x="124" y="50" width="20" height="15" rx="3" fill="#f87171"/><rect x="151" y="51" width="20" height="14" rx="3" fill="#f87171"/>
      <text x="103" y="82" fill="#8b98a5" font-size="7.8" text-anchor="middle">probability mass is spread across many alternatives</text>
    </g>
  </g>

  <!-- Limits -->
  <g transform="translate(541 337)">
    <text x="97" y="-10" fill="#fbbf24" font-size="10.5" font-weight="700" text-anchor="middle">COMPARE LIKE WITH LIKE</text>
    <rect width="195" height="98" rx="12" fill="#17150f" stroke="#745d1d" stroke-width="1.5"/>
    <g transform="translate(13 13)" fill="#8b98a5" font-size="8">
      <circle cx="4" cy="6" r="3" fill="#fbbf24"/><text x="14" y="9">same evaluation corpus</text>
      <circle cx="4" cy="24" r="3" fill="#fbbf24"/><text x="14" y="27">same tokenization and units</text>
      <circle cx="4" cy="42" r="3" fill="#fbbf24"/><text x="14" y="45">same normalization and context</text>
      <path d="M2 56l9 9M11 56L2 65" stroke="#f87171" stroke-width="2"/><text x="18" y="64" fill="#fca5a5">not factuality or usefulness</text>
      <text x="84" y="78" fill="#fde68a" font-size="7.7" text-anchor="middle">validate downstream task quality separately</text>
    </g>
  </g>

  <text x="380" y="458" fill="#6b7684" font-size="11" text-anchor="middle">Perplexity measures likelihood—not factuality, usefulness, safety, calibration, or downstream task quality.</text>
</svg>
```lexity is the standard *intrinsic* measure of how well a language model predicts text, and the cleanest way to understand it is as the model's average *branching factor*: at each token, how many equally-likely choices does the model effectively think it is choosing between? A perplexity of 10 means the model is, on average, as uncertain as if it were picking uniformly among 10 options for every next token. Lower is better — a perfect model that always assigned probability 1 to the correct token would have a perplexity of 1. That single number, tracked over a training run, is the heartbeat of language-model pretraining, and it comes directly from the loss the model is already optimizing.\n\n**Perplexity is just the exponential of the cross-entropy loss, which is why it costs nothing to compute.** A language model is trained to maximize the probability it assigns to the real next token, and the cross-entropy loss is the average negative log-probability it assigns to the true tokens of a held-out text. Perplexity is simply that loss exponentiated — raise e (or 2) to the average cross-entropy and you get perplexity. So the quantity the optimizer is already minimizing *is* perplexity in log space; there is no separate evaluation to run. This tight coupling is exactly why perplexity is the natural training-time metric: it is the loss, re-expressed on a scale that has an intuitive meaning.\n\n**That meaning is uncertainty, and it doubles as a measure of compression.** Because cross-entropy is measured in bits (or nats), perplexity is directly tied to *bits per token* — the number of bits you would need, on average, to encode the next token given the model's predictions. A lower-perplexity model is literally a better compressor of the text, which is the deep reason perplexity tracks language-modeling quality: predicting text well and compressing it well are the same problem. This is also why "perplexity equals effective vocabulary size" is a fair intuition — it is the size of the uniform distribution that would leave the model equally surprised.\n\n**Its fatal limitation is that perplexity is only comparable within the same tokenizer and data, and it does not measure usefulness.** Perplexity is computed per token, so a model with a different vocabulary or tokenizer chops the text into different units and produces numbers that cannot be compared to another model's — a smaller perplexity across tokenizers can be an artifact of tokenization, not better modeling. It is also purely *intrinsic*: it rewards assigning high probability to the reference text, which is not the same as being helpful, truthful, or good at a downstream task. A model can have excellent perplexity and still fail at reasoning, follow instructions poorly, or hallucinate. This is why perplexity anchors *pretraining* but is complemented by task benchmarks and human preference for judging a finished model.\n\n| Property | What it means |\n|---|---|\n| Definition | exp(cross-entropy loss) — the average per-token surprise |\n| Interpretation | Effective branching factor / uniform choices per token |\n| Direction | Lower is better; a perfect model scores 1 |\n| Ties to | Bits per token; text compression quality |\n| Key limitation | Tokenizer-dependent; measures fit, not usefulness |\n\n\nThe unhelpful way to meet perplexity is as an opaque number on a training dashboard that should go down. The useful way is to hold onto its one plain meaning — the average number of choices the model feels it is guessing among for each token — and let everything else follow from it. Because it is the exponential of the cross-entropy the model already minimizes, it is free to compute and tracks training directly; because uncertainty and compression are the same thing, a lower-perplexity model is a better compressor of language; and because it is measured per token against a reference, it cannot be compared across tokenizers and says nothing about whether the model is actually useful. Read perplexity through a how-surprised-is-the-model-at-each-token lens rather than a mysterious-loss-number lens, and it becomes both the most natural metric to watch during pretraining and one you know better than to trust alone.