<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Mixed Precision Training — FP16, BF16, and FP8</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">keep a master copy in FP32, compute in lower precision — 2× speed, half memory, same accuracy</text>

  <!-- === TOP: How mixed precision works === -->
  <rect x="25" y="62" width="710" height="138" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="380" y="82" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Mixed Precision Training Loop</text>

  <!-- FP32 master weights -->
  <rect x="45" y="95" width="120" height="35" rx="4" fill="#1a1520" stroke="#a78bfa" stroke-width="1"/>
  <text x="105" y="112" fill="#c4b5fd" font-size="8.5" text-anchor="middle" font-weight="600">Master (FP32)</text>
  <text x="105" y="125" fill="#6b7684" font-size="7.5" text-anchor="middle">full precision copy</text>

  <!-- Cast down -->
  <path d="M168,112 L195,112" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
  <polygon points="193,109 199,112 193,115" fill="#8b98a5"/>
  <text x="181" y="104" fill="#6b7684" font-size="6.5" text-anchor="middle">cast ↓</text>

  <!-- FP16/BF16 forward pass -->
  <rect x="202" y="95" width="120" height="35" rx="4" fill="#0f1a2a" stroke="#60a5fa" stroke-width="1"/>
  <text x="262" y="112" fill="#93c5fd" font-size="8.5" text-anchor="middle" font-weight="600">Forward (FP16)</text>
  <text x="262" y="125" fill="#6b7684" font-size="7.5" text-anchor="middle">fast tensor cores</text>

  <!-- Loss + scale -->
  <path d="M325,112 L352,112" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
  <polygon points="350,109 356,112 350,115" fill="#8b98a5"/>

  <rect x="359" y="95" width="100" height="35" rx="4" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1"/>
  <text x="409" y="112" fill="#fbbf24" font-size="8.5" text-anchor="middle" font-weight="600">Loss × scale</text>
  <text x="409" y="125" fill="#6b7684" font-size="7.5" text-anchor="middle">prevent underflow</text>

  <!-- Backward FP16 -->
  <path d="M462,112 L489,112" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
  <polygon points="487,109 493,112 487,115" fill="#8b98a5"/>

  <rect x="496" y="95" width="120" height="35" rx="4" fill="#0f1a2a" stroke="#60a5fa" stroke-width="1"/>
  <text x="556" y="112" fill="#93c5fd" font-size="8.5" text-anchor="middle" font-weight="600">Backward (FP16)</text>
  <text x="556" y="125" fill="#6b7684" font-size="7.5" text-anchor="middle">grads in half precision</text>

  <!-- Unscale + update FP32 -->
  <path d="M619,112 L646,112" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
  <polygon points="644,109 650,112 644,115" fill="#8b98a5"/>

  <rect x="653" y="95" width="65" height="35" rx="4" fill="#14261f" stroke="#34d399" stroke-width="1"/>
  <text x="685" y="108" fill="#6ee7b7" font-size="8" text-anchor="middle">Unscale</text>
  <text x="685" y="121" fill="#6ee7b7" font-size="8" text-anchor="middle">÷ scale</text>

  <!-- Loop back arrow -->
  <path d="M685,133 C685,165 105,165 105,133" fill="none" stroke="#a78bfa" stroke-width="0.8" stroke-dasharray="3,2"/>
  <text x="380" y="158" fill="#c4b5fd" font-size="8" text-anchor="middle">update master FP32 weights with unscaled FP16 gradients</text>

  <!-- Why loss scaling -->
  <text x="380" y="185" fill="#fbbf24" font-size="8.5" text-anchor="middle">Loss scaling: multiply loss by 1024-65536 so small gradients don't underflow to zero in FP16 (5.96e-8 min subnormal)</text>

  <!-- === MIDDLE LEFT: Format comparison === -->
  <rect x="25" y="208" width="350" height="120" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="226" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Numeric Format Comparison</text>

  <text x="45" y="248" fill="#c4b5fd" font-size="8.5" font-weight="600">FP32:</text>
  <text x="85" y="248" fill="#8b98a5" font-size="8.5">8 exp + 23 mantissa (baseline, 4 bytes)</text>
  <text x="45" y="264" fill="#60a5fa" font-size="8.5" font-weight="600">FP16:</text>
  <text x="85" y="264" fill="#8b98a5" font-size="8.5">5 exp + 10 mant (narrow range, needs loss scale)</text>
  <text x="45" y="280" fill="#34d399" font-size="8.5" font-weight="600">BF16:</text>
  <text x="85" y="280" fill="#8b98a5" font-size="8.5">8 exp + 7 mant (same range as FP32, less precise)</text>
  <text x="45" y="296" fill="#fbbf24" font-size="8.5" font-weight="600">FP8 (E4M3):</text>
  <text x="130" y="296" fill="#8b98a5" font-size="8.5">4 exp + 3 mant (H100 inference, 1 byte)</text>
  <text x="45" y="312" fill="#f87171" font-size="8.5" font-weight="600">INT8/INT4:</text>
  <text x="120" y="312" fill="#8b98a5" font-size="8.5">post-training quantization (inference only)</text>
  <text x="45" y="326" fill="#6b7684" font-size="8">BF16 is the default for LLM training (no loss scaling needed)</text>

  <!-- === MIDDLE RIGHT: Hardware support === -->
  <rect x="390" y="208" width="345" height="120" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="562" y="226" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Hardware Tensor Core Support</text>

  <text x="410" y="248" fill="#60a5fa" font-size="8.5" font-weight="600">V100 (2017):</text>
  <text x="500" y="248" fill="#8b98a5" font-size="8.5">FP16 tensor cores (125 TFLOPS)</text>
  <text x="410" y="266" fill="#34d399" font-size="8.5" font-weight="600">A100 (2020):</text>
  <text x="500" y="266" fill="#8b98a5" font-size="8.5">FP16 + BF16 + TF32 (312 TFLOPS)</text>
  <text x="410" y="284" fill="#fbbf24" font-size="8.5" font-weight="600">H100 (2022):</text>
  <text x="500" y="284" fill="#8b98a5" font-size="8.5">+ FP8 (1979 TFLOPS peak!)</text>
  <text x="410" y="302" fill="#c4b5fd" font-size="8.5" font-weight="600">B200 (2024):</text>
  <text x="500" y="302" fill="#8b98a5" font-size="8.5">+ FP4 (planned for inference)</text>
  <text x="410" y="320" fill="#6b7684" font-size="8">Each gen doubles FLOPS by halving precision support</text>

  <!-- === BOTTOM: Practical guide === -->
  <rect x="25" y="338" width="710" height="65" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="356" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Practical Usage</text>

  <text x="130" y="378" fill="#34d399" font-size="9" text-anchor="middle" font-weight="600">Training (default)</text>
  <text x="130" y="392" fill="#8b98a5" font-size="8" text-anchor="middle">BF16 + FP32 master</text>

  <text x="310" y="378" fill="#60a5fa" font-size="9" text-anchor="middle" font-weight="600">PyTorch AMP</text>
  <text x="310" y="392" fill="#8b98a5" font-size="8" text-anchor="middle">autocast + GradScaler</text>

  <text x="490" y="378" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="600">Inference (H100)</text>
  <text x="490" y="392" fill="#8b98a5" font-size="8" text-anchor="middle">FP8 (2× throughput vs BF16)</text>

  <text x="660" y="378" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="600">Edge / mobile</text>
  <text x="660" y="392" fill="#8b98a5" font-size="8" text-anchor="middle">INT8/INT4 (NPU)</text>

  <!-- Key insight -->
  <rect x="25" y="411" width="710" height="22" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="426" fill="#fbbf24" font-size="9" text-anchor="middle">BF16 won: same exponent range as FP32 (no loss scaling), 2× speed, 0.5× memory — the universal LLM training format.</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">Mixed precision is free performance: tensor cores run 2-8× faster in lower precision with negligible quality loss.</text>
</svg>
```d-precision training is the standard recipe that lets modern models train in half the memory and roughly twice the throughput without losing accuracy. The idea is simple to state and subtle to get right: do the heavy compute — the matrix multiplies in the forward and backward pass — in a 16-bit format that the hardware's tensor cores chew through fast, while keeping a full-precision copy of the things that must stay accurate. Every large model today is trained this way, and the two failure modes it has to defend against — underflow of tiny gradients and drift of slowly-accumulating weights — are exactly what the recipe is built around.\n\n**The core trick is a full-precision master copy of the weights.** You keep the authoritative weights in FP32, cast a 16-bit copy for each step's forward and backward pass, compute the gradients in 16-bit, and then apply the update to the FP32 master weights. This matters because a weight update is often many times smaller than the weight itself; in pure 16-bit, that tiny increment rounds away to nothing and training silently stalls. Accumulating the update into an FP32 master copy preserves it. Reductions like the loss and the gradient accumulation are likewise done in FP32.\n\n**FP16 and BF16 make opposite trade-offs with the same 16 bits.** FP16 spends 5 bits on the exponent and 10 on the mantissa: good precision, but a narrow dynamic range, so small gradients fall below the smallest representable value and underflow to zero. BF16 spends 8 exponent bits — the same range as FP32 — and only 7 on the mantissa: coarser precision, but it covers the full FP32 range, so gradients almost never underflow. That single difference is why BF16 has largely won for training: it needs no special handling, whereas FP16 requires loss scaling to be usable.\n\n**Loss scaling is how you make FP16 safe.** Before the backward pass you multiply the loss by a large constant S, which shifts the entire gradient distribution up out of the FP16 underflow region; after backprop, and before the optimizer step, you divide the gradients back down by S. *Dynamic* loss scaling automates the choice of S: it pushes S up until a gradient overflows to infinity, then backs off and skips that step, continually tracking the largest safe value. BF16's wide range means you can usually skip loss scaling entirely.\n\n**The payoff is why it is universal.** Sixteen-bit matrix multiplies run at roughly twice the rate of FP32 on tensor-core hardware, and the activations stored for the backward pass take half the memory — often the difference between a model fitting on a device or not. NVIDIA's TF32 is a related middle ground that keeps FP32 range with reduced mantissa for the matmul inputs, and FP8 pushes the same idea further for the largest training runs. In every case the principle is identical: compute cheap, but keep a precise master copy so the small quantities survive.\n\n| Format | Exponent / mantissa bits | Dynamic range | Loss scaling? | Role |\n|---|---|---|---|---|\n| FP32 | 8 / 23 | Full | n/a | Master weights, reductions |\n| TF32 | 8 / 10 | FP32 range | No | Matmul inputs (NVIDIA) |\n| BF16 | 8 / 7 | FP32 range | Usually no | Default training compute |\n| FP16 | 5 / 10 | Narrow | Yes | Training compute (needs scaling) |\n| FP8 | 4-5 / 2-3 | Very narrow | Yes (per-tensor) | Largest-scale training |\n\n```svg\n<svg viewBox="0 0 820 500" xmlns="http://www.w3.org/2000/svg" font-family="Segoe UI, Helvetica, Arial, sans-serif">\n  <rect x="0" y="0" width="820" height="500" fill="#0d1117"/>\n  <text x="30" y="38" fill="#e6edf3" font-size="21" font-weight="700">Mixed precision: compute cheap, keep a precise master</text>\n  <text x="30" y="60" fill="#8b98a5" font-size="13">16-bit matmuls for speed and memory; an FP32 master copy so the small quantities never round away.</text>\n\n  <!-- Panel 1: bit layouts -->\n  <text x="30" y="94" fill="#58a6ff" font-size="14" font-weight="700">1 - Same 16 bits, opposite trade-off</text>\n  <text x="45" y="120" fill="#8b98a5" font-size="13">FP32</text>\n  <rect x="90" y="108" width="14" height="16" fill="#db61a2"/>\n  <rect x="106" y="108" width="90" height="16" fill="#d29922"/>\n  <rect x="198" y="108" width="180" height="16" fill="#58a6ff"/>\n  <text x="120" y="121" fill="#0d1117" font-size="13">8 exp</text>\n  <text x="250" y="121" fill="#0d1117" font-size="13">23 mantissa</text>\n  <text x="45" y="148" fill="#8b98a5" font-size="13">BF16</text>\n  <rect x="90" y="136" width="14" height="16" fill="#db61a2"/>\n  <rect x="106" y="136" width="90" height="16" fill="#d29922"/>\n  <rect x="198" y="136" width="56" height="16" fill="#58a6ff"/>\n  <text x="120" y="149" fill="#0d1117" font-size="13">8 exp</text>\n  <text x="205" y="149" fill="#0d1117" font-size="13">7 mant</text>\n  <text x="262" y="149" fill="#3fb950" font-size="12">full range, no loss scaling</text>\n  <text x="45" y="176" fill="#8b98a5" font-size="13">FP16</text>\n  <rect x="90" y="164" width="14" height="16" fill="#db61a2"/>\n  <rect x="106" y="164" width="56" height="16" fill="#d29922"/>\n  <rect x="164" y="164" width="112" height="16" fill="#58a6ff"/>\n  <text x="118" y="177" fill="#0d1117" font-size="13">5 exp</text>\n  <text x="195" y="177" fill="#0d1117" font-size="13">10 mantissa</text>\n  <text x="282" y="177" fill="#f85149" font-size="12">narrow range, needs loss scaling</text>\n  <text x="90" y="200" fill="#8b98a5" font-size="12">more exponent = more range;  more mantissa = more precision</text>\n\n  <!-- Panel 2: the loop -->\n  <text x="30" y="234" fill="#58a6ff" font-size="14" font-weight="700">2 - The mixed-precision training loop</text>\n  <rect x="45" y="250" width="150" height="40" rx="6" fill="#0d1117" stroke="#3fb950"/>\n  <text x="58" y="268" fill="#3fb950" font-size="12" font-weight="700">FP32 master weights</text>\n  <text x="58" y="283" fill="#8b98a5" font-size="12">the authoritative copy</text>\n  <line x1="195" y1="270" x2="245" y2="270" stroke="#484f58"/><text x="196" y="263" fill="#8b98a5" font-size="13">cast</text>\n  <rect x="245" y="250" width="130" height="40" rx="6" fill="#0d1117" stroke="#58a6ff"/>\n  <text x="258" y="268" fill="#58a6ff" font-size="12" font-weight="700">16-bit forward</text>\n  <text x="258" y="283" fill="#8b98a5" font-size="12">fast tensor-core matmul</text>\n  <line x1="375" y1="270" x2="425" y2="270" stroke="#484f58"/>\n  <rect x="425" y="250" width="120" height="40" rx="6" fill="#0d1117" stroke="#d29922"/>\n  <text x="438" y="268" fill="#d29922" font-size="12" font-weight="700">loss x S</text>\n  <text x="438" y="283" fill="#8b98a5" font-size="12">scale up</text>\n  <line x1="545" y1="270" x2="595" y2="270" stroke="#484f58"/>\n  <rect x="595" y="250" width="185" height="40" rx="6" fill="#0d1117" stroke="#58a6ff"/>\n  <text x="608" y="268" fill="#58a6ff" font-size="12" font-weight="700">16-bit backward</text>\n  <text x="608" y="283" fill="#8b98a5" font-size="12">gradients computed in 16-bit</text>\n  <line x1="687" y1="290" x2="687" y2="310" stroke="#484f58"/>\n  <line x1="687" y1="310" x2="120" y2="310" stroke="#484f58"/>\n  <line x1="120" y1="310" x2="120" y2="290" stroke="#484f58"/>\n  <text x="300" y="325" fill="#d29922" font-size="13">gradients / S  (unscale)  ->  optimizer updates the FP32 master weights</text>\n\n  <!-- Panel 3: loss scaling -->\n  <text x="30" y="358" fill="#58a6ff" font-size="14" font-weight="700">3 - Loss scaling rescues tiny gradients</text>\n  <line x1="60" y1="455" x2="410" y2="455" stroke="#484f58"/>\n  <line x1="150" y1="360" x2="150" y2="465" stroke="#f85149" stroke-width="1.5" stroke-dasharray="4 3"/>\n  <text x="60" y="470" fill="#f85149" font-size="12">FP16 underflow floor (anything left of this rounds to 0)</text>\n  <path d="M70,455 Q110,400 150,455" fill="#f85149" opacity="0.35"/>\n  <text x="66" y="392" fill="#f85149" font-size="12">before: mass under the floor</text>\n  <path d="M230,455 Q290,385 350,455" fill="#3fb950" opacity="0.4"/>\n  <text x="240" y="378" fill="#3fb950" font-size="12">after x S: shifted into range</text>\n  <text x="180" y="418" fill="#8b98a5" font-size="16">-&gt;</text>\n\n  <rect x="440" y="360" width="350" height="110" rx="10" fill="#161b22" stroke="#30363d"/>\n  <text x="458" y="386" fill="#e6edf3" font-size="13" font-weight="700">Why it is universal</text>\n  <text x="458" y="410" fill="#3fb950" font-size="12">~2x throughput on tensor cores</text>\n  <text x="458" y="430" fill="#58a6ff" font-size="12">~half the activation memory</text>\n  <text x="458" y="450" fill="#d29922" font-size="12">near-zero accuracy loss</text>\n  <text x="458" y="466" fill="#8b98a5" font-size="12">the FP32 master copy is what makes it safe</text>\n</svg>\n```\n\nThe shallow reading of mixed precision is "use fewer bits to go faster." That misses the whole engineering problem, which is that not every number in training can afford fewer bits. The weight updates and the reductions need range and precision the 16-bit formats cannot give them, so the technique is really about *sorting* the numbers: heavy matmuls go cheap, the master weights and accumulations stay precise, and loss scaling shuttles the gradient distribution into whatever range the compute format can represent. Read mixed precision through a keep-a-precise-master-copy-while-computing-cheap lens rather than a just-use-fewer-bits lens, and the choice between BF16 and FP16, and the need for loss scaling, follow directly from one question: does this number need dynamic range, or precision, or both?