FP8 is an 8-bit floating-point number format used to store and multiply the weights, activations, and sometimes gradients of large neural networks at half the width of FP16. It comes in two variants standardized by the OCP: E4M3, which spends four bits on the exponent and three on the mantissa for more precision over a narrower range, and E5M2, which spends five on the exponent for wider dynamic range at coarser resolution. Because eight bits cannot span a whole tensor's values, FP8 is always paired with a scale factor that shifts its small window onto the real data.\n\nEight bits force a direct trade between range and precision. A floating-point number splits its bits into sign, exponent, and mantissa: exponent bits set how many powers of two the format can reach (dynamic range), mantissa bits set how finely it resolves values within each power (precision). At 16 bits you needn't choose much, but at 8 the budget is brutal. E4M3 keeps three mantissa bits for finer steps but reaches only about 2 to the power of a few tens of range; E5M2 matches FP16's exponent range but is left with two mantissa bits. Unlike INT8, whose step size is fixed across the whole range, FP8's floating exponent preserves relative precision across scales.\n\nEach format is matched to what it holds. In practice the forward pass uses E4M3 for weights and activations, whose magnitudes cluster in a limited band where precision matters more than reach, while E5M2 carries gradients in the backward pass, where values span a huge range and the occasional large outlier must not overflow. This is why FP8 training pipelines quote both formats together. The narrow width is what buys the speed: modern tensor cores run FP8 matmuls at roughly double the FP16 rate, and the tensors take half the memory and bandwidth, which is exactly the pressure point for large-model inference.\n\n| | E4M3 | E5M2 | INT8 |\n|---|---|---|---|\n| Bits (S/E/M) | 1 / 4 / 3 | 1 / 5 / 2 | 1 / 0 / 7 |\n| Strength | precision | dynamic range | fixed-point speed |\n| Typical use | weights, activations | gradients | quantized inference |\n| Step size | relative (floating) | relative (floating) | uniform (fixed) |\n| Needs a scale | yes (per-tensor/block) | yes | yes (+ zero-point) |\n| Vs FP16 | ~2× math, ½ memory | ~2× math, ½ memory | ~2× math, ½ memory |\n\n``svg\n\n``\n\nA scale factor makes the narrow window usable. Because FP8's representable range is far too small to hold an entire tensor, each tensor (or each block or channel of it) is divided by a scale derived from its maximum absolute value, or amax, so the largest element lands near the top of FP8's range and the rest fill the window instead of underflowing to zero. The FP8 matmul then accumulates its products in FP16 or FP32, and the result is multiplied back by the scale. Choosing that scale is the whole game: per-tensor scaling is cheapest, per-block scaling is tighter and more robust to outliers, and delayed scaling reuses a running amax history to avoid an extra pass. Getting it wrong shows up as overflow to infinity or silent underflow, which is why formats like MXFP8 bake fine-grained block scales into the standard.\n\nRead FP8 through a quant lens rather than a 'smaller numbers' lens: the number it moves is bits-per-element, halved from FP16, which converts almost directly into 2× arithmetic throughput and half the memory-bandwidth and capacity cost, the binding constraints on both training and inference. The design questions are all about the exponent/mantissa split and the scale: pick E4M3 or E5M2 by whether a tensor needs precision or range, and pick per-tensor, per-block, or delayed scaling by how far the tensor's amax outruns its typical value, since the technique only wins while the accuracy lost to eight bits stays smaller than the throughput you gain.