fp4

FP4 and the microscaling (MX) formats are how modern accelerators push numeric precision below 8 bits without the model falling apart. A four-bit float has only about sixteen representable values, so the naive approach of scaling a whole tensor by one number stops working: a handful of outliers force a scale so coarse that the many small values underneath them round to zero. The MX formats solve this with block floating point — a small group of elements shares a local scale — and it is now an open industry standard rather than one vendor's trick.\n\n**Below FP8, a single per-tensor scale runs out of range.** FP8 already leans on a scale factor per tensor to place its limited exponent window over the actual data. Halve the bits again to FP4 (the E2M1 layout: one sign bit, two exponent bits, one mantissa bit) and the representable set is tiny, while the spread of magnitudes inside a real weight or activation tensor is not. One shared scale has to cover both the largest outlier and the smallest meaningful value, and it cannot: set it for the outlier and the small values vanish; set it for the small values and the outlier saturates. Precision at four bits is really a dynamic-range problem, not a rounding problem.\n\n**Microscaling fixes the range problem by giving every small block its own scale.** In an MX format a block of elements — 32 in the OCP standard — shares one scale factor stored as an 8-bit power-of-two exponent (the E8M0 type), and each element is a low-precision value: FP8, FP6, FP4, or INT8. Because the scale is local to 32 numbers instead of the whole tensor, it tracks the local dynamic range, so an outlier in one block no longer crushes the small values in the next. This is classic block floating point, standardized: MXFP8, MXFP6, MXFP4, and MXINT8 differ only in the element type sitting under that shared block scale.\n\n**The cost of the block scale is small, which is the whole point.** The scale adds bits, but amortized across the block they nearly disappear: an MXFP4 value costs its four element bits plus eight scale bits spread over 32 elements, or 4.25 bits each. NVIDIA's NVFP4 on Blackwell chooses a tighter arrangement — blocks of 16 with an FP8 (E4M3) block scale plus a per-tensor FP32 scale — landing near 4.5 effective bits but with noticeably better accuracy than plain MXFP4, because a finer block and a higher-precision scale capture the distribution more faithfully. Either way you store weights and activations at roughly four bits while keeping enough local range to stay usable.\n\n**The payoff is bandwidth and math throughput, and the discipline is knowing what to keep in higher precision.** Four-bit data is half the memory and half the bandwidth of FP8, and Blackwell-class tensor cores execute FP4 natively at about twice the FP8 rate, so FP4 is both smaller and faster rather than a storage-only trick. The catch is accuracy: the robust recipe keeps the sensitive operations — attention softmax, normalization layers, and often the first and last layers — in higher precision, and quantizes the bulk of the matmuls to FP4. This is the floating-point cousin of INT4 methods like AWQ and GPTQ, the difference being that MXFP4 and NVFP4 are native hardware number formats rather than integer weights that must be dequantized in software.\n\n| Format | Element | Block | Scale | Effective bits / value |\n|---|---|---|---|---|\n| FP8 (per-tensor) | E4M3 / E5M2 | whole tensor | 1 × FP32 | ~8 |\n| **MXFP8** | E4M3 / E5M2 | 32 | E8M0 (8-bit) | 8.25 |\n| **MXFP4** | E2M1 (4-bit) | 32 | E8M0 (8-bit) | 4.25 |\n| **NVFP4** | E2M1 (4-bit) | 16 | E4M3 + tensor FP32 | ~4.5 |\n| INT4 (AWQ / GPTQ) | int4 | 64–128 group | FP16 | ~4.1–4.25 |\n\n```svg\n \n```\n\nRead FP4 and the MX formats through a *local-dynamic-range* lens rather than a *fewer-bits* lens: the reason four-bit inference works at all is not better rounding but the block scale, which lets 32 (or 16) neighboring values share a range so outliers stop erasing the small numbers around them — and because Blackwell executes these formats natively, the same weights become both half the size and roughly twice the throughput of FP8.