Low precision means representing weights, activations, and gradients in fewer bits than FP32 — FP16, BF16, FP8, FP4, or INT8 — to shrink memory footprint and run more math per second on specialized hardware. The choice is never just 'how many bits' but how those bits split between range and precision.\n\nA float is a sign, an exponent, and a mantissa — and each format spends its bits differently. The exponent sets dynamic range (how large and small a value can be); the mantissa sets precision (how finely values are resolved). FP32 has 8 exponent and 23 mantissa bits. BF16 keeps all 8 exponent bits but drops to 7 mantissa, so it matches FP32's range while sacrificing precision — which is why it trains stably as a near drop-in. FP16 keeps 10 mantissa but only 5 exponent bits, so it is precise but underflows without loss scaling.\n\nBelow 16 bits the split gets sharper. FP8 comes in two flavors: E4M3 leans on precision for forward passes, E5M2 leans on range for gradients. FP4 has just 16 representable levels, so it only works with fine-grained block scales that restore local range. INT8 abandons the float split entirely — a uniform grid plus one scale factor — which is cheapest to multiply but least forgiving of outliers.\n\n| Format | Bits | Exp / Mant | Strength | Main risk |\n|---|---|---|---|---|\n| FP32 | 32 | 8 / 23 | baseline accuracy | 4x the memory & bandwidth |\n| FP16 | 16 | 5 / 10 | precise | narrow range, needs loss scaling |\n| BF16 | 16 | 8 / 7 | FP32 range, stable | coarser mantissa |\n| FP8 (E4M3/E5M2) | 8 | 4/3 or 5/2 | fast train + infer | tight range, per-tensor scales |\n| FP4 / INT4 | 4 | 2/1 or integer | max compression | needs block/group scaling |\n\n``svg\n\n``\n\nRange and precision fail in different ways, so mixed precision is the norm. Practical pipelines keep a high-precision master copy for the optimizer, compute the heavy matmuls in BF16 or FP8, and reserve FP32 for accumulation and sensitive reductions. The art is matching each format's bit-split to the tensor's statistics: wide-dynamic-range gradients want exponent bits, tightly clustered weights want mantissa bits or a uniform integer grid.\n\nRead low precision through a quant lens rather than an accuracy-loss lens: every bit removed multiplies effective bandwidth and compute throughput, so per the roofline it directly moves a memory-bound kernel toward its compute roof. The engineering question is not 'is FP8 lossy' but which field — range or precision — a given tensor can afford to shorten before error crosses tolerance, measured rather than assumed.