quantization

Quantization stores and computes a neural network in fewer bits — mapping high-precision FP32 weights and activations onto a small set of low-precision levels (INT8, INT4, FP8) so the model takes less memory and runs faster.\n\n**It is a mapping from a continuous range onto a discrete grid.** A quantizer picks a scale s and a zero-point z, then represents each real value x as the integer q = round(x / s) + z and recovers an approximation x' = s(q - z). INT8 gives 256 levels, INT4 only 16. The art is choosing s — and whether to use one scale per tensor, per channel, or per block — so the grid lands where the weights actually cluster.\n\n**Why it pays off.** Cutting FP32 to INT8 shrinks the model 4x and cuts memory traffic 4x, which — per the roofline — is exactly what helps a memory-bound LLM. The multiplies also become cheap integer or low-precision operations that tensor cores run several times faster. The cost is rounding error: too few levels, or a few outlier values stretching the scale, and accuracy drops.\n\n| Approach | When quantized | Accuracy | Cost | Typical use |\n|---|---|---|---|---|\n| PTQ (post-training) | after training | good with care | minutes | most INT8 deploys |\n| QAT (quant-aware) | during training | best | full retrain | aggressive low-bit |\n| Weight-only | weights only, acts FP16 | high | simple | LLM inference (GPTQ, AWQ) |\n| Dynamic | activations at runtime | good | slight overhead | activation-sensitive nets |\n\n```svg\n\n```\n\n**Bit-width is a spectrum, and outliers set the floor.** INT8 is routine; INT4 and FP4 need care because a handful of outlier activations dominate the range and inflate the rounding error for everything else. Per-channel scales, group-wise quantization, and outlier-aware methods (SmoothQuant, AWQ, GPTQ) exist precisely to keep the grid useful as the bit-width shrinks.\n\nRead quantization through a quant lens rather than an accuracy-table lens: it is a knob on arithmetic intensity and model footprint, not merely a compression trick. Every bit removed multiplies effective bandwidth and tensor-core throughput, so the real question is how far levels can drop before rounding error crosses the task's tolerance — a measured bytes-versus-error trade, not an assumed one.