what is model quantization
**Quantization is the practice of storing and computing a model's weights and activations in fewer bits than the 16 or 32 bits they were trained in, trading a small amount of numerical precision for a large amount of memory and bandwidth headroom.** A model trained in FP16 or BF16 keeps every weight as a 16-bit floating-point number, which is precise but expensive to move: every one of those bytes has to travel from memory to the compute units for every matrix multiply. Quantizing the same weights down to INT8, FP8, or even INT4 means each one now takes half, a quarter, or an eighth as many bytes to store and move — and since the earlier discussion of LLM inference showed decode is usually bottlenecked on memory bandwidth rather than raw math, cutting the bytes-per-weight is often the single most effective lever available for making inference faster.
**The core trick is representing a wide range of real numbers using a small number of integer buckets, which only works because neural network weights are forgiving of small errors.** A 16-bit float can represent an enormous range of values with fine-grained precision; an 8-bit integer only has 256 distinct values to work with. Quantization solves this by picking a scale factor for a group of weights — mapping their real range onto the available integer buckets — then storing just the integers plus that one scale factor, reconstructing an approximation of the original values when needed. Because a trained network's weights are statistically redundant and individual neurons rarely depend on razor-precise values, this approximation usually costs only a small, often unnoticeable, drop in output quality.
```svg
```
**Weight-only quantization is the most common starting point, and activation quantization is the harder, riskier extension of the same idea.** Weight-only quantization compresses the static model parameters but still computes in higher precision, which shrinks memory footprint and bandwidth demand without touching numerical behavior during the actual matrix multiply. Quantizing activations too — the intermediate values flowing between layers as a request is processed — compounds the savings further, since now both operands of every multiply are smaller, but activations vary token to token in ways weights don't, making them far more sensitive to a poorly chosen scale factor. This is why FP8 has gained particular traction for inference: it keeps a wider dynamic range than INT8 at the same bit width, tolerating activation variability better while still halving memory versus FP16.
| Format | Bits per Value | Typical Memory Savings vs. FP16 | Typical Accuracy Impact |
|---|---|---|---|
| INT8 | 8 | ~2x | Usually near-lossless with calibration |
| FP8 | 8 | ~2x | Near-lossless, better handles activation outliers |
| INT4 | 4 | ~4x | Small measurable accuracy drop, needs finer grouping |
```flowchart
st=>start: Trained model weights arrive in FP16/BF16
group=>operation: Weights grouped into small blocks for per-group scale factors
calibrate=>operation: Calibration pass measures each group's real value range
convert=>operation: Values mapped onto integer buckets using each group's scale factor
store=>operation: Quantized integers plus scale factors stored, replacing full-precision weights
serve=>operation: Inference reconstructs approximate values on the fly during compute
pass=>end: Model serves requests using a fraction of the original memory bandwidth
st->group->calibrate->convert->store->serve->pass
```
**None of this matters in the abstract — it matters because memory bandwidth, not raw compute, is what limits how fast an accelerator can serve decode.** The earlier discussion of prefill and decode showed decode re-reads the entire KV cache and full weight set on every single generated token, so if those weights and that cache take a quarter as many bytes to move, decode can run proportionally faster or serve proportionally more concurrent users on the same hardware. That's why AI accelerator designers care as much about native INT8/FP8/INT4 matrix-multiply support and memory bandwidth as they do about peak FLOPs — a chip that can't quantize efficiently is leaving real serving throughput on the table, regardless of how impressive its full-precision compute numbers look.