quantization-aware training
Quantization-aware training is a technique that simulates the effects of reduced numerical precision during a neural network's training process itself, letting the model adapt to eventual quantization and preserve more accuracy than converting an already-trained model afterward would.
```flowchart
{
"rows": [
{ "type": "nodes", "items": [
{ "title": "Model quantized after training loses some accuracy", "sub": "reduced precision applied to weights it never trained with", "tone": "neutral" }
]},
{ "type": "arrow" },
{ "type": "group", "title": "Quantization effects simulated during training itself", "items": [
{ "title": "Model learns to work well despite reduced precision", "sub": "weights adapt around quantization's specific rounding effects", "tone": "blue" }
]},
{ "type": "arrow" },
{ "type": "nodes", "items": [
{ "title": "Quantized model retains accuracy much closer to full precision", "sub": "better results than quantizing an already-finished model", "tone": "green" }
]}
]
}
```
**Quantization-aware training exists because converting an already-trained, full-precision model to lower numerical precision after the fact can meaningfully hurt accuracy, since the model's weights were never optimized with that eventual precision reduction in mind.** Since a model trained with simulated quantization effects present throughout training can adjust its weights specifically to remain accurate despite that reduced precision, quantization-aware training inserts operations that mimic quantization's rounding and precision-limiting effects during the training process itself, letting the model learn weight values that work well under quantization rather than being caught off guard by it afterward.
```svg
```
```svg
```
| Aspect | Post-training quantization | Quantization-aware training |
|---|---|---|
| When precision reduction is introduced | After training completes | Simulated throughout training |
| Typical accuracy retention | Lower | Higher |
| Training cost | None additional | Extra training required |
| Common use | Quick deployment, less accuracy-sensitive cases | Accuracy-critical deployed models |
**Quantization-aware training typically works by inserting fake quantization operations into the training computation graph, which round values to simulate lower precision during the forward pass while still allowing normal gradient-based updates during the backward pass.** Because gradients generally can't flow properly through a true, non-differentiable rounding operation, quantization-aware training uses techniques that simulate quantization's rounding behavior in the forward pass while approximating a usable gradient for the backward pass, letting standard training procedures continue to function while still exposing the model to quantization's effects.
**Quantization-aware training requires more computational cost and training time than simply quantizing an already-trained model, since it typically involves additional training steps beyond the model's original training process.** Because quantization-aware training generally starts from an already-trained model and then performs additional fine-tuning steps with simulated quantization active, it takes more total time and computation than post-training quantization's comparatively quick conversion process, a cost that's often considered worthwhile when the resulting accuracy improvement matters for the deployment use case.
**Quantization-aware training is particularly valuable for aggressive quantization levels, such as reducing precision to very few bits, where post-training quantization's accuracy loss tends to become more severe.** Because the accuracy gap between post-training quantization and quantization-aware training tends to widen as the target precision gets more aggressive, quantization-aware training becomes an increasingly important technique specifically for deployment scenarios targeting very low bit-width representations, where preserving as much accuracy as possible matters most.
Read quantization-aware training through a rehearsal lens: rather than a performer discovering only on the actual performance day that they'll have less rehearsal time than they trained for, quantization-aware training is like rehearsing under those exact constrained conditions from the start, so the eventual performance under real conditions comes out far closer to what was practiced.