quantization aware training qat

**Quantization-Aware Training (QAT)** is the **model compression technique that simulates reduced numerical precision (INT8/INT4) during the forward pass of training, allowing the network to adapt its weights to quantization noise before deployment — producing models that run 2-4x faster on integer hardware with minimal accuracy loss compared to their full-precision counterparts**. **Why Quantization Matters** A 7-billion-parameter model in FP16 requires 14 GB just for weights. Quantizing to INT4 drops that to 3.5 GB, fitting on a single consumer GPU. Beyond memory savings, integer arithmetic (INT8 multiply-accumulate) executes 2-4x faster and draws less power than floating-point on every major accelerator architecture (NVIDIA Tensor Cores, Qualcomm Hexagon, Apple Neural Engine). **Post-Training Quantization (PTQ) vs. QAT** - **PTQ**: Quantizes a fully-trained FP32/FP16 model after the fact using a small calibration dataset to determine per-tensor or per-channel scale factors. Fast and simple, but accuracy degrades significantly below INT8, especially for models with wide activation ranges or outlier channels. - **QAT**: Inserts "fake quantization" nodes into the training graph that round activations and weights to the target integer grid during the forward pass, but use straight-through estimators to pass gradients backward in full precision. The model learns to place its weight distributions within the quantization grid, actively minimizing the rounding error. **Implementation Architecture** 1. **Fake Quantize Nodes**: Placed after each weight tensor and after each activation layer. They compute round(clamp(x / scale, -qmin, qmax)) * scale, simulating the information loss of integer representation while keeping the computation in floating-point for gradient flow. 2. **Scale and Zero-Point Calibration**: Per-channel weight quantization uses the actual min/max of each output channel. Activation quantization uses exponential moving averages of observed ranges during training. 3. **Fine-Tuning Duration**: QAT typically requires only 10-20% of original training epochs — not a full retrain. The model has already converged; QAT adjusts weight distributions to accommodate quantization bins. **When to Choose What** - **PTQ** is sufficient for INT8 on most vision and language models where activation distributions are well-behaved. - **QAT** becomes essential at INT4 and below, for models with outlier activation channels (common in LLMs), and when even 0.5% accuracy loss is unacceptable. Quantization-Aware Training is **the precision tool that closes the gap between theoretical hardware throughput and real-world model efficiency** — teaching the model to live within the integer grid rather than fighting it at deployment time.