model quantization int8 inference

**Model Quantization** is **the neural network compression technique that converts floating-point weights and activations to lower-precision integer representations (INT8, INT4, or binary) — reducing model size by 2-8×, accelerating inference by 2-4× on quantization-friendly hardware, and enabling deployment on edge devices with limited memory and compute**. **Quantization Fundamentals:** - **Uniform Quantization**: maps continuous FP32 range [rmin, rmax] to discrete integer values — q = round((r - rmin) / scale), where scale = (rmax - rmin) / (2^bits - 1); dequantization recovers approximate float: r ≈ q × scale + zero_point - **Symmetric vs. Asymmetric**: symmetric quantization centers range around zero (zero_point = 0) — simpler computation but wastes range for non-negative activations (ReLU outputs); asymmetric uses full integer range for any distribution - **Per-Tensor vs. Per-Channel**: per-tensor uses single scale/zero_point for entire tensor — per-channel quantization uses different scales per output channel; per-channel achieves 0.5-1% better accuracy for weights with varying magnitude distributions - **Dynamic vs. Static**: dynamic quantization computes activation ranges at runtime — adds overhead but handles varying input distributions; static quantization calibrates ranges offline on representative dataset **Post-Training Quantization (PTQ):** - **Weight-Only Quantization**: quantize only weights to INT8/INT4, keep activations in FP16 — simplest approach; reduces model size without modifying inference pipeline; effective for memory-bound models (LLMs) - **Weight + Activation Quantization**: quantize both weights and activations for full INT8 inference — requires calibration dataset (100-1000 representative samples) to determine activation ranges; achieves 2-4× speedup on INT8-capable hardware - **GPTQ**: second-order weight quantization for LLMs — quantizes weights column-by-column using Hessian information to minimize quantization error; achieves INT4 weight quantization with minimal accuracy loss for 100B+ parameter models - **AWQ (Activation-Aware Weight Quantization)**: identifies salient weight channels based on activation magnitudes — protects important weights from aggressive quantization; outperforms GPTQ for INT4 LLM quantization **Quantization-Aware Training (QAT):** - **Fake Quantization**: simulate quantization during training by quantizing-then-dequantizing in forward pass — backward pass uses straight-through estimator (STE) to pass gradients through non-differentiable rounding operation - **Trained Scale Parameters**: learn optimal quantization ranges during training rather than calibrating post-hoc — result: model weights adapt to quantization-friendly distributions; typically 0.5-2% better accuracy than PTQ - **Mixed-Precision QAT**: different layers quantized at different bit-widths — sensitivity analysis determines which layers tolerate INT4 vs. requiring INT8; first and last layers often kept at higher precision - **Distillation-Assisted QAT**: use full-precision model as teacher during QAT — student matches teacher's output distribution, recovering accuracy lost from quantization; combines benefits of distillation and quantization **Model quantization is the most deployment-impactful compression technique — INT8 quantization is now standard practice for inference serving, and INT4 quantization is rapidly maturing for LLM deployment, enabling models that previously required multiple GPUs to run on a single GPU or even edge devices.**