model quantization inference

**Model Quantization** is the **compression technique that reduces neural network weight and activation precision from 32-bit or 16-bit floating point to lower bit-widths (INT8, INT4, FP8, even 1-2 bits) — shrinking model size by 2-8×, reducing memory bandwidth requirements proportionally, and enabling faster inference on hardware with specialized low-precision compute units, making it essential for deploying large language models on consumer GPUs and edge devices**. **Quantization Fundamentals** Quantization maps continuous float values to discrete integer levels: x_q = round(x / scale) + zero_point, where scale = (max-min)/(2^b-1) for b-bit quantization. Dequantization recovers an approximation: x ≈ (x_q - zero_point) × scale. The quantization error depends on bit-width and the distribution of values. **Post-Training Quantization (PTQ)** Quantize a pre-trained model without retraining: - **Weight-Only Quantization**: Quantize weights to INT4/INT8; activations remain in FP16. During matrix multiplication, weights are dequantized on-the-fly. Reduces memory (model fits on fewer GPUs) but computational savings are limited. Standard for LLM deployment. - **Weight + Activation Quantization**: Both weights and activations are quantized. Enables integer-only computation on specialized hardware (INT8 Tensor Cores). Requires calibration data to determine activation ranges. **LLM Quantization Methods** - **GPTQ**: Layer-wise quantization using the Optimal Brain Quantizer framework. For each layer, quantize weights to INT4 while minimizing the output error using Hessian information (second-order approximation). Processes one layer at a time, updating remaining weights to compensate for quantization error. Achieves INT4 with <1% perplexity degradation for most LLMs. - **AWQ (Activation-Aware Weight Quantization)**: Identifies "salient" weights (those multiplied by large activations) and scales them up before quantization to reduce their quantization error. Simple channel-wise scaling achieves better quality than GPTQ with faster quantization time. - **GGUF/llama.cpp Quantization**: Multiple quantization formats (Q4_K_M, Q5_K_S, Q8_0) optimized for CPU inference. Mixed-precision: more important layers (attention) get higher precision; less important layers (some FFN) get lower precision. Enables LLM inference on laptops and phones. - **FP8 (Floating-Point 8-bit)**: E4M3 (4 exponent, 3 mantissa) format preserves the dynamic range of floats while reducing precision. Native hardware support on H100 and later GPUs. 2× throughput vs. FP16 with minimal quality loss. Becoming the default training and inference precision. **Quantization-Aware Training (QAT)** Simulate quantization during training using straight-through estimators for gradient computation. The model learns to be robust to quantization effects. Higher quality than PTQ at the same bit-width but requires full training infrastructure. Used for INT4 and lower where PTQ quality degrades significantly. **Extreme Quantization (1-2 bits)** - **BitNet**: Binary or ternary weights ({-1, 0, +1}). Replaces multiplications with additions. 10-100× computational savings but significant quality loss for general tasks. Potentially viable for specialized inference hardware. - **1.58-bit (1, 0, -1)**: BitNet b1.58 uses ternary weights achieving surprisingly strong performance when the model is trained from scratch at this precision. Model Quantization is **the compression technology that makes large AI models deployable** — the mathematical mapping from high-precision to low-precision that trades a controlled amount of accuracy for dramatic reductions in memory, bandwidth, and compute, enabling the gap between model capability and hardware availability to be bridged economically.