Model Quantization is the compression technique that reduces neural network weight and activation precision from 32-bit or 16-bit floating point to lower bit-widths (INT8, INT4, FP8, even 1-2 bits) — shrinking model size by 2-8×, reducing memory bandwidth requirements proportionally, and enabling faster inference on hardware with specialized low-precision compute units, making it essential for deploying large language models on consumer GPUs and edge devices.

Quantization Fundamentals

Quantization maps continuous float values to discrete integer levels: x_q = round(x / scale) + zero_point, where scale = (max-min)/(2^b-1) for b-bit quantization. Dequantization recovers an approximation: x ≈ (x_q - zero_point) × scale. The quantization error depends on bit-width and the distribution of values.

Post-Training Quantization (PTQ)

Quantize a pre-trained model without retraining:

• Weight-Only Quantization: Quantize weights to INT4/INT8; activations remain in FP16. During matrix multiplication, weights are dequantized on-the-fly. Reduces memory (model fits on fewer GPUs) but computational savings are limited. Standard for LLM deployment.
• Weight + Activation Quantization: Both weights and activations are quantized. Enables integer-only computation on specialized hardware (INT8 Tensor Cores). Requires calibration data to determine activation ranges.

LLM Quantization Methods

• GPTQ: Layer-wise quantization using the Optimal Brain Quantizer framework. For each layer, quantize weights to INT4 while minimizing the output error using Hessian information (second-order approximation). Processes one layer at a time, updating remaining weights to compensate for quantization error. Achieves INT4 with <1% perplexity degradation for most LLMs.

• AWQ (Activation-Aware Weight Quantization): Identifies "salient" weights (those multiplied by large activations) and scales them up before quantization to reduce their quantization error. Simple channel-wise scaling achieves better quality than GPTQ with faster quantization time.

• GGUF/llama.cpp Quantization: Multiple quantization formats (Q4_K_M, Q5_K_S, Q8_0) optimized for CPU inference. Mixed-precision: more important layers (attention) get higher precision; less important layers (some FFN) get lower precision. Enables LLM inference on laptops and phones.

• FP8 (Floating-Point 8-bit): E4M3 (4 exponent, 3 mantissa) format preserves the dynamic range of floats while reducing precision. Native hardware support on H100 and later GPUs. 2× throughput vs. FP16 with minimal quality loss. Becoming the default training and inference precision.

Quantization-Aware Training (QAT)

Simulate quantization during training using straight-through estimators for gradient computation. The model learns to be robust to quantization effects. Higher quality than PTQ at the same bit-width but requires full training infrastructure. Used for INT4 and lower where PTQ quality degrades significantly.

Extreme Quantization (1-2 bits)

• BitNet: Binary or ternary weights ({-1, 0, +1}). Replaces multiplications with additions. 10-100× computational savings but significant quality loss for general tasks. Potentially viable for specialized inference hardware.
• 1.58-bit (1, 0, -1): BitNet b1.58 uses ternary weights achieving surprisingly strong performance when the model is trained from scratch at this precision.

Model Quantization is the compression technology that makes large AI models deployable — the mathematical mapping from high-precision to low-precision that trades a controlled amount of accuracy for dramatic reductions in memory, bandwidth, and compute, enabling the gap between model capability and hardware availability to be bridged economically.