Model Quantization is the inference optimization technique that reduces the numerical precision of neural network weights and activations from 32-bit or 16-bit floating-point to lower bit-widths (8-bit, 4-bit, or even 2-bit integers) — shrinking model memory footprint by 2-8x, accelerating computation on hardware with integer execution units, and enabling deployment of large models on resource-constrained devices with minimal quality degradation.

Why Quantize

A 70B parameter model in FP16 requires 140 GB of memory — exceeding the capacity of any single consumer GPU. Quantizing to 4-bit reduces this to ~35 GB, fitting on a single 48GB GPU. Beyond memory, integer arithmetic is 2-4x faster than floating-point on most hardware, and reduced memory bandwidth (the primary bottleneck for LLM inference) directly increases tokens-per-second.

Post-Training Quantization (PTQ)

Quantize a pre-trained model without retraining:

• Round-to-Nearest (RTN): Simply round each weight to the nearest quantized value. Works well at INT8; significant quality loss at INT4.
• GPTQ: Uses approximate second-order information (Hessian) to quantize weights one at a time, adjusting remaining weights to compensate for the quantization error. Achieves near-lossless INT4 weight quantization for LLMs.
• AWQ (Activation-Aware Weight Quantization): Identifies the small fraction (~1%) of weight channels that are critical for maintaining accuracy (those corresponding to large activation magnitudes) and protects them with per-channel scaling before quantization.
• SqueezeLLM / QuIP: Use non-uniform quantization and incoherence processing to push quality at extreme (2-3 bit) compression.

Quantization-Aware Training (QAT)

Simulate quantization during training by inserting fake-quantization nodes that round weights/activations during the forward pass but pass gradients through using the straight-through estimator. The model learns to be robust to quantization noise, consistently outperforming PTQ at the same bit-width but requiring a full training run.

Quantization Formats

Mixed-Precision and Group Quantization

Rather than quantizing all weights to the same precision, modern methods use group quantization (quantize in blocks of 32-128 weights with per-group scale factors) and mixed precision (keep sensitive layers at higher precision). This provides fine-grained control over the accuracy-compression tradeoff.

Model Quantization is the compression technique that made billion-parameter AI accessible on consumer hardware — proving that neural networks are massively over-precise and that most of their intelligence survives dramatic precision reduction.