Quantization for LLMs

What is Quantization? Quantization reduces the numerical precision of model weights from 32-bit or 16-bit floating point to lower bit widths (8-bit, 4-bit, or even 2-bit integers), dramatically reducing memory usage and improving inference speed.

Quantization Methods Comparison

Popular Quantization Techniques

GPTQ (GPT Quantization)

• Post-training quantization using second-order optimization
• Widely supported in transformers library
• Good for GPU inference

AWQ (Activation-aware Weight Quantization)

• Preserves salient weights based on activation patterns
• Generally better quality than GPTQ at same bit width
• Best for production deployments

GGUF (llama.cpp format)

• Flexible quantization levels (Q2_K to Q8_0)
• Optimized for CPU inference
• Popular for local LLM deployment

Practical Example A 70B parameter model:

• FP16: 140GB VRAM (needs 2x A100 80GB)
• INT8: 70GB VRAM (fits on 1x A100 80GB)
• INT4: 35GB VRAM (fits on 1x A100 40GB)

When to Use Quantization

• Production inference: Almost always use INT8 or INT4
• Development/training: Keep FP16/BF16
• Edge deployment: Use aggressive quantization (4-bit or lower)