post training quantization
**Post-Training Quantization (PTQ)** is the **model compression technique that reduces the numerical precision of neural network weights and activations after training is complete** — without requiring retraining or fine-tuning, converting float32/bfloat16 models to int8, int4, or lower precision to reduce memory footprint by 2–8× and increase inference throughput by 1.5–4× on hardware with quantized compute support, at a small accuracy cost that modern algorithms minimize through careful calibration.
**Why LLMs Need Specialized PTQ**
- Standard PTQ (per-tensor, per-channel) works well for CNNs but struggles with LLMs.
- LLM activations contain **outliers**: a few channels have 100× larger values than others.
- Naively quantizing these outliers causes massive accuracy loss.
- Solution: per-channel/group quantization, outlier-aware methods, weight-only quantization.
**GPTQ (Frantar et al., 2022)**
- Applies Optimal Brain Quantization (OBQ) row-by-row to transformer weight matrices.
- Quantizes weights to int4 using second-order Hessian information → minimizes quantization error.
- Key insight: Quantize one weight at a time, update remaining weights to compensate for error.
- Speed: Quantizes 175B GPT model in ~4 hours on a single GPU.
- Result: int4 GPTQ quality ≈ int8 naive quantization for most LLMs.
```python
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
quantize_config = BaseQuantizeConfig(
bits=4, # int4
group_size=128, # quantize in groups of 128 weights
desc_act=False, # disable activation order for speed
)
model = AutoGPTQForCausalLM.from_pretrained(model_path, quantize_config)
model.quantize(calibration_data) # Calibrate on ~128 samples
```
**AWQ (Activation-aware Weight Quantization)**
- Observes that a small fraction (~1%) of weights are "salient" — high activation scale → large quantization error if rounded.
- Solution: Scale salient weights up before quantization → scale activations down to compensate.
- Math: (s·W)·(X/s) = W·X but (s·W) quantizes more accurately since s > 1.
- No retraining: Only ~1% of weights are scaled, rest are straightforward int4.
- Result: AWQ generally outperforms GPTQ at very low bit-widths (< 4 bit).
**SmoothQuant**
- Problem: Activation outliers make int8 activation quantization difficult.
- Solution: Transfer quantization difficulty from activations to weights via per-channel scaling.
- Math: Y = (Xdiag(s)⁻¹)·(diag(s)W) where s smooths activation dynamic range.
- Enables W8A8 (int8 weights + int8 activations) → uses tensor core INT8 arithmetic → 1.6–2× faster than FP16.
**Quantization Granularity**
| Granularity | Description | Accuracy | Overhead |
|-------------|-------------|----------|----------|
| Per-tensor | Single scale for entire tensor | Lowest | Minimal |
| Per-channel | Scale per output channel | Good | Small |
| Per-group | Scale per 64/128 weights | Better | Moderate |
| Per-token (act) | Scale per activation token | Best | Runtime |
**Key Metrics and Trade-offs**
- **Perplexity delta**: int4 GPTQ: +0.2–0.5 perplexity on WikiText2 vs FP16 baseline.
- **Memory reduction**: FP16 (2 bytes) → INT4 (0.5 bytes) = 4× reduction.
- **Throughput**: INT4 weight-only: 1.5–2.5× faster generation (memory bandwidth limited).
- **W8A8**: 1.5–2× faster for batch inference (compute-limited scenarios).
**Calibration Data**
- PTQ requires small calibration dataset (128–512 samples) to compute activation statistics.
- Quality matters: calibration data should match downstream task distribution.
- Common: WikiText, C4, or task-specific examples.
Post-training quantization is **the practical gateway to deploying state-of-the-art LLMs on accessible hardware** — by compressing 70B parameter models from 140GB in FP16 to 35GB in INT4 without costly retraining, PTQ methods like GPTQ and AWQ have made it possible to run frontier-scale models on single workstation GPUs, democratizing LLM inference and enabling the local AI ecosystem that powers privacy-preserving, offline-capable AI applications.