model quantization basics

**Model Quantization** — reducing the numerical precision of model weights and activations (e.g., FP32 → INT8 → INT4) to decrease model size, memory usage, and inference latency. **Precision Levels** | Format | Bits | Size Reduction | Accuracy Impact | |---|---|---|---| | FP32 | 32 | 1x (baseline) | None | | FP16/BF16 | 16 | 2x | Minimal | | INT8 | 8 | 4x | Small | | INT4 | 4 | 8x | Moderate | | INT2/Binary | 2/1 | 16-32x | Significant | **Methods** - **Post-Training Quantization (PTQ)**: Quantize a pre-trained model without retraining. Fast and easy. Some accuracy loss - **Quantization-Aware Training (QAT)**: Simulate quantization during training so the model learns to be robust to it. Better accuracy but requires full training - **GPTQ**: PTQ method optimized for large language models (row-by-row quantization) - **AWQ (Activation-Aware)**: Protect important weights from quantization error **Hardware Support** - NVIDIA Tensor Cores: INT8 and INT4 acceleration - Apple Neural Engine: INT8 - Qualcomm Hexagon: INT8/INT4 - Intel AMX: INT8/BF16 **Practical Impact** - LLaMA 70B FP16: 140GB → INT4: 35GB (fits on single GPU) - 2-4x inference speedup with INT8 on supported hardware **Quantization** is essential for deploying large models in production — it's how billion-parameter models run on consumer devices.