quant

**Quantization for LLMs** **What is Quantization?** Quantization reduces the numerical precision of model weights from 32-bit or 16-bit floating point to lower bit widths (8-bit, 4-bit, or even 2-bit integers), dramatically reducing memory usage and improving inference speed. **Quantization Methods Comparison** | Method | Bits | Memory Reduction | Quality Impact | Speed | |--------|------|------------------|----------------|-------| | FP16 | 16 | 2x baseline | None | Good | | INT8 | 8 | 4x baseline | Minimal | Fast | | GPTQ | 4 | 8x baseline | Small | Fast | | AWQ | 4 | 8x baseline | Smaller | Fast | | GGUF | 2-8 | Variable | Varies | CPU-friendly | | FP8 | 8 | 2x baseline | None (H100) | Native | **Popular Quantization Techniques** **GPTQ (GPT Quantization)** - Post-training quantization using second-order optimization - Widely supported in transformers library - Good for GPU inference **AWQ (Activation-aware Weight Quantization)** - Preserves salient weights based on activation patterns - Generally better quality than GPTQ at same bit width - Best for production deployments **GGUF (llama.cpp format)** - Flexible quantization levels (Q2_K to Q8_0) - Optimized for CPU inference - Popular for local LLM deployment **Practical Example** A 70B parameter model: - FP16: 140GB VRAM (needs 2x A100 80GB) - INT8: 70GB VRAM (fits on 1x A100 80GB) - INT4: 35GB VRAM (fits on 1x A100 40GB) **When to Use Quantization** - **Production inference**: Almost always use INT8 or INT4 - **Development/training**: Keep FP16/BF16 - **Edge deployment**: Use aggressive quantization (4-bit or lower)