int4

INT4 (4-bit integer) quantization aggressively compresses model weights to 4 bits per parameter, achieving 8× memory reduction versus FP32 and enabling large models to run on consumer hardware. Methods: (1) GPTQ (post-training, layer-wise quantization using Hessian information to minimize error—one-shot, fast), (2) AWQ (Activation-aware Weight Quantization—protects salient weights based on activation magnitudes), (3) GGUF Q4_K_M (k-quant with mixed precision—important weights get more bits), (4) NF4 (4-bit NormalFloat used in QLoRA—information-theoretically optimal for normally distributed weights). Memory examples: 7B model FP16=14GB → INT4=3.5GB (fits on 4GB GPU); 70B model FP16=140GB → INT4=35GB (fits on single GPU). Quality: perplexity increase typically 0.1-0.5 points for well-calibrated 4-bit vs. FP16 on large models (>7B). Below 4-bit (2-3 bit): significant quality degradation for most tasks. Inference: INT4 requires dequantization to FP16 for compute (memory savings, not compute speedup on standard hardware). W4A16 (4-bit weights, 16-bit activations) is the practical sweet spot for LLM deployment.