awq

AWQ (Activation-Aware Weight Quantization) achieves high-quality 4-bit weight quantization by identifying and preserving salient weights based on activation patterns, outperforming uniform quantization while enabling efficient inference. Key insight: not all weights equally important; weights multiplied by large activations (salient channels) matter more for model output; protecting these weights during quantization preserves quality. Method: analyze activation statistics to identify salient channels; scale these channels to protect from quantization error; scale back after quantization. Per-channel scaling: learned scales protect important weights; scales absorbed into adjacent layers for zero runtime overhead. No retraining: AWQ works post-training; analyze activations on calibration data, compute scales, quantize weights—fast process. Weight-only quantization: quantizes weights to 4-bit but keeps activations in higher precision; balanced approach for memory-bound inference. Comparison to GPTQ: AWQ is simpler and faster to apply; GPTQ uses reconstruction optimization. Quality: 4-bit AWQ approaches 16-bit quality on many models; minimal perplexity increase. Deployment: efficient kernels (CUDA, TensorRT-LLM) for fast 4-bit inference. Combining with other techniques: AWQ weights work with speculative decoding, KV cache optimization, and other inference optimizations. AWQ makes 4-bit quantization practical for production LLM deployment.