smoothquant

SmoothQuant is a quantization technique that migrates the quantization difficulty from activations (which have outliers) to weights (which are well-behaved), enabling accurate INT8 quantization of both activations and weights in large language models. The challenge: LLM activations have outlier channels with values 10-100× larger than others; quantizing activations uniformly causes large errors on outlier channels. Key insight: weights are smooth (no outliers); activations have outliers concentrated in specific channels; can rebalance difficulty by scaling. SmoothQuant operation: multiply activations by scaling factor s, divide weights by s (mathematically equivalent); choose s to equalize quantization difficulty. Scaling factor: per-channel s computed from activation/weight magnitude ratios; α hyperparameter controls balance between weight and activation quantization difficulty. Result: scaled activations have no outliers (quantizes well), scaled weights slightly less smooth but still quantizable; overall INT8 accuracy much better. No retraining: apply SmoothQuant using calibration data to estimate outlier channels; fast post-training quantization. Integration: compatible with standard quantized inference kernels; no specialized hardware needed. Performance: enables W8A8 (8-bit weights and activations) with minimal accuracy loss on LLMs where activation-only quantization fails. Comparison: better than activation-only quantization, simpler than mixed-precision. SmoothQuant makes full INT8 inference practical for transformer models.