model compression
**Model compression refers to a family of techniques that reduce the size, memory footprint, and computational cost of neural networks while preserving as much accuracy as possible.** As transformer-based models have scaled to hundreds of billions of parameters, deploying them efficiently — on cloud GPUs, edge devices, or mobile phones — requires shrinking the model, speeding up inference, or both. The four primary compression strategies are quantization (reducing numerical precision), pruning (removing unnecessary weights), knowledge distillation (training a smaller model to mimic a larger one), and low-rank factorization (approximating weight matrices with lower-dimensional representations). These techniques are not mutually exclusive; production deployments routinely combine two or three of them to achieve the required latency, throughput, and memory targets.
**Quantization replaces high-precision floating-point weights and activations with lower-precision representations.** The most common path is FP32 to FP16 (half precision), which halves memory and doubles throughput on GPUs with tensor cores, with negligible accuracy loss for most models. INT8 quantization reduces memory by 4x compared to FP32 and is widely supported by inference frameworks like TensorRT, ONNX Runtime, and vLLM. Aggressive 4-bit quantization (INT4, NF4) has become practical for large language models thanks to techniques like GPTQ, AWQ, and the QLoRA approach that fine-tunes in 4-bit. The key challenge is that not all layers tolerate the same quantization level — attention projection matrices and the first and last layers are typically more sensitive than feed-forward layers. Post-training quantization (PTQ) applies quantization after training with a small calibration dataset, while quantization-aware training (QAT) simulates low-precision arithmetic during training, producing models that are more robust to precision reduction but requiring full retraining. Weight-only quantization reduces memory without changing activation precision, which is particularly effective for memory-bandwidth-bound inference where the bottleneck is loading weights from HBM rather than computing on them.
**Pruning removes weights, neurons, attention heads, or entire layers that contribute minimally to model output.** Unstructured pruning zeros out individual weights based on magnitude or gradient information, achieving high sparsity ratios (90 percent or more) but requiring sparse matrix formats and hardware support to translate sparsity into actual speedup. Structured pruning removes entire rows, columns, attention heads, or layers, producing dense sub-networks that run efficiently on standard hardware without special sparse kernels. The lottery ticket hypothesis (Frankle and Carlin, 2019) showed that dense networks contain sparse sub-networks that, when trained in isolation from the same initialization, match the full model's accuracy. SparseGPT demonstrated that large language models can be pruned to 50-60 percent sparsity in one shot with minimal accuracy degradation by solving a layer-wise reconstruction problem. NVIDIA's Ampere and Hopper GPUs support 2:4 structured sparsity natively, where exactly two of every four weights are zero, providing a 2x speedup on sparse tensor cores.
**Knowledge distillation trains a smaller student model to reproduce the behavior of a larger teacher model.** Rather than training on hard labels alone, the student learns from the teacher's soft probability distributions (logits), which contain richer information about inter-class relationships and uncertainty. Hinton et al. (2015) introduced the temperature-scaled softmax that smooths the teacher's output distribution, making subtle probability differences more visible to the student. For large language models, distillation can target next-token probability distributions, intermediate hidden states, or attention patterns. The student typically has fewer layers, smaller hidden dimensions, or fewer attention heads than the teacher. DistilBERT achieved 97 percent of BERT's performance with 40 percent fewer parameters and 60 percent faster inference. For modern LLMs, distillation is often combined with synthetic data generation: the teacher generates high-quality training examples that the student learns from, effectively transferring the teacher's reasoning capability into a smaller architecture.
**Low-rank factorization and parameter-efficient methods reduce model size by exploiting redundancy in weight matrices.** LoRA (Low-Rank Adaptation) freezes the original model weights and adds small trainable low-rank matrices to each layer, reducing the number of trainable parameters by 10-100x during fine-tuning. While LoRA was designed for efficient fine-tuning rather than compression per se, the merged LoRA weights can be combined with quantization (QLoRA) for deployment. Matrix factorization decomposes a weight matrix W of dimensions m x n into two smaller matrices of dimensions m x r and r x n, where the rank r is much smaller than min(m, n). Tensor decomposition extends this idea to multi-dimensional weight tensors. These methods are particularly effective for attention projection matrices and embedding layers, where significant redundancy exists.
| Technique | Memory reduction | Speedup | Accuracy impact | Hardware requirements | Best for |
|---|---|---|---|---|---|
| FP16 quantization | 2x | 1.5-2x | Negligible | Tensor cores (standard GPUs) | Universal baseline |
| INT8 quantization (PTQ) | 4x | 2-3x | Less than 1 percent loss typical | INT8 tensor cores | Cloud inference |
| INT4 quantization (GPTQ/AWQ) | 8x | 3-4x | 1-3 percent loss, task-dependent | INT4 support or dequantize on-the-fly | LLM serving on consumer GPUs |
| Unstructured pruning (90 percent) | Up to 10x | Requires sparse hardware | Variable; retraining helps | Sparse tensor cores or custom accelerators | Research, specialized hardware |
| Structured pruning (heads/layers) | 1.5-3x | 1.5-3x (no special hardware) | Moderate; depends on redundancy | Standard hardware | Edge deployment |
| Knowledge distillation | 3-10x (smaller architecture) | 3-10x | 1-5 percent loss typical | Standard hardware | Mobile, latency-critical applications |
| LoRA (fine-tuning) | Trainable params reduced 10-100x | Training speedup only | Comparable to full fine-tuning | Standard hardware | Efficient fine-tuning and adaptation |
```svg
```
**Hardware-aware compression tailors the optimization strategy to the target accelerator's capabilities.** NVIDIA GPUs with tensor cores achieve peak throughput at specific precision formats (FP16, INT8, FP8 on Hopper), so quantizing to a precision the hardware cannot accelerate provides no speedup. Similarly, structured pruning that removes entire channels or attention heads produces smaller dense matrices that benefit directly from existing GEMM kernels, while unstructured sparsity requires dedicated sparse matrix support available only on Ampere and later architectures. Mobile NPUs from Qualcomm, Apple, and MediaTek each support different precision formats and operator sets, so compression for on-device deployment must account for the specific hardware target. The emerging practice is to profile the model on target hardware, identify the bottleneck (compute-bound vs memory-bandwidth-bound), and select compression techniques accordingly — quantization primarily helps memory-bound workloads while pruning and distillation help compute-bound ones.
**The economics of model compression directly impact AI deployment at scale.** Serving a 70-billion-parameter model in FP16 requires at least 140 GB of GPU memory — two A100 80GB GPUs or one H100 — at a cloud cost of roughly 4-8 dollars per hour. Quantizing to INT4 reduces the memory footprint to approximately 35 GB, fitting on a single GPU and cutting serving costs by half or more. For consumer applications, compression determines whether a model runs locally on a laptop (16 GB RAM limits models to roughly 13 billion parameters in 4-bit) or requires cloud inference. Knowledge distillation can reduce a 405-billion-parameter model to an 8-billion-parameter student that runs on a single consumer GPU while retaining 85-90 percent of the teacher's capability. These economics drive the rapid adoption of compression techniques across the industry, making them as important to practical AI deployment as the model architecture itself.