Model Compression Techniques are the family of methods that reduce neural network size, memory footprint, and computational cost while preserving accuracy — including pruning (removing unnecessary weights or neurons), quantization (reducing precision), knowledge distillation (training smaller models), and architecture search for efficient designs, enabling deployment on resource-constrained devices and reducing inference costs.

Magnitude-Based Pruning:

• Unstructured Pruning: removes individual weights with smallest absolute values; prune weights where |w| < threshold or keep top-k% by magnitude; achieves high compression ratios (90-95% sparsity) with minimal accuracy loss but requires sparse matrix operations for speedup; standard dense hardware doesn't accelerate unstructured sparsity
• Structured Pruning: removes entire channels, filters, or layers rather than individual weights; maintains dense computation that runs efficiently on standard hardware; typical compression: 30-50% of channels removed with 1-3% accuracy loss; directly reduces FLOPs and memory without specialized kernels
• Iterative Magnitude Pruning (IMP): train → prune lowest magnitude weights → retrain → repeat; gradual pruning over multiple iterations preserves accuracy better than one-shot pruning; Han et al. (2015) achieved 90% sparsity on AlexNet with minimal accuracy loss
• Pruning Schedule: pruning rate typically follows cubic schedule: s_t = s_f + (s_i - s_f)(1 - t/T)³ where s_i is initial sparsity, s_f is final sparsity, t is current step, T is total steps; gradual pruning allows the network to adapt to increasing sparsity

Lottery Ticket Hypothesis:

• Core Idea: dense networks contain sparse subnetworks (winning tickets) that, when trained in isolation from initialization, match the full network's performance; finding these subnetworks enables training sparse models from scratch rather than pruning dense models
• Winning Ticket Identification: train dense network, prune to sparsity s, rewind weights to initialization (or early training checkpoint), retrain the sparse mask; the resulting sparse network achieves comparable accuracy to the original dense network
• Implications: suggests that much of a network's capacity is redundant; the critical factor is finding the right sparse connectivity pattern, not the final weight values; challenges the necessity of overparameterization for training
• Practical Limitations: finding winning tickets requires training the full dense network first (no computational savings during search); works well at moderate sparsity (50-80%) but breaks down at extreme sparsity (>95%); more of a scientific insight than a practical compression method

Structured Pruning Methods:

• Channel Pruning: removes entire convolutional filters/channels based on importance metrics; importance measured by L1/L2 norm of filter weights, activation statistics, or gradient-based sensitivity; directly reduces FLOPs and memory with no specialized hardware needed
• Layer Pruning: removes entire layers from deep networks; surprisingly, many layers can be removed with minimal accuracy loss; BERT can drop 25-50% of layers with <2% accuracy degradation; requires careful selection of which layers to remove (middle layers often more redundant than early/late)
• Attention Head Pruning: removes entire attention heads in Transformers; many heads are redundant or attend to similar patterns; pruning 20-40% of heads typically has minimal impact; enables faster attention computation and reduced KV cache memory
• Width Pruning: reduces hidden dimensions uniformly across all layers; simpler than selective channel pruning but less efficient (removes capacity uniformly rather than targeting redundant channels)

Dynamic and Adaptive Pruning:

• Dynamic Sparse Training: maintains constant sparsity throughout training by periodically removing low-magnitude weights and growing new connections; RigL (Rigging the Lottery) grows weights with largest gradient magnitudes; enables training sparse networks from scratch without dense pre-training
• Gradual Magnitude Pruning (GMP): increases sparsity gradually during training following a schedule; used in TensorFlow Model Optimization Toolkit; simpler than iterative pruning (single training run) but typically achieves lower compression ratios
• Movement Pruning: prunes weights that move toward zero during training rather than weights with small magnitude; considers weight trajectory, not just current value; achieves better accuracy-sparsity trade-offs for Transformers
• Soft Pruning: uses continuous relaxation of binary masks (differentiable pruning); learns pruning masks via gradient descent; L0 regularization encourages sparsity; enables end-to-end pruning without iterative train-prune cycles

Pruning for Specific Architectures:

• Transformer Pruning: attention heads, FFN intermediate dimensions, and entire layers can be pruned; structured pruning of FFN (removing rows/columns) is most effective; CoFi (Coarse-to-Fine Pruning) achieves 50% compression with <1% accuracy loss on BERT
• CNN Pruning: filter pruning is standard; early layers are more sensitive (contain low-level features); later layers are more redundant; pruning ratios typically vary by layer (10-30% early, 50-70% late)
• LLM Pruning: SparseGPT enables one-shot pruning of LLMs to 50-60% sparsity with minimal perplexity increase; Wanda (Pruning by Weights and Activations) uses activation statistics to identify important weights; enables running 70B models with 50% fewer parameters

Combining Compression Techniques:

• Pruning + Quantization: prune to 50% sparsity, then quantize to INT8; achieves 8-10× compression with 1-2% accuracy loss; order matters — typically prune first, then quantize
• Pruning + Distillation: prune the teacher model, then distill to a smaller student; combines structural compression (pruning) with capacity transfer (distillation); achieves better accuracy than pruning alone
• AutoML for Compression: neural architecture search finds optimal pruning ratios per layer; NetAdapt, AMC (AutoML for Model Compression) automatically determine layer-wise compression policies; achieves better accuracy-efficiency trade-offs than uniform pruning

Model compression techniques are essential for democratizing AI deployment — enabling state-of-the-art models to run on smartphones, embedded devices, and edge hardware by removing the 50-90% of parameters that contribute minimally to accuracy, making advanced AI accessible beyond datacenter-scale infrastructure.