Neural Network Pruning Methods are the algorithmic approaches for identifying and removing redundant parameters or structures from trained networks — using criteria such as weight magnitude, gradient information, activation statistics, or learned importance scores to determine which components can be eliminated with minimal impact on model performance, enabling systematic compression beyond simple magnitude thresholding.

Gradient-Based Pruning:

• Taylor Expansion Pruning: approximates the change in loss when removing a parameter using first-order Taylor expansion; importance I(w) ≈ |∂L/∂w · w| = |gradient · weight|; removes parameters with smallest importance score; captures both magnitude and gradient information
• Hessian-Based Pruning (Optimal Brain Damage): uses second-order information; importance I(w) ≈ 0.5 · ∂²L/∂w² · w²; accounts for curvature of loss landscape; more accurate than first-order but computationally expensive (requires Hessian diagonal)
• Fisher Information Pruning: uses Fisher information matrix to estimate parameter importance; I(w) = F_ii · w² where F_ii is diagonal Fisher; approximates expected gradient magnitude; more stable than instantaneous gradients
• Movement Pruning: prunes weights moving toward zero during fine-tuning; importance based on weight trajectory: I(w) = w · Σ_t ∂L/∂w_t; considers optimization dynamics rather than static weight values; particularly effective for Transformer fine-tuning

Activation-Based Pruning:

• Activation Magnitude Pruning: removes channels/neurons with consistently small activations; importance I(channel_i) = mean(|A_i|) over dataset; identifies channels that contribute little to network output; requires forward passes on representative data
• Activation Variance Pruning: removes channels with low activation variance; low variance indicates the channel produces similar outputs regardless of input; such channels provide limited discriminative information
• Wanda (Weights and Activations): combines weight magnitude and activation statistics; importance I(w_ij) = |w_ij| · ||a_j||² where a_j is input activation; prunes weights that are both small and receive small activations; enables one-shot LLM pruning with minimal perplexity increase
• Batch Normalization Scaling Factors: for networks with BatchNorm, the scaling factor γ indicates channel importance; channels with small γ contribute less to output; Network Slimming prunes channels with smallest γ values

Learned Pruning Masks:

• L0 Regularization: adds L0 penalty (count of non-zero weights) to loss; relaxed to continuous approximation using hard concrete distribution; learns binary masks via gradient descent; end-to-end differentiable pruning
• Gumbel-Softmax Pruning: uses Gumbel-Softmax trick to learn discrete pruning decisions; enables gradient-based optimization of discrete masks; temperature annealing gradually sharpens soft masks to hard binary decisions
• Variational Dropout: interprets dropout as variational inference; learns per-weight dropout rates; weights with high dropout rates are pruned; automatically discovers optimal sparsity pattern
• Lottery Ticket Rewinding: identifies winning tickets by training, pruning, and rewinding to early checkpoint (not initialization); rewinding to iteration 1000-5000 often works better than iteration 0; enables finding trainable sparse subnetworks

Structured Pruning Algorithms:

• ThiNet: prunes channels by analyzing their contribution to next layer's activations; solves optimization problem to find channels whose removal minimally affects next layer; greedy layer-by-layer pruning
• Channel Pruning via LASSO: formulates channel selection as LASSO regression problem; minimizes reconstruction error of next layer's input subject to L1 penalty; automatically determines number of channels to prune per layer
• Discrimination-Aware Channel Pruning: preserves channels that maximize class discrimination; uses Fisher criterion or class separation metrics; maintains discriminative power while reducing redundancy
• AutoML for Pruning (AMC): reinforcement learning agent learns layer-wise pruning ratios; reward is accuracy under resource constraint (FLOPs, latency); discovers non-uniform pruning policies that outperform uniform pruning

Dynamic and Adaptive Pruning:

• Dynamic Network Surgery: alternates between pruning (removing small weights) and splicing (recovering important pruned weights); allows recovery from incorrect pruning decisions; maintains sparsity while refining mask
• RigL (Rigging the Lottery): maintains constant sparsity throughout training; periodically drops smallest-magnitude weights and grows weights with largest gradient magnitudes; enables training sparse networks from scratch without dense pre-training
• Soft Threshold Reparameterization (STR): reparameterizes weights as w = s · θ where s is soft-thresholded; s = sign(α) · max(|α| - λ, 0); learns α via gradient descent; threshold λ controls sparsity; enables end-to-end sparse training
• Gradual Pruning: increases sparsity following schedule s_t = s_f · (1 - (1 - t/T)³); smooth transition from dense to sparse; allows network to adapt gradually; more stable than one-shot pruning

Pruning for Specific Objectives:

• Latency-Aware Pruning: prunes to minimize actual inference latency rather than FLOPs; uses hardware-specific latency lookup tables; accounts for memory access patterns, parallelism, and hardware-specific optimizations
• Energy-Aware Pruning: optimizes for energy consumption; memory access dominates energy cost; structured pruning (reducing memory footprint) more effective than unstructured (same memory, sparse compute)
• Accuracy-Preserving Pruning: binary search for maximum sparsity that maintains accuracy within threshold; conservative but guarantees performance; used when accuracy is critical
• Compression-Rate Targeting: prunes to achieve specific compression ratio; adjusts pruning threshold to hit target sparsity; useful for deployment with fixed memory budgets

Evaluation and Validation:

• Sensitivity Analysis: measures accuracy drop when pruning each layer independently; identifies sensitive layers (prune less) and robust layers (prune more); guides non-uniform pruning strategies
• Pruning Ratio Search: grid search or evolutionary search over per-layer pruning ratios; expensive but finds optimal compression-accuracy trade-off; can be amortized across multiple models
• Fine-Tuning Strategies: learning rate for fine-tuning typically 0.1-0.01× original training rate; longer fine-tuning (50-100 epochs) recovers more accuracy; knowledge distillation during fine-tuning further improves recovery
• Iterative vs One-Shot: iterative pruning (prune 20% → retrain → prune 20% → ...) achieves higher compression than one-shot (prune 80% once) but requires multiple training runs; one-shot preferred for efficiency if accuracy is acceptable

Neural network pruning methods represent the algorithmic sophistication behind model compression — moving beyond naive magnitude thresholding to principled approaches that consider gradients, activations, learned importance, and task-specific objectives, enabling practitioners to systematically compress models while preserving the capabilities that matter for their specific applications.