Structured Pruning is the model compression technique that removes entire structural units (channels, attention heads, layers, or filter groups) from neural networks — unlike unstructured pruning that zeros individual weights in a sparse pattern requiring specialized hardware, structured pruning eliminates complete computational blocks to produce smaller dense models that run faster on standard GPUs, CPUs, and mobile hardware without sparse matrix support, typically achieving 2-4× speedup with less than 1% accuracy loss when combined with fine-tuning.

What Is Structured Pruning?

• Definition: The systematic removal of entire structural components from a neural network — channels (filters) in CNNs, attention heads in transformers, or complete layers — producing a smaller dense model that requires no special sparse computation support.
• Structured vs. Unstructured: Unstructured pruning zeros individual weights (e.g., 90% sparsity) but the model retains its original dimensions and requires sparse matrix libraries for speedup. Structured pruning physically removes dimensions, producing a genuinely smaller model with standard dense operations.
• Importance Scoring: Each structural unit is assigned an importance score — based on weight magnitude (L1/L2 norm), gradient information (Taylor expansion), activation statistics, or learned gating parameters — and the least important units are removed.
• Fine-Tuning Recovery: After pruning, the model is fine-tuned on the original training data to recover accuracy lost from removing structure — typically 10-30 epochs of fine-tuning recovers most or all of the original accuracy.

Pruning Granularity Levels

Structured Pruning Methods

• Magnitude-Based: Rank channels/heads by L1 or L2 norm of their weights — remove the smallest. Simple and effective but doesn't account for interaction effects between channels.
• Taylor Expansion: Estimate each unit's contribution to the loss function using first-order Taylor approximation — importance = |activation × gradient|. More accurate than magnitude alone.
• Learned Pruning (Gating): Add learnable gate parameters (0 or 1) to each structural unit — train the gates with sparsity regularization so the network learns which units to remove. Methods include Network Slimming (batch norm scaling factors) and differentiable pruning.
• Sensitivity Analysis: Prune each layer independently and measure accuracy impact — layers with low sensitivity can be pruned aggressively while sensitive layers are preserved.

Structured Pruning for Transformers

• Head Pruning: Remove attention heads that contribute least to model output — research shows 20-40% of heads in BERT and GPT models can be removed with minimal accuracy loss.
• Width Pruning: Reduce the hidden dimension of feed-forward layers — the FFN layers (4× hidden size) contain significant redundancy and respond well to structured pruning.
• Layer Dropping: Remove entire transformer layers — deeper models often have redundant layers, and removing 25-50% of layers from over-parameterized models maintains most task performance.
• Depth + Width Combined: Jointly optimize which layers to remove and how much to slim remaining layers — achieving better compression than either approach alone.

Tools and Frameworks

• PyTorch Pruning: torch.nn.utils.prune provides structured pruning utilities — channel pruning, LN-structured pruning with custom importance criteria.
• Neural Network Intelligence (NNI): Microsoft's AutoML toolkit with structured pruning algorithms — FPGM, Taylor, activation-based pruning with automatic fine-tuning.
• Torch-Pruning (DepGraph): Dependency graph-based structural pruning — automatically handles complex architectures with skip connections and shared layers.
• NVIDIA ASP: Automatic Sparsity for 2:4 structured sparsity on Ampere+ GPUs — hardware-accelerated semi-structured pruning.

Structured pruning is the practical model compression technique that delivers real inference speedups on commodity hardware — removing entire channels, heads, and layers to produce smaller dense models that run 2-4× faster without requiring specialized sparse computation support, making it the go-to approach for deploying large models on resource-constrained devices.