gradient-based pruning

**Gradient-Based Pruning** is a **more principled pruning criterion** — using gradient information (or second-order derivatives) to estimate the impact of removing a weight on the loss function, rather than relying on magnitude alone. **What Is Gradient-Based Pruning?** - **Idea**: A weight is important if removing it causes a large increase in loss. - **First-Order (Taylor)**: Importance $approx |w cdot partial L / partial w|$ (weight times gradient). - **Second-Order (OBS/OBD)**: Uses the Hessian to estimate the curvature of the loss landscape around each weight. - **Fisher Information**: Uses the Fisher matrix as an approximation to the Hessian. **Why It Matters** - **Accuracy**: Can identify important small weights that magnitude pruning would incorrectly remove. - **Layer Sensitivity**: Naturally adapts pruning ratios per layer based on gradient flow. - **Cost**: More expensive than magnitude pruning (requires backward pass), but more precise. **Gradient-Based Pruning** is **informed surgery** — using diagnostic information about the network's health to decide what to remove.