gradient flow preservation

**Gradient Flow Preservation** is a **design principle for pruning and sparse training** — ensuring that removing weights does not disrupt the backpropagation signal, keeping gradient magnitudes stable across layers to prevent training collapse. **What Is Gradient Flow Preservation?** - **Problem**: Aggressive pruning can create "dead zones" where gradients vanish, causing layers to stop learning. - **Metrics**: Checking the Jacobian singular values, layer-wise gradient norms, or signal propagation theory. - **Solutions**: - **Balanced Pruning**: Ensure each layer retains a minimum number of connections. - **Skip Connections**: ResNet-style shortcut connections maintain gradient highways even if main path is heavily pruned. - **Dynamic Regrowth**: DST methods (RigL) regrow connections in gradient-starved regions. **Why It Matters** - **Trainability**: A pruned network that can't propagate gradients is useless regardless of its theoretical capacity. - **Depth Sensitivity**: Deeper networks are more fragile. Preserving flow is critical for 100+ layer architectures. **Gradient Flow Preservation** is **keeping the neural highway open** — ensuring that information can flow backward for learning no matter how sparse the network becomes.