sparse model topology updates

**Dynamic Sparse Training (DST)** is a **training paradigm where the sparse network topology changes during training** — allowing connections to be pruned and regrown dynamically, so the network can discover the optimal sparse structure while training. **What Is DST?** - **Key Difference from Pruning**: Pruning starts dense and removes. DST starts sparse and rearranges. - **Algorithm (SET/RigL)**: 1. Initialize a sparse random network. 2. Train for $Delta T$ steps. 3. Drop: Remove connections with smallest magnitude. 4. Grow: Add new connections with largest gradient. 5. Repeat. - **Budget**: Total number of non-zero weights stays constant throughout. **Why It Matters** - **Training Efficiency**: Never allocates memory for dense matrices. The FLOPs budget is always sparse. - **Performance**: RigL matches dense training accuracy at 90% sparsity. - **Exploration**: Allows the network to explore different topologies and find better sparse structures. **Dynamic Sparse Training** is **neural plasticity** — mimicking the brain's ability to rewire connections based on experience.