Dynamic Sparse Training (DST) is a training paradigm where the sparse network topology changes during training — allowing connections to be pruned and regrown dynamically, so the network can discover the optimal sparse structure while training.

What Is DST?

• Key Difference from Pruning: Pruning starts dense and removes. DST starts sparse and rearranges.
• Algorithm (SET/RigL):

1. Initialize a sparse random network. 2. Train for $Delta T$ steps. 3. Drop: Remove connections with smallest magnitude. 4. Grow: Add new connections with largest gradient. 5. Repeat.

• Budget: Total number of non-zero weights stays constant throughout.

Why It Matters

• Training Efficiency: Never allocates memory for dense matrices. The FLOPs budget is always sparse.
• Performance: RigL matches dense training accuracy at 90% sparsity.
• Exploration: Allows the network to explore different topologies and find better sparse structures.

Dynamic Sparse Training is neural plasticity — mimicking the brain's ability to rewire connections based on experience.