rigging the lottery

**Rigging the Lottery (RigL)** is a **state-of-the-art Dynamic Sparse Training algorithm** — that uses gradient information to intelligently regrow pruned connections, achieving dense-network-level accuracy while training with a fixed sparse computational budget. **What Is RigL?** - **Key Innovation**: Use the *gradient magnitude* of currently-zero (inactive) weights to decide which connections to grow back. - **Algorithm**: 1. Drop: Remove $k$ active weights with smallest magnitude. 2. Grow: Activate $k$ inactive weights with largest gradient (gradient tells us "this connection *would* have been useful"). 3. Maintain constant sparsity. - **Paper**: Evci et al. (2020, Google Brain). **Why It Matters** - **Performance**: First sparse training method to match dense baselines on ImageNet at 90% sparsity. - **Efficiency**: 3-5x training FLOPs savings vs dense training. - **Principled**: The gradient-based grow criterion is theoretically motivated. **RigL** is **intelligent network rewiring** — using gradient signals as a compass to navigate the space of sparse architectures during training.