winning tickets

**Winning Tickets** are the **specific sparse sub-networks identified by the Lottery Ticket Hypothesis** — sub-networks that, when trained from their original random initialization, achieve comparable performance to the full dense network. **What Are Winning Tickets?** - **Definition**: A mask $m$ over weights $ heta_0$ such that training $m odot heta_0$ achieves accuracy $geq$ training $ heta_0$ in $leq$ iterations. - **Properties**: - **Initialization Dependent**: The ticket only works with its *original* random init, not a new random init. - **Transferable**: Tickets found on one task often transfer to related tasks. - **Stable**: Late Rewinding (resetting to iteration $k$ instead of $0$) improves stability for large networks. **Why They Matter** - **Sparse Training**: If we can identify tickets early, we can train only the essential connections from the start. - **Generalization**: Winning tickets often generalize better (fewer parameters = less overfitting). - **Hardware**: Could enable training directly on edge devices if tickets are found cheaply. **Winning Tickets** are **the diamonds in the rough** — proving that neural network training is really a search problem for the right sparse structure.