gradient-based nas

**Gradient-Based NAS** is a **family of NAS methods that reformulate the architecture search as a continuous optimization problem** — making architecture parameters differentiable and optimizable via gradient descent, dramatically reducing search cost compared to RL or evolutionary approaches. **How Does Gradient-Based NAS Work?** - **Continuous Relaxation**: Replace discrete architecture choices with continuous weights (softmax over operations). - **Bilevel Optimization**: Alternately optimize architecture weights $alpha$ and network weights $w$. - **Methods**: DARTS, ProxylessNAS, FBNet, SNAS. - **Speed**: 1-4 GPU-days vs. 1000+ for RL-based methods. **Why It Matters** - **Efficiency**: Orders of magnitude faster than RL or evolutionary NAS. - **Simplicity**: Standard gradient descent — no specialized RL or EA machinery needed. - **Challenges**: Architecture collapse, weight entanglement, and the gap between continuous relaxation and discrete final architecture. **Gradient-Based NAS** is **turning architecture search into gradient descent** — the insight that made neural architecture search practical for everyday use.