differentiable architecture search

**DARTS** (Differentiable Architecture Search) is a **gradient-based NAS method that makes the architecture search differentiable** — by relaxing the discrete architecture choice into a continuous optimization problem, enabling efficient search using standard gradient descent in orders of magnitude less time. **How Does DARTS Work?** - **Mixed Operations**: Each edge in the search graph has all possible operations running in parallel, weighted by architecture parameters $alpha$. - **Softmax**: $ar{o}(x) = sum_k frac{exp(alpha_k)}{sum_j exp(alpha_j)} cdot o_k(x)$ - **Bilevel Optimization**: Alternate between optimizing architecture weights $alpha$ and network weights $w$. - **Discretization**: After search, select the operation with highest $alpha$ on each edge. **Why It Matters** - **Speed**: 1-4 GPU-days vs. 1000+ GPU-days for RL-based NAS. - **Simplicity**: Standard gradient descent — no RL controllers or evolutionary populations needed. - **Limitation**: Prone to architecture collapse (all edges converge to skip connections or parameter-free ops). **DARTS** is **gradient descent for architecture design** — searching the space of possible networks as smoothly as training the weights of a single network.