DARTS (Differentiable Architecture Search) is a gradient-based NAS method that makes the architecture search differentiable — by relaxing the discrete architecture choice into a continuous optimization problem, enabling efficient search using standard gradient descent in orders of magnitude less time.

How Does DARTS Work?

• Mixed Operations: Each edge in the search graph has all possible operations running in parallel, weighted by architecture parameters $alpha$.
• Softmax: $ar{o}(x) = sum_k frac{exp(alpha_k)}{sum_j exp(alpha_j)} cdot o_k(x)$
• Bilevel Optimization: Alternate between optimizing architecture weights $alpha$ and network weights $w$.
• Discretization: After search, select the operation with highest $alpha$ on each edge.

Why It Matters

• Speed: 1-4 GPU-days vs. 1000+ GPU-days for RL-based NAS.
• Simplicity: Standard gradient descent — no RL controllers or evolutionary populations needed.
• Limitation: Prone to architecture collapse (all edges converge to skip connections or parameter-free ops).

DARTS is gradient descent for architecture design — searching the space of possible networks as smoothly as training the weights of a single network.