Efficient Neural Architecture Search (NAS) is the automated discovery of optimal neural network architectures using weight-sharing, one-shot, or differentiable methods that reduce the search cost from thousands of GPU-days to a few GPU-hours — making architecture optimization practical for real-world deployment rather than requiring the massive computational budgets of early NAS approaches like NASNet that trained and evaluated thousands of independent networks.

The Evolution from Brute-Force to Efficient NAS

Early NAS (Zoph & Le 2017) used reinforcement learning to sample architectures and trained each from scratch to evaluate fitness — requiring 48,000 GPU-hours for CIFAR-10. This was computationally prohibitive for most organizations and larger datasets.

One-Shot / Weight-Sharing NAS

The key breakthrough was the supernet concept: train a single over-parameterized network (supernet) that contains all candidate architectures as sub-networks. Each sub-network (subnet) shares weights with the supernet.

Supernet (one-time training cost):
  Layer 1: [conv3x3 | conv5x5 | sep_conv3x3 | skip_connect | none]
  Layer 2: [conv3x3 | conv5x5 | sep_conv3x3 | skip_connect | none]
  ...
  
Search: Sample subnets → evaluate using inherited weights → rank
Result: Best subnet architecture found without retraining

Methods include:

• ENAS: Controller RNN samples subnets; shared weights updated via REINFORCE.
• Once-for-All (OFA): Progressive shrinking trains a supernet supporting variable depth/width/resolution — deploy any subnet without retraining.
• BigNAS: Single-stage training with sandwich sampling (largest + smallest + random subnets per step).

Differentiable NAS (DARTS)

DARTS relaxes the discrete architecture choice into continuous weights (architecture parameters α) optimized via gradient descent alongside network weights:

# Mixed operation: weighted sum of all candidate ops
output = sum(softmax(alpha[i]) * op_i(x) for i, op_i in enumerate(ops))

# Bi-level optimization:
#   Inner loop: update network weights w on training data
#   Outer loop: update architecture params α on validation data
# After search: discretize by selecting argmax(α) per edge

DARTS searches in hours but suffers from performance collapse — skip connections dominate because they are easiest to optimize. Fixes include: DARTS+ (auxiliary skip penalty), Fair DARTS (sigmoid instead of softmax), P-DARTS (progressive depth increase).

Hardware-Aware NAS

Modern NAS optimizes for deployment constraints jointly with accuracy:

Zero-Shot / Training-Free NAS

The frontier eliminates even supernet training — using proxy metrics computed at initialization (Jacobian covariance, gradient flow, linear region count) to score architectures in seconds.

Efficient NAS has democratized architecture optimization — by reducing search costs from GPU-years to GPU-hours or even minutes, weight-sharing and differentiable methods have made neural architecture discovery an accessible and practical tool for both researchers and practitioners deploying models across diverse hardware targets.