neural architecture search efficiency
**Efficient Neural Architecture Search (NAS)** is the **automated discovery of optimal neural network architectures using weight-sharing, one-shot, or differentiable methods that reduce the search cost from thousands of GPU-days to a few GPU-hours** — making architecture optimization practical for real-world deployment rather than requiring the massive computational budgets of early NAS approaches like NASNet that trained and evaluated thousands of independent networks.
**The Evolution from Brute-Force to Efficient NAS**
Early NAS (Zoph & Le 2017) used reinforcement learning to sample architectures and trained each from scratch to evaluate fitness — requiring 48,000 GPU-hours for CIFAR-10. This was computationally prohibitive for most organizations and larger datasets.
**One-Shot / Weight-Sharing NAS**
The key breakthrough was the **supernet** concept: train a single over-parameterized network (supernet) that contains all candidate architectures as sub-networks. Each sub-network (subnet) shares weights with the supernet.
```
Supernet (one-time training cost):
Layer 1: [conv3x3 | conv5x5 | sep_conv3x3 | skip_connect | none]
Layer 2: [conv3x3 | conv5x5 | sep_conv3x3 | skip_connect | none]
...
Search: Sample subnets → evaluate using inherited weights → rank
Result: Best subnet architecture found without retraining
```
Methods include:
- **ENAS**: Controller RNN samples subnets; shared weights updated via REINFORCE.
- **Once-for-All (OFA)**: Progressive shrinking trains a supernet supporting variable depth/width/resolution — deploy any subnet without retraining.
- **BigNAS**: Single-stage training with sandwich sampling (largest + smallest + random subnets per step).
**Differentiable NAS (DARTS)**
DARTS relaxes the discrete architecture choice into continuous weights (architecture parameters α) optimized via gradient descent alongside network weights:
```python
# Mixed operation: weighted sum of all candidate ops
output = sum(softmax(alpha[i]) * op_i(x) for i, op_i in enumerate(ops))
# Bi-level optimization:
# Inner loop: update network weights w on training data
# Outer loop: update architecture params α on validation data
# After search: discretize by selecting argmax(α) per edge
```
DARTS searches in hours but suffers from **performance collapse** — skip connections dominate because they are easiest to optimize. Fixes include: **DARTS+** (auxiliary skip penalty), **Fair DARTS** (sigmoid instead of softmax), **P-DARTS** (progressive depth increase).
**Hardware-Aware NAS**
Modern NAS optimizes for deployment constraints jointly with accuracy:
| Method | Constraint | Approach |
|--------|-----------|----------|
| MnasNet | Latency on mobile | RL with latency reward |
| FBNet | FLOPs/latency | Differentiable + LUT |
| ProxylessNAS | Target hardware | Latency loss in objective |
| EfficientNet | Compound scaling | NAS for base + scaling rules |
**Zero-Shot / Training-Free NAS**
The frontier eliminates even supernet training — using proxy metrics computed at initialization (Jacobian covariance, gradient flow, linear region count) to score architectures in seconds.
**Efficient NAS has democratized architecture optimization** — by reducing search costs from GPU-years to GPU-hours or even minutes, weight-sharing and differentiable methods have made neural architecture discovery an accessible and practical tool for both researchers and practitioners deploying models across diverse hardware targets.