neural architecture search nas efficiency

**Neural Architecture Search (NAS) Efficiency Methods** is **a set of techniques that reduce the computational cost of automated architecture discovery from thousands of GPU-days to single GPU-hours** — transforming NAS from a prohibitively expensive research curiosity into a practical tool for designing high-performance neural networks. **Early NAS and the Cost Problem** The original NAS (Zoph and Le, 2017) used reinforcement learning to search over architectures, requiring 22,400 GPU-hours (≈$40K in cloud compute) to find a single CNN architecture for CIFAR-10. NASNet extended this to ImageNet but cost 48,000 GPU-hours. Each candidate architecture was trained from scratch to convergence before evaluation, making the search combinatorially explosive. This motivated efficient alternatives that share computation across candidates. **One-Shot NAS and Supernet Training** - **Supernet concept**: A single over-parameterized network (supernet) encodes all candidate architectures as subnetworks within a shared weight space - **Weight sharing**: All candidate architectures share parameters; evaluating a candidate requires only a forward pass through the relevant subnetwork - **Single training run**: The supernet is trained once (typically 100-200 epochs), then candidates are evaluated by inheriting supernet weights - **Path sampling**: During supernet training, random paths (subnetworks) are sampled each batch, approximating joint training of all candidates - **Cost reduction**: From thousands of GPU-days to 1-4 GPU-days for complete search **DARTS: Differentiable Architecture Search** - **Continuous relaxation**: DARTS (Liu et al., 2019) replaces discrete architecture choices with continuous softmax weights over operations (convolution, pooling, skip connection) - **Bilevel optimization**: Architecture parameters (α) optimized on validation loss; network weights (w) optimized on training loss via alternating gradient descent - **Search cost**: Approximately 1.5 GPU-days on CIFAR-10 (1000x cheaper than original NAS) - **Collapse problem**: DARTS tends to converge to parameter-free operations (skip connections, pooling) due to optimization bias—addressed by DARTS+, FairDARTS, and progressive shrinking - **Cell-based search**: Discovers normal and reduction cells that are stacked to form the final architecture **Progressive and Predictor-Based Methods** - **Progressive NAS (PNAS)**: Grows architectures incrementally from simple to complex, pruning unpromising candidates early - **Predictor-based NAS**: Trains a surrogate model (MLP, GNN, or Gaussian process) to predict architecture performance from encoding - **Zero-cost proxies**: Evaluate architectures at initialization without training using metrics like Jacobian covariance, synaptic saliency, or gradient norm - **Hardware-aware NAS**: Jointly optimizes accuracy and latency/FLOPs/energy using multi-objective search (e.g., MnasNet, FBNet, EfficientNet) **Search Space Design** - **Cell-based**: Search within a repeatable cell structure; stack cells to form network (NASNet, DARTS) - **Network-level**: Search over depth, width, resolution, and connectivity patterns (EfficientNet compound scaling) - **Operation set**: Typically includes 3x3/5x5 convolutions, depthwise separable convolutions, dilated convolutions, skip connections, and zero (no connection) - **Macro search**: Full topology discovery including branching and merging paths - **Hierarchical search**: Multi-level search combining cell-level and network-level decisions **Practical Deployment and Recent Advances** - **Once-for-All (OFA)**: Trains a single supernet supporting elastic depth, width, kernel size, and resolution; extracts specialized subnets for different hardware targets without retraining - **NAS benchmarks**: NAS-Bench-101, NAS-Bench-201, and NAS-Bench-301 provide precomputed results for reproducible NAS research - **AutoML frameworks**: Auto-PyTorch, NNI (Microsoft), and AutoGluon integrate NAS into end-to-end pipelines - **Transferability**: Architectures found on proxy tasks (CIFAR-10) often transfer well to larger datasets (ImageNet) via scaling **Efficient NAS methods have democratized architecture design, enabling practitioners to discover hardware-optimized networks in hours rather than weeks, making automated architecture engineering a standard component of the modern deep learning workflow.**