neural architecture search nas

**Neural Architecture Search (NAS)** is **the automated process of discovering optimal neural network architectures by searching over a defined space of possible layer types, connections, and hyperparameters — replacing manual architecture design with algorithmic optimization that has produced architectures matching or exceeding human-designed networks on image classification, detection, and language tasks**. **Search Space Design:** - **Cell-Based Search**: search for optimal cell (small computational block) and stack cells into full architecture; normal cells preserve spatial dimensions, reduction cells downsample; dramatically reduces search space vs searching full architectures directly - **Operations**: candidate operations within each cell edge: convolution (3×3, 5×5, depthwise separable), pooling (max, avg), skip connection, zero (no connection); each edge selects one operation from the candidate set - **Macro Architecture**: number of cells, channel width schedule, and cell connectivity are either fixed (cell-based NAS) or searched (hierarchical NAS); macro search is more flexible but exponentially larger search space - **Hardware-Aware Search**: search space constrained by target hardware (latency, memory, FLOPs); lookup tables mapping operations to measured latency on target device enable hardware-aware objective optimization **Search Strategies:** - **Reinforcement Learning NAS**: controller (RNN) generates architecture description as sequence of tokens; architecture is trained and evaluated; reward (validation accuracy) updates the controller via REINFORCE; Zoph & Le (2017) original approach — effective but requires thousands of GPU-hours - **DARTS (Differentiable NAS)**: relaxes discrete architecture choices to continuous weights using softmax over operations on each edge; jointly optimizes architecture weights (which operations to keep) and network weights (operation parameters) via gradient descent; 1-4 GPU-days vs thousands for RL-NAS - **One-Shot NAS (Supernet)**: train a single supernet containing all possible architectures; evaluate candidate architectures by inheriting supernet weights; search reduces to selecting paths through the pretrained supernet — decouples training from search, enabling millions of architecture evaluations - **Evolutionary NAS**: population of architectures mutated (change operations, add/remove connections) and evaluated; tournament selection retains best performers; naturally parallelizable across many GPUs; AmoebaNet achieved SOTA on ImageNet **Efficiency Improvements:** - **Weight Sharing**: all architectures in the search space share weights; avoids training each candidate from scratch; supernet training cost equivalent to training one large network — 1000× cheaper than independent training - **Proxy Tasks**: evaluate architectures on smaller datasets (CIFAR-10 instead of ImageNet), fewer epochs (50 instead of 300), or reduced channel widths; rankings transfer approximately across scales for relative architecture comparison - **Predictor-Based Search**: train a neural predictor that estimates architecture accuracy from its encoding; enables rapid evaluation of millions of candidates without actual training; predictors trained on hundreds of fully-evaluated architectures - **Zero-Cost Proxies**: score architectures at initialization (no training) using gradient signals, Jacobian statistics, or linear region counts; 10000× faster than training-based evaluation but less reliable for fine-grained architecture ranking **Notable Discoveries:** - **EfficientNet**: compound scaling of depth, width, and resolution discovered by NAS; EfficientNet-B0 to B7 family achieved SOTA ImageNet accuracy with significantly fewer parameters and FLOPs than prior architectures - **NASNet/AmoebaNet**: among first NAS-discovered architectures competitive with human-designed networks; transferred from CIFAR-10 search to ImageNet by stacking discovered cells - **Once-for-All (OFA)**: single supernet supporting 10^19 subnets; extract specialized architectures for different hardware targets without retraining — deploy the same supernet to phone, tablet, and server - **Hardware-Optimal Architectures**: NAS consistently discovers architectures that differ from human intuition — favoring asymmetric structures, unusual operation combinations, and hardware-specific optimizations invisible to manual design Neural architecture search is **the automation of the most creative aspect of deep learning engineering — systematically exploring architectural possibilities that human designers would never consider, producing hardware-efficient architectures that define the performance frontier for vision, language, and multimodal AI models**.