neural architecture search nas

**Neural Architecture Search (NAS)** is the **automated machine learning technique that discovers optimal neural network architectures by searching over a defined design space — replacing manual architecture engineering with algorithmic exploration of layer types, connections, depths, and widths to find designs that maximize accuracy, minimize latency, or optimize any specified objective on target hardware**. **The Search Space** NAS operates over a structured design space defining what architectures are possible: - **Cell-Based Search**: Design a repeating cell (normal cell for feature extraction, reduction cell for downsampling) that is stacked to form the full network. Dramatically reduces search space compared to searching the entire architecture. - **Operation Set**: The building blocks within each cell — convolution 3x3, 5x5, dilated convolution, depthwise separable convolution, skip connection, pooling, zero (no connection). - **Macro Search**: Search over the overall network structure — number of layers, channel widths, resolution changes, skip connection patterns. **Search Strategies** - **Reinforcement Learning (RL)**: A controller RNN generates architecture descriptions (sequences of tokens). Architectures are trained and evaluated; the accuracy serves as the reward signal. The controller learns to generate better architectures. NASNet (Google, 2018) used 500 GPUs for 4 days — effective but extremely expensive. - **Evolutionary Search**: Maintain a population of architectures. Apply mutations (add/remove layers, change operations) and crossover. Select the fittest (highest accuracy) for the next generation. AmoebaNet matched NASNet quality with comparable search cost. - **Differentiable NAS (DARTS)**: Make the discrete architecture choice differentiable by maintaining a continuous probability distribution over operations. Jointly optimize architecture weights and network weights via gradient descent. Reduces search cost from thousands of GPU-days to a single GPU-day. - **One-Shot / Weight Sharing**: Train a single "supernet" containing all possible architectures. Each architecture is a subgraph. Search selects the best subgraph based on supernet performance. OFA (Once-for-All) trains one supernet that supports thousands of sub-networks for different hardware constraints. **Hardware-Aware NAS** Modern NAS optimizes for both accuracy and hardware efficiency: - **Latency-Aware**: Include measured inference latency on target hardware (mobile phone, edge TPU, server GPU) in the objective function. MNASNet and EfficientNet used hardware-aware search to find architectures that are Pareto-optimal on accuracy vs. latency. - **Multi-Objective**: Optimize accuracy, latency, parameter count, and energy consumption simultaneously. The result is a Pareto frontier of architectures offering different trade-offs. **Key Results** - **EfficientNet** (2019): NAS-discovered scaling coefficients for width, depth, and resolution that outperformed all manually-designed architectures at every FLOP budget. - **FBNet** (Facebook): Hardware-aware NAS producing models 20% more efficient than MobileNetV2 on mobile devices. Neural Architecture Search is **the automation of neural network design** — replacing human intuition about architecture with systematic, objective-driven search that consistently discovers designs matching or surpassing the best hand-crafted architectures at any efficiency target.