Neural Architecture Search (NAS) is the automated machine learning technique that discovers optimal neural network architectures by searching over a defined design space — replacing manual architecture engineering with algorithmic exploration of layer types, connections, depths, and widths to find designs that maximize accuracy, minimize latency, or optimize any specified objective on target hardware.

The Search Space

NAS operates over a structured design space defining what architectures are possible:

• Cell-Based Search: Design a repeating cell (normal cell for feature extraction, reduction cell for downsampling) that is stacked to form the full network. Dramatically reduces search space compared to searching the entire architecture.
• Operation Set: The building blocks within each cell — convolution 3x3, 5x5, dilated convolution, depthwise separable convolution, skip connection, pooling, zero (no connection).
• Macro Search: Search over the overall network structure — number of layers, channel widths, resolution changes, skip connection patterns.

Search Strategies

• Reinforcement Learning (RL): A controller RNN generates architecture descriptions (sequences of tokens). Architectures are trained and evaluated; the accuracy serves as the reward signal. The controller learns to generate better architectures. NASNet (Google, 2018) used 500 GPUs for 4 days — effective but extremely expensive.
• Evolutionary Search: Maintain a population of architectures. Apply mutations (add/remove layers, change operations) and crossover. Select the fittest (highest accuracy) for the next generation. AmoebaNet matched NASNet quality with comparable search cost.
• Differentiable NAS (DARTS): Make the discrete architecture choice differentiable by maintaining a continuous probability distribution over operations. Jointly optimize architecture weights and network weights via gradient descent. Reduces search cost from thousands of GPU-days to a single GPU-day.
• One-Shot / Weight Sharing: Train a single "supernet" containing all possible architectures. Each architecture is a subgraph. Search selects the best subgraph based on supernet performance. OFA (Once-for-All) trains one supernet that supports thousands of sub-networks for different hardware constraints.

Hardware-Aware NAS

Modern NAS optimizes for both accuracy and hardware efficiency:

• Latency-Aware: Include measured inference latency on target hardware (mobile phone, edge TPU, server GPU) in the objective function. MNASNet and EfficientNet used hardware-aware search to find architectures that are Pareto-optimal on accuracy vs. latency.
• Multi-Objective: Optimize accuracy, latency, parameter count, and energy consumption simultaneously. The result is a Pareto frontier of architectures offering different trade-offs.

Key Results

• EfficientNet (2019): NAS-discovered scaling coefficients for width, depth, and resolution that outperformed all manually-designed architectures at every FLOP budget.
• FBNet (Facebook): Hardware-aware NAS producing models 20% more efficient than MobileNetV2 on mobile devices.

Neural Architecture Search is the automation of neural network design — replacing human intuition about architecture with systematic, objective-driven search that consistently discovers designs matching or surpassing the best hand-crafted architectures at any efficiency target.