Neural Architecture Search (NAS) is the automated machine learning technique that discovers optimal neural network architectures by searching over a defined design space — systematically evaluating thousands of candidate architectures (layer types, connections, dimensions, activation functions) using reinforcement learning, evolutionary algorithms, or gradient-based methods to find designs that outperform human-crafted architectures on target metrics including accuracy, latency, and model size.

Why Automate Architecture Design

The number of possible neural network configurations is astronomically large. Human experts design architectures through intuition and incremental experimentation, but this process is slow (months per architecture) and biased toward known patterns. NAS explores the design space systematically, often discovering non-obvious configurations that outperform the best human designs.

Search Space

The search space defines what architectures NAS can discover:

• Cell-Based: Search for a repeating cell (normal cell and reduction cell) that is stacked to form the full network. This reduces the search space dramatically while producing transferable designs.
• Layer-Wise: Search over the type, size, and connections of each individual layer. More flexible but exponentially larger search space.
• Typical Choices: Convolution kernel sizes (3x3, 5x5, 7x7), skip connections, pooling types, attention mechanisms, channel widths, expansion ratios, activation functions.

Search Strategies

• RL-Based (NASNet): A controller RNN generates architecture descriptions. Each architecture is trained and evaluated, and the controller is updated via REINFORCE to generate better architectures. Extremely expensive — the original NAS paper used 800 GPUs for 28 days.
• Evolutionary (AmoebaNet): Maintain a population of architectures. Mutate the best performers (add/remove layers, change operations) and select based on fitness. Matches RL quality with simpler implementation.
• One-Shot / Weight Sharing (ENAS, DARTS): Train a single supernet containing all possible architectures as subgraphs. Architecture search becomes selecting which subgraph performs best, reducing search cost from thousands of GPU-days to a single GPU-day.
• DARTS (Differentiable): Makes the architecture selection continuous and differentiable — architecture choice is parameterized by continuous weights optimized through gradient descent alongside the network weights.

Hardware-Aware NAS

Modern NAS optimizes for deployment constraints alongside accuracy:

• Latency Prediction: A lookup table or predictor model estimates the inference latency of each candidate on the target hardware (mobile CPU, GPU, TPU, edge NPU).
• Multi-Objective: Pareto-optimal architectures are found that balance accuracy vs. latency, model size, or energy consumption.
• EfficientNet/EfficientDet: Landmark architectures discovered by NAS that achieved state-of-the-art accuracy at every compute budget, outperforming all hand-designed alternatives.

Neural Architecture Search is the meta-learning approach that turns architecture design from art into optimization — letting algorithms discover neural network designs that no human would conceive but that consistently outperform the best expert-crafted models.