neural architecture search

Neural Architecture Search (NAS) automatically discovers optimal neural network architectures, replacing manual design with algorithmic search over structure, connectivity, and operations to find architectures that maximize performance on target tasks. Three components: search space (what architectures are possible—operations, connections, cell structures), search algorithm (how to explore the space—RL, evolutionary, gradient-based), and evaluation strategy (how to measure architecture quality—full training, weight sharing, predictors). Search evolution: early NAS (NASNet, 2017) used thousands of GPU-hours; modern methods achieve similar results in GPU-hours through weight sharing (one-shot methods), performance prediction, and efficient search spaces. Key methods: reinforcement learning (controller generates architectures, reward from validation accuracy), evolutionary algorithms (population-based mutation and selection), differentiable/gradient-based (DARTS—continuous relaxation, gradient descent on architecture), and predictor-based (train surrogate model to predict performance). Search spaces: macro (entire network structure) versus micro (cell design, then stacking). Cost: from 30,000 GPU-hours (early) to single GPU-hours (modern efficient methods). NAS has discovered competitive architectures (EfficientNet, RegNet) and is now practical for customizing architectures to specific tasks, hardware, and constraints.