neural architecture search

**Neural Architecture Search (NAS)** — using algorithms to automatically discover optimal neural network architectures instead of relying on human design, a key branch of AutoML. **The Problem** - Architecture design is manual and requires expert intuition - Huge design space: Number of layers, filter sizes, connections, attention heads, activation functions - Humans can't explore all possibilities **Search Strategies** - **Reinforcement Learning NAS**: A controller network proposes architectures; reward = validation accuracy. Original method (Google, 2017). Cost: 800 GPU-days - **Evolutionary NAS**: Mutate and evolve a population of architectures. Similar cost to RL approach - **Differentiable NAS (DARTS)**: Make architecture choices continuous and differentiable → use gradient descent to search. Cost: 1-4 GPU-days (1000x cheaper) - **One-Shot NAS**: Train a single supernet containing all candidate architectures, then extract the best subnet **Notable Results** - **NASNet**: Found architectures better than human-designed ResNet - **EfficientNet**: NAS-designed CNN that set ImageNet records - **MnasNet**: NAS for mobile — Pareto-optimal speed vs accuracy **Limitations** - Search space must be carefully defined by humans - Results often aren't dramatically better than well-designed manual architectures - Reproducibility challenges **NAS** demonstrated that machines can design neural networks — but the community has shifted toward scaling known architectures rather than searching for new ones.