neural architecture search
**Neural Architecture Search (NAS)** is the **automated process of discovering optimal neural network architectures for a given task** — replacing manual architecture design with algorithmic search over the space of possible layers, connections, and operations, having discovered architectures like EfficientNet and NASNet that outperform human-designed networks.
**NAS Components**
| Component | Description | Examples |
|-----------|------------|----------|
| Search Space | Set of possible architectures | Layer types, connections, channels |
| Search Strategy | How to explore the space | RL, evolutionary, gradient-based |
| Performance Estimation | How to evaluate candidates | Full training, weight sharing, proxy tasks |
**Search Strategies**
**Reinforcement Learning (NASNet, 2017)**
- Controller RNN generates architecture description tokens.
- Architecture is trained, accuracy becomes the reward signal.
- Controller is updated via REINFORCE/PPO.
- Cost: Original NASNet used 500 GPUs × 4 days = 2000 GPU-days.
**Evolutionary (AmoebaNet)**
- Population of architectures maintained.
- Mutation: Randomly change one operation or connection.
- Selection: Keep the fittest (highest accuracy) architectures.
- Advantage: Naturally parallel, no gradient computation for search.
**Gradient-Based (DARTS)**
- Represent architecture as a continuous relaxation: weighted sum of all possible operations.
- Architecture weights optimized via backpropagation alongside network weights.
- After search: Discretize — keep the highest-weighted operation at each edge.
- Cost: Single GPU, 1-4 days — orders of magnitude cheaper than RL-based NAS.
**One-Shot / Supernet Methods**
- Train a single supernet containing all possible architectures as subnetworks.
- Each training step: Sample a random subnetwork and update its weights.
- After training: Evaluate subnetworks without retraining.
- Used by: Once-for-All (OFA), BigNAS, FBNetV2.
**Notable NAS-Discovered Architectures**
| Architecture | Method | Achievement |
|-------------|--------|------------|
| NASNet | RL | First NAS to match human design on ImageNet |
| EfficientNet | RL + scaling | SOTA ImageNet accuracy/efficiency |
| DARTS cells | Gradient | Competitive results in hours, not days |
| MnasNet | RL (mobile) | Optimized for mobile latency |
**Hardware-Aware NAS**
- Objective: Maximize accuracy subject to latency/FLOPs/energy constraints.
- Latency lookup table per operation per target hardware.
- Multi-objective optimization: Pareto frontier of accuracy vs. efficiency.
Neural architecture search is **the foundation of automated machine learning (AutoML)** — while manual architecture design still produces breakthrough innovations, NAS has proven that algorithmic search can discover efficient, high-performing architectures that generalize across tasks and hardware targets.