neural architecture search nas

**Neural Architecture Search (NAS)** is **the automated machine learning technique that algorithmically discovers optimal neural network architectures for a given task — replacing manual architecture design with systematic exploration of topology, layer types, connectivity patterns, and hyperparameters to find designs that outperform human-designed networks**. **Search Space Design:** - **Cell-Based Search**: define a DAG cell structure with learnable operations on each edge — discovered cell is stacked/repeated to build full network; reduces search space from exponential (full network) to manageable (single cell with ~10 edges) - **Operation Candidates**: each edge can be one of K operations — typical choices: 3×3 conv, 5×5 conv, dilated conv, depthwise separable conv, max pool, avg pool, skip connection, zero (no connection) - **Macro Search**: directly search for full network topology including layer count, widths, and skip connections — larger search space but can discover fundamentally novel architectures - **Hierarchical Search**: search at multiple granularities — inner cell structure, cell connectivity, and network-level design (number of cells, reduction placement) each searched at appropriate level **Search Strategies:** - **Reinforcement Learning (NASNet)**: controller RNN generates architecture descriptions, trained with REINFORCE using validation accuracy as reward — found NASNet achieving state-of-the-art ImageNet accuracy but required 48,000 GPU-hours - **Evolutionary (AmoebaNet)**: maintain population of architectures, mutate best performers, evaluate offspring — tournament selection with aging removes stagnant individuals; comparable to RL-based search at similar compute cost - **Differentiable (DARTS)**: relax discrete architecture choices to continuous weights over all operations — optimize architecture parameters via gradient descent simultaneously with network weights; reduces search from thousands of GPU-hours to single GPU-day - **One-Shot/Supernet**: train a single overparameterized network containing all candidate operations — individual architectures are sub-networks evaluated by inheriting weights from the supernet; enables evaluating thousands of architectures without training each from scratch **Efficiency Improvements:** - **Weight Sharing**: all architectures in the search space share weights from a common supernet — eliminates the need to train each candidate independently; reduces search cost by 1000× - **Predictor-Based**: train a performance predictor (neural network or Gaussian process) on evaluated architectures — use predictor to score unseen architectures without expensive training; focuses evaluation on promising candidates - **Hardware-Aware NAS**: include latency, FLOPs, or energy as objectives alongside accuracy — multi-objective optimization produces Pareto-optimal architectures balancing accuracy with deployment constraints - **Zero-Cost Proxies**: estimate architecture quality at initialization (before training) using gradient statistics — enables evaluating millions of candidates in minutes; examples include synflow, NASWOT, and jacob_cov scores **Neural Architecture Search represents the automation of the last major manual component in deep learning pipelines — while early NAS methods required enormous compute budgets, modern efficient NAS techniques discover architectures in hours that match or exceed years of expert human design effort.**