neural architecture search nas

**Neural Architecture Search (NAS)** is the **automated machine learning technique for discovering optimal neural network architectures within defined search spaces — using gradient-based (DARTS), evolutionary, or reinforcement learning strategies to balance accuracy and efficiency constraints**. **NAS Search Space and Strategy:** - Search space definition: cell-based (repeated motifs), chain-structured (sequential layers), macro (entire architecture); defines architectural decisions - Search strategy: reinforcement learning (RNN controller generates architectures), evolutionary algorithms (mutation/crossover), gradient-based (DARTS) - Architecture encoding: RNN controller or differentiable operations enable efficient exploration; alternatives use graph representations - Objective function: accuracy + latency/energy/model size; hardware-aware NAS trades off multiple constraints **DARTS (Differentiable Architecture Search):** - Continuous relaxation: replace discrete operation choice with continuous mixture; enable gradient descent through architecture search - Bilevel optimization: inner loop trains network weights; outer loop optimizes architecture parameters via gradient descent - One-shot paradigm: single supernetwork contains all operations; weight sharing across candidate architectures → efficient search - Computational efficiency: 4 GPU-days vs thousands of GPU-days for reinforcement learning NAS; enables broader adoption **EfficientNet and Compound Scaling:** - NAS-discovered baseline: EfficientNet-B0 found via NAS; better accuracy-latency tradeoff than hand-designed networks - Compound scaling: systematically scale depth, width, resolution with fixed ratios (discovered via grid search over scaling factors) - EfficientNet family: B0-B7 provides range of model sizes; B0 (5.3M params) → B7 (66M params); consistent accuracy gains - State-of-the-art accuracy: competitive with larger models (ResNet-152, AmoebaNet) while being much faster **NAS Applications and Variants:** - Hardware-aware NAS: optimize for specific hardware targets (mobile CPU/GPU, edge TPUs); latency-aware search objectives - ProxylessNAS: removes proxy task requirement; directly searches on target task; more flexible and accurate - One-shot NAS: weight sharing accelerates search; evaluated model inherits supernet weights; enables NAS on modest compute - NAS for transformers: architecture search discovers optimal transformer depths, widths, attention heads for different data sizes **Search Cost Reduction:** - Early stopping: stop training unpromising architectures; identify good architectures faster - Performance prediction: train small proxy tasks; predict full-scale performance without full training - Evolutionary search: population-based search with mutations/crossover; parallelizable across multiple workers - Transfer learning: reuse architectures across similar domains; transfer-friendly NAS **NAS automates the tedious manual design process — discovering architectures tailored to specific accuracy-efficiency tradeoffs that often outperform hand-designed networks across vision, language, and multimodal domains.**