Neural Architecture Search (NAS) is the automated machine learning technique for discovering optimal neural network architectures within defined search spaces — using gradient-based (DARTS), evolutionary, or reinforcement learning strategies to balance accuracy and efficiency constraints.

NAS Search Space and Strategy:

• Search space definition: cell-based (repeated motifs), chain-structured (sequential layers), macro (entire architecture); defines architectural decisions
• Search strategy: reinforcement learning (RNN controller generates architectures), evolutionary algorithms (mutation/crossover), gradient-based (DARTS)
• Architecture encoding: RNN controller or differentiable operations enable efficient exploration; alternatives use graph representations
• Objective function: accuracy + latency/energy/model size; hardware-aware NAS trades off multiple constraints

DARTS (Differentiable Architecture Search):

• Continuous relaxation: replace discrete operation choice with continuous mixture; enable gradient descent through architecture search
• Bilevel optimization: inner loop trains network weights; outer loop optimizes architecture parameters via gradient descent
• One-shot paradigm: single supernetwork contains all operations; weight sharing across candidate architectures → efficient search
• Computational efficiency: 4 GPU-days vs thousands of GPU-days for reinforcement learning NAS; enables broader adoption

EfficientNet and Compound Scaling:

• NAS-discovered baseline: EfficientNet-B0 found via NAS; better accuracy-latency tradeoff than hand-designed networks
• Compound scaling: systematically scale depth, width, resolution with fixed ratios (discovered via grid search over scaling factors)
• EfficientNet family: B0-B7 provides range of model sizes; B0 (5.3M params) → B7 (66M params); consistent accuracy gains
• State-of-the-art accuracy: competitive with larger models (ResNet-152, AmoebaNet) while being much faster

NAS Applications and Variants:

• Hardware-aware NAS: optimize for specific hardware targets (mobile CPU/GPU, edge TPUs); latency-aware search objectives
• ProxylessNAS: removes proxy task requirement; directly searches on target task; more flexible and accurate
• One-shot NAS: weight sharing accelerates search; evaluated model inherits supernet weights; enables NAS on modest compute
• NAS for transformers: architecture search discovers optimal transformer depths, widths, attention heads for different data sizes

Search Cost Reduction:

• Early stopping: stop training unpromising architectures; identify good architectures faster
• Performance prediction: train small proxy tasks; predict full-scale performance without full training
• Evolutionary search: population-based search with mutations/crossover; parallelizable across multiple workers
• Transfer learning: reuse architectures across similar domains; transfer-friendly NAS

NAS automates the tedious manual design process — discovering architectures tailored to specific accuracy-efficiency tradeoffs that often outperform hand-designed networks across vision, language, and multimodal domains.