neural architecture search nas

**Neural Architecture Search (NAS) with Weight Sharing** is **a computationally efficient paradigm for automated network design that trains a single overparameterized supernet encompassing all candidate architectures, enabling evaluation of thousands of designs without training each from scratch** — reducing the search cost from thousands of GPU-days to a single training run while maintaining competitive accuracy with expert-designed architectures. **Supernet Training Fundamentals:** - **Supernetwork Construction**: Build an overparameterized network where each layer contains all candidate operations (convolutions, pooling, skip connections, identity mappings) - **Path Sampling**: During each training step, randomly sample a sub-architecture (path) from the supernet and update only its weights - **Weight Inheritance**: Child architectures inherit trained weights from the shared supernet, avoiding independent training - **Search Space Definition**: Specify the set of candidate operations, connectivity patterns, and architectural constraints defining the design space - **Evaluation Protocol**: Rank candidate architectures by their validation accuracy using inherited supernet weights as a proxy for independently trained performance **Key NAS Approaches:** - **One-Shot NAS**: Train the supernet once, then search by evaluating sampled sub-networks using inherited weights without additional training - **DARTS (Differentiable Architecture Search)**: Relax discrete architecture choices into continuous variables optimized by gradient descent alongside network weights - **FairNAS**: Address weight coupling bias by ensuring all operations receive equal training updates during supernet training - **ProxylessNAS**: Directly search on the target task and hardware platform, eliminating proxy dataset and latency model approximations - **Once-for-All (OFA)**: Train a single supernet that supports deployment across diverse hardware platforms with different latency and memory constraints - **EfficientNAS**: Combine progressive shrinking with knowledge distillation to improve supernet training quality **Weight Sharing Challenges:** - **Weight Coupling**: Shared weights may not accurately represent independently trained weights, leading to ranking inconsistencies among candidate architectures - **Supernet Training Instability**: Balancing training across exponentially many sub-networks can cause optimization difficulties and gradient interference - **Search Space Bias**: The supernet's architecture and training hyperparameters may inadvertently favor certain operations over others - **Ranking Correlation**: The correlation between supernet-based evaluation and standalone training performance (Kendall's tau) varies significantly across search spaces - **Depth Imbalance**: Deeper paths in the supernet receive fewer gradient updates, biasing the search toward shallower architectures **Hardware-Aware NAS:** - **Latency Prediction**: Build lookup tables or lightweight predictors mapping architectural choices to measured inference latency on target hardware - **Multi-Objective Optimization**: Jointly optimize accuracy and hardware metrics (latency, energy, memory) using Pareto-optimal search strategies - **Platform-Specific Search**: Architectures found for mobile GPUs differ substantially from those optimal for server GPUs or edge TPUs - **Quantization-Aware NAS**: Search for architectures that maintain accuracy under low-bit quantization (INT8, INT4) **Practical Deployment:** - **Search Cost**: Weight-sharing NAS reduces costs from 3,000+ GPU-days (early NAS methods) to 1–10 GPU-days - **Transfer Learning**: Architectures discovered on proxy tasks (CIFAR-10) often transfer well to larger benchmarks (ImageNet) but not always to domain-specific tasks - **Reproducibility**: Results are sensitive to supernet training recipes, search algorithms, and random seeds, necessitating careful ablation studies NAS with weight sharing has **democratized automated architecture design by making the search process practical on standard academic compute budgets — though careful attention to weight coupling, ranking fidelity, and hardware-aware objectives remains essential for discovering architectures that genuinely outperform expert-designed baselines in real-world deployments**.