Neural Architecture Search (NAS) with Weight Sharing is a computationally efficient paradigm for automated network design that trains a single overparameterized supernet encompassing all candidate architectures, enabling evaluation of thousands of designs without training each from scratch — reducing the search cost from thousands of GPU-days to a single training run while maintaining competitive accuracy with expert-designed architectures.

Supernet Training Fundamentals:

• Supernetwork Construction: Build an overparameterized network where each layer contains all candidate operations (convolutions, pooling, skip connections, identity mappings)
• Path Sampling: During each training step, randomly sample a sub-architecture (path) from the supernet and update only its weights
• Weight Inheritance: Child architectures inherit trained weights from the shared supernet, avoiding independent training
• Search Space Definition: Specify the set of candidate operations, connectivity patterns, and architectural constraints defining the design space
• Evaluation Protocol: Rank candidate architectures by their validation accuracy using inherited supernet weights as a proxy for independently trained performance

Key NAS Approaches:

• One-Shot NAS: Train the supernet once, then search by evaluating sampled sub-networks using inherited weights without additional training
• DARTS (Differentiable Architecture Search): Relax discrete architecture choices into continuous variables optimized by gradient descent alongside network weights
• FairNAS: Address weight coupling bias by ensuring all operations receive equal training updates during supernet training
• ProxylessNAS: Directly search on the target task and hardware platform, eliminating proxy dataset and latency model approximations
• Once-for-All (OFA): Train a single supernet that supports deployment across diverse hardware platforms with different latency and memory constraints
• EfficientNAS: Combine progressive shrinking with knowledge distillation to improve supernet training quality

Weight Sharing Challenges:

• Weight Coupling: Shared weights may not accurately represent independently trained weights, leading to ranking inconsistencies among candidate architectures
• Supernet Training Instability: Balancing training across exponentially many sub-networks can cause optimization difficulties and gradient interference
• Search Space Bias: The supernet's architecture and training hyperparameters may inadvertently favor certain operations over others
• Ranking Correlation: The correlation between supernet-based evaluation and standalone training performance (Kendall's tau) varies significantly across search spaces
• Depth Imbalance: Deeper paths in the supernet receive fewer gradient updates, biasing the search toward shallower architectures

Hardware-Aware NAS:

• Latency Prediction: Build lookup tables or lightweight predictors mapping architectural choices to measured inference latency on target hardware
• Multi-Objective Optimization: Jointly optimize accuracy and hardware metrics (latency, energy, memory) using Pareto-optimal search strategies
• Platform-Specific Search: Architectures found for mobile GPUs differ substantially from those optimal for server GPUs or edge TPUs
• Quantization-Aware NAS: Search for architectures that maintain accuracy under low-bit quantization (INT8, INT4)

Practical Deployment:

• Search Cost: Weight-sharing NAS reduces costs from 3,000+ GPU-days (early NAS methods) to 1–10 GPU-days
• Transfer Learning: Architectures discovered on proxy tasks (CIFAR-10) often transfer well to larger benchmarks (ImageNet) but not always to domain-specific tasks
• Reproducibility: Results are sensitive to supernet training recipes, search algorithms, and random seeds, necessitating careful ablation studies

NAS with weight sharing has democratized automated architecture design by making the search process practical on standard academic compute budgets — though careful attention to weight coupling, ranking fidelity, and hardware-aware objectives remains essential for discovering architectures that genuinely outperform expert-designed baselines in real-world deployments.