supernet training
**Supernet Training** is a **neural architecture search paradigm that trains a single over-parameterized network (supernet) containing all candidate architectures simultaneously by randomly activating different subnetworks (subnets) at each training step — amortizing architecture search cost across the entire search space so any subnet can be extracted and evaluated for free by inheriting the supernet's weights without additional training** — the architectural backbone of modern efficient NAS methods including Once-for-All (OFA), Slimmable Networks, and hardware-aware neural architecture search pipelines that produce deployment-ready models for thousands of different hardware targets from a single training run.
**What Is Supernet Training?**
- **Supernet**: An over-parameterized master network whose architecture space encompasses all candidate networks in the search space — every possible combination of layer widths, depths, kernel sizes, and connection choices forms a valid subnet.
- **Weight Sharing**: Each subnet inherits its weights directly from the matching positions in the supernet — no separate training per architecture.
- **Sandwiching (Progressive Shrinking)**: During training, the supernet is trained by sampling subnets at different complexity levels each batch — largest, smallest, and random medium-sized subnets. This prevents large subnets from dominating weight updates.
- **Search Phase**: After supernet training, evolutionary search, random search, or predictor-guided search identifies the best subnet for a target constraint (FLOPs, latency, memory) without retraining — just inherited weights.
- **Deployment**: The selected subnet is extracted, optionally fine-tuned for a few epochs, and deployed.
**Architectures and Variants**
| Method | Supernet Strategy | Key Feature |
|--------|-------------------|-------------|
| **ENAS** | Random subgraph sampling + RL controller | One of the first weight-sharing NAS |
| **DARTS** | Continuous relaxation of architecture weights | Gradient-based architecture optimization |
| **Once-for-All (OFA)** | Progressive shrinking curriculum | Single supernet for 1,000+ hardware targets |
| **Slimmable Networks** | Unified width-switching at runtime | Multiple width configurations without NAS |
| **AttentiveNAS** | Pareto-optimal search with accuracy/FLOPs | Production deployment with hardware constraints |
| **BigNAS** | Single-stage supernet with in-place distillation | Simplified supernet training without separate finetuning |
**The Once-for-All (OFA) Paradigm**
OFA (Cai et al., MIT, 2020) is the most successful supernet training approach for production deployment:
- **Decouple Training and Search**: Train the supernet once; search and deploy specialized subnets instantly for any device.
- **Progressive Shrinking**: Train largest architecture first, then progressively enable smaller architectures — preventing weight conflicts.
- **Search Space**: Kernel sizes (3, 5, 7), depths (2–4 per block), widths (3–6 channels per group) — 10^19 possible network configurations in one supernet.
- **Result**: 40× faster deployment than training from scratch per target, enabling device-specific model deployment at industrial scale.
**Challenges in Supernet Training**
- **Weight Coupling**: Optimal weights for large subnets may differ from optimal weights for small subnets — the supernet learns a compromise.
- **Ranking Inconsistency**: Subnets ranked highly by supernet weights may not rank equally after standalone training.
- **Training Stability**: Equal gradient weighting across subnets of very different sizes causes instability — addressed by loss normalization and sampling schedules.
- **Search Space Coverage**: Ensuring all parts of the search space receive sufficient training signal requires careful sampling strategies.
Supernet Training is **the industrialization of neural architecture search** — the framework that transforms architecture optimization from a research experiment into a practical engineering tool, enabling companies to produce deployment-optimized models for thousands of hardware targets from a single carefully trained master network.