once-for-all networks

**Once-for-All (OFA)** is a **NAS approach that trains a single large "supernet" that supports many sub-networks** — enabling deployment of different-sized architectures for different hardware targets without re-training, by simply selecting the appropriate sub-network. **How Does OFA Work?** - **Progressive Shrinking**: Train the supernet with progressively smaller sub-networks (first full model, then reduced depth, then reduced width, then reduced kernel size and resolution). - **Elastic Dimensions**: Supports variable depth (layer count), width (channel count), kernel size, and input resolution. - **Deployment**: Given a hardware constraint, search for the best sub-network within the trained supernet. - **Paper**: Cai et al. (2020). **Why It Matters** - **Train Once**: A single training run produces models for every deployment scenario (cloud, mobile, IoT, edge). - **Massive Efficiency**: Eliminates re-training for each target -> 10-100x reduction in total NAS compute. - **Practical**: Enables rapid customization of models for new hardware without ML expertise. **Once-for-All** is **the universal donor network** — one model that contains optimized sub-networks for every possible deployment target.