once for all

Once-for-All (OFA) trains a single supernet containing all possible subnetworks sharing weights, enabling efficient neural architecture search by extracting specialized subnets for specific hardware constraints without retraining. Supernet concept: train one network that contains all architectures in search space as subnetworks; weights are shared—small networks use subset of large network's weights. OFA training: progressive shrinking—train largest network first, then gradually enable smaller networks, using knowledge distillation from larger to smaller subnetworks (in-place distillation). Search dimensions: depth (number of layers), width (channel counts), kernel size (convolution sizes), and resolution (input size). Subnet extraction: given target constraints (latency, memory, FLOPs), search for subnet configuration meeting constraints while maximizing accuracy—search is fast since weights are already trained. Accuracy-latency trade-off: single OFA supernet produces family of networks spanning different efficiency points; Pareto-optimal for various hardware. Hardware-specific: extract subnet optimized for specific device (different subnets for mobile vs server). Benefits: train once, deploy many variants; dramatically reduces NAS compute compared to training each architecture separately. OFA demonstrated that elastic networks can match specialized architecture performance while providing flexible deployment options.