slimmable networks

**Slimmable Networks** are **neural networks trained to execute at multiple preset width configurations** — a single model that can run at 0.25×, 0.5×, 0.75×, or 1.0× width, allowing runtime selection of the accuracy-efficiency trade-off without retraining. **Slimmable Training** - **Switchable Batch Norm**: Each width uses its own batch normalization statistics (separate running means/variances). - **Training**: For each mini-batch, randomly select a width and train at that width — all widths share the same weights. - **Inference**: Select the width at runtime based on the available computation budget. - **Width Configs**: Typically 4 preset widths, but can be extended to more. **Why It Matters** - **One Model, Many Budgets**: Deploy a single model that adapts to varying computational resources at runtime. - **No Retraining**: Switch between accuracy levels without retraining or storing multiple models. - **Device Heterogeneity**: Different devices run the same model at different widths matching their hardware capability. **Slimmable Networks** are **the adjustable-width neural network** — one model trained to operate at multiple efficiency levels, selected at runtime.