compound scaling

**Compound Scaling** is the **principled method of jointly scaling a neural network's depth, width, and resolution using a fixed ratio** — introduced in EfficientNet, showing that balanced scaling outperforms scaling any single dimension. **How Does Compound Scaling Work?** - **Three Dimensions**: Depth ($d$), Width ($w$), Resolution ($r$). - **Constraint**: $alpha cdot eta^2 cdot gamma^2 approx 2$ (doubles FLOPs per unit increase in $phi$). - **Grid Search**: Find optimal $alpha, eta, gamma$ on a small model (B0). Then scale with $phi$. - **Result**: $d = alpha^phi, w = eta^phi, r = gamma^phi$. **Why It Matters** - **Balanced Growth**: Networks that only grow deeper (ResNet-1000) or only wider (Wide-ResNet) hit diminishing returns. Compound scaling avoids this. - **Universal**: The principle applies beyond EfficientNet — any architecture benefits from balanced scaling. - **Design Rule**: Provides a concrete recipe for scaling any base architecture. **Compound Scaling** is **the growth formula for neural networks** — a mathematical recipe ensuring balanced development across all dimensions.