zero-cost proxies

**Zero-Cost Proxies** are **metrics that estimate the performance of a neural architecture without any training** — computed in a single forward/backward pass at initialization, enabling architecture ranking in seconds instead of hours. **What Are Zero-Cost Proxies?** - **Examples**: - **SynFlow**: Sum of product of all parameters' absolute values (measures signal propagation). - **NASWOT**: Log-determinant of the neural tangent kernel at initialization. - **GradNorm**: Norm of gradients at initialization. - **Fisher**: Fisher information of the network at initialization. - **Cost**: One forward + one backward pass = seconds per architecture. **Why It Matters** - **Speed**: Evaluate 10,000 architectures in minutes (vs. days for one-shot, weeks for full training). - **Pre-Filtering**: Use zero-cost proxies to prune the search space before expensive evaluation. - **Limitation**: Correlation with trained accuracy is imperfect (0.5-0.8 Spearman rank), but improving. **Zero-Cost Proxies** are **instant architecture critics** — predicting network performance at birth, before a single weight update.