evonorm

**EvoNorm** is a **family of normalization-activation layers discovered by automated search** — using evolutionary algorithms to find novel combinations of normalization and activation operations that outperform hand-designed ones like BN-ReLU or GN-ReLU. **How Was EvoNorm Discovered?** - **Search Space**: Primitive operations (mean, variance, sigmoid, multiplication, max, etc.) combined in computation graphs. - **Objective**: Maximize validation accuracy on ImageNet with various architectures. - **Results**: EvoNorm-B0 (batch-dependent, replaces BN-ReLU), EvoNorm-S0 (batch-independent, replaces GN-ReLU). - **Paper**: Liu et al. (2020). **Why It Matters** - **Beyond Hand-Design**: Demonstrates that automated search can discover normalization layers humans haven't considered. - **Performance**: EvoNorm-S0 matches BatchNorm+ReLU accuracy while being batch-independent. - **Joint Design**: Searches normalization and activation together, finding synergies that separate design misses. **EvoNorm** is **evolved normalization** — normalization-activation layers discovered by evolution rather than human intuition.