feature learning regime

**Feature Learning Regime** is the **operating mode where neural networks actively learn useful internal representations during training** — as opposed to the lazy regime where features remain random. This is the regime where deep learning achieves its remarkable empirical success. **What Is Feature Learning?** - **Condition**: Networks with practical width, learning rate, and initialization (not the infinite-width NTK limit). - **Feature Evolution**: Hidden representations change significantly during training, adapting to the data. - **Beyond NTK**: NTK theory describes lazy training. Feature learning is the more complex, nonlinear regime. - **Muᵖ Parameterization**: The maximal update parameterization (muP) provably enables feature learning at any width. **Why It Matters** - **Performance**: Feature learning is what makes deep learning work. Lazy training networks underperform. - **Representation**: The ability to learn hierarchical features (edges -> textures -> objects) is deep learning's key advantage. - **Theory Gap**: Feature learning is theoretically harder to analyze, creating a gap between NTK theory and practice. **Feature Learning** is **the real revolution of deep learning** — the regime where networks actually learn the right internal representations, not just linearly combine random features.