metainit

**MetaInit** is a **meta-learning-based initialization method that uses gradient descent to find weight initializations that minimize the curvature of the loss landscape** — searching for starting points where training dynamics will be most favorable. **How Does MetaInit Work?** - **Objective**: Find initial weights $ heta_0$ that minimize the trace of the Hessian $ ext{tr}(H( heta_0))$ (surrogate for loss landscape curvature). - **Process**: Use gradient descent on the initialization itself — not on the loss, but on a meta-objective about the loss landscape. - **Effect**: Produces starting points in flat, well-conditioned regions of the loss landscape. - **Paper**: Dauphin & Schoenholz (2019). **Why It Matters** - **Principled**: Directly optimizes the quantity that determines training difficulty (curvature). - **BatchNorm-Free**: Can enable training of deep networks without BatchNorm by finding better starting points. - **Theory**: Connects initialization to the loss landscape geometry literature (flat vs. sharp minima). **MetaInit** is **learning how to start** — using meta-learning to find the optimal initial conditions for neural network training.