mode connectivity

**Mode Connectivity** is the **observation that distinct minima of neural network loss functions are often connected by low-loss paths** — meaning there exist smooth trajectories in parameter space between different trained solutions without crossing high-loss barriers. **What Is Mode Connectivity?** - **Discovery**: Garipov et al. (2018) showed that independently trained networks can be connected by simple curves (quadratic Bezier) with near-constant loss along the path. - **Linear Connectivity**: A stronger form where a straight line between two minima has low loss everywhere. - **Loss Barriers**: Traditional view (convex optimization) expected high barriers between minima. Mode connectivity shows the landscape is more benign. **Why It Matters** - **Ensemble Understanding**: Explains why model averaging and snapshot ensembles work well. - **Landscape Geometry**: Reveals that the loss landscape has a connected low-loss manifold, not isolated valleys. - **Training**: Models trained with different initializations find solutions in the same "basin." **Mode Connectivity** is **the hidden highways between solutions** — low-loss tunnels connecting apparently different minima in the vast parameter landscape.