linear mode connectivity

**Linear Mode Connectivity** is a **stronger form of mode connectivity where two trained networks are connected by a straight line (linear interpolation) in parameter space with no loss barrier** — meaning $mathcal{L}(alpha heta_1 + (1-alpha) heta_2) leq max(mathcal{L}( heta_1), mathcal{L}( heta_2))$ for all $alpha in [0, 1]$. **What Is Linear Mode Connectivity?** - **Test**: Interpolate weights: $ heta_alpha = alpha heta_A + (1-alpha) heta_B$, evaluate loss at each $alpha$. - **Connected**: If no loss barrier exists along this line, the two solutions are linearly mode connected. - **Result**: Models trained from the same initialization (or with shared early training) are typically linearly connected. **Why It Matters** - **Model Merging**: Linearly connected models can be averaged for free ensemble performance (model soups). - **Federated Learning**: If local models are linearly connected, simple averaging works for aggregation. - **Git Re-Basin**: Techniques like permutation alignment can make independently trained models linearly connected. **Linear Mode Connectivity** is **the alignment test for neural networks** — two models that can be linearly interpolated without degradation live in the same loss basin.