Linear Mode Connectivity is a stronger form of mode connectivity where two trained networks are connected by a straight line (linear interpolation) in parameter space with no loss barrier — meaning $mathcal{L}(alpha heta_1 + (1-alpha) heta_2) leq max(mathcal{L}( heta_1), mathcal{L}( heta_2))$ for all $alpha in [0, 1]$.

What Is Linear Mode Connectivity?

• Test: Interpolate weights: $ heta_alpha = alpha heta_A + (1-alpha) heta_B$, evaluate loss at each $alpha$.
• Connected: If no loss barrier exists along this line, the two solutions are linearly mode connected.
• Result: Models trained from the same initialization (or with shared early training) are typically linearly connected.

Why It Matters

• Model Merging: Linearly connected models can be averaged for free ensemble performance (model soups).
• Federated Learning: If local models are linearly connected, simple averaging works for aggregation.
• Git Re-Basin: Techniques like permutation alignment can make independently trained models linearly connected.

Linear Mode Connectivity is the alignment test for neural networks — two models that can be linearly interpolated without degradation live in the same loss basin.