Mode Interpolation (Linear Mode Connectivity) is a model merging technique based on the observation that fine-tuned models from the same pre-trained checkpoint are connected by a linear path of low loss — enabling simple weight interpolation between models.

How Does Mode Interpolation Work?

• Two Models: $ heta_A$ and $ heta_B$, both fine-tuned from the same pre-trained $ heta_0$.
• Interpolate: $ heta_alpha = (1-alpha) heta_A + alpha heta_B$ for $alpha in [0, 1]$.
• Low Loss Path: The loss along the interpolation path is roughly constant (linear mode connectivity).
• Paper: Frankle et al. (2020), Neyshabur et al. (2020).

Why It Matters

• Model Soup: Linear mode connectivity is the theoretical foundation for model soup working.
• Multi-Task: Interpolating between task-specific models creates multi-task models.
• Pre-Training Matters: Models fine-tuned from different random initializations are NOT linearly connected — shared pre-training is key.

Mode Interpolation is the straight line between fine-tuned models — the remarkable finding that models from the same checkpoint live in the same loss valley.