mode interpolation

**Mode Interpolation** (Linear Mode Connectivity) is a **model merging technique based on the observation that fine-tuned models from the same pre-trained checkpoint are connected by a linear path of low loss** — enabling simple weight interpolation between models. **How Does Mode Interpolation Work?** - **Two Models**: $ heta_A$ and $ heta_B$, both fine-tuned from the same pre-trained $ heta_0$. - **Interpolate**: $ heta_alpha = (1-alpha) heta_A + alpha heta_B$ for $alpha in [0, 1]$. - **Low Loss Path**: The loss along the interpolation path is roughly constant (linear mode connectivity). - **Paper**: Frankle et al. (2020), Neyshabur et al. (2020). **Why It Matters** - **Model Soup**: Linear mode connectivity is the theoretical foundation for model soup working. - **Multi-Task**: Interpolating between task-specific models creates multi-task models. - **Pre-Training Matters**: Models fine-tuned from different random initializations are NOT linearly connected — shared pre-training is key. **Mode Interpolation** is **the straight line between fine-tuned models** — the remarkable finding that models from the same checkpoint live in the same loss valley.