model soup

**Model Soup** is a **model merging technique that averages the weights of multiple fine-tuned models** — taking several models fine-tuned with different hyperparameters from the same pre-trained checkpoint and averaging their parameters, often outperforming the best individual model. **How Does Model Soup Work?** - **Fine-Tune**: Train multiple models from the same pre-trained checkpoint with different hyperparameters (learning rate, augmentation, etc.). - **Average**: $ heta_{soup} = frac{1}{K}sum_k heta_k$ (simple weight averaging). - **Greedy Soup**: Iteratively add models to the soup only if they improve validation accuracy. - **Paper**: Wortsman et al. (2022). **Why It Matters** - **Free Accuracy**: Outperforms the best individual model without additional inference cost. - **CLIP**: Greedy model soup of CLIP fine-tunes achieved SOTA on ImageNet (2022). - **No Ensemble Cost**: Unlike model ensembles ($K imes$ compute at inference), model soup has the same cost as one model. **Model Soup** is **the recipe for better models** — averaging multiple fine-tuned models into one that is better than any individual ingredient.