weight averaging
**Weight averaging** is a **model combination technique that averages parameters from multiple trained models** — creating merged models that often outperform individual components through ensemble-like effects.
**What Is Weight Averaging?**
- **Definition**: Average corresponding weights from multiple models.
- **Formula**: w_merged = (w_A + w_B) / 2, or weighted average.
- **Requirement**: Models must share same architecture.
- **Result**: Single model combining capabilities.
- **No Training**: Merge without additional compute.
**Why Weight Averaging Matters**
- **Improved Performance**: Often beats individual models.
- **Combine Strengths**: Merge specialist models.
- **Regularization**: Averaging smooths weight space.
- **Community**: Foundation of Stable Diffusion model merging.
- **Efficiency**: No training required.
**Averaging Methods**
- **Simple Average**: (A + B) / 2.
- **Weighted Average**: α*A + (1-α)*B, control contribution.
- **SLERP**: Spherical interpolation in weight space.
- **Task Arithmetic**: Add/subtract task-specific directions.
**When It Works**
- Models trained on same architecture.
- Models fine-tuned from same base.
- Similar training data distributions.
- Complementary specializations.
**Example**
```python
merged = {}
for key in model_a.keys():
merged[key] = 0.7 * model_a[key] + 0.3 * model_b[key]
```
Weight averaging is the **simplest and often effective model merging** — combining capabilities without training.