fisher-weighted averaging

**Fisher-Weighted Averaging** is a **model merging technique that weights each parameter by its Fisher information** — parameters that are more important for a task (higher Fisher information) are weighted more heavily during averaging, preserving critical task-specific knowledge. **How Does Fisher-Weighted Averaging Work?** - **Fisher Information**: $F_i = mathbb{E}[( abla_{ heta_i} log p(y|x, heta))^2]$ — measures how sensitive the loss is to each parameter. - **Weighted Average**: $ heta_{merged,i} = frac{sum_k F_i^{(k)} cdot heta_i^{(k)}}{sum_k F_i^{(k)}}$ (Fisher-weighted). - **Intuition**: If parameter $i$ is crucial for task $A$ but unimportant for task $B$, use task $A$'s value. - **Paper**: Matena & Raffel (2022). **Why It Matters** - **Importance-Weighted**: Not all parameters are equally important — Fisher weighting respects this. - **Better Than Uniform**: Outperforms simple averaging by preserving each task's critical parameters. - **EWC Connection**: Related to Elastic Weight Consolidation, using Fisher information to prevent catastrophic forgetting. **Fisher-Weighted Averaging** is **importance-aware merging** — using information theory to determine which task's version of each parameter matters most.