dare

**DARE** (Drop and Rescale) is a **model merging technique that randomly drops (zeros out) a fraction of fine-tuned parameter changes and rescales the remaining ones** — reducing parameter interference between merged models while preserving the overall magnitude of task-specific updates. **How Does DARE Work?** - **Task Vector**: Compute $ au = heta_{fine} - heta_{pre}$ (the fine-tuning delta). - **Drop**: Randomly set a fraction $p$ of $ au$'s elements to zero (Bernoulli mask). - **Rescale**: Multiply remaining elements by $1/(1-p)$ to maintain expected magnitude. - **Merge**: Average the dropped-and-rescaled task vectors from multiple models. - **Paper**: Yu et al. (2024). **Why It Matters** - **Less Interference**: Dropping parameters reduces overlap and conflict between task vectors. - **Better Merging**: DARE + TIES or DARE + simple averaging significantly outperforms naive averaging. - **LLM Merging**: Widely used in the open-source LLM community for merging fine-tuned models. **DARE** is **dropout for model merging** — randomly sparsifying task vectors before merging to reduce destructive interference between models.