dense-to-sparse conversion
**Dense-to-sparse conversion** is the **process of transforming a pretrained dense model into an MoE-style sparse model by expanding and routing selected layers** - it reuses existing learned representations to reduce full sparse pretraining cost.
**What Is Dense-to-sparse conversion?**
- **Definition**: Upcycling workflow that clones or factorizes dense feed-forward blocks into multiple experts.
- **Initialization Goal**: Preserve useful dense-model knowledge while enabling expert specialization.
- **Router Introduction**: Add gating modules and load-balancing objectives to control token assignment.
- **Scope Choice**: Usually applied to specific transformer layers rather than every layer at once.
**Why Dense-to-sparse conversion Matters**
- **Cost Savings**: Avoids training very large sparse models from random initialization.
- **Faster Ramp-Up**: Starts from a strong checkpoint with already learned general capabilities.
- **Practical Scaling**: Lets teams increase capacity with manageable incremental training budgets.
- **Risk Reduction**: Dense baseline offers fallback if sparse conversion underperforms.
- **Deployment Speed**: Shortens timeline from architecture idea to usable sparse model.
**How It Is Used in Practice**
- **Checkpoint Expansion**: Duplicate dense MLP weights into multiple expert slots with controlled perturbation.
- **Router Warmup**: Train routing gradually while monitoring expert utilization and quality drift.
- **Stabilization Phase**: Apply balancing losses and schedule adjustments until specialization becomes healthy.
Dense-to-sparse conversion is **a pragmatic path to large-capacity MoE systems** - upcycling dense checkpoints can deliver sparse benefits with significantly lower training investment.