sparse upcycling

**Sparse Upcycling** is the **model scaling technique that converts a pre-trained dense transformer into a Mixture of Experts (MoE) model by replicating the feed-forward network (FFN) layers into multiple experts and adding a learned router — leveraging the full pre-training investment while dramatically increasing model capacity at modest additional training cost** — the proven methodology (used by Mixtral and Switch Transformer variants) for creating high-capacity sparse models without the prohibitive cost of training them from scratch. **What Is Sparse Upcycling?** - **Definition**: Taking a fully pre-trained dense transformer and converting it into a sparse MoE model by: (1) copying each FFN layer into N expert copies, (2) adding a gating/routing network, and (3) continuing training with sparse expert activation — transforming a dense 7B model into a sparse 47B model (8 experts × 7B FFN). - **Initialization from Dense Weights**: Experts are initialized as copies of the original dense FFN — ensuring the starting point has the full quality of the pre-trained model rather than random initialization. - **Sparse Activation**: During inference, only top-k experts (typically k=1 or k=2) are activated per token — total parameters increase dramatically but active parameters (and FLOPs) increase only modestly. - **Continued Pre-Training**: After conversion, the model is trained for additional steps to allow experts to specialize and the router to learn meaningful routing patterns. **Why Sparse Upcycling Matters** - **Leverages Pre-Training Investment**: Pre-training a 7B model costs $1M+; upcycling reuses this investment entirely — the upcycled model starts from full pre-trained quality and only needs additional training for expert specialization. - **5–10× Cheaper Than Fresh MoE Training**: Training a 47B MoE from scratch requires compute comparable to a 47B dense model; upcycling from a 7B dense model requires only 10–20% of that compute for continued training. - **Proven at Scale**: Mixtral-8x7B (likely upcycled from Mistral-7B) demonstrated that sparse upcycled models match or exceed dense models 3× their active parameter count — 47B total parameters performing at 70B dense quality. - **Incremental Scaling**: Organizations can progressively scale their models — train a dense 7B, upcycle to 8×7B MoE, and later upcycle further — avoiding the all-or-nothing bet of training massive models from scratch. - **Expert Specialization**: Despite starting from identical copies, experts naturally specialize during continued training — some become coding experts, others language experts, others reasoning experts. **Sparse Upcycling Process** **Step 1 — Dense Model Selection**: - Start with a well-trained dense transformer (e.g., Llama-7B, Mistral-7B). - The dense model provides the attention layers (shared across all experts) and FFN layers (replicated into experts). **Step 2 — Expert Initialization**: - Copy the FFN weights from each transformer layer into N experts (typically N=4, 8, or 16). - Add a lightweight router network (linear layer projecting hidden_dim → N expert scores). - Attention layers remain shared — only FFN layers become sparse. **Step 3 — Continued Pre-Training**: - Train with top-k expert routing (k=1 or k=2 active experts per token). - Load balancing loss encourages uniform expert utilization. - Training duration: 10–20% of original pre-training compute. **Step 4 — Expert Specialization Verification**: - Analyze routing patterns to confirm experts have developed different specializations. - Verify that different token types preferentially route to different experts. **Upcycling Economics** | Approach | Total Parameters | Active Parameters | Training Cost (vs. Dense) | |----------|-----------------|-------------------|--------------------------| | **Dense 7B** | 7B | 7B | 1.0× (baseline) | | **Upcycled 8×7B MoE** | 47B | 13B | 1.1–1.2× | | **Fresh MoE 8×7B** | 47B | 13B | 5–8× | | **Dense 70B** | 70B | 70B | 10× | Sparse Upcycling is **the capital-efficient path to model scaling** — transforming the economics of large model development by proving that sparse capacity can be grafted onto proven dense foundations rather than grown from seed, enabling organizations to achieve frontier-model quality at a fraction of the compute investment.