mixture of experts moe

**Mixture of Experts (MoE)** is the **sparse architecture paradigm where each input token is routed to only a small subset (typically 1-2) of many parallel "expert" sub-networks within each layer — enabling models with trillions of total parameters while activating only a fraction per token, achieving dramatically better quality-per-FLOP than equivalent dense models**. **The Core Idea** A dense Transformer applies every parameter to every token. An MoE layer replaces the single feed-forward network (FFN) with N parallel FFN experts (e.g., 8, 16, or 64) and a lightweight gating network that decides which expert(s) each token should use. If only 2 of 64 experts fire per token, the active computation is ~32x smaller than a dense model with the same total parameter count. **Gating and Routing** - **Top-K Routing**: The gating network computes a score for each expert given the input token embedding. The top-K experts (typically K=1 or K=2) are selected, and their outputs are weighted by the softmax of their gate scores. - **Switch Transformer**: Routes each token to exactly one expert (K=1), maximizing sparsity. The simplified routing reduces communication overhead and improves training stability. - **Expert Choice Routing**: Instead of each token choosing experts, each expert selects its top-K tokens from the batch. This naturally balances load across experts but requires global coordination. **Load Balancing** Without intervention, the gating network tends to collapse — sending most tokens to a few "popular" experts while others receive no traffic (expert dropout). Mitigation strategies include auxiliary load-balancing losses that penalize uneven expert utilization, noise injection into gate scores during training, and capacity factors that cap the maximum tokens per expert. **Scaling Results** - **GShard** (2020): 600B parameter MoE with 2048 experts, trained with automatic sharding across TPUs. - **Switch Transformer** (2021): Demonstrated that scaling to 1.6T parameters with simplified top-1 routing achieves 4x speedup over dense T5 at equivalent quality. - **Mixtral 8x7B** (2024): 8 experts of 7B parameters each, with top-2 routing. Despite having ~47B total parameters, each forward pass activates only ~13B — matching or exceeding Llama 2 70B quality at ~3x lower inference cost. - **DeepSeek-V2/V3**: Multi-head latent attention combined with fine-grained MoE (256 routed experts), pushing the efficiency frontier further. **Infrastructure Challenges** MoE models require expert parallelism — different experts reside on different GPUs, and all-to-all communication routes tokens to their assigned experts. This communication overhead can dominate training time if not carefully optimized with techniques like expert buffering, hierarchical routing, and capacity-aware placement. Mixture of Experts is **the architecture that broke the linear relationship between model quality and inference cost** — proving that bigger models can actually be cheaper to run by activating only the knowledge each token needs.