mixture of experts moe

**Mixture of Experts (MoE)** is the **neural network architecture that routes each input token through only a subset of specialized sub-networks (experts) selected by a learned gating mechanism — enabling models with trillions of parameters while keeping per-token computation constant, because only 1-2 experts out of hundreds are activated for any given input**. **The Scaling Dilemma MoE Solves** Dense transformer models scale by increasing width (hidden dimension) and depth (layers), but compute cost grows proportionally with parameter count. A 1.8T parameter dense model would require enormous FLOPs per token. MoE decouples parameter count from compute cost: a 1.8T MoE model with 128 experts and top-2 routing activates only ~28B parameters per token — the same compute as a 28B dense model but with access to a much larger knowledge capacity. **Architecture** In a typical MoE transformer, every other feed-forward network (FFN) layer is replaced with an MoE layer: - **Experts**: N identical FFN sub-networks (e.g., N=8, 64, or 128), each with independent parameters. - **Router (Gating Network)**: A lightweight linear layer that takes the token representation as input and outputs a probability distribution over experts. The top-K experts (typically K=1 or K=2) are selected per token. - **Combination**: The outputs of the selected experts are weighted by their gating probabilities and summed. **Load Balancing Challenge** Without constraints, the router tends to collapse — sending all tokens to a few popular experts while others remain unused. This wastes capacity and creates compute imbalance across devices (each expert is placed on a different GPU). Solutions: - **Auxiliary Load Balancing Loss**: An additional loss term that penalizes uneven expert utilization, encouraging the router to distribute tokens evenly. - **Expert Capacity Factor**: Each expert has a maximum number of tokens it can process per batch. Overflow tokens are either dropped or routed to a shared fallback expert. - **Token Choice vs. Expert Choice**: In expert-choice routing, each expert selects its top-K tokens rather than each token selecting its top-K experts — guaranteeing perfect load balance. **Training Infrastructure** MoE layers require expert parallelism: experts are distributed across GPUs, and all-to-all communication shuffles tokens to their assigned expert's GPU and back. This all-to-all pattern is bandwidth-intensive and requires careful overlap with computation. Frameworks like Megatron-LM and DeepSpeed-MoE provide optimized implementations combining data, tensor, expert, and pipeline parallelism. **Notable MoE Models** - **Switch Transformer** (Google): Top-1 routing with simplified load balancing. Demonstrated 7x training speedup over dense T5 at equivalent compute. - **Mixtral 8x7B** (Mistral): 8 experts per layer, top-2 routing. 46.7B total parameters but ~13B active per token. Outperforms LLaMA 2 70B at much lower inference cost. - **DeepSeek-V2/V3**: MoE with fine-grained experts (up to 256) and shared expert layers for common knowledge. Mixture of Experts is **the architectural paradigm that breaks the linear relationship between model capacity and inference cost** — enabling foundation models to store vastly more knowledge in their parameters while maintaining practical serving latency and throughput.