mixture of experts moe

**Mixture of Experts (MoE)** is the **sparse model architecture that replaces each dense feed-forward layer with multiple parallel "expert" sub-networks and a learned gating function that routes each input token to only K of N experts (typically K=1-2 out of N=8-128) — enabling models with trillion-parameter total capacity while maintaining the per-token compute cost of a much smaller dense model, because only a fraction of parameters are activated for each input**. **Why MoE Scales Efficiently** A dense 175B model requires 175B parameters of computation per token. An MoE model with 8 experts of 22B each has 176B total parameters but activates only 1-2 experts (22-44B) per token. The model has the capacity to specialize different experts for different input types while keeping inference cost comparable to a 22-44B dense model. **Architecture** In a transformer MoE layer: 1. **Gating Network**: A small linear layer maps each token's hidden state to a score for each expert: g(x) = softmax(W_g · x). The top-K experts with highest scores are selected. 2. **Expert Computation**: Each selected expert processes the token through its own feed-forward network (two linear layers with activation). Different experts can specialize in different token types. 3. **Combination**: The outputs of the K selected experts are weighted by their gating scores and summed: output = Σ g_k(x) · Expert_k(x). **Routing Challenges** - **Load Imbalance**: Without regularization, the gating network tends to route most tokens to a few "popular" experts, leaving others underutilized. An auxiliary load-balancing loss penalizes uneven expert utilization, encouraging uniform routing. - **Expert Collapse**: In extreme imbalance, unused experts stop learning and become permanently dead. Hard-coded routing constraints (capacity factor limiting tokens per expert) prevent this. - **Token Dropping**: When an expert exceeds its capacity budget, excess tokens are either dropped (skipping the MoE layer) or routed to a secondary expert. Dropped tokens lose representational quality. **Key Models** - **Switch Transformer (Google, 2021)**: K=1 routing (only one expert per token), N=128 experts. Demonstrated 4-7x training speedup over dense T5 at equivalent compute. - **Mixtral 8x7B (Mistral, 2023)**: 8 experts, K=2 routing. 46.7B total parameters but 12.9B active per token. Matches or exceeds Llama 2 70B quality at fraction of compute. - **DeepSeek-V3 (2024)**: 256 experts with auxiliary-loss-free routing and multi-token prediction. 671B total / 37B active parameters. **Inference Challenges** MoE models require all N experts in memory even though only K are active per token. A 8x22B MoE needs the same memory as a 176B dense model. Expert parallelism distributes experts across GPUs, but the dynamic routing makes load balancing across GPUs non-trivial. Expert offloading (storing inactive experts on CPU/NVMe) enables single-GPU inference at the cost of latency. Mixture of Experts is **the architecture that breaks the linear relationship between model capacity and compute cost** — proving that a model can know vastly more than it uses for any single input, selecting the relevant expertise on the fly.