mixture of experts
**Mixture of Experts (MoE)** is the **model architecture that uses a gating network to dynamically route each input to a sparse subset of specialized "expert" sub-networks** — enabling models with dramatically more total parameters (and thus more capacity) while keeping per-input computation constant, allowing models like Mixtral 8x7B and GPT-4 to achieve superior performance without proportionally increasing inference cost.
**Core Architecture**
- **Experts**: N parallel feed-forward networks (e.g., N=8 or N=64), each potentially specializing in different input types.
- **Router/Gate**: A network that assigns each token to the top-K experts (typically K=1 or K=2).
- **Sparse Activation**: Only K out of N experts process each input → computation scales with K, not N.
**Routing (Gating)**
$G(x) = TopK(Softmax(W_g \cdot x))$
- Linear layer projects input to N scores (one per expert).
- Softmax normalizes scores to probabilities.
- TopK selects the K highest-scoring experts.
- Output: Weighted sum of selected expert outputs, weighted by gate probabilities.
**Parameter vs. Compute Scaling**
| Model | Total Params | Active Params/Token | Experts | Top-K |
|-------|-------------|--------------------|---------|---------|
| Mixtral 8x7B | 47B | ~13B | 8 | 2 |
| Switch Transformer | 1.6T | ~100B | 128 | 1 |
| GPT-4 (rumored) | ~1.8T | ~220B | 16 | 2 |
| DeepSeek-MoE | 145B | ~22B | 64 | 6 |
**Load Balancing Challenge**
- Without intervention: Router sends most tokens to a few "popular" experts → others idle.
- **Auxiliary load balancing loss**: Penalty for uneven expert utilization.
- $L_{balance} = N \cdot \sum_{i=1}^N f_i \cdot p_i$ where f_i = fraction of tokens to expert i, p_i = average gate probability.
- **Expert capacity**: Token buffer per expert — overflow tokens dropped or re-routed.
**Training Challenges**
- **Instability**: Routing decisions are discrete → training can be unstable.
- **Expert collapse**: All experts converge to similar behavior → no specialization.
- **Communication overhead**: In distributed training, tokens must be sent to the GPU holding each expert (all-to-all communication).
**Sparse vs. Dense Trade-offs**
- **Advantage**: More parameters → more knowledge capacity at same inference cost.
- **Disadvantage**: Higher memory footprint (all experts in memory), communication overhead, less efficient on small batches.
- **When to use MoE**: Large-scale pretraining where parameter count matters more than memory efficiency.
Mixture of experts is **the dominant scaling strategy for frontier language models** — by decoupling parameter count from per-token computation, MoE enables models to store more knowledge and handle more diverse tasks while maintaining economically viable inference costs.