top-k expert selection
**Top-K Expert Selection** is the gating mechanism in Mixture-of-Experts (MoE) transformer architectures that routes each input token to only the K highest-scoring expert networks (typically K=1 or K=2 out of dozens to hundreds of experts), enabling massive model capacity while maintaining computational cost proportional to the active subset rather than the total number of experts. The gating network produces a probability distribution over all experts, and only the top-K experts process each token.
**Why Top-K Expert Selection Matters in AI/ML:**
Top-K expert selection is the **fundamental efficiency mechanism** that makes MoE architectures practical, enabling models with trillions of parameters to run with the FLOPs budget of a much smaller dense model.
• **Sparse activation** — With K=2 and 64 experts, each token activates only ~3% of total parameters, providing 10-30× more model capacity than a dense model with equivalent computational cost per forward pass
• **Gating function** — A learned linear layer followed by softmax produces expert scores: g(x) = softmax(W_g · x), and the top-K scores select which experts process the token; remaining experts contribute zero computation
• **Load balancing** — Auxiliary loss terms (importance loss, load loss) encourage the gating network to distribute tokens evenly across experts, preventing "expert collapse" where few experts receive all traffic while others remain undertrained
• **Expert capacity** — Each expert has a fixed buffer size (capacity factor × tokens/experts); tokens exceeding capacity are dropped or routed to overflow experts, requiring careful capacity planning for training stability
• **Noise injection** — Adding tunable Gaussian noise to gating logits before top-K selection (as in Switch Transformer, GShard) improves exploration during training and promotes more uniform expert utilization
| Architecture | K Value | Experts | Active Params | Total Params |
|-------------|---------|---------|--------------|--------------|
| Switch Transformer | 1 | 128 | ~0.8% | 1.6T |
| GShard | 2 | 2048 | ~0.1% | 600B |
| Mixtral 8×7B | 2 | 8 | 25% | 47B |
| GLaM | 2 | 64 | ~3% | 1.2T |
| ST-MoE | 1 | 32 | ~3% | 269B |
**Top-K expert selection is the architectural innovation that makes trillion-parameter MoE models computationally feasible, enabling each token to leverage massive model capacity while only paying the computational cost of K active experts, fundamentally changing the scaling relationship between model size and inference cost.**