expert capacity
**Expert Capacity** is the maximum number of tokens that can be routed to any single expert within a Mixture-of-Experts (MoE) layer during a single forward pass, defined as the capacity factor (CF) multiplied by the average number of tokens per expert (total tokens / number of experts). Expert capacity acts as a hard buffer limit that prevents memory overflow and ensures balanced computation, but tokens exceeding an expert's capacity are dropped and passed through residual connections without expert processing.
**Why Expert Capacity Matters in AI/ML:**
Expert capacity is a **critical design parameter** that balances computational efficiency, memory usage, and model quality in MoE architectures—too low causes excessive token dropping, too high wastes memory and computation.
• **Capacity factor tuning** — CF = 1.0 means each expert has exactly enough buffer for perfectly balanced routing; practical values range from 1.0-1.5 to accommodate routing imbalance; Switch Transformer uses CF = 1.0-1.25 with auxiliary load balancing
• **Token dropping** — When more tokens are routed to an expert than its capacity allows, overflow tokens skip expert processing and pass through the residual connection, degrading quality proportional to the drop rate; well-tuned models target <1% token dropping
• **Memory planning** — Expert capacity directly determines the memory allocated per expert for activation storage during the forward pass; capacity × hidden_dim × batch determines the expert buffer size in GPU memory
• **Batch size interaction** — Larger batch sizes provide better statistical averaging of routing decisions, reducing per-expert load variance and allowing lower capacity factors; small batches require higher CF to avoid excessive dropping
• **Dynamic capacity** — Advanced implementations (e.g., Megablocks, FlexMoE) use variable-length expert buffers to eliminate fixed capacity constraints, processing exactly the tokens routed to each expert without dropping or waste
| Capacity Factor | Token Drop Rate | Memory Usage | Best For |
|----------------|-----------------|-------------|----------|
| 1.0 | 5-20% | Minimum | Memory-constrained training |
| 1.25 | 1-5% | Moderate | Standard training |
| 1.5 | <1% | Higher | Quality-critical applications |
| 2.0 | ~0% | 2× minimum | Small-batch inference |
| Dynamic | 0% | Variable | Advanced implementations |
**Expert capacity is the key parameter governing the efficiency-quality tradeoff in MoE architectures, determining how many tokens each expert can process per batch and directly controlling the token-dropping rate that impacts model quality, memory consumption, and computational efficiency of sparse expert models.**