expert capacity factor
**Expert Capacity Factor** is the **hyperparameter in Mixture of Experts (MoE) models that controls the maximum number of tokens each expert can process per batch** — calculated as (total tokens / number of experts) × capacity factor, where a factor of 1.0 means each expert handles its fair share and values above 1.0 (typically 1.25-1.5) provide buffer space for uneven routing, with tokens that exceed an expert's capacity being dropped (not processed) or routed to a secondary expert.
**What Is Expert Capacity Factor?**
- **Definition**: A multiplier that determines the buffer size for each expert in a MoE layer — if there are 1024 tokens and 8 experts, the fair share is 128 tokens per expert. A capacity factor of 1.25 sets each expert's buffer to 160 tokens, providing 25% headroom for routing imbalance.
- **The Routing Problem**: MoE routers don't distribute tokens perfectly evenly — popular experts receive more tokens than unpopular ones. Without capacity limits, a single expert could receive all tokens, defeating the purpose of parallelism.
- **Token Dropping**: When an expert's buffer is full, additional tokens routed to that expert are "dropped" — they skip the expert computation entirely and pass through via the residual connection only. Dropped tokens lose the benefit of expert processing.
- **Padding Waste**: Experts that receive fewer tokens than their capacity have empty buffer slots that consume compute but produce no useful output — higher capacity factors increase this wasted computation.
**Capacity Factor Tradeoffs**
| Factor | Buffer Size | Token Dropping | Compute Waste | Quality |
|--------|-----------|---------------|--------------|---------|
| 1.0 | Exact fair share | High (any imbalance drops) | Minimal | Lower (many drops) |
| 1.25 | 25% buffer | Moderate | Low | Good (standard) |
| 1.5 | 50% buffer | Low | Moderate | Better |
| 2.0 | 100% buffer | Very low | High | Best (but wasteful) |
| ∞ (no limit) | Unlimited | None | Variable | Best quality, worst efficiency |
**Capacity Factor in Practice**
- **Switch Transformer (Google)**: Uses capacity factor 1.0-1.25 with auxiliary load balancing loss — the load balancing loss encourages even routing, reducing the need for large capacity buffers.
- **Mixtral (Mistral)**: Uses top-2 routing without explicit capacity limits — relies on the router learning balanced distributions during training.
- **GShard**: Introduced the capacity factor concept with a default of 2.0 — prioritizing quality over compute efficiency in early MoE research.
- **Expert Choice Routing**: An alternative approach where experts choose their top-k tokens (instead of tokens choosing experts) — guarantees perfect load balance and eliminates the need for capacity factors entirely.
**Expert capacity factor is the buffer-sizing knob that balances token processing quality against compute efficiency in MoE models** — setting it too low drops tokens and hurts quality, setting it too high wastes compute on empty buffer slots, with the optimal value (typically 1.25) depending on how well the router distributes tokens across experts.