capacity factor tuning

**Capacity factor tuning** is the **calibration of per-expert token buffer headroom relative to average expected load in MoE routing** - it balances overflow risk, memory use, and communication efficiency. **What Is Capacity factor tuning?** - **Definition**: Setting a multiplier on average tokens per expert to determine maximum accepted tokens each step. - **Formula Context**: Capacity is typically proportional to total routed tokens divided by number of experts times a factor. - **Low Factor Behavior**: Minimal headroom improves efficiency but increases token dropping during load spikes. - **High Factor Behavior**: More slack reduces drops but increases memory footprint and transfer volume. **Why Capacity factor tuning Matters** - **Quality Protection**: Excess drops can harm convergence and degrade final model performance. - **Throughput Impact**: Overly large capacity wastes bandwidth and may slow dispatch and combine phases. - **Stability Control**: Proper headroom prevents frequent overflow oscillations. - **Cost Efficiency**: Right-sized buffers reduce unnecessary infrastructure overhead. - **Operational Predictability**: Balanced settings produce smoother latency distributions across steps. **How It Is Used in Practice** - **Baseline Sweep**: Benchmark several factor values under representative sequence and batch regimes. - **Metric Pairing**: Track drop rate, utilization skew, and step latency together during tuning. - **Adaptive Policy**: Revisit factor after router behavior shifts during later training phases. Capacity factor tuning is **a high-leverage MoE systems parameter** - disciplined calibration is necessary to preserve quality while keeping sparse execution efficient.