token dropping in moe

**Token dropping in MoE** is the **overflow handling behavior where tokens exceeding expert capacity are skipped or routed through fallback paths** - it protects system stability when router assignments temporarily exceed per-expert processing limits. **What Is Token dropping in MoE?** - **Definition**: Capacity-control mechanism that limits tokens processed by each expert per step. - **Trigger Condition**: Occurs when router assignments exceed configured expert capacity factor. - **Fallback Modes**: Dropped tokens may pass through residual paths, backup experts, or deferred handling logic. - **Systems Context**: Relevant in top-k routing schemes where load spikes can be highly uneven. **Why Token dropping in MoE Matters** - **Stability Protection**: Prevents runtime failures from expert buffer overflow. - **Throughput Control**: Keeps step latency predictable under routing imbalance. - **Quality Risk**: Excessive drops can hurt model accuracy and gradient quality. - **Capacity Planning**: Drop rate is a key signal for tuning expert count and routing policy. - **Operational Monitoring**: Persistent dropping indicates load-balancing or architecture issues. **How It Is Used in Practice** - **Metric Tracking**: Monitor drop fraction by layer, expert, and training phase. - **Router Tuning**: Adjust capacity factors, auxiliary losses, and routing temperature to reduce overflow. - **Fallback Design**: Implement robust residual or backup routing to limit quality degradation. Token dropping in MoE is **an important safeguard but also a diagnostic signal** - controlled low drop rates indicate healthy routing and efficient expert utilization.