shared expert in moe

**Shared expert in MoE** is the **always-active expert path that processes every token alongside routed sparse experts** - it provides a stable general-purpose representation channel while specialist experts handle token-specific patterns. **What Is Shared expert in MoE?** - **Definition**: A designated expert that receives all tokens regardless of router top-k assignment. - **Architectural Role**: Acts as a dense backbone inside an MoE block to preserve common language and reasoning signals. - **Routing Interaction**: Shared output is combined with routed expert outputs before residual integration. - **Deployment Pattern**: Common in large MoE systems where pure sparse routing can be unstable early in training. **Why Shared expert in MoE Matters** - **Training Stability**: Guarantees each token has a reliable processing path even when routing is noisy. - **Knowledge Retention**: Captures broad capabilities such as syntax and generic semantics that should not depend on expert selection. - **Drop Mitigation**: Reduces quality loss when capacity limits or imbalance cause routed-token pressure. - **Convergence Support**: Helps routers specialize gradually without catastrophic dependence on early routing decisions. - **Production Robustness**: Improves consistency under variable loads and imperfect expert utilization. **How It Is Used in Practice** - **Block Design**: Add one shared FFN expert per MoE layer and merge it with sparse expert outputs. - **Weight Tuning**: Calibrate mixing coefficients so shared and routed paths contribute appropriately. - **Monitoring**: Track whether shared path over-dominates, which can hide needed expert specialization. Shared expert in MoE is **a practical stability anchor for sparse transformer architectures** - it balances specialist routing with dependable general-purpose computation.