mixture of experts hierarchical

**Hierarchical MoE** is the **multi-stage routing architecture that selects expert groups first and individual experts second** - it scales sparse expert systems by reducing routing search complexity and communication fan-out. **What Is Hierarchical MoE?** - **Definition**: A tree-like expert selection design with coarse routing followed by fine routing. - **Routing Stages**: Stage one picks an expert cluster, and stage two selects top experts within that cluster. - **Scale Objective**: Supports very large expert counts without evaluating every expert for every token. - **System Structure**: Often aligns expert groups with topology boundaries such as node or rack locality. **Why Hierarchical MoE Matters** - **Scalability**: Reduces router compute and metadata overhead as expert count grows into the thousands. - **Communication Efficiency**: Limits token traffic to selected groups instead of global all-to-all to every expert shard. - **Specialization Depth**: Enables coarse domain grouping plus fine-grained specialist behavior inside each group. - **Operational Control**: Easier to reason about load distribution at group and expert levels. - **Cost Containment**: Makes large sparse models more feasible on real cluster budgets. **How It Is Used in Practice** - **Group Construction**: Partition experts by capacity and expected feature domains before training. - **Router Training**: Train coarse and fine routers jointly with balancing losses at both levels. - **Telemetry**: Monitor group-level skew and expert-level skew separately to detect collapse quickly. Hierarchical MoE is **a key architecture for scaling sparse models beyond flat routing limits** - staged selection improves both system efficiency and manageability at large expert counts.