token dispatch

**Token dispatch** is the **routing-stage data movement that groups and sends token representations to their assigned experts** - it transforms router decisions into contiguous expert-ready batches for efficient sparse computation. **What Is Token dispatch?** - **Definition**: Permutation and transfer process that maps tokens from source order to destination-expert order. - **Core Steps**: Build routing indices, pack token buffers by expert, and transmit shards to expert-owner ranks. - **Memory Objective**: Create contiguous blocks so expert GEMM kernels run with high efficiency. - **Execution Layer**: Implemented with fused permutation kernels, communication collectives, and metadata tables. **Why Token dispatch Matters** - **Step-Time Share**: Dispatch can consume large runtime when token counts or expert groups are large. - **Bandwidth Use**: Packing quality affects payload efficiency and network overhead. - **Compute Readiness**: Poor dispatch layouts degrade expert kernel throughput. - **Scalability**: Dispatch bottlenecks limit gains from adding more experts or devices. - **Stability**: Deterministic dispatch logic is required for correct token-to-output mapping. **How It Is Used in Practice** - **Kernel Optimization**: Use high-throughput pack and permutation kernels to reduce staging overhead. - **Metadata Design**: Maintain compact index structures for fast combine reversal. - **End-to-End Profiling**: Separate dispatch latency from expert compute to target the right bottleneck. Token dispatch is **a foundational MoE runtime primitive** - efficient token packing and transfer directly determine sparse execution performance.