token combine

**Token combine** is the **post-expert reconstruction stage that returns routed outputs to original token order and merges multi-expert contributions** - it closes the MoE routing loop and restores sequence-consistent activations. **What Is Token combine?** - **Definition**: Inverse mapping process that gathers expert outputs and places them back at source token indices. - **Combination Logic**: Applies router weights to blend outputs when top-k routing uses multiple experts. - **Data Dependency**: Relies on dispatch metadata to ensure exact correspondence between tokens and expert results. - **Runtime Position**: Executed after expert compute and before downstream transformer operations. **Why Token combine Matters** - **Correctness**: Any index or weighting error corrupts token representations and model quality. - **Latency Contribution**: Combine can become a hidden bottleneck in large expert-parallel groups. - **Memory Traffic**: Inefficient scatter and gather patterns increase HBM and network overhead. - **Numerical Integrity**: Weighted merge precision influences stability in mixed-precision training. - **Pipeline Balance**: Fast combine is required so expert compute gains are not canceled downstream. **How It Is Used in Practice** - **Inverse Indexing**: Store compact permutation maps during dispatch for exact reconstruction. - **Fused Operations**: Merge gather and weighting steps to reduce extra memory passes. - **Validation Suite**: Test token-order restoration and top-k weighting parity against reference implementation. Token combine is **a critical correctness and performance stage in MoE execution** - robust reconstruction logic ensures sparse routing produces usable and efficient transformer activations.