expert parallelism

**Expert parallelism** is a distributed computing strategy specifically designed for **Mixture of Experts (MoE)** models, where different **expert sub-networks** are placed on **different GPUs**. This allows the model to scale to enormous sizes while keeping the compute cost per token manageable. **How Expert Parallelism Works** - **Expert Assignment**: In an MoE layer, each token is routed to a small subset of experts (typically **2 out of 8–64** experts) by a learned **gating network**. - **Physical Distribution**: Different experts reside on different GPUs. When a token is routed to a specific expert, the token's data is sent to the GPU hosting that expert via **all-to-all communication**. - **Parallel Computation**: Multiple experts process their assigned tokens simultaneously across different GPUs, then results are gathered back. **Comparison with Other Parallelism Strategies** - **Data Parallelism**: Replicates the entire model on each GPU, processes different data. Doesn't help with model size. - **Tensor Parallelism**: Splits individual layers across GPUs. High communication overhead but fine-grained. - **Pipeline Parallelism**: Splits the model into sequential stages across GPUs. Can cause **pipeline bubbles**. - **Expert Parallelism**: Uniquely suited for MoE — splits the model along the **expert dimension**, with communication only needed for token routing. **Challenges** - **Load Balancing**: If the gating network sends too many tokens to experts on the same GPU, that GPU becomes a bottleneck. **Auxiliary load-balancing losses** are used during training to encourage even distribution. - **All-to-All Communication**: The token shuffling between GPUs requires high-bandwidth interconnects (**NVLink, InfiniBand**) to avoid becoming a bottleneck. - **Token Dropping**: When an expert receives more tokens than its capacity, excess tokens may be dropped, requiring careful capacity factor tuning. **Real-World Usage** Models like **Mixtral 8×7B**, **GPT-4** (rumored MoE), and **Switch Transformer** use expert parallelism to achieve very large effective model sizes while only activating a fraction of parameters per token, making both training and inference more efficient.