expert parallel

Expert parallelism distributes Mixture-of-Experts (MoE) model experts across different GPUs, enabling sparse activation with massive total parameter counts. Architecture: router network selects top-k experts per token (typically k=1 or 2), each GPU holds subset of experts and processes only routed tokens. Communication: all-to-all collective sends tokens to assigned expert GPUs, gathers results back. Benefits: scale model parameters without proportional compute increase (e.g., Switch Transformer: 1.6T parameters, activates ~1/128 per token). Challenges: load balancing (some experts overloaded), communication overhead (all-to-all bandwidth), and expert dropout (unused experts). Solutions: auxiliary load-balancing loss, capacity factors (limit tokens per expert), and expert choice routing (experts select tokens). Comparison: tensor parallelism (split layers), pipeline parallelism (split stages), expert parallelism (split experts). Used in: GShard, Switch Transformer, Mixtral, and GPT-4 (rumored). Essential for training trillion-parameter models efficiently.