mixture of experts moe architecture

**Mixture of Experts (MoE)** is **the conditional computation architecture that routes each input token to a subset of specialized expert sub-networks rather than processing through all parameters — enabling models with massive parameter counts (hundreds of billions) while maintaining inference cost comparable to much smaller dense models by activating only 1-2 experts per token**. **MoE Architecture:** - **Expert Networks**: each expert is a standard feed-forward network (FFN) with identical architecture but independent parameters; a Switch Transformer layer replaces the single FFN with E experts (typically 8-128), each containing the same hidden dimension - **Gating Network (Router)**: a learned linear layer that takes the input token embedding and produces a probability distribution over experts; top-K experts (K=1 or K=2) are selected per token based on highest gating scores - **Sparse Activation**: with E=64 experts and K=2, each token uses 2/64 = 3.1% of the total parameters; total model capacity scales with E while per-token compute scales with K — decoupling capacity from compute cost - **Expert FFN Placement**: MoE layers typically replace every other FFN layer in a Transformer; alternating dense and MoE layers provides a balance between shared representations (dense layers) and specialized processing (MoE layers) **Routing Mechanisms:** - **Top-K Routing**: select K experts with highest router logits; weight their outputs by normalized softmax probability; original Shazeer et al. (2017) approach used Top-2 routing with noisy gating - **Expert Choice Routing**: instead of tokens choosing experts, each expert selects its top-K tokens based on router scores; guarantees perfect load balance (each expert processes exactly the same number of tokens) but some tokens may be dropped or processed by fewer experts - **Token Dropping**: when an expert receives more tokens than its capacity buffer allows, excess tokens are dropped (assigned to a residual connection); capacity factor C (typically 1.0-1.5) determines buffer size as C × (total_tokens / num_experts) - **Auxiliary Load Balancing Loss**: additional training loss penalizing uneven token distribution across experts; fraction of tokens assigned to each expert should approximate 1/E for uniform distribution; loss coefficient typically 0.01-0.1 to avoid overwhelming the main training objective **Training Challenges:** - **Load Imbalance**: without auxiliary loss, the majority of tokens route to a few "popular" experts while others receive minimal traffic (expert collapse); severe imbalance wastes capacity and starves unused experts of gradient signal - **Expert Parallelism**: experts distributed across GPUs require all-to-all communication to route tokens to their assigned expert's GPU; communication volume = batch_size × hidden_dim × 2 (send + receive); bandwidth-intensive for large models - **Training Instability**: router gradients can be noisy; expert competition creates reinforcement loops (popular experts improve faster, attracting more tokens); dropout on router logits and jitter noise stabilize training - **Batch Size Sensitivity**: each expert sees batch_size/E effective tokens; larger global batch sizes ensure each expert receives sufficient gradient signal per step; MoE models typically require 4-8× larger batch sizes than equivalent dense models **Production Models:** - **Mixtral 8×7B**: 8 experts with 7B parameters each, Top-2 routing; total 47B parameters but only 13B active per token; matches or exceeds Llama 2 70B while being 6× faster at inference - **Switch Transformer**: Top-1 routing to simplify training; scaled to 1.6 trillion parameters with 2048 experts; demonstrated that scaling expert count improves sample efficiency - **GPT-4 (Rumored)**: believed to use MoE architecture with ~16 experts; 1.8T total parameters with ~220B active per forward pass; demonstrates MoE viability at the frontier of AI capability - **DeepSeek-V2/V3**: MoE with fine-grained expert segmentation (256+ experts, Top-6 routing); achieved competitive performance with significantly reduced training cost Mixture of Experts is **the architectural innovation that breaks the linear relationship between model capacity and inference cost — enabling the training of models with hundreds of billions of parameters at a fraction of the computational cost of equivalent dense models, fundamentally changing the economics of scaling AI systems**.