Mixture of Experts (MoE) Language Models is the sparse routing architecture where each token is routed to subset of experts through learned gating — achieving high parameter count with reasonable compute by activating only subset of total experts per forward pass.

Sparse MoE Gating Mechanism:

• Expert routing: learned gating network routes each input token to top-K experts (typically K=2 or K=4) based on highest gate scores
• Switch Transformer: simplified MoE with K=1 (each token routed to single expert); reduced routing overhead and expert imbalance
• Expert capacity: each expert handles fixed batch tokens per forward pass; exceeding capacity requires auxiliary loss or dropping tokens
• Gating function: softmax(linear_projection(token_representation)) → sparse selection; alternative sparse gating functions exist

Load Balancing and Training:

• Expert load imbalance problem: some experts may receive disproportionate token assignments; underutilized capacity
• Auxiliary loss: added to training loss to encourage balanced expert utilization; loss_balance = cv²(router_probs) encouraging uniform distribution
• Token-to-expert assignment: learned mapping encourages specialization while maintaining balance; dynamic routing during training
• Dropout in routing: regularization to prevent collapse to single expert; improve generalization

Scaling and Efficiency:

• Parameter efficiency: Mixtral (46.7B total, 12.9B active) matches or exceeds dense 70B models with significantly reduced compute
• Compute efficiency: active parameter count determines FLOPs; sparse routing enables efficient scaling to trillion-parameter models
• Communication overhead: MoE requires all-to-all communication in distributed training for expert specialization
• Memory requirements: expert parameters stored across devices; token routing induces load imbalance affecting device utilization

Mixtral and Architectural Variants:

• Mixtral-8x7B: 8 experts, 2 selected per token; mixture of smaller specialists more interpretable than single large network
• Expert specialization: different experts learn distinct knowledge domains (language-specific, task-specific, linguistic feature-specific)
• Compared to dense models: MoE provides parameter scaling without proportional compute increase; useful for resource-constrained deployments

Mixture-of-Experts models leverage sparse routing to activate only necessary experts per token — enabling efficient scaling to massive parameter counts while maintaining computational efficiency superior to equivalent dense models.