Mixture of Experts (MoE) is the model architecture that uses a gating network to dynamically route each input to a sparse subset of specialized "expert" sub-networks — enabling models with dramatically more total parameters (and thus more capacity) while keeping per-input computation constant, allowing models like Mixtral 8x7B and GPT-4 to achieve superior performance without proportionally increasing inference cost.

Core Architecture

• Experts: N parallel feed-forward networks (e.g., N=8 or N=64), each potentially specializing in different input types.
• Router/Gate: A network that assigns each token to the top-K experts (typically K=1 or K=2).
• Sparse Activation: Only K out of N experts process each input → computation scales with K, not N.

Routing (Gating)

$G(x) = TopK(Softmax(W_g \cdot x))$

• Linear layer projects input to N scores (one per expert).
• Softmax normalizes scores to probabilities.
• TopK selects the K highest-scoring experts.
• Output: Weighted sum of selected expert outputs, weighted by gate probabilities.

Parameter vs. Compute Scaling

Load Balancing Challenge

• Without intervention: Router sends most tokens to a few "popular" experts → others idle.
• Auxiliary load balancing loss: Penalty for uneven expert utilization.
• $L_{balance} = N \cdot \sum_{i=1}^N f_i \cdot p_i$ where f_i = fraction of tokens to expert i, p_i = average gate probability.
• Expert capacity: Token buffer per expert — overflow tokens dropped or re-routed.

Training Challenges

• Instability: Routing decisions are discrete → training can be unstable.
• Expert collapse: All experts converge to similar behavior → no specialization.
• Communication overhead: In distributed training, tokens must be sent to the GPU holding each expert (all-to-all communication).

Sparse vs. Dense Trade-offs

• Advantage: More parameters → more knowledge capacity at same inference cost.
• Disadvantage: Higher memory footprint (all experts in memory), communication overhead, less efficient on small batches.
• When to use MoE: Large-scale pretraining where parameter count matters more than memory efficiency.

Mixture of experts is the dominant scaling strategy for frontier language models — by decoupling parameter count from per-token computation, MoE enables models to store more knowledge and handle more diverse tasks while maintaining economically viable inference costs.