Mixture of Experts (MoE) is the neural architecture pattern that replaces dense feedforward layers with multiple specialized expert networks, activating only a sparse subset of experts per input token via learned routing — enabling models to scale to trillions of parameters while maintaining constant per-token compute cost, as demonstrated by Switch Transformer (1.6T parameters), GLaM (1.2T), and GPT-4's rumored MoE architecture that achieves GPT-3-level quality at 10-20× lower training cost.

MoE Architecture Components:

• Expert Networks: typically 8-256 identical feedforward networks (experts) replace each dense FFN layer; each expert has 2-8B parameters in large models; experts specialize during training to handle different input patterns, linguistic structures, or knowledge domains without explicit supervision
• Router/Gating Network: lightweight network (typically single linear layer + softmax) that computes expert selection scores for each token; top-k routing selects k experts (usually k=1 or k=2) with highest scores; router trained end-to-end with expert networks via gradient descent
• Load Balancing: auxiliary loss term encourages uniform expert utilization to prevent collapse where few experts dominate; typical formulation: L_aux = α × Σ(f_i × P_i) where f_i is fraction of tokens routed to expert i, P_i is router probability for expert i; α=0.01-0.1
• Expert Capacity: maximum tokens per expert per batch to enable efficient batched computation; capacity factor C (typically 1.0-1.25) determines buffer size; tokens exceeding capacity are either dropped (with residual connection) or routed to next-best expert

Routing Strategies and Variants:

• Top-1 Routing (Switch Transformer): each token routed to single expert with highest score; maximizes sparsity (1/N experts active per token for N experts); simplest implementation but sensitive to load imbalance; achieves 7× speedup vs dense model at same quality
• Top-2 Routing (GShard, GLaM): each token routed to 2 experts; improves training stability and model quality at 2× compute cost vs top-1; weighted combination of expert outputs using normalized router scores; reduces sensitivity to router errors
• Expert Choice Routing: experts select top-k tokens rather than tokens selecting experts; guarantees perfect load balance; used in Google's V-MoE (Vision MoE) and recent language models; eliminates need for auxiliary load balancing loss
• Soft MoE: all experts process all tokens but with weighted combinations; eliminates discrete routing decisions; higher compute cost but improved gradient flow; used in some vision transformers where token count is manageable

Scaling and Efficiency:

• Parameter Scaling: MoE enables 10-100× parameter increase vs dense models at same compute budget; Switch Transformer: 1.6T parameters with 2048 experts, each token sees ~1B parameters (equivalent to dense 1B model compute)
• Training Efficiency: GLaM (1.2T parameters, 64 experts) matches GPT-3 (175B dense) quality using 1/3 training FLOPs and 1/2 energy; Switch Transformer achieves 4× pre-training speedup vs T5-XXL at same quality
• Inference Efficiency: sparse activation reduces inference cost proportionally to sparsity; top-1 routing with 64 experts uses 1/64 of parameters per token; critical for serving trillion-parameter models within latency budgets
• Communication Overhead: in distributed training, expert parallelism requires all-to-all communication to route tokens to expert-assigned devices; becomes bottleneck at high expert counts; hierarchical MoE and expert replication mitigate this

Implementation and Deployment Challenges:

• Load Imbalance: without careful tuning, few experts handle most tokens while others remain idle; auxiliary loss, expert capacity limits, and expert choice routing address this; monitoring per-expert utilization critical during training
• Training Instability: router can collapse early in training, routing all tokens to few experts; higher learning rates for router, router z-loss (penalizes large logits), and expert dropout improve stability
• Memory Requirements: storing N experts requires N× memory vs dense model; expert parallelism distributes experts across devices; at extreme scale (2048 experts), each device holds subset of experts
• Fine-tuning Challenges: MoE models can be difficult to fine-tune on downstream tasks; expert specialization may not transfer; techniques include freezing router, fine-tuning subset of experts, or adding task-specific experts

Mixture of Experts is the breakthrough architecture that decouples model capacity from computation cost — enabling the trillion-parameter models that define the current frontier of AI capabilities while remaining trainable and deployable within practical compute and memory budgets, fundamentally changing the economics of scaling language models.

AI Accelerator Architecture — Compute, Memory, and Interconnect. Modern AI chips are purpose-built for matrix multiplication: a systolic array or tensor core computes thousands of multiply-accumulate (MAC) operations per cycle, fed by a memory hierarchy (registers → SRAM → HBM) connected through a network-on-chip (NoC) that determines whether the compute units starve or stay busy. The single metric that captures this interaction is the roofline model: peak performance (TFLOPS) vs memory bandwidth (TB/s), where the arithmetic intensity of the workload (FLOPs/byte) determines which resource limits throughput.

Tensor Cores — The Matrix Multiply Unit. NVIDIA tensor cores perform 4$\times$4 matrix multiply-accumulate (D = A$\times$B + C) in a single clock cycle at mixed precision (FP16 inputs, FP32 accumulate). The H100 has 528 tensor cores across 132 SMs, delivering 989 TFLOPS at FP16 or 1,979 TFLOPS at FP8 — a 3$\times$ generational improvement over A100 (312 TFLOPS FP16). Programming tensor cores requires structuring data in tile-friendly layouts (16$\times$16 or 32$\times$8 fragments) via CUDA WMMA or MMA PTX instructions. Utilization typically reaches 60–80% in production training (compute-bound GEMM) but drops to 10–30% during inference decode (memory-bound, limited by KV cache reads). AMD CDNA3 Matrix Cores and Google TPU v5 MXUs provide equivalent functionality at comparable TFLOPS/W.

KV Cache and Inference Efficiency. During autoregressive LLM inference, each generated token requires reading the full key-value cache of all prior tokens — creating a memory-bandwidth bottleneck where arithmetic intensity drops to 1–5 FLOPs/byte (far left of the roofline). A 70B-parameter model at sequence length 4096 stores 40 GB of KV cache in HBM; generating each token reads 40 GB at 3.35 TB/s = 12 ms latency per token — regardless of compute capacity. Solutions: PagedAttention (vLLM) eliminates KV cache fragmentation; multi-query attention (MQA/GQA) reduces KV size by 8$\times$; speculative decoding verifies 4–8 draft tokens per forward pass, increasing effective throughput 2–4$\times$; continuous batching (Orca) amortizes KV reads across multiple sequences in flight.

Network-on-Chip (NoC) for AI Accelerators. The NoC connects hundreds of compute tiles (tensor cores, memory controllers, I/O ports) through a mesh, ring, or hierarchical topology — and its bisection bandwidth determines the maximum data rate for all-reduce operations during distributed training. An H100 has a 12$\times$11 crossbar connecting 132 SMs, 6 HBM3 stacks, and 18 NVLink ports. The total internal bandwidth exceeds 30 TB/s. For multi-chip training, NVLink 4.0 provides 900 GB/s chip-to-chip (18 links $\times$ 50 GB/s each) while PCIe 5.0 adds 128 GB/s for host communication. The NoC design determines whether the GPU can keep all tensor cores fed during a 2048-GPU training run where each iteration requires an all-reduce of 1–10 GB of gradients across the fabric.

Mixture of Experts (MoE) — Hardware Implications. MoE models (GPT-4, Mixtral, Switch Transformer) activate only 2–8 experts per token out of 64–256 total, reducing compute by 10–30$\times$ relative to a dense model of equivalent capacity — but at the cost of massive memory footprint (every expert's weights must reside in HBM) and irregular memory access patterns that stress the NoC and memory controller. A Mixtral 8$\times$7B model has 46.7B total parameters but only 12.9B active per token; the challenge is that expert routing is data-dependent and unpredictable, causing load imbalance across GPU SMs and across nodes in distributed inference. Hardware solutions include expert parallelism (each GPU holds a subset of experts), capacity factors limiting expert overload, and all-to-all communication patterns that require high bisection bandwidth.