Mixture of Experts (MoE) is the sparse-activation architecture that scales a neural network to trillions of parameters while keeping per-token compute fixed — each input activates only a small subset of "expert" sub-networks selected by a learned router, so total model capacity grows without proportional growth in inference FLOPs. GPT-4, Mixtral 8×7B, Switch Transformer, DeepSeek-V2, and Grok all use MoE layers to achieve frontier accuracy at a fraction of the cost of an equivalently-sized dense model.

The core idea — conditional computation. In a dense Transformer, every token passes through every FFN parameter. In an MoE Transformer, the standard FFN block is replaced by $N$ parallel expert FFNs plus a lightweight gating (router) network. For each token, the router selects the top-$k$ experts (typically $k = 1$ or $k = 2$), and only those experts run. If $N = 64$ and $k = 2$, the model has 64× the parameters of one expert but only 2× the compute per token — a ~32× parameter-to-FLOP leverage ratio.

Router design. The router $G(x)$ maps a token embedding $x \in \mathbb{R}^d$ to a probability distribution over experts:

where $W_g \in \mathbb{R}^{N \times d}$ is a learned matrix and $\epsilon$ is optional noise for exploration during training. The top-$k$ entries of $G(x)$ select which experts fire; the corresponding softmax weights become the mixture coefficients for combining expert outputs:

Load balancing — the critical auxiliary loss. Without intervention, training collapses: a few popular experts attract most tokens, receive the strongest gradients, and become even more popular (expert collapse). The fix is an auxiliary loss that penalizes uneven load:

where $f_i$ is the fraction of tokens actually routed to expert $i$ and $p_i$ is the mean router probability assigned to expert $i$ across the batch. Minimizing $\mathcal{L}_{\text{aux}}$ pushes the router toward uniform dispatch. Typical $\alpha$: 0.01–0.1.

Capacity factor and token dropping. Each expert can process at most $C = \text{capacity\_factor} \times T/N$ tokens per batch (where $T$ = total tokens). Tokens that overflow are either dropped (Switch Transformer, capacity factor ≈ 1.25) or re-routed to a shared fallback expert. DeepSeek-V2 eliminates dropping entirely with a "shared expert" that all tokens pass through, plus routed experts for specialization.

Training — expert parallelism. MoE layers require a collective all-to-all communication: tokens are gathered at the GPU hosting their assigned expert, processed, then scattered back. This is the defining bottleneck of MoE training at scale. A typical layout: data-parallel across most of the model, expert-parallel across the MoE FFN. With $P$ GPUs and $N$ experts, each GPU holds $N/P$ experts and receives tokens routed to them from all other GPUs.

Inference — why MoE is hard on hardware. Although only top-$k$ experts compute per token, all $N$ experts must reside in memory (HBM) because the router's selections are input-dependent and change every token. This means:

• Memory scales with total parameters (not active parameters). A 1.8T-parameter MoE at fp16 needs ~3.6 TB of HBM — requiring multi-node inference.
• Compute scales with active parameters ($k$ experts × expert size). The arithmetic intensity is low (small matrix per expert), making MoE decode memory-bandwidth-bound even more severely than dense models.
• Expert offloading (expert-to-CPU/SSD): exploits the sparsity by keeping only hot experts in HBM and paging cold ones on demand — but latency spikes when a token routes to a cold expert.

Chip-design implications. An MoE-optimized accelerator needs: (1) massive HBM capacity to hold all experts (HBM3E 6-stack or 8-stack configurations), (2) very high memory bandwidth (the decode bottleneck), (3) fast all-to-all interconnect between chips for expert parallelism (NVLink, UALink, or custom mesh), and (4) a small low-latency router engine that can select experts before launching the main compute — a pattern the CFS Inference Simulator models at /infer.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Mixture of Experts — Sparse Conditional Compute</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">only activate 2 of N experts per token — get large model capacity at small per-token compute cost</text>

  <!-- === MAIN: MoE layer data flow === -->
  <text x="380" y="74" fill="#e6edf3" font-size="12" font-weight="600" text-anchor="middle">MoE FFN Layer (replaces dense FFN in transformer block)</text>

  <!-- Input token -->
  <rect x="30" y="115" width="65" height="35" rx="4" fill="#0f2a1f" stroke="#34d399" stroke-width="1.2"/>
  <text x="62" y="130" fill="#6ee7b7" font-size="9" text-anchor="middle">token</text>
  <text x="62" y="143" fill="#6b7684" font-size="7.5" text-anchor="middle">hidden (d)</text>

  <!-- Arrow to router -->
  <path d="M97,132 L125,132" fill="none" stroke="#34d399" stroke-width="1.2"/>
  <polygon points="123,129 129,132 123,135" fill="#34d399"/>

  <!-- Router / Gating network -->
  <rect x="130" y="105" width="90" height="55" rx="5" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.4"/>
  <text x="175" y="122" fill="#fbbf24" font-size="10" text-anchor="middle" font-weight="600">Router</text>
  <text x="175" y="136" fill="#8b98a5" font-size="8" text-anchor="middle">W_g × x → scores</text>
  <text x="175" y="150" fill="#8b98a5" font-size="7.5" text-anchor="middle">softmax → top-k</text>

  <!-- Router output: gate weights -->
  <text x="175" y="172" fill="#fbbf24" font-size="8" text-anchor="middle">g₁=0.7, g₅=0.3</text>

  <!-- Arrows from router to selected experts -->
  <path d="M222,120 L268,100" fill="none" stroke="#f59e0b" stroke-width="1.2"/>
  <path d="M222,145 L268,206" fill="none" stroke="#f59e0b" stroke-width="1.2"/>
  <!-- Dashed arrows to non-selected (inactive) -->
  <path d="M222,125 L268,130" fill="none" stroke="#6b7684" stroke-width="0.5" stroke-dasharray="2,2" opacity="0.4"/>
  <path d="M222,130 L268,157" fill="none" stroke="#6b7684" stroke-width="0.5" stroke-dasharray="2,2" opacity="0.4"/>
  <path d="M222,140 L268,184" fill="none" stroke="#6b7684" stroke-width="0.5" stroke-dasharray="2,2" opacity="0.4"/>
  <path d="M222,148 L268,233" fill="none" stroke="#6b7684" stroke-width="0.5" stroke-dasharray="2,2" opacity="0.4"/>

  <!-- Expert bank (8 experts) -->
  <!-- Expert 0 (ACTIVE) -->
  <rect x="270" y="86" width="75" height="30" rx="4" fill="#1c1633" stroke="#a78bfa" stroke-width="1.4"/>
  <text x="307" y="105" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="600">Expert 0</text>
  <!-- Expert 1 (inactive) -->
  <rect x="270" y="120" width="75" height="26" rx="4" fill="#14202c" stroke="#3a4453" stroke-width="0.7" opacity="0.5"/>
  <text x="307" y="137" fill="#6b7684" font-size="8" text-anchor="middle">Expert 1</text>
  <!-- Expert 2 (inactive) -->
  <rect x="270" y="150" width="75" height="26" rx="4" fill="#14202c" stroke="#3a4453" stroke-width="0.7" opacity="0.5"/>
  <text x="307" y="167" fill="#6b7684" font-size="8" text-anchor="middle">Expert 2</text>
  <!-- Expert 3 (inactive) -->
  <rect x="270" y="180" width="75" height="26" rx="4" fill="#14202c" stroke="#3a4453" stroke-width="0.7" opacity="0.5"/>
  <text x="307" y="197" fill="#6b7684" font-size="8" text-anchor="middle">Expert 3</text>
  <!-- Expert 4 (ACTIVE) -->
  <rect x="270" y="210" width="75" height="30" rx="4" fill="#1c1633" stroke="#a78bfa" stroke-width="1.4"/>
  <text x="307" y="229" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="600">Expert 4</text>
  <!-- Expert 5-7 (inactive, faded) -->
  <rect x="270" y="244" width="75" height="22" rx="3" fill="#14202c" stroke="#3a4453" stroke-width="0.5" opacity="0.3"/>
  <text x="307" y="258" fill="#6b7684" font-size="7" text-anchor="middle" opacity="0.5">... Expert 5-7</text>

  <!-- Active indicator -->
  <circle cx="262" cy="101" r="4" fill="#34d399"/>
  <circle cx="262" cy="225" r="4" fill="#34d399"/>
  <text x="355" y="101" fill="#34d399" font-size="8">active (top-2)</text>
  <text x="355" y="225" fill="#34d399" font-size="8">active</text>

  <!-- Expert outputs combined -->
  <path d="M347,101 L388,155" fill="none" stroke="#a78bfa" stroke-width="1"/>
  <path d="M347,225 L388,165" fill="none" stroke="#a78bfa" stroke-width="1"/>

  <!-- Weighted sum -->
  <rect x="390" y="140" width="75" height="40" rx="4" fill="#1a1520" stroke="#c4b5fd" stroke-width="1.2"/>
  <text x="427" y="157" fill="#c4b5fd" font-size="8.5" text-anchor="middle">Weighted Sum</text>
  <text x="427" y="170" fill="#8b98a5" font-size="7.5" text-anchor="middle">g₁·E₀ + g₅·E₄</text>

  <!-- Output -->
  <path d="M467,160 L495,160" fill="none" stroke="#e6edf3" stroke-width="1.2"/>
  <polygon points="493,157 499,160 493,163" fill="#e6edf3"/>
  <rect x="500" y="145" width="60" height="30" rx="4" fill="#14202c" stroke="#e6edf3" stroke-width="1"/>
  <text x="530" y="164" fill="#e6edf3" font-size="9" text-anchor="middle">output</text>

  <!-- === RIGHT: Why MoE works === -->
  <rect x="580" y="82" width="150" height="185" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="655" y="100" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Why MoE</text>

  <text x="595" y="120" fill="#34d399" font-size="8.5">Total params: huge</text>
  <text x="595" y="134" fill="#34d399" font-size="8.5">Active params: small</text>
  <text x="595" y="150" fill="#8b98a5" font-size="8">DeepSeek-V3:</text>
  <text x="595" y="163" fill="#8b98a5" font-size="8">671B total, 37B active</text>
  <text x="595" y="180" fill="#8b98a5" font-size="8">Mixtral 8×7B:</text>
  <text x="595" y="193" fill="#8b98a5" font-size="8">47B total, 13B active</text>
  <text x="595" y="213" fill="#6b7684" font-size="8">= dense-model quality</text>
  <text x="595" y="226" fill="#6b7684" font-size="8">at 3-5x less compute</text>
  <text x="595" y="246" fill="#fbbf24" font-size="8">GPT-4: rumored 8×220B</text>
  <text x="595" y="259" fill="#6b7684" font-size="7.5">(~1.8T total, 220B active)</text>

  <!-- === BOTTOM: Challenges + production === -->
  <rect x="30" y="282" width="340" height="105" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="300" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Engineering Challenges</text>

  <text x="50" y="320" fill="#f87171" font-size="9">Load balancing: some experts get all tokens</text>
  <text x="50" y="336" fill="#8b98a5" font-size="8.5">Fix: auxiliary loss, expert capacity, z-loss</text>
  <text x="50" y="354" fill="#f87171" font-size="9">Communication: experts on different GPUs</text>
  <text x="50" y="370" fill="#8b98a5" font-size="8.5">Fix: expert parallelism (all-to-all dispatch)</text>
  <text x="50" y="386" fill="#6b7684" font-size="8">Memory: all experts in VRAM even if unused</text>

  <!-- Production MoE models -->
  <rect x="385" y="282" width="345" height="105" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="557" y="300" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Production MoE Models (2024-25)</text>

  <text x="405" y="320" fill="#fbbf24" font-size="9" font-weight="600">DeepSeek-V3: 256 experts, top-8, 671B</text>
  <text x="405" y="336" fill="#c4b5fd" font-size="9">Mixtral 8×7B: 8 experts, top-2, 47B</text>
  <text x="405" y="352" fill="#60a5fa" font-size="9">Grok-1: 8 experts, 314B (xAI)</text>
  <text x="405" y="368" fill="#34d399" font-size="9">Qwen-MoE: 60 experts, top-4</text>
  <text x="405" y="384" fill="#6b7684" font-size="8.5">DBRX: 16 experts, top-4 (Databricks)</text>

  <!-- Bottom insight -->
  <text x="380" y="410" fill="#e6edf3" font-size="9.5" text-anchor="middle" font-weight="600">Each expert learns to specialize: code expert, math expert, language expert — the router learns who to call</text>
  <text x="380" y="428" fill="#8b98a5" font-size="9" text-anchor="middle">Inference: same latency as dense 37B, but quality of dense 200B+ — the free lunch of sparse models</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">MoE = hire many specialists, but only consult two per question. Huge knowledge, small per-query cost.</text>
</svg>

The MoE scaling law. Empirically, an MoE model with $N$ experts and active parameters $A$ performs roughly like a dense model of size $A \cdot N^{0.3}$ in terms of loss — better than $A$ alone, but not as good as a dense model of size $A \cdot N$. The exponent varies (0.2–0.4) depending on routing quality and expert granularity. This makes MoE the dominant architecture for cost-efficient frontier models: you get 80% of the benefit of a model 5–10× larger at only the inference cost of the active slice.

Fine-grained vs coarse-grained experts. Early MoE (Switch, Mixtral) used 8–128 experts each the size of a full FFN. DeepSeek-V2 and later designs shrink expert size dramatically (e.g. 256 experts, each 1/16 the FFN width) so more experts can be selected per token ($k = 6$–8) without increasing total compute — this gives smoother routing, less load imbalance, and better generalization because each token assembles a more nuanced combination.

What MoE changes for the hardware stack. The shift from dense to MoE fundamentally re-weights the hardware bottleneck hierarchy: memory capacity and bandwidth matter more than peak FLOPS, inter-chip interconnect bandwidth becomes the training limiter (all-to-all), and the router decision latency is on the critical path for every single token. This is why the CFS platform models MoE workloads across the HBM (/hbm), KV-cache (/kvcache), and inference (/infer) simulators — each captures a different facet of the MoE serving challenge.