Mixture of Experts (MoE) is the sparse-activation architecture that scales a neural network to trillions of parameters while keeping per-token compute fixed — each input activates only a small subset of "expert" sub-networks selected by a learned router, so total model capacity grows without proportional growth in inference FLOPs. GPT-4, Mixtral 8×7B, Switch Transformer, DeepSeek-V2, and Grok all use MoE layers to achieve frontier accuracy at a fraction of the cost of an equivalently-sized dense model.

The core idea — conditional computation. In a dense Transformer, every token passes through every FFN parameter. In an MoE Transformer, the standard FFN block is replaced by $N$ parallel expert FFNs plus a lightweight gating (router) network. For each token, the router selects the top-$k$ experts (typically $k = 1$ or $k = 2$), and only those experts run. If $N = 64$ and $k = 2$, the model has 64× the parameters of one expert but only 2× the compute per token — a ~32× parameter-to-FLOP leverage ratio.

Router design. The router $G(x)$ maps a token embedding $x \in \mathbb{R}^d$ to a probability distribution over experts:

where $W_g \in \mathbb{R}^{N \times d}$ is a learned matrix and $\epsilon$ is optional noise for exploration during training. The top-$k$ entries of $G(x)$ select which experts fire; the corresponding softmax weights become the mixture coefficients for combining expert outputs:

Load balancing — the critical auxiliary loss. Without intervention, training collapses: a few popular experts attract most tokens, receive the strongest gradients, and become even more popular (expert collapse). The fix is an auxiliary loss that penalizes uneven load:

where $f_i$ is the fraction of tokens actually routed to expert $i$ and $p_i$ is the mean router probability assigned to expert $i$ across the batch. Minimizing $\mathcal{L}_{\text{aux}}$ pushes the router toward uniform dispatch. Typical $\alpha$: 0.01–0.1.

Capacity factor and token dropping. Each expert can process at most $C = \text{capacity\_factor} \times T/N$ tokens per batch (where $T$ = total tokens). Tokens that overflow are either dropped (Switch Transformer, capacity factor ≈ 1.25) or re-routed to a shared fallback expert. DeepSeek-V2 eliminates dropping entirely with a "shared expert" that all tokens pass through, plus routed experts for specialization.

Training — expert parallelism. MoE layers require a collective all-to-all communication: tokens are gathered at the GPU hosting their assigned expert, processed, then scattered back. This is the defining bottleneck of MoE training at scale. A typical layout: data-parallel across most of the model, expert-parallel across the MoE FFN. With $P$ GPUs and $N$ experts, each GPU holds $N/P$ experts and receives tokens routed to them from all other GPUs.

Inference — why MoE is hard on hardware. Although only top-$k$ experts compute per token, all $N$ experts must reside in memory (HBM) because the router's selections are input-dependent and change every token. This means:

• Memory scales with total parameters (not active parameters). A 1.8T-parameter MoE at fp16 needs ~3.6 TB of HBM — requiring multi-node inference.
• Compute scales with active parameters ($k$ experts × expert size). The arithmetic intensity is low (small matrix per expert), making MoE decode memory-bandwidth-bound even more severely than dense models.
• Expert offloading (expert-to-CPU/SSD): exploits the sparsity by keeping only hot experts in HBM and paging cold ones on demand — but latency spikes when a token routes to a cold expert.

Chip-design implications. An MoE-optimized accelerator needs: (1) massive HBM capacity to hold all experts (HBM3E 6-stack or 8-stack configurations), (2) very high memory bandwidth (the decode bottleneck), (3) fast all-to-all interconnect between chips for expert parallelism (NVLink, UALink, or custom mesh), and (4) a small low-latency router engine that can select experts before launching the main compute — a pattern the CFS Inference Simulator models at /infer.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Mixture of Experts — Sparse Conditional Compute</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">route each token to k-of-N experts: total parameters scale without per-token FLOPs scaling</text>

  <!-- === TOP: MoE layer architecture === -->
  <rect x="25" y="62" width="710" height="155" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="380" y="82" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Inside One MoE Layer (replaces dense FFN)</text>

  <!-- Input hidden state -->
  <rect x="40" y="118" width="70" height="30" rx="4" fill="#14261f" stroke="#34d399" stroke-width="0.9"/>
  <text x="75" y="136" fill="#6ee7b7" font-size="8.5" text-anchor="middle">hidden h</text>

  <!-- Arrow to router -->
  <path d="M113,133 L145,133" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="143,130 149,133 143,136" fill="#8b98a5"/>

  <!-- Router / Gate -->
  <rect x="152" y="108" width="95" height="50" rx="5" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.1"/>
  <text x="199" y="126" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="600">Router (Gate)</text>
  <text x="199" y="140" fill="#a1701a" font-size="8" text-anchor="middle">W_g · h → logits</text>
  <text x="199" y="152" fill="#6b7684" font-size="7.5" text-anchor="middle">softmax → top-k</text>

  <!-- Arrows from router to experts -->
  <path d="M250,118 L305,96" fill="none" stroke="#f59e0b" stroke-width="0.8"/>
  <path d="M250,128 L305,126" fill="none" stroke="#f59e0b" stroke-width="0.8"/>
  <path d="M250,148 L305,158" fill="none" stroke="#8b98a5" stroke-width="0.5" stroke-dasharray="2,2"/>
  <path d="M250,153 L305,180" fill="none" stroke="#8b98a5" stroke-width="0.5" stroke-dasharray="2,2"/>

  <!-- Experts (N total, k active) -->
  <rect x="308" y="86" width="100" height="26" rx="3" fill="#14261f" stroke="#34d399" stroke-width="0.9"/>
  <text x="358" y="103" fill="#6ee7b7" font-size="8.5" text-anchor="middle">Expert 1 (FFN) ✓</text>

  <rect x="308" y="116" width="100" height="26" rx="3" fill="#14261f" stroke="#34d399" stroke-width="0.9"/>
  <text x="358" y="133" fill="#6ee7b7" font-size="8.5" text-anchor="middle">Expert 2 (FFN) ✓</text>

  <rect x="308" y="148" width="100" height="26" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.7"/>
  <text x="358" y="165" fill="#6b7684" font-size="8.5" text-anchor="middle">Expert 3 (idle)</text>

  <rect x="308" y="178" width="100" height="26" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.7"/>
  <text x="358" y="195" fill="#6b7684" font-size="8.5" text-anchor="middle">Expert N (idle)</text>

  <text x="358" y="212" fill="#6b7684" font-size="7.5" text-anchor="middle">top-k=2 of N=8 active</text>

  <!-- Weighted sum -->
  <path d="M411,99 L445,118" fill="none" stroke="#34d399" stroke-width="0.8"/>
  <path d="M411,129 L445,125" fill="none" stroke="#34d399" stroke-width="0.8"/>

  <rect x="448" y="108" width="80" height="30" rx="4" fill="#1a1520" stroke="#a78bfa" stroke-width="0.9"/>
  <text x="488" y="122" fill="#c4b5fd" font-size="8" text-anchor="middle">Σ g_i · Expert_i(h)</text>
  <text x="488" y="134" fill="#6b7684" font-size="7" text-anchor="middle">weighted combination</text>

  <!-- Output -->
  <path d="M531,123 L560,123" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="558,120 564,123 558,126" fill="#8b98a5"/>
  <rect x="567" y="108" width="60" height="30" rx="4" fill="#14261f" stroke="#34d399" stroke-width="0.9"/>
  <text x="597" y="126" fill="#6ee7b7" font-size="8.5" text-anchor="middle">output</text>

  <!-- Load balance note -->
  <text x="560" y="170" fill="#f87171" font-size="8.5" text-anchor="middle">aux loss penalizes uneven routing</text>
  <text x="560" y="184" fill="#6b7684" font-size="8" text-anchor="middle">L_balance = α · N · Σ f_i · P_i</text>

  <!-- === MIDDLE LEFT: Parameter efficiency === -->
  <rect x="25" y="225" width="280" height="108" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="165" y="243" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Parameter Efficiency</text>

  <text x="45" y="264" fill="#8b98a5" font-size="9">Mixtral 8×7B:</text>
  <text x="45" y="280" fill="#60a5fa" font-size="8.5">  Total params: 47B (8 experts × 7B FFN each)</text>
  <text x="45" y="296" fill="#34d399" font-size="8.5">  Active params: ~13B per token (top-2)</text>
  <text x="45" y="312" fill="#fbbf24" font-size="8.5">  Perf ≈ Llama-2 70B at 1/5th the FLOPs</text>
  <text x="45" y="326" fill="#6b7684" font-size="8">  Trade: need all 47B in memory (VRAM cost)</text>

  <!-- === MIDDLE RIGHT: Communication in distributed MoE === -->
  <rect x="320" y="225" width="415" height="108" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="527" y="243" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Distributed MoE: Expert Parallelism</text>

  <text x="340" y="264" fill="#60a5fa" font-size="8.5">Each GPU holds a subset of experts</text>
  <text x="340" y="280" fill="#fbbf24" font-size="8.5">All-to-all shuffle: route tokens to correct GPU</text>
  <text x="340" y="296" fill="#f87171" font-size="8.5">Communication bottleneck: 2× all-to-all per layer</text>
  <text x="340" y="312" fill="#8b98a5" font-size="8.5">Capacity factor limits max tokens per expert</text>
  <text x="340" y="326" fill="#34d399" font-size="8.5">DeepSeek-V2: shared experts reduce all-to-all volume</text>

  <!-- === BOTTOM: Models and comparison === -->
  <rect x="25" y="343" width="710" height="68" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="361" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Production MoE Models</text>

  <text x="100" y="383" fill="#c4b5fd" font-size="9.5" text-anchor="middle" font-weight="600">Mixtral 8×7B</text>
  <text x="100" y="397" fill="#8b98a5" font-size="8" text-anchor="middle">top-2, 47B total, 13B active</text>

  <text x="260" y="383" fill="#fbbf24" font-size="9.5" text-anchor="middle" font-weight="600">GPT-4 (rumored)</text>
  <text x="260" y="397" fill="#8b98a5" font-size="8" text-anchor="middle">8×220B, 16 experts</text>

  <text x="420" y="383" fill="#34d399" font-size="9.5" text-anchor="middle" font-weight="600">DeepSeek-V3</text>
  <text x="420" y="397" fill="#8b98a5" font-size="8" text-anchor="middle">256 experts, top-8, 671B</text>

  <text x="580" y="383" fill="#60a5fa" font-size="9.5" text-anchor="middle" font-weight="600">Switch Transformer</text>
  <text x="580" y="397" fill="#8b98a5" font-size="8" text-anchor="middle">top-1, 1.6T params (Google)</text>

  <!-- Key insight -->
  <rect x="25" y="419" width="710" height="20" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="433" fill="#fbbf24" font-size="9" text-anchor="middle">MoE decouples model capacity (total params) from inference cost (active params) — scale knowledge without scaling FLOPs.</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">MoE is how frontier models stay smart without being slow: more parameters, same compute per token.</text>
</svg>

The MoE scaling law. Empirically, an MoE model with $N$ experts and active parameters $A$ performs roughly like a dense model of size $A \cdot N^{0.3}$ in terms of loss — better than $A$ alone, but not as good as a dense model of size $A \cdot N$. The exponent varies (0.2–0.4) depending on routing quality and expert granularity. This makes MoE the dominant architecture for cost-efficient frontier models: you get 80% of the benefit of a model 5–10× larger at only the inference cost of the active slice.

Fine-grained vs coarse-grained experts. Early MoE (Switch, Mixtral) used 8–128 experts each the size of a full FFN. DeepSeek-V2 and later designs shrink expert size dramatically (e.g. 256 experts, each 1/16 the FFN width) so more experts can be selected per token ($k = 6$–8) without increasing total compute — this gives smoother routing, less load imbalance, and better generalization because each token assembles a more nuanced combination.

What MoE changes for the hardware stack. The shift from dense to MoE fundamentally re-weights the hardware bottleneck hierarchy: memory capacity and bandwidth matter more than peak FLOPS, inter-chip interconnect bandwidth becomes the training limiter (all-to-all), and the router decision latency is on the critical path for every single token. This is why the CFS platform models MoE workloads across the HBM (/hbm), KV-cache (/kvcache), and inference (/infer) simulators — each captures a different facet of the MoE serving challenge.