Mixture of Experts (MoE) is the sparse-activation architecture that scales a neural network to trillions of parameters while keeping per-token compute fixed — each input activates only a small subset of "expert" sub-networks selected by a learned router, so total model capacity grows without proportional growth in inference FLOPs. GPT-4, Mixtral 8×7B, Switch Transformer, DeepSeek-V2, and Grok all use MoE layers to achieve frontier accuracy at a fraction of the cost of an equivalently-sized dense model.

The core idea — conditional computation. In a dense Transformer, every token passes through every FFN parameter. In an MoE Transformer, the standard FFN block is replaced by $N$ parallel expert FFNs plus a lightweight gating (router) network. For each token, the router selects the top-$k$ experts (typically $k = 1$ or $k = 2$), and only those experts run. If $N = 64$ and $k = 2$, the model has 64× the parameters of one expert but only 2× the compute per token — a ~32× parameter-to-FLOP leverage ratio.

Router design. The router $G(x)$ maps a token embedding $x \in \mathbb{R}^d$ to a probability distribution over experts:

$$G(x) = \text{softmax}(W_g \cdot x + \epsilon)$$

where $W_g \in \mathbb{R}^{N \times d}$ is a learned matrix and $\epsilon$ is optional noise for exploration during training. The top-$k$ entries of $G(x)$ select which experts fire; the corresponding softmax weights become the mixture coefficients for combining expert outputs:

$$y = \sum_{i \in \text{TopK}(G(x))} G(x)_i \cdot E_i(x)$$

Load balancing — the critical auxiliary loss. Without intervention, training collapses: a few popular experts attract most tokens, receive the strongest gradients, and become even more popular (expert collapse). The fix is an auxiliary loss that penalizes uneven load:

$$\mathcal{L}_{\text{aux}} = \alpha \cdot N \cdot \sum_{i=1}^{N} f_i \cdot p_i$$

where $f_i$ is the fraction of tokens actually routed to expert $i$ and $p_i$ is the mean router probability assigned to expert $i$ across the batch. Minimizing $\mathcal{L}_{\text{aux}}$ pushes the router toward uniform dispatch. Typical $\alpha$: 0.01–0.1.

Capacity factor and token dropping. Each expert can process at most $C = \text{capacity\_factor} \times T/N$ tokens per batch (where $T$ = total tokens). Tokens that overflow are either dropped (Switch Transformer, capacity factor ≈ 1.25) or re-routed to a shared fallback expert. DeepSeek-V2 eliminates dropping entirely with a "shared expert" that all tokens pass through, plus routed experts for specialization.

Training — expert parallelism. MoE layers require a collective all-to-all communication: tokens are gathered at the GPU hosting their assigned expert, processed, then scattered back. This is the defining bottleneck of MoE training at scale. A typical layout: data-parallel across most of the model, expert-parallel across the MoE FFN. With $P$ GPUs and $N$ experts, each GPU holds $N/P$ experts and receives tokens routed to them from all other GPUs.

Inference — why MoE is hard on hardware. Although only top-$k$ experts compute per token, all $N$ experts must reside in memory (HBM) because the router's selections are input-dependent and change every token. This means:

• Memory scales with total parameters (not active parameters). A 1.8T-parameter MoE at fp16 needs ~3.6 TB of HBM — requiring multi-node inference.
• Compute scales with active parameters ($k$ experts × expert size). The arithmetic intensity is low (small matrix per expert), making MoE decode memory-bandwidth-bound even more severely than dense models.
• Expert offloading (expert-to-CPU/SSD): exploits the sparsity by keeping only hot experts in HBM and paging cold ones on demand — but latency spikes when a token routes to a cold expert.

Chip-design implications. An MoE-optimized accelerator needs: (1) massive HBM capacity to hold all experts (HBM3E 6-stack or 8-stack configurations), (2) very high memory bandwidth (the decode bottleneck), (3) fast all-to-all interconnect between chips for expert parallelism (NVLink, UALink, or custom mesh), and (4) a small low-latency router engine that can select experts before launching the main compute — a pattern the CFS Inference Simulator models at /infer.

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 780 420" font-family="system-ui,sans-serif">
  <rect width="780" height="420" fill="#1a1a17" rx="12"/>
  <text x="390" y="30" fill="#c9c3f2" font-size="15" text-anchor="middle" font-weight="600">Mixture of Experts — token routing through a Transformer MoE layer</text>

  <!-- Input tokens -->
  <text x="60" y="70" fill="#8a8a86" font-size="12">Input tokens</text>
  <rect x="30" y="80" width="90" height="28" fill="#2a4a2a" stroke="#6fbf6f" stroke-width="1.2" rx="4"/>
  <text x="75" y="99" fill="#bff0bf" font-size="11" text-anchor="middle">token₁</text>
  <rect x="30" y="115" width="90" height="28" fill="#2a4a2a" stroke="#6fbf6f" stroke-width="1.2" rx="4"/>
  <text x="75" y="134" fill="#bff0bf" font-size="11" text-anchor="middle">token₂</text>
  <rect x="30" y="150" width="90" height="28" fill="#2a4a2a" stroke="#6fbf6f" stroke-width="1.2" rx="4"/>
  <text x="75" y="169" fill="#bff0bf" font-size="11" text-anchor="middle">token₃</text>
  <rect x="30" y="185" width="90" height="28" fill="#2a4a2a" stroke="#6fbf6f" stroke-width="1.2" rx="4"/>
  <text x="75" y="204" fill="#bff0bf" font-size="11" text-anchor="middle">token₄</text>

  <!-- Router -->
  <rect x="170" y="100" width="100" height="100" fill="#4a3a6a" stroke="#9a8adf" stroke-width="1.5" rx="8"/>
  <text x="220" y="140" fill="#d4c8ff" font-size="12" text-anchor="middle">Router</text>
  <text x="220" y="158" fill="#8a8a86" font-size="10" text-anchor="middle">G(x) = softmax(Wg·x)</text>
  <text x="220" y="175" fill="#8a8a86" font-size="10" text-anchor="middle">select top-k</text>

  <!-- arrows tokens → router -->
  <line x1="120" y1="94" x2="170" y2="130" stroke="#6fbf6f" stroke-width="1" marker-end="url(#arr)"/>
  <line x1="120" y1="129" x2="170" y2="140" stroke="#6fbf6f" stroke-width="1"/>
  <line x1="120" y1="164" x2="170" y2="155" stroke="#6fbf6f" stroke-width="1"/>
  <line x1="120" y1="199" x2="170" y2="170" stroke="#6fbf6f" stroke-width="1"/>

  <!-- Expert blocks -->
  <rect x="340" y="55" width="120" height="50" fill="#3a3a37" stroke="#e0913a" stroke-width="1.3" rx="5"/>
  <text x="400" y="78" fill="#e8d44d" font-size="11" text-anchor="middle">Expert 1 (FFN)</text>
  <text x="400" y="95" fill="#8a8a86" font-size="10" text-anchor="middle">W₁ · ReLU · W₂</text>

  <rect x="340" y="120" width="120" height="50" fill="#3a3a37" stroke="#e0913a" stroke-width="1.3" rx="5"/>
  <text x="400" y="143" fill="#e8d44d" font-size="11" text-anchor="middle">Expert 2 (FFN)</text>
  <text x="400" y="160" fill="#8a8a86" font-size="10" text-anchor="middle">W₁ · ReLU · W₂</text>

  <rect x="340" y="185" width="120" height="50" fill="#5a4a3a" stroke="#e0913a" stroke-width="2" rx="5"/>
  <text x="400" y="208" fill="#e8d44d" font-size="11" text-anchor="middle">Expert 3 (FFN)</text>
  <text x="400" y="225" fill="#e0913a" font-size="10" text-anchor="middle">★ selected</text>

  <text x="400" y="258" fill="#6f6f6a" font-size="16" text-anchor="middle">⋮</text>

  <rect x="340" y="270" width="120" height="50" fill="#5a4a3a" stroke="#e0913a" stroke-width="2" rx="5"/>
  <text x="400" y="293" fill="#e8d44d" font-size="11" text-anchor="middle">Expert N (FFN)</text>
  <text x="400" y="310" fill="#e0913a" font-size="10" text-anchor="middle">★ selected</text>

  <!-- routing arrows from router to experts -->
  <line x1="270" y1="125" x2="340" y2="210" stroke="#9a8adf" stroke-width="1.5" stroke-dasharray="4,2"/>
  <line x1="270" y1="160" x2="340" y2="295" stroke="#9a8adf" stroke-width="1.5" stroke-dasharray="4,2"/>
  <text x="300" y="115" fill="#9a8adf" font-size="10">route</text>

  <!-- Weighted sum -->
  <circle cx="530" cy="240" r="28" fill="#2a2a28" stroke="#bff0e4" stroke-width="1.5"/>
  <text x="530" y="236" fill="#bff0e4" font-size="11" text-anchor="middle">Σ</text>
  <text x="530" y="252" fill="#8a8a86" font-size="9" text-anchor="middle">weighted</text>

  <!-- expert outputs → sum -->
  <line x1="460" y1="210" x2="502" y2="235" stroke="#e0913a" stroke-width="1.2"/>
  <line x1="460" y1="295" x2="502" y2="248" stroke="#e0913a" stroke-width="1.2"/>

  <!-- Output -->
  <rect x="590" y="222" width="130" height="36" fill="#2a4a4a" stroke="#6fafaf" stroke-width="1.3" rx="5"/>
  <text x="655" y="245" fill="#bff0e4" font-size="11" text-anchor="middle">MoE layer output</text>
  <line x1="558" y1="240" x2="590" y2="240" stroke="#6fafaf" stroke-width="1.2"/>

  <!-- Bottom annotation -->
  <rect x="30" y="340" width="720" height="65" fill="#1f1f1c" stroke="#3a3a37" stroke-width="1" rx="6"/>
  <text x="50" y="362" fill="#d4d4d0" font-size="12">Key insight: N experts in memory, only top-k compute per token</text>
  <text x="50" y="382" fill="#8a8a86" font-size="11">• Parameters scale ×N (capacity) but FLOPs scale ×k (active) → ~N/k leverage ratio</text>
  <text x="50" y="399" fill="#8a8a86" font-size="11">• All experts must be in HBM (input-dependent routing) → memory-bound inference, demands high BW</text>

  <!-- arrow marker -->
  <defs><marker id="arr" markerWidth="6" markerHeight="6" refX="5" refY="3" orient="auto"><path d="M0,0 L6,3 L0,6" fill="none" stroke="#6fbf6f" stroke-width="1"/></marker></defs>
</svg>

The MoE scaling law. Empirically, an MoE model with $N$ experts and active parameters $A$ performs roughly like a dense model of size $A \cdot N^{0.3}$ in terms of loss — better than $A$ alone, but not as good as a dense model of size $A \cdot N$. The exponent varies (0.2–0.4) depending on routing quality and expert granularity. This makes MoE the dominant architecture for cost-efficient frontier models: you get 80% of the benefit of a model 5–10× larger at only the inference cost of the active slice.

Fine-grained vs coarse-grained experts. Early MoE (Switch, Mixtral) used 8–128 experts each the size of a full FFN. DeepSeek-V2 and later designs shrink expert size dramatically (e.g. 256 experts, each 1/16 the FFN width) so more experts can be selected per token ($k = 6$–8) without increasing total compute — this gives smoother routing, less load imbalance, and better generalization because each token assembles a more nuanced combination.

What MoE changes for the hardware stack. The shift from dense to MoE fundamentally re-weights the hardware bottleneck hierarchy: memory capacity and bandwidth matter more than peak FLOPS, inter-chip interconnect bandwidth becomes the training limiter (all-to-all), and the router decision latency is on the critical path for every single token. This is why the CFS platform models MoE workloads across the HBM (/hbm), KV-cache (/kvcache), and inference (/infer) simulators — each captures a different facet of the MoE serving challenge.