Mixture of Experts (MoE) is the sparse-activation architecture that scales a neural network to trillions of parameters while keeping per-token compute fixed — each input activates only a small subset of "expert" sub-networks selected by a learned router, so total model capacity grows without proportional growth in inference FLOPs. GPT-4, Mixtral 8×7B, Switch Transformer, DeepSeek-V2, and Grok all use MoE layers to achieve frontier accuracy at a fraction of the cost of an equivalently-sized dense model.

The core idea — conditional computation. In a dense Transformer, every token passes through every FFN parameter. In an MoE Transformer, the standard FFN block is replaced by $N$ parallel expert FFNs plus a lightweight gating (router) network. For each token, the router selects the top-$k$ experts (typically $k = 1$ or $k = 2$), and only those experts run. If $N = 64$ and $k = 2$, the model has 64× the parameters of one expert but only 2× the compute per token — a ~32× parameter-to-FLOP leverage ratio.

Router design. The router $G(x)$ maps a token embedding $x \in \mathbb{R}^d$ to a probability distribution over experts:

where $W_g \in \mathbb{R}^{N \times d}$ is a learned matrix and $\epsilon$ is optional noise for exploration during training. The top-$k$ entries of $G(x)$ select which experts fire; the corresponding softmax weights become the mixture coefficients for combining expert outputs:

Load balancing — the critical auxiliary loss. Without intervention, training collapses: a few popular experts attract most tokens, receive the strongest gradients, and become even more popular (expert collapse). The fix is an auxiliary loss that penalizes uneven load:

where $f_i$ is the fraction of tokens actually routed to expert $i$ and $p_i$ is the mean router probability assigned to expert $i$ across the batch. Minimizing $\mathcal{L}_{\text{aux}}$ pushes the router toward uniform dispatch. Typical $\alpha$: 0.01–0.1.

Capacity factor and token dropping. Each expert can process at most $C = \text{capacity\_factor} \times T/N$ tokens per batch (where $T$ = total tokens). Tokens that overflow are either dropped (Switch Transformer, capacity factor ≈ 1.25) or re-routed to a shared fallback expert. DeepSeek-V2 eliminates dropping entirely with a "shared expert" that all tokens pass through, plus routed experts for specialization.

Training — expert parallelism. MoE layers require a collective all-to-all communication: tokens are gathered at the GPU hosting their assigned expert, processed, then scattered back. This is the defining bottleneck of MoE training at scale. A typical layout: data-parallel across most of the model, expert-parallel across the MoE FFN. With $P$ GPUs and $N$ experts, each GPU holds $N/P$ experts and receives tokens routed to them from all other GPUs.

Inference — why MoE is hard on hardware. Although only top-$k$ experts compute per token, all $N$ experts must reside in memory (HBM) because the router's selections are input-dependent and change every token. This means:

• Memory scales with total parameters (not active parameters). A 1.8T-parameter MoE at fp16 needs ~3.6 TB of HBM — requiring multi-node inference.
• Compute scales with active parameters ($k$ experts × expert size). The arithmetic intensity is low (small matrix per expert), making MoE decode memory-bandwidth-bound even more severely than dense models.
• Expert offloading (expert-to-CPU/SSD): exploits the sparsity by keeping only hot experts in HBM and paging cold ones on demand — but latency spikes when a token routes to a cold expert.

Chip-design implications. An MoE-optimized accelerator needs: (1) massive HBM capacity to hold all experts (HBM3E 6-stack or 8-stack configurations), (2) very high memory bandwidth (the decode bottleneck), (3) fast all-to-all interconnect between chips for expert parallelism (NVLink, UALink, or custom mesh), and (4) a small low-latency router engine that can select experts before launching the main compute — a pattern the CFS Inference Simulator models at /infer.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Expert Parallelism Moe Technical Microarchitecture</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">Detailed Domain Pipeline, Architectural Blocks &amp; Engineering Performance Optimization (ID 13675)</text>
  
  
    <!-- NEURAL NETWORK FLOW (3 Grid Panels) -->
    <g transform="translate(25, 75)">
      <rect width="220" height="325" fill="#161b22" stroke="#30363d" stroke-width="1.5" rx="8"/>
      <text x="110" y="25" fill="#60a5fa" font-size="12" font-weight="700" text-anchor="middle">1. Input &amp; Embeddings</text>
      <rect x="15" y="45" width="190" height="80" fill="#0d1117" stroke="#30363d" rx="4"/>
      <text x="105" y="70" fill="#93c5fd" font-size="11" font-weight="700" text-anchor="middle">Token / Feature Tensor</text>
      <text x="105" y="90" fill="#8b98a5" font-size="9" text-anchor="middle">Input Shape: [B, SeqLen, D_model]</text>
      <text x="105" y="110" fill="#3fb950" font-size="9" font-weight="700" text-anchor="middle">High Precision FP16/BF16</text>
      <rect x="15" y="145" width="190" height="135" fill="#0d1117" stroke="#1d4ed8" rx="4"/>
      <text x="105" y="170" fill="#e6edf3" font-size="11" font-weight="700" text-anchor="middle">Positional Encoding</text>
      <text x="105" y="195" fill="#8b98a5" font-size="9" text-anchor="middle">RoPE / Sinusoidal Projection</text>
      <text x="105" y="220" fill="#8b98a5" font-size="9" text-anchor="middle">Preserves Sequence Order</text>
      <text x="105" y="250" fill="#93c5fd" font-size="9" font-weight="700" text-anchor="middle">Multi-Modal Fusion Ready</text>
    </g>
    <g transform="translate(270, 75)">
      <rect width="220" height="325" fill="#161b22" stroke="#30363d" stroke-width="1.5" rx="8"/>
      <text x="110" y="25" fill="#60a5fa" font-size="12" font-weight="700" text-anchor="middle">2. Transformer / Residual Block</text>
      <rect x="15" y="45" width="190" height="85" fill="#0d1117" stroke="#60a5fa" stroke-width="1.5" rx="4"/>
      <text x="105" y="70" fill="#ffffff" font-size="11" font-weight="700" text-anchor="middle">Multi-Head Self-Attention</text>
      <text x="105" y="90" fill="#93c5fd" font-size="9" text-anchor="middle">Softmax(QK^T / sqrt(d)) * V</text>
      <text x="105" y="110" fill="#3fb950" font-size="9" font-weight="700" text-anchor="middle">FlashAttention-2 Kernel</text>
      <path d="M 15 87 L -10 87 L -10 230 L 15 230" fill="none" stroke="#3fb950" stroke-width="2" stroke-dasharray="3"/>
      <rect x="15" y="150" width="190" height="85" fill="#0d1117" stroke="#30363d" rx="4"/>
      <text x="105" y="175" fill="#d2a8ff" font-size="11" font-weight="700" text-anchor="middle">Feed-Forward MLP (SwiGLU)</text>
      <text x="105" y="195" fill="#8b98a5" font-size="9" text-anchor="middle">Hidden Dim: 4x D_model</text>
      <text x="105" y="215" fill="#3fb950" font-size="9" font-weight="700" text-anchor="middle">RMSNorm Pre-Layer Normalization</text>
    </g>
    <g transform="translate(515, 75)">
      <rect width="220" height="325" fill="#161b22" stroke="#30363d" stroke-width="1.5" rx="8"/>
      <text x="110" y="25" fill="#60a5fa" font-size="12" font-weight="700" text-anchor="middle">3. Head &amp; Loss Optimization</text>
      <rect x="15" y="45" width="190" height="80" fill="#0d1117" stroke="#30363d" rx="4"/>
      <text x="105" y="70" fill="#58a6ff" font-size="11" font-weight="700" text-anchor="middle">Prediction Head</text>
      <text x="105" y="90" fill="#8b98a5" font-size="9" text-anchor="middle">Linear Projection to Vocab/Classes</text>
      <text x="105" y="110" fill="#3fb950" font-size="9" font-weight="700" text-anchor="middle">Softmax Probability Vector</text>
      <rect x="15" y="145" width="190" height="135" fill="#0d1117" stroke="#30363d" rx="4"/>
      <text x="105" y="170" fill="#f87171" font-size="11" font-weight="700" text-anchor="middle">Cross-Entropy Loss &amp; Autodiff</text>
      <text x="105" y="195" fill="#8b98a5" font-size="9" text-anchor="middle">Backward Pass &amp; Gradient Clipping</text>
      <text x="105" y="220" fill="#8b98a5" font-size="9" text-anchor="middle">AdamW Weight Update (β1, β2)</text>
      <text x="105" y="250" fill="#3fb950" font-size="9" font-weight="700" text-anchor="middle">Stable Convergence Standard</text>
    </g>
  
  <!-- Key insight bar -->
  <rect x="25" y="415" width="710" height="22" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="430" fill="#fbbf24" font-size="9" font-weight="700" text-anchor="middle">Key Insight: Optimal Expert Parallelism Moe architecture balances performance throughput, systemic latency, and physical constraints.</text>
  
  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">Technical specification &amp; verification reference for Expert Parallelism Moe (Row ID 13675)</text>
</svg>

The MoE scaling law. Empirically, an MoE model with $N$ experts and active parameters $A$ performs roughly like a dense model of size $A \cdot N^{0.3}$ in terms of loss — better than $A$ alone, but not as good as a dense model of size $A \cdot N$. The exponent varies (0.2–0.4) depending on routing quality and expert granularity. This makes MoE the dominant architecture for cost-efficient frontier models: you get 80% of the benefit of a model 5–10× larger at only the inference cost of the active slice.

Fine-grained vs coarse-grained experts. Early MoE (Switch, Mixtral) used 8–128 experts each the size of a full FFN. DeepSeek-V2 and later designs shrink expert size dramatically (e.g. 256 experts, each 1/16 the FFN width) so more experts can be selected per token ($k = 6$–8) without increasing total compute — this gives smoother routing, less load imbalance, and better generalization because each token assembles a more nuanced combination.

What MoE changes for the hardware stack. The shift from dense to MoE fundamentally re-weights the hardware bottleneck hierarchy: memory capacity and bandwidth matter more than peak FLOPS, inter-chip interconnect bandwidth becomes the training limiter (all-to-all), and the router decision latency is on the critical path for every single token. This is why the CFS platform models MoE workloads across the HBM (/hbm), KV-cache (/kvcache), and inference (/infer) simulators — each captures a different facet of the MoE serving challenge.