what is mixture of experts
**Mixture-of-experts (MoE) is an architecture where a model has far more total parameters than a comparably-sized dense model, but only activates a small fraction of them for any given token — trading a fixed compute cost per token for a much larger pool of specialized knowledge to draw from.** Every model discussed so far in this series — the transformer itself, its attention layers, the KV cache — has been "dense": every parameter participates in processing every single token. MoE breaks that assumption. Instead of one large feed-forward block per layer, an MoE layer contains many parallel feed-forward blocks called experts, plus a small router network that looks at each token and decides which handful of experts should process it. A model might have dozens of experts totaling hundreds of billions of parameters, while any single token only ever touches two or four of them.
**The router is the piece that makes the whole idea work, and it's trained alongside the experts rather than hand-designed.** For every token arriving at an MoE layer, the router computes a score for each expert and picks the top few — commonly the top 2 — to actually process that token; the results from those selected experts are combined (often weighted by the router's own confidence scores) into the layer's output, while every other expert simply doesn't run for that token. Nothing forces experts to specialize in a human-interpretable way like "math expert" or "code expert" — specialization emerges from training, as gradient descent finds it useful for different experts to become good at different kinds of tokens or contexts. The router itself is small and cheap compared to the experts it's choosing between, so its own compute cost is close to negligible.
```svg
```
**This creates a specific, important mismatch: MoE reduces compute per token far more than it reduces memory footprint, which is the opposite tradeoff from quantization and GQA/MQA.** Every technique earlier in this series — quantization, grouped/multi-query attention — shrinks how many bytes have to be stored and moved. MoE doesn't shrink storage at all: every expert's weights still have to live somewhere in memory, ready to be selected by the router at any moment, since which experts get picked can change from token to token and can't be predicted in advance. What MoE actually saves is compute — only a fraction of the total parameters do any math for a given token — so an MoE model can have the knowledge capacity of a much larger dense model while running each token through roughly the compute cost of a much smaller one. Serving an MoE model well means having enough memory to hold every expert, even though most of them sit idle on any single forward pass.
| Technique | Reduces | Leaves Unchanged | Category |
|---|---|---|---|
| Quantization | Bytes per stored weight/cache value | Total number of parameters | Memory footprint |
| Grouped/Multi-Query Attention | Number of separate KV sets stored | Total attention compute | Memory footprint |
| Speculative Decoding | Number of expensive large-model passes | Total parameter count | Throughput scheduling |
| Mixture of Experts | Compute (FLOPs) run per token | Total memory needed to store all experts | Compute sparsity |
```flowchart
st=>start: Token arrives at an MoE layer
score=>operation: Small router network scores every available expert for this token
select=>operation: Top-scoring experts (commonly top 2) selected to process the token
compute=>operation: Only the selected experts run their feed-forward computation on this token
idle=>operation: All other experts remain idle for this token, though their weights still occupy memory
combine=>operation: Selected experts' outputs combined, weighted by router confidence
pass=>end: Combined result becomes this layer's output for the token
st->score->select->compute->idle->combine->pass
```
**For AI accelerator design, MoE shifts the engineering problem from pure compute density toward memory capacity and fast, flexible data movement.** Because a chip serving an MoE model has to keep every expert resident and ready, even though most compute engines sit idle for any given token, total memory capacity and the interconnect bandwidth to route tokens to whichever experts the router picks become just as important as raw FLOPs — arguably more so than for a dense model of equivalent total size. That's precisely why MoE has become one of the dominant architecture choices for the largest current models: it lets a system scale total knowledge capacity largely independently of the per-token compute budget, provided the underlying hardware has the memory capacity and routing bandwidth to keep every expert within reach.