mixture of experts
A mixture of experts is a neural network architecture in which a gating network routes each input to only a small subset of specialized "expert" subnetworks rather than passing every input through the entire model, letting the overall model hold far more total parameters while keeping the computation per input manageable.
```flowchart
{
"rows": [
{ "type": "nodes", "items": [
{ "title": "Input arrives needing to be processed by the model", "sub": "full dense model would activate every parameter for it", "tone": "neutral" }
]},
{ "type": "arrow" },
{ "type": "group", "title": "Gating network routes input to a few relevant experts", "items": [
{ "title": "Only a small subset of expert subnetworks activated", "sub": "most of the model's total parameters stay unused for this input", "tone": "blue" }
]},
{ "type": "arrow" },
{ "type": "nodes", "items": [
{ "title": "Output combines the activated experts' results", "sub": "large total capacity, modest per-input compute cost", "tone": "green" }
]}
]
}
```
**Mixture of experts architectures exist because simply making a dense neural network bigger, activating every single parameter for every single input, causes computational cost to grow directly with model size, eventually making further scaling impractically expensive.** Since routing each individual input to only a handful of specialized expert subnetworks, rather than activating the entire model, lets total parameter count grow largely independently of per-input computational cost, a mixture of experts architecture uses a gating network to select a small, relevant subset of experts for each input, achieving far greater total model capacity without a proportional increase in the compute needed to process any single input.
```svg
```
```svg
```
| Aspect | Dense neural network | Mixture of experts |
|---|---|---|
| Parameters activated per input | All of them | A small selected subset |
| Total model capacity achievable | Limited by compute budget | Can be much larger |
| Per-input compute cost | Scales with total size | Scales with active experts only |
| Common use | Smaller or simpler models | Very large-scale models |
**A mixture of experts model's gating network is trained alongside the experts themselves, learning which experts tend to be most useful for which kinds of inputs, though the resulting specialization isn't necessarily aligned with human-interpretable categories.** Because the gating network's routing decisions are learned purely to minimize the overall model's training objective rather than to match any predefined notion of what each expert should specialize in, the patterns different experts end up handling can be subtle or not easily describable in simple human terms, even though the routing itself proves effective at improving the model's performance.
**Mixture of experts models introduce their own engineering challenges around balancing how evenly inputs get routed across the available experts, since routing too many inputs to too few experts wastes the unused experts' capacity and can slow training.** Because a gating network left unconstrained can learn to favor a small handful of experts disproportionately, leaving many other experts rarely used and undertrained, practical mixture of experts implementations typically include mechanisms that encourage more balanced routing across the full set of available experts.
**Mixture of experts architectures have become particularly important for scaling large language models, since they let total model capacity grow substantially while the compute cost of processing each individual token stays comparatively controlled.** Because language modeling at very large scale runs into serious computational cost constraints as dense model size increases, mixture of experts approaches have become an increasingly common technique for continuing to scale total model capacity while keeping the per-token computational cost from growing at the same steep rate.
Read mixture of experts through a hospital-specialist lens: rather than sending every patient through every single specialist regardless of their condition, a well-run hospital routes each patient to just the relevant specialists, letting the hospital maintain a huge range of expertise overall while keeping each individual patient's visit efficient and focused.