mixture

Mixture of Experts (MoE) is a neural-network architecture in which many specialist subnetworks (experts) exist in a layer, but a lightweight router activates only a few of them for each token. This decouples a model's parameter count from its per-token compute: the network can hold enormous capacity in memory while each token pays for just the handful of experts it actually uses.\n\n**A router picks a sparse subset of experts per token.** In a dense feed-forward layer, every token flows through the same weights. An MoE layer replaces that single block with N experts plus a small gating network. For each token, the router scores the experts and selects the top-k (often 1 or 2 of many), runs only those, and combines their outputs weighted by the gate scores. The other experts do no work for that token, so the layer is conditionally — sparsely — activated.\n\n**Capacity grows without growing per-token FLOPs.** Because only k of N experts run, adding experts increases total parameters (and therefore model capacity) while the compute per token stays roughly fixed. A model can hold hundreds of billions or trillions of parameters yet activate only a few billion per token. That is the whole appeal: MoE buys representational capacity at the cost of memory to store all experts, not at the cost of proportionally more math on every token.\n\n| Property | Dense model | MoE model |\n|---|---|---|\n| Experts per layer | 1 | N (e.g. 8-256) |\n| Active per token | all weights | top-k (e.g. 1-2) |\n| Params vs compute | coupled | decoupled |\n| Memory footprint | one FFN | all experts resident |\n| Main challenge | scaling FLOPs | routing & load balance |\n\n```svg\n\n```\n\n**The hard part is routing, balance, and memory.** Sparsity introduces problems a dense model never has: the router must spread tokens evenly or a few popular experts get overloaded while others starve, so training adds a load-balancing (auxiliary) loss and an expert-capacity limit that drops or reroutes overflow tokens. At serving time all experts must be resident in memory even though only a few run, and distributing experts across GPUs (expert parallelism) requires all-to-all communication to shuffle tokens to their assigned experts and back. MoE trades dense FLOPs for a routing-and-memory engineering problem.\n\nRead MoE through a quant lens rather than a 'smart routing' lens: the numbers that matter are total parameters versus active parameters per token, and the memory bandwidth to stream the chosen experts' weights. Per the roofline, MoE lowers arithmetic intensity — fewer FLOPs per byte of weights moved — so decoding is often bound by loading expert weights from HBM, not by the multiply-adds. The design question is how many experts you can store, how few you activate, and how evenly the router balances them: a measured capacity-versus-bandwidth budget, not just 'pick the best expert.'