Mixture-of-Depths (MoD) is a transformer efficiency technique built on a simple observation: a standard transformer spends exactly the same amount of computation on every token, whether that token is a throwaway "the" or a pivotal technical term that the whole prediction hinges on. MoD breaks that uniformity by letting the model choose, at every layer, which tokens are worth the full cost of that layer's attention and feed-forward computation and which can simply skip it and ride the residual connection through unchanged. It is conditional computation along the depth axis of the network — hence the name — and it is the depth-wise cousin of Mixture-of-Experts, which does the same trick along the width axis.\n\nA dense transformer wastes compute by treating every token identically. Every position flows through every block, paying the identical FLOP cost for self-attention and the MLP, regardless of how much processing that position actually needs. But language is not uniform: some tokens are trivially predictable from local context and some require deep, many-layer reasoning. Spending a fixed, maximal budget on all of them means the easy tokens are massively over-served while the compute that could have gone to hard tokens is spread thin. MoD is the attempt to reallocate that fixed budget toward the tokens that need it.\n\nMoD puts a router before each block that admits only the top-k tokens; the rest take the residual shortcut. At every MoD layer a small learned router scores each token, and only the highest-scoring fraction — a fixed capacity, say 12.5% or 50% of the sequence — is passed through the block's attention and MLP. The non-selected tokens bypass the block entirely via the identity residual, arriving at the next layer unchanged. The crucial engineering choice is that the capacity is static: exactly k tokens are processed per block, known ahead of time, so the compute graph has a fixed shape and batches efficiently on a GPU or TPU. This is what separates MoD from classic early-exit / adaptive-depth schemes, where each token dynamically decides when to stop — flexible in theory but a nightmare to batch because different sequences finish at different layers.\n\nMoD, Mixture-of-Experts, and early exit are three different answers to "where do we save compute?" Mixture-of-Experts routes each token to a few of many parallel expert MLPs — it saves compute along width, activating only a sparse slice of a very large parameter count while keeping full depth. MoD routes along depth, keeping the same parameters but letting most tokens skip most layers, cutting FLOPs per token. Early exit varies depth dynamically per token, which maximizes flexibility but sacrifices the static, hardware-friendly shape MoD preserves. Because MoE and MoD save on orthogonal axes, they compose: a "MoDE" block can route across experts and across depth at once, stacking both savings.\n\n| Technique | Saves compute along | Compute shape | Parameters |\n|---|---|---|---|\n| Dense transformer | Nothing — every token, every layer | Static | Fully used |\n| Mixture-of-Experts | Width (parallel experts) | Static, sparse | Many, sparsely activated |\n| Mixture-of-Depths | Depth (skip layers) | Static, fixed capacity | Same as dense |\n| Early exit / adaptive depth | Depth (dynamic per token) | Dynamic, hard to batch | Same as dense |\n\n``svg\n\n``\n\nThe unhelpful way to think about Mixture-of-Depths is as yet another niche efficiency hack layered onto the transformer. The useful way is to see it as answering a question the dense architecture never asks: not how to process a token but whether this token, at this layer, is worth processing at all. By giving every block a router with a fixed capacity, MoD reallocates a constant compute budget toward the tokens that need deep processing and lets the easy ones coast on the residual — capturing much of the benefit of dynamic early-exit while keeping the static, batch-friendly compute shape that hardware demands. Set beside Mixture-of-Experts, which sparsifies the network's width, MoD sparsifies its depth, and the two combine cleanly. Read Mixture-of-Depths through a spend-compute-only-where-it's-needed lens rather than an every-token-deserves-every-layer lens, and the router, the fixed capacity, and the residual shortcut stop looking like tricks and become the natural machinery for buying back the compute a uniform transformer throws away.