mixture of depths

Mixture-of-Depths (MoD) is a transformer efficiency technique built on a simple observation: a standard transformer spends exactly the same amount of computation on every token, whether that token is a throwaway "the" or a pivotal technical term that the whole prediction hinges on. MoD breaks that uniformity by letting the model *choose*, at every layer, which tokens are worth the full cost of that layer's attention and feed-forward computation and which can simply skip it and ride the residual connection through unchanged. It is conditional computation along the *depth* axis of the network — hence the name — and it is the depth-wise cousin of Mixture-of-Experts, which does the same trick along the *width* axis.\n\n**A dense transformer wastes compute by treating every token identically.** Every position flows through every block, paying the identical FLOP cost for self-attention and the MLP, regardless of how much processing that position actually needs. But language is not uniform: some tokens are trivially predictable from local context and some require deep, many-layer reasoning. Spending a fixed, maximal budget on all of them means the easy tokens are massively over-served while the compute that could have gone to hard tokens is spread thin. MoD is the attempt to reallocate that fixed budget toward the tokens that need it.\n\n**MoD puts a router before each block that admits only the top-k tokens; the rest take the residual shortcut.** At every MoD layer a small learned router scores each token, and only the highest-scoring fraction — a fixed *capacity*, say 12.5% or 50% of the sequence — is passed through the block's attention and MLP. The non-selected tokens bypass the block entirely via the identity residual, arriving at the next layer unchanged. The crucial engineering choice is that the capacity is *static*: exactly k tokens are processed per block, known ahead of time, so the compute graph has a fixed shape and batches efficiently on a GPU or TPU. This is what separates MoD from classic *early-exit* / adaptive-depth schemes, where each token dynamically decides when to stop — flexible in theory but a nightmare to batch because different sequences finish at different layers.\n\n**MoD, Mixture-of-Experts, and early exit are three different answers to "where do we save compute?"** Mixture-of-Experts routes each token to a few of many parallel expert MLPs — it saves compute along *width*, activating only a sparse slice of a very large parameter count while keeping full depth. MoD routes along *depth*, keeping the same parameters but letting most tokens skip most layers, cutting FLOPs per token. Early exit varies depth *dynamically* per token, which maximizes flexibility but sacrifices the static, hardware-friendly shape MoD preserves. Because MoE and MoD save on orthogonal axes, they compose: a "MoDE" block can route across experts *and* across depth at once, stacking both savings.\n\n| Technique | Saves compute along | Compute shape | Parameters |\n|---|---|---|---|\n| Dense transformer | Nothing — every token, every layer | Static | Fully used |\n| Mixture-of-Experts | Width (parallel experts) | Static, sparse | Many, sparsely activated |\n| Mixture-of-Depths | Depth (skip layers) | Static, fixed capacity | Same as dense |\n| Early exit / adaptive depth | Depth (dynamic per token) | Dynamic, hard to batch | Same as dense |\n\n```svg\n\n```\n\nThe unhelpful way to think about Mixture-of-Depths is as yet another niche efficiency hack layered onto the transformer. The useful way is to see it as answering a question the dense architecture never asks: not *how* to process a token but *whether* this token, at this layer, is worth processing at all. By giving every block a router with a fixed capacity, MoD reallocates a constant compute budget toward the tokens that need deep processing and lets the easy ones coast on the residual — capturing much of the benefit of dynamic early-exit while keeping the static, batch-friendly compute shape that hardware demands. Set beside Mixture-of-Experts, which sparsifies the network's *width*, MoD sparsifies its *depth*, and the two combine cleanly. Read Mixture-of-Depths through a spend-compute-only-where-it's-needed lens rather than an every-token-deserves-every-layer lens, and the router, the fixed capacity, and the residual shortcut stop looking like tricks and become the natural machinery for buying back the compute a uniform transformer throws away.