mixture of depths
**Mixture of Depths (MoD)** is the **dynamic computation technique for transformers that allows individual tokens to skip certain transformer layers** — allocating compute resources proportionally to token "difficulty" rather than uniformly processing every token through every layer, achieving 50% compute reduction with minimal quality loss by routing easy tokens (function words, whitespace, common patterns) through fewer layers while hard tokens (rare words, complex reasoning steps) receive full depth processing.
**Motivation: Uniform Compute is Wasteful**
- Standard transformers: Every token passes through every layer → fixed compute per sequence.
- Observation: Not all tokens are equally hard. "the", "and", punctuation rarely need 32+ layers of processing.
- Mixture of Experts (MoE): Routes tokens to different FFN experts (same depth, different width).
- MoD: Routes tokens to different depth levels → same width, different depth → complementary to MoE.
**MoD Mechanism**
- At each transformer layer, a lightweight router (linear projection → top-k selection) decides:
- **Include**: Token passes through this layer's attention + FFN.
- **Skip**: Token bypasses this layer via residual connection (identity transformation).
```
For each layer l:
router_scores = linear(token_embedding) # scalar per token
top_k_mask = topk(router_scores, k=S*C) # select capacity C fraction
full_tokens = tokens[top_k_mask] # process these through attention+FFN
skip_tokens = tokens[~top_k_mask] # bypass via residual
output = combine(processed_full, skip_tokens_unchanged)
```
**Capacity and Routing**
- **Capacity C**: Fraction of tokens processed at each layer (e.g., C=0.125 = 12.5% of tokens).
- **k selection**: Causal attention requires reordering-safe routing (cannot use future tokens to route).
- **Auxiliary router**: Small predictor trained alongside main model to predict skip/process per token.
- **Training**: Joint optimization of router + transformer parameters → routers learn which tokens are "hard".
**Results (Raposo et al., 2024)**
- 12.5% capacity MoD model matches isoFLOP baseline on language modeling.
- At same wall-clock time: MoD is faster (fewer FLOPs per forward pass).
- At same FLOPs: MoD achieves lower perplexity (better allocation of compute).
- Combined MoD+MoE: Additive benefits — tokens routed in both expert and depth dimensions.
**What Gets Skipped?**
- Empirically, frequent function words, whitespace, simple punctuation tend to skip.
- Complex semantic tokens, rare words, tokens at key decision points tend to be processed fully.
- Pattern emerges without supervision — router learns from language modeling loss alone.
**Comparison with Related Methods**
| Method | What Routes | Savings |
|--------|------------|--------|
| MoE | Which expert (same depth) | Width compute |
| MoD | Which depth (same width) | Depth compute |
| Early Exit | Stop at intermediate layer | Trailing layers |
| Adaptive Span | Attention span per head | Attention compute |
**Practical Challenges**
- Batch efficiency: Skipped tokens create irregular compute → harder to batch uniformly.
- KV cache: Skipped layers don't write to KV cache → cache layout changes per token.
- Implementation: Requires custom CUDA kernels or sparse computation frameworks.
Mixture of Depths is **the principled answer to the observation that transformers waste enormous compute treating all tokens equally** — by learning to allocate depth proportional to token complexity, MoD achieves the theoretical ideal of adaptive compute allocation in an end-to-end differentiable framework, pointing toward a future where transformer inference cost is proportional to content complexity rather than sequence length, making long-context reasoning dramatically more efficient without architectural changes.