Mixture of Depths (MoD) is the dynamic computation technique for transformers that allows individual tokens to skip certain transformer layers — allocating compute resources proportionally to token "difficulty" rather than uniformly processing every token through every layer, achieving 50% compute reduction with minimal quality loss by routing easy tokens (function words, whitespace, common patterns) through fewer layers while hard tokens (rare words, complex reasoning steps) receive full depth processing.

Motivation: Uniform Compute is Wasteful

• Standard transformers: Every token passes through every layer → fixed compute per sequence.
• Observation: Not all tokens are equally hard. "the", "and", punctuation rarely need 32+ layers of processing.
• Mixture of Experts (MoE): Routes tokens to different FFN experts (same depth, different width).
• MoD: Routes tokens to different depth levels → same width, different depth → complementary to MoE.

MoD Mechanism

• At each transformer layer, a lightweight router (linear projection → top-k selection) decides:
• Include: Token passes through this layer's attention + FFN.
• Skip: Token bypasses this layer via residual connection (identity transformation).

For each layer l:
  router_scores = linear(token_embedding)  # scalar per token
  top_k_mask = topk(router_scores, k=S*C)  # select capacity C fraction
  
  full_tokens = tokens[top_k_mask]          # process these through attention+FFN
  skip_tokens = tokens[~top_k_mask]         # bypass via residual
  
  output = combine(processed_full, skip_tokens_unchanged)

Capacity and Routing

• Capacity C: Fraction of tokens processed at each layer (e.g., C=0.125 = 12.5% of tokens).
• k selection: Causal attention requires reordering-safe routing (cannot use future tokens to route).
• Auxiliary router: Small predictor trained alongside main model to predict skip/process per token.
• Training: Joint optimization of router + transformer parameters → routers learn which tokens are "hard".

Results (Raposo et al., 2024)

• 12.5% capacity MoD model matches isoFLOP baseline on language modeling.
• At same wall-clock time: MoD is faster (fewer FLOPs per forward pass).
• At same FLOPs: MoD achieves lower perplexity (better allocation of compute).
• Combined MoD+MoE: Additive benefits — tokens routed in both expert and depth dimensions.

What Gets Skipped?

• Empirically, frequent function words, whitespace, simple punctuation tend to skip.
• Complex semantic tokens, rare words, tokens at key decision points tend to be processed fully.
• Pattern emerges without supervision — router learns from language modeling loss alone.

Comparison with Related Methods

Practical Challenges

• Batch efficiency: Skipped tokens create irregular compute → harder to batch uniformly.
• KV cache: Skipped layers don't write to KV cache → cache layout changes per token.
• Implementation: Requires custom CUDA kernels or sparse computation frameworks.

Mixture of Depths is the principled answer to the observation that transformers waste enormous compute treating all tokens equally — by learning to allocate depth proportional to token complexity, MoD achieves the theoretical ideal of adaptive compute allocation in an end-to-end differentiable framework, pointing toward a future where transformer inference cost is proportional to content complexity rather than sequence length, making long-context reasoning dramatically more efficient without architectural changes.