Mixture of Depths (MoD) is the adaptive computation architecture that dynamically allocates transformer layer processing based on input token complexity — allowing easy tokens to skip layers and save compute while difficult tokens receive full-depth processing — the depth-axis complement to Mixture of Experts (width variation) that reduces inference FLOPs by 20–50% with minimal quality degradation by recognizing that not all tokens require equal computational investment.

What Is Mixture of Depths?

• Definition: A transformer architecture modification where a learned router at each layer decides whether each token should be processed by that layer or skip directly to the next layer via a residual connection — dynamically varying the effective depth per token.
• Per-Token Routing: Unlike early exit (which stops computation for the entire sequence), MoD operates at token granularity — within a single sequence, function words may skip 60% of layers while technical terms use all layers.
• Learned Routing: The router is a lightweight network (linear layer + sigmoid) trained jointly with the main model — learning which tokens benefit from additional processing at each layer.
• Capacity Budget: A fixed compute budget per layer limits the number of tokens processed — e.g., only 50% of tokens pass through each layer's attention and FFN, while the rest skip via residual.

Why Mixture of Depths Matters

• 20–50% FLOPs Reduction: By skipping layers for easy tokens, total compute decreases substantially — enabling faster inference without architecture changes.
• Quality Preservation: The router learns to allocate computation where it matters — model quality drops <1% even when 50% of layer operations are skipped.
• Complementary to MoE: MoE varies width (which expert processes a token); MoD varies depth (how many layers process a token) — combining both enables 2D adaptive computation.
• Batch Efficiency: In a batch, different tokens take different paths — but the total compute per layer is bounded by the capacity budget, enabling predictable throughput.
• Training Efficiency: MoD models train faster per FLOP than equivalent dense models — the adaptive computation acts as implicit regularization.

MoD Architecture

Router Mechanism:

• Each layer has a lightweight router: r(x) = σ(W_r · x + b_r) producing a routing score per token.
• Tokens with scores above a threshold (or top-k tokens) are processed by the layer.
• Skipped tokens pass through via the residual connection: output = input (no transformation).

Training:

• Router trained jointly with model weights using straight-through estimator for gradient flow through discrete routing decisions.
• Auxiliary load-balancing loss encourages the router to use the full capacity budget rather than routing all tokens through or none.
• Capacity factor (e.g., C=0.5) sets the fraction of tokens processed per layer during training.

Inference:

• Router decisions are made in real-time — no fixed skip patterns.
• Easy tokens (common words, punctuation) naturally learn to skip most layers.
• Complex tokens (domain-specific terms, reasoning-critical words) receive full processing.

MoD Performance

Mixture of Depths is the recognition that computational difficulty varies token-by-token — enabling transformers to invest their compute budget where it matters most, achieving the efficiency gains of model compression without the permanent quality loss, by making depth itself a dynamic, learned property of the inference process.