mixture of depths (mod)
**Mixture of Depths (MoD)** is the **adaptive computation architecture that dynamically allocates transformer layer processing based on input token complexity — allowing easy tokens to skip layers and save compute while difficult tokens receive full-depth processing** — the depth-axis complement to Mixture of Experts (width variation) that reduces inference FLOPs by 20–50% with minimal quality degradation by recognizing that not all tokens require equal computational investment.
**What Is Mixture of Depths?**
- **Definition**: A transformer architecture modification where a learned router at each layer decides whether each token should be processed by that layer or skip directly to the next layer via a residual connection — dynamically varying the effective depth per token.
- **Per-Token Routing**: Unlike early exit (which stops computation for the entire sequence), MoD operates at token granularity — within a single sequence, function words may skip 60% of layers while technical terms use all layers.
- **Learned Routing**: The router is a lightweight network (linear layer + sigmoid) trained jointly with the main model — learning which tokens benefit from additional processing at each layer.
- **Capacity Budget**: A fixed compute budget per layer limits the number of tokens processed — e.g., only 50% of tokens pass through each layer's attention and FFN, while the rest skip via residual.
**Why Mixture of Depths Matters**
- **20–50% FLOPs Reduction**: By skipping layers for easy tokens, total compute decreases substantially — enabling faster inference without architecture changes.
- **Quality Preservation**: The router learns to allocate computation where it matters — model quality drops <1% even when 50% of layer operations are skipped.
- **Complementary to MoE**: MoE varies width (which expert processes a token); MoD varies depth (how many layers process a token) — combining both enables 2D adaptive computation.
- **Batch Efficiency**: In a batch, different tokens take different paths — but the total compute per layer is bounded by the capacity budget, enabling predictable throughput.
- **Training Efficiency**: MoD models train faster per FLOP than equivalent dense models — the adaptive computation acts as implicit regularization.
**MoD Architecture**
**Router Mechanism**:
- Each layer has a lightweight router: r(x) = σ(W_r · x + b_r) producing a routing score per token.
- Tokens with scores above a threshold (or top-k tokens) are processed by the layer.
- Skipped tokens pass through via the residual connection: output = input (no transformation).
**Training**:
- Router trained jointly with model weights using straight-through estimator for gradient flow through discrete routing decisions.
- Auxiliary load-balancing loss encourages the router to use the full capacity budget rather than routing all tokens through or none.
- Capacity factor (e.g., C=0.5) sets the fraction of tokens processed per layer during training.
**Inference**:
- Router decisions are made in real-time — no fixed skip patterns.
- Easy tokens (common words, punctuation) naturally learn to skip most layers.
- Complex tokens (domain-specific terms, reasoning-critical words) receive full processing.
**MoD Performance**
| Configuration | FLOPs (vs. Dense) | Quality (vs. Dense) | Throughput Gain |
|---------------|-------------------|--------------------:|----------------|
| **C=0.75** (75% processed) | 78% | 99.5% | 1.25× |
| **C=0.50** (50% processed) | 55% | 98.8% | 1.7× |
| **C=0.25** (25% processed) | 35% | 96.5% | 2.5× |
Mixture of Depths is **the recognition that computational difficulty varies token-by-token** — enabling transformers to invest their compute budget where it matters most, achieving the efficiency gains of model compression without the permanent quality loss, by making depth itself a dynamic, learned property of the inference process.