mixture of depths advanced

**Mixture of Depths (MoD)** is a **dynamic computation architecture for transformer models that routes individual tokens through a variable number of layers based on processing difficulty, using lightweight router networks at each layer to decide whether a token requires full self-attention and feed-forward computation or can skip directly to the next layer via a residual connection** — reducing average inference FLOPs by 30–50% with minimal quality degradation by acknowledging that not every token in a sequence requires the same amount of neural processing. **What Is Mixture of Depths?** - **Definition**: MoD adds a binary routing decision at each transformer layer: for each incoming token, a small router network (typically a single linear projection + sigmoid) outputs a score indicating whether the token should be fully processed by that layer or bypass it via the residual stream. Tokens that bypass a layer incur near-zero compute for that layer. - **Complementary to MoE**: Mixture of Experts (MoE) varies the width of computation — selecting which expert (sub-network) processes each token at a given layer. MoD varies the depth — selecting how many layers each token traverses. The two approaches are orthogonal and can be combined for compound efficiency gains. - **Token-Level Granularity**: The routing decision is made independently for each token at each layer, creating a unique computation path through the network for every token in every sequence. Common words and predictable continuations exit early, while rare words and complex reasoning steps receive full-depth processing. **Why Mixture of Depths Matters** - **Inference Efficiency**: In standard transformers, every token passes through every layer — but empirical analysis shows that many tokens converge to their final representation well before the last layer. MoD eliminates this wasted computation, reducing average FLOPs per token by 30–50% depending on input complexity. - **Variable Difficulty**: Natural language has enormous variation in processing difficulty. The word "the" in "the cat sat on the mat" requires minimal contextual processing, while "bank" in "I need to bank on the river bank near the bank" requires deep contextual disambiguation. MoD allocates compute proportionally to this difficulty variation. - **Latency Reduction**: For autoregressive generation where tokens are processed sequentially, reducing the average number of layers per token directly reduces wall-clock latency — critical for interactive applications where users perceive generation speed. - **Scaling Efficiency**: MoD enables training larger (deeper) models while maintaining the same inference budget as smaller models, because the average effective depth is less than the total depth. This allows models to store more knowledge in additional layers while only accessing those layers when needed. **Router Architecture and Training** - **Router Design**: Typically a single linear layer that projects the token's hidden state to a scalar, followed by a sigmoid activation. The output is thresholded to produce a binary route/skip decision, or used as a soft weight for differentiable training via Gumbel-Softmax. - **Capacity Control**: An auxiliary loss encourages balanced routing — preventing collapse where the router learns to skip all layers (trivial solution) or route all tokens through all layers (no efficiency gain). Typical targets set a compute budget (e.g., "process 50% of tokens at each layer"). - **Training Strategy**: MoD models are often trained from scratch with the routing mechanism, or initialized from a dense pretrained model with routers added and fine-tuned. End-to-end training learns both the layer parameters and the routing policy jointly. **Mixture of Depths** is **dynamic depth allocation** — the architectural recognition that different tokens require fundamentally different amounts of neural processing, enabling transformers to invest computation where it matters most while saving resources on the easy predictions.