mixture of depths adaptive compute

**Mixture of Depths and Adaptive Computation** are the **neural network techniques that dynamically allocate different amounts of computation to different inputs based on their difficulty — allowing easy inputs to exit the network early or skip layers while hard inputs receive the full computational treatment, reducing average inference cost by 30-60% with minimal accuracy loss by avoiding wasteful computation on simple examples**. **The Uniform Computation Problem** Standard neural networks apply the same computation to every input regardless of difficulty. A trivially classifiable image (clear photo of a cat) receives the same 100+ layer processing as an ambiguous, occluded scene. This wastes compute on easy examples that could be resolved with a fraction of the network. **Early Exit** Add classification heads at intermediate layers. If the model is "confident enough" at an early layer, output the prediction and skip remaining layers: - **Confidence Threshold**: Exit when the maximum softmax probability exceeds a threshold (e.g., 0.95). Easy examples exit early; hard examples propagate deeper. - **BranchyNet / SDN (Shallow-Deep Networks)**: Train auxiliary classifiers at multiple intermediate points. Average depth reduction: 30-50% at <1% accuracy cost. - **For LLMs**: CALM (Confident Adaptive Language Modeling) routes tokens through variable numbers of Transformer layers. Function words ("the", "is") exit early; content-bearing tokens receive full processing. **Mixture of Depths (MoD)** Each Transformer layer has a router that decides, for each token, whether to process it through the full self-attention + FFN computation or to skip the layer entirely (pass through via residual connection only): - A lightweight router (single linear layer) produces a routing score for each token. - Top-K tokens (by routing score) are processed; remaining tokens skip. - Training: the router is trained jointly with the model using a straight-through estimator. - Result: 12.5% of tokens might skip a given layer → 12.5% compute savings at that layer, compounding across all layers. **Adaptive Computation Time (ACT)** Graves (2016) proposed a halting mechanism where each position has a learned probability of halting at each step. Computation continues until the cumulative halting probability exceeds a threshold. A ponder cost regularizer encourages the model to halt as early as possible, balancing accuracy against computational cost. **Universal Transformers** Apply the same Transformer layer repeatedly (shared weights) with ACT controlling the number of iterations per position. Positions requiring more "thinking" receive more iterations. Combines the parameter efficiency of weight sharing with input-adaptive depth. **Token Merging (ToMe)** For Vision Transformers: merge similar tokens across the sequence to reduce token count progressively through layers. Bipartite matching identifies the most similar token pairs; they are averaged into single tokens. Reduces FLOPs by 30-50% with <0.5% accuracy loss on ImageNet. **Practical Benefits** - **Inference Cost Reduction**: 30-60% average FLOPS savings with <1% quality degradation on most benchmarks. - **Latency Improvement**: Particularly impactful for streaming/real-time applications where average latency matters more than worst-case. - **Proportional to Task Difficulty**: Simple queries (factual recall, formatting) are fast; complex queries (multi-step reasoning, analysis) receive full computation. Adaptive Computation is **the efficiency paradigm that makes neural network inference proportional to problem difficulty** — breaking the assumption that every input deserves equal computational investment and instead allocating compute where it matters most, matching the intuition that thinking harder should be reserved for harder problems.