input-dependent computation

**Input-Dependent Computation** is the **paradigm where the computational graph or resource allocation of a neural network changes dynamically based on the input** — the model "decides" how much and what type of computation to apply to each input, enabling efficient and flexible inference. **Forms of Input-Dependent Computation** - **Routing**: Mixture of Experts (MoE) — route each input to a subset of expert networks. - **Gating**: Conditional computation gates decide which modules to activate per input. - **Attention**: Self-attention dynamically weighs which features to focus on per input. - **Resolution**: Choose input or feature map resolution based on input complexity. **Why It Matters** - **Computational Efficiency**: Not all inputs need the same computation — input-dependent allocation saves resources. - **Expressivity**: The model can allocate specialized computation (different experts) for different input types. - **Scaling**: MoE models scale to trillions of parameters while keeping per-input FLOPs constant. **Input-Dependent Computation** is **compute on demand** — dynamically choosing what and how much to compute based on each individual input.