conditional computation

**Conditional Computation** is the **execution of only a subset of a neural network's parameters for each input** — using gating mechanisms to selectively activate modules, layers, or experts, reducing the average computational cost while maintaining model capacity. **Conditional Computation Methods** - **Gated Layers**: Binary gates decide whether to execute each layer — skip unnecessary layers per input. - **Mixture of Experts**: Route inputs to a subset of experts based on a learned gating function. - **SkipNet**: Train a policy network to decide which residual blocks to skip for each input. - **Stochastic Depth**: Randomly skip layers during training (regularization), deterministically during inference. **Why It Matters** - **Decoupled Capacity and Compute**: A model with 10B parameters can use only 1B per input — large capacity, small cost. - **Sparse Models**: Conditional computation enables sparse, efficient models that scale beyond dense network limits. - **Switch Transformers**: Google's Switch Transformer uses conditional computation to scale to trillion-parameter models. **Conditional Computation** is **activating only what's needed** — selectively executing network components based on each input for massive efficiency gains.