Conditional Computation is the neural network design paradigm where only a fraction of the model's total parameters are activated for any given input, fundamentally decoupling model capacity (total knowledge stored) from inference cost (FLOPs per prediction) — enabling the construction of trillion-parameter models that access only the relevant 1–2% of parameters per query, transforming the scaling economics of large language models by allowing knowledge to grow without proportional compute growth.

What Is Conditional Computation?

• Definition: Conditional computation refers to any mechanism that selectively activates subsets of a neural network's parameters based on the input, rather than executing all parameters for every input. The key insight is that different inputs require different knowledge and different processing — a question about chemistry should activate chemistry-relevant parameters while leaving biology parameters dormant.
• Capacity vs. Cost: In a dense (standard) neural network, capacity equals cost — a 70B parameter model requires 70B parameter multiplications per forward pass. Conditional computation breaks this relationship — a 1T parameter MoE model might activate only 20B parameters per token, achieving 50x the capacity at the same inference cost as a 20B dense model.
• Sparsity: Conditional computation creates dynamic sparsity — different parameters are active for different inputs, but the overall activation pattern is sparse (few parameters active out of many total). This contrasts with static sparsity (weight pruning) where the same parameters are always zero.

Why Conditional Computation Matters

• Scaling Beyond Dense Limits: Dense models face a fundamental scaling wall — doubling parameters doubles inference cost, memory requirements, and serving costs. Conditional computation enables continued scaling of model knowledge and capability without proportional cost increase, making trillion-parameter models economically viable for production deployment.
• Specialization: Conditional activation enables implicit specialization — different parameter subsets learn to handle different domains, languages, or task types. Analysis of trained MoE models shows that specific experts specialize in specific topics (one expert handles code, another handles medical text) without explicit supervision, driven purely by the routing mechanism's optimization.
• Memory vs. Compute Trade-off: Conditional computation trades memory (storing all parameters) for reduced compute (activating few parameters). With modern hardware where memory is relatively cheap but compute (FLOP/s) is the bottleneck, this trade-off is highly favorable for large-scale deployment.
• Production Economics: The economic argument is compelling — serving a 1T parameter MoE model costs roughly the same as serving a 50–100B dense model (same active parameter count) but achieves quality comparable to a much larger dense model. This directly reduces the cost-per-query for LLM services.

Conditional Computation Implementations

Conditional Computation is the massive library paradigm — storing a million books of knowledge across trillions of parameters but reading only the one relevant page per query, enabling AI systems to be simultaneously vast in knowledge and efficient in execution.