Adaptive Computation Time (ACT) is a mechanism introduced by Alex Graves that allows recurrent neural networks and transformers to learn how many computational steps to perform for each input element, rather than using a fixed number of steps. ACT adds a learned "halting probability" at each step, and computation continues until the cumulative halting probability exceeds a threshold (typically 1.0), with a ponder cost penalty that encourages the model to halt early when additional computation is unnecessary.

Why ACT Matters in AI/ML: ACT provides learned, input-dependent computation allocation that enables models to automatically dedicate more processing to difficult inputs and less to easy ones, achieving better quality-efficiency tradeoffs than fixed-depth architectures.

• Halting mechanism — At each computational step t, the model outputs a scalar halting probability h_t ∈ (0,1); the model halts at step N when the cumulative sum Σh_t first exceeds 1.0, with the remainder R_N = 1 - Σ_{t=1}^{N-1} h_t weighting the final step's contribution • Ponder cost — An auxiliary loss term L_ponder = Σ R_t penalizes unnecessary computation, encouraging the model to halt as early as possible while still achieving accurate predictions; the ponder cost coefficient balances efficiency vs. accuracy • Variable depth per position — In sequence models, each position can halt independently, meaning some tokens receive 2 computational steps while others receive 20, naturally allocating capacity where the input is most complex • Differentiable computation budget — Unlike hard early-exit thresholds, ACT is fully differentiable through the halting mechanism, allowing end-to-end gradient-based training of both the computation and halting networks • Universal Transformer integration — ACT combined with weight-sharing transformers (Universal Transformers) creates models that iterate transformer layers until convergence, with each position deciding independently when to stop

Adaptive Computation Time is a foundational mechanism for input-dependent computation that enables neural networks to learn their own computational budget per input, automatically allocating more processing steps to difficult examples and fewer to easy ones, achieving superior efficiency-accuracy tradeoffs through end-to-end differentiable halting decisions.