AI Compute

Compute, in AI, means floating-point operations — FLOPs — the raw arithmetic a training run or inference request actually consumes, and it functions as the field's basic unit of currency: every model's cost, training time, and even its eventual capability can be tied back to how much compute went into producing it. OpenAI's own 2018 analysis "AI and Compute" quantified just how fast that currency had been inflating: the amount of compute used to train the largest AI models had been doubling roughly every 3.4 months between 2012 and 2018, a pace dramatically faster than Moore's Law's historic doubling of transistor density roughly every two years — meaning compute spent on frontier training runs was growing an order of magnitude faster than the hardware improvements underneath it, funded instead by larger and larger clusters.

The standard shortcut for estimating training compute without simulating the whole run is a single approximation used throughout the scaling-laws literature. Total training FLOPs comes out to roughly six times the number of parameters times the number of training tokens — the factor of six accounts for the forward pass, the backward pass, and the extra arithmetic the backward pass needs relative to the forward pass — which is exactly the relationship Hoffmann and colleagues' Chinchilla paper used to compare different model-size-and-data allocations at a fixed compute budget, the IsoFLOP method covered under scaling-laws.

The theoretical peak FLOPs a GPU or TPU advertises and the FLOPs a real training run actually achieves are two very different numbers, and the gap between them is tracked explicitly as Model FLOPs Utilization. Frontier training runs typically achieve somewhere in the 30 to 50 percent range of a cluster's theoretical peak throughput, with the rest lost to communication overhead between accelerators, memory bandwidth limits, and the time the hardware spends waiting rather than computing — which means a cluster's advertised peak FLOPs figure overstates the usable compute for an actual training run by roughly two to three times, a gap large enough that MFU is reported as a standard metric in most major model-training papers rather than treated as an afterthought.

Compute budgets are allocated deliberately, not just spent until money runs out, which is exactly the discipline the "compute-optimal" framing from scaling-laws is trying to enforce. Given a fixed compute budget — a number decided in advance by what a lab can afford or has access to — the Chinchilla-style question is how to split that budget between model size and training data to reach the lowest possible loss, rather than defaulting to the largest model the budget can technically support; this distinction between "spending compute" and "spending it optimally" is precisely why frontier labs run the kind of IsoFLOP sweeps described under scaling-laws before committing a full training budget to one specific model size.

At the scale of a modern frontier model, compute is measured in units large enough that intuition stops working, and translating those units into something familiar is worth doing once. A single modern AI accelerator delivers on the order of hundreds of teraFLOPs per second at usable precision, and training a frontier-scale model can consume an aggregate on the order of 10^25 total floating-point operations — a number so large that labs report it in scientific notation rather than any everyday unit, and the only way to make it concrete is to divide it back down into GPU-hours or dollars, which is exactly the conversion the flowchart above exists to make explicit.

Compute's real role in the field is as the resource every other tradeoff ultimately gets measured against: more compute buys a bigger model or more training data, per the scaling laws it feeds into; more compute efficiency — a higher MFU, a better compute-optimal allocation — buys the same capability for less of it; and the entire history of frontier AI since 2012 is, in large part, a history of compute spending growing far faster than the hardware generating it, funded by clusters rather than chips.