Scaling laws are the empirical power-law relationships that predict how a language model's loss falls as you add parameters, training data, and compute. They are the reason frontier model building shifted from guesswork to forecasting: before spending millions on a training run, labs can extrapolate from small runs and predict, with surprising accuracy, how good the final model will be. Scaling laws are the quantitative backbone of the "just make it bigger" era — and, just as importantly, the tool that told the field when bigger was the wrong move.\n\n``svg\n\n``\n\nThe core finding is that loss follows a power law. Kaplan and colleagues at OpenAI showed in 2020 that test loss decreases as a clean power-law function of model size, dataset size, and compute — appearing as straight lines on log-log axes across many orders of magnitude. Because the relationship is so smooth, a handful of small, cheap training runs can be fit to a curve and extrapolated to predict the loss of a run thousands of times larger. This predictability is what makes massive investments defensible.\n\nChinchilla corrected the recipe. In 2022, Hoffmann and colleagues at DeepMind re-ran the analysis more carefully and found that the earlier work had over-weighted model size relative to data. For a fixed compute budget, parameters and training tokens should be scaled in roughly equal proportion — about twenty tokens per parameter. Their 70B-parameter Chinchilla model, trained on far more data, beat the 280B-parameter Gopher despite being four times smaller. The lesson: most large models of that era were badly undertrained.\n\nCompute-optimal is not the same as deployment-optimal. The Chinchilla frontier minimizes training loss for a given compute budget, where compute is approximately six times parameters times tokens. But inference cost scales with parameter count, not training tokens, so if a model will serve billions of queries it pays to make it smaller and train it well past the compute-optimal point. This is why models like Llama are deliberately "over-trained" relative to Chinchilla — trading extra training compute for cheaper, faster inference.\n\nThe functional form makes the trade-offs explicit. Loss is modeled as an irreducible floor plus two shrinking terms — one that falls with parameters, one that falls with data. The floor is the entropy of the data itself, which no amount of scale can beat; the other two terms decay as power laws with their own exponents. Fitting these constants on small runs lets a lab read off the optimal split of a budget between a bigger model and more data, and predict the payoff before committing.\n\nScaling laws guide but do not guarantee. Power laws eventually bend, high-quality training data is finite (the looming "data wall"), and smooth improvements in loss do not translate cleanly into smooth improvements on downstream tasks — some capabilities appear to emerge abruptly at scale. Loss is predictable; usefulness is messier. The frontier of the field is now as much about data quality, better objectives, and inference-aware scaling as about simply buying more compute.\n\n| Quantity | Symbol | Scaling-law role | Real-world constraint |\n|---|---|---|---|\n| Parameters | N | loss falls as 1/N^α | memory and per-query inference cost |\n| Training tokens | D | loss falls as 1/D^β | supply of high-quality data |\n| Compute | C ≈ 6ND | sets the achievable frontier | budget, time, energy |\n| Chinchilla ratio | D / N ≈ 20 | the compute-optimal split | shifts higher when inference dominates |\n\nRead scaling through a compute-allocation lens rather than a bigger-is-better lens: the real insight is not that adding parameters helps, but that a fixed compute budget has an optimal split between model size and data — and that the whole curve is predictable enough to plan around before the expensive run begins.\n