Neural Scaling Laws Emergent Abilities
# Neural Scaling Laws & Emergent Abilities
## 1. Introduction & Motivation
One of the most consequential empirical discoveries in modern machine learning is that a neural network's test loss, as a function of model size, dataset size, or training compute, follows remarkably smooth and predictable power-law curves over many orders of magnitude. This observation, formalized by Kaplan et al. (2020) and refined by Hoffmann et al. (2022, the "Chinchilla" paper), transformed language model development from a largely empirical, trial-and-error process into something closer to an engineering discipline with genuine predictive power: given a fixed compute budget, scaling laws tell practitioners, in advance and without running the full-scale experiment, roughly how large a model and how much data to use to minimize loss, and what loss to expect once trained.
This predictability matters enormously in practice because frontier model training runs cost many millions of dollars and cannot be iterated on the way smaller experiments can — a wrong architectural or data decision discovered only after a full-scale run is enormously expensive. Scaling laws let researchers fit trends on cheap, small-scale experiments and extrapolate with reasonable confidence to the much larger regime they actually care about, fundamentally changing how compute budgets are allocated between model size and dataset size.
A closely related and more contested phenomenon is that of "emergent abilities" — capabilities that appear to arise suddenly, at a particular model scale, rather than improving smoothly the way the underlying loss does. Early reports (Wei et al., 2022) catalogued dozens of tasks (multi-step arithmetic, certain reasoning benchmarks) where small models perform at chance level and only past some scale threshold does performance suddenly and sharply improve. Whether this is a genuine, qualitative shift in model capability or an artifact of the specific metrics used to measure it (Schaefer et al., 2023) is one of the more interesting ongoing debates connecting scaling laws to the practical evaluation of large language models.
## 2. Core Concepts & Theory
A neural scaling law is an empirical power-law relationship between a model's test loss $L$ and a resource variable $X$ (parameter count $N$, dataset size $D$, or compute $C$), typically of the form $L(X) = a X^{-\alpha} + L_\infty$, where $\alpha$ is the scaling exponent, $a$ is a constant, and $L_\infty$ is an "irreducible loss" floor representing the entropy of the data distribution itself (the loss a perfect predictor could not go below, due to genuine randomness or ambiguity in the data). The exponent $\alpha$ determines how quickly returns diminish: a larger $\alpha$ means loss falls faster per unit of additional resource, while small $\alpha$ implies scaling that resource alone yields only modest gains.
Kaplan et al.'s original scaling laws studied parameter count, dataset size, and compute largely independently, and their headline recommendation — that compute-optimal training should heavily prioritize model size over dataset size — was later revised by the Chinchilla paper, which found that Kaplan et al.'s experiments had under-trained their large models relative to the available compute (holding dataset size too small), and that a substantially more balanced scaling of parameters and data yields lower loss for the same total compute. This is the origin of the now widely quoted "Chinchilla-optimal" ratio of roughly 20 training tokens per parameter, though the precise optimal ratio depends on the specific loss function fit and continues to be refined by subsequent work (e.g., research suggesting the optimal ratio shifts further toward more data as inference costs, not just training costs, are taken into account).
"Emergent abilities" refers to the empirical observation that certain downstream task metrics — as opposed to the smoothly decreasing pretraining loss itself — appear flat (near chance performance) across a wide range of small-to-medium model scales, then rise sharply once a threshold scale is crossed. This behavior is distinct from, and considerably more contested than, the smooth power-law scaling of pretraining loss, which is essentially universally observed and uncontroversial.
## 3. Mathematical Formulation
The joint scaling law used in Chinchilla-style compute-optimal analysis models loss as a function of both parameters $N$ and training tokens $D$:
$
L(N, D) = \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} + E
$
where $E$ is the irreducible entropy floor, and $A, B, \alpha, \beta$ are fit empirically from a grid of training runs at varying $(N, D)$. Given a fixed compute budget, approximated for transformer training as $C \approx 6ND$ (FLOPs per token times tokens, times 2 for the forward-plus-backward pass, times roughly the parameter count), the compute-optimal allocation is found by minimizing $L(N, D)$ subject to $C = 6ND$. Substituting $D = C / (6N)$ and setting $\partial L / \partial N = 0$ yields closed-form power-law scaling for the optimal parameter and data allocation:
$
N^\star(C) \propto C^{\frac{\beta}{\alpha+\beta}}, \qquad D^\star(C) \propto C^{\frac{\alpha}{\alpha+\beta}}
$
This is a central, testable prediction of compute-optimal scaling theory: both the optimal model size and optimal dataset size should themselves grow as power laws in the total compute budget, with exponents determined entirely by the ratio of the two underlying scaling exponents $\alpha$ and $\beta$.
For a metric like exact-match sequence accuracy, if a model gets each of $L$ output tokens correct independently with probability $p$ (a simplification, but illustrative), the sequence-level exact-match accuracy is
$
ext{Acc}_{ ext{exact}} = p^{L}
$
Because this is a highly nonlinear (and, for large $L$, extremely sharp) function of the smoothly-improving per-token probability $p$, even a gradual, continuous improvement in $p$ as a function of scale can translate into an apparently sudden, threshold-like jump in $ ext{Acc}_{ ext{exact}}$ — this is the mathematical core of the "emergent abilities as a measurement artifact" argument.
## 4. Advanced Theory & Extensions
Broken neural scaling laws (BNSL, Caballero et al.) extend the simple power-law form to piecewise or smoothly-interpolated multi-segment power laws, since empirically many real scaling curves show one or more distinct "breaks" — regions where the effective scaling exponent changes — rather than following a single power law across all scales. This is important because extrapolating a single power law fit at small scale can be substantially wrong if the true curve has a break beyond the fitted range, a genuine practical risk for scaling law-based compute planning.
The "mirage" argument for emergent abilities (Schaefer et al., 2023) shows, both mathematically and empirically, that many reported emergent abilities disappear or become smooth and predictable when the same underlying model outputs are scored with a smoother, more information-preserving metric (e.g., token-level cross-entropy or edit distance) instead of a sharp, nonlinear, discontinuous one (e.g., exact-match accuracy on multi-step arithmetic, which fails completely unless every single digit is correct). This does not settle the debate entirely — some argue there remain genuine capability thresholds not fully explained by metric choice — but it substantially reframes the phenomenon as, at minimum, partly a property of how success is measured rather than purely a property of the model.
Data scaling laws have also been extended to account for data quality and repetition: research on data-constrained scaling (Muennighoff et al.) shows that repeating training data has diminishing and eventually near-zero marginal value compared to fresh data, with the effective value of repeated tokens itself following a predictable decay curve, which matters increasingly as high-quality text data becomes a scarcer resource than compute for the largest training runs. Downstream (task-specific) scaling laws are also generally noisier and less universal than pretraining loss scaling laws, since downstream performance depends on the interaction between pretraining distribution and the specific task distribution, which is not captured by a single scalar loss.
## 5. Computational Considerations
Fitting a reliable scaling law requires training a grid of models across a meaningful range of scales (often spanning 3+ orders of magnitude in compute) to get a stable exponent estimate, which itself represents a significant compute investment before the "payoff" extrapolation to full scale — this up-front cost is precisely what makes scaling-law research the province of only the most compute-resourced labs, even though the resulting laws are published and usable by everyone. The compute-optimal allocation prediction is only as good as the accuracy of the fitted $A, B, \alpha, \beta$ constants, and small errors in these fitted exponents can translate into meaningfully suboptimal (N, D) choices at the much larger extrapolated scale, which is part of why the exact Chinchilla-optimal ratio has been revised and debated in follow-up work.
Because $N^\star(C)$ and $D^\star(C)$ are power laws with exponents that are generally close to (but not exactly) 0.5 each, compute-optimal training under the standard formulation roughly balances growth in model size and dataset size — a useful rule of thumb, but one that shifts once additional considerations (such as amortized inference cost across the model's deployment lifetime, which favors smaller models trained on relatively more data than pure training-compute optimality would suggest) are incorporated into the objective being optimized.
## 6. Practical Implementation Strategies
In practice, scaling-law fitting is done by training a suite of models at varying sizes and token counts, recording final (or IsoFLOP-curve-minimum) validation loss for each, and fitting the joint power-law form via nonlinear least squares (typically in log-space for numerical stability, or via the full nonlinear form using something like L-BFGS, since the additive irreducible-loss term $E$ makes a pure log-log linear fit only an approximation). A standard technique, used in the Chinchilla paper, is the "IsoFLOP profile" approach: for each of several fixed compute budgets, train models across a range of (N, D) combinations that all consume that same compute, plot loss against N, and read off the minimum — repeating this across several compute budgets directly traces out the empirical $N^\star(C)$ curve without needing to assume the parametric form in advance.
When applying scaling laws to real training decisions, practitioners should fit on data spanning as wide a compute range as feasible (extrapolating a law fit over less than one order of magnitude of compute is considerably riskier than extrapolating one fit over three or more), and should sanity-check predictions against at least one held-out larger-scale run before committing a full training budget to a scaling-law-derived configuration, given the real possibility of a "break" beyond the fitted range.
## 7. Benchmark Datasets & Evaluation
Pretraining loss scaling laws are typically evaluated on standard large-scale text corpora (Common Crawl-derived datasets, The Pile, and successors like RedPajama and Dolma), using held-out validation cross-entropy loss as the primary, smooth metric. Downstream and "emergent ability" evaluations draw on established multi-task benchmark suites (BIG-Bench, MMLU, GSM8K for arithmetic and reasoning), specifically because these benchmarks include tasks with genuinely nonlinear, all-or-nothing scoring criteria (exact-match, multi-step correctness) that are exactly the kind of metric implicated in the emergent-abilities-as-mirage argument.
Rigorous evaluation of the emergence question requires reporting results under multiple metrics for the same task and model checkpoints — both a sharp, discontinuous metric (like exact-match) and a smoother, partial-credit metric (like token-edit-distance or per-token log-likelihood) — since the central empirical claim of the "mirage" hypothesis is specifically about how the shape of the observed scaling curve changes (from sharp jump to smooth improvement) purely as a function of which metric is used to score the identical underlying model outputs.
## 8. Key Challenges & Limitations
Scaling laws are empirical fits, not physical laws, and their extrapolation reliability degrades the further one extrapolates beyond the fitted data range, with documented cases of "breaks" where the effective exponent shifts at larger scale in ways not predictable from the smaller-scale fit alone. The exact numerical value of the Chinchilla-optimal token-to-parameter ratio has itself been challenged and revised in follow-up analyses questioning some of the original paper's fitting methodology, illustrating that even well-regarded scaling law results carry meaningful methodological uncertainty and should be treated as a strong prior rather than an exact prescription.
The emergent-abilities debate remains genuinely unresolved: while the metric-artifact explanation accounts for a substantial fraction of reported emergent behaviors, it does not obviously explain every case, and some researchers argue that certain qualitative shifts in model behavior (such as reliable multi-step tool use or in-context learning of genuinely novel task formats) are not fully reducible to a sharp-versus-smooth metric argument. Data scaling faces its own limitation distinct from compute or parameter scaling: the supply of high-quality, non-synthetic text data is finite, and multiple research groups have projected that the highest-quality data sources could be exhausted for further scaling within the coming years, motivating active research into synthetic data generation and more data-efficient training methods as a response to this constraint.
## 9. Hyperparameter Tuning
Fitting the power-law exponents $\alpha, \beta$ (and the irreducible loss $E$) is itself a nonlinear regression problem sensitive to the fitting procedure: the Chinchilla paper's own re-analyses by later researchers highlighted that using a robust loss (such as Huber loss on log-loss residuals) rather than plain least squares, and appropriately weighting IsoFLOP data points, can materially shift the fitted optimal token-to-parameter ratio, underscoring that scaling-law hyperparameter choices are not merely bookkeeping but can change the resulting compute-allocation recommendation.
For practical training runs guided by a fitted scaling law, the learning rate schedule and batch size must themselves be adjusted appropriately as model and data scale change (per separate, complementary scaling relationships for optimal batch size and learning rate as functions of compute), since a scaling law fit for loss as a function of (N, D) implicitly assumes near-optimal training hyperparameters were used at every scale in the fitting data — a poorly tuned smaller-scale run can bias the fitted exponents and lead to systematically wrong extrapolated predictions.
## 10. Real-World Applications & Case Studies
The Chinchilla paper's findings directly influenced the design of subsequent frontier language models, many of which shifted toward increasing training-token counts substantially relative to parameter count compared to earlier Kaplan-era models, reflecting the industry-wide adoption of Chinchilla-optimal (or, increasingly, inference-cost-adjusted "beyond Chinchilla-optimal," i.e., smaller-model-more-data) allocation strategies. GPT-4-class and subsequent model development reportedly relies heavily on scaling-law extrapolation from smaller-scale experiments to make architecture, data mixture, and compute allocation decisions well before committing to the full-scale training run, given the enormous cost of any full-scale mistake.
The emergent-abilities discussion has had direct practical consequences for evaluation methodology across the industry: model providers increasingly report results across multiple model scales and multiple metric formulations for the same benchmark, specifically to give a clearer picture of whether observed capability improvements are gradual or genuinely threshold-like, informing decisions about which capabilities are safe to expect (or must be specifically tested for) at a given deployment scale.
## 11. Integration with Other Methods
Scaling laws integrate directly with the test-time compute scaling research area (see Chain-of-Thought & Test-Time Compute Scaling): recent work explores joint scaling laws that treat inference-time compute (number of samples, reasoning length) as an additional resource variable alongside training-time parameters and data, asking how a fixed total compute budget should be split between training a larger model and spending more compute per query at inference time — a natural extension of the classical training-time-only scaling law framework. Scaling laws also inform data curation and mixture research: since the dataset-size scaling exponent $\beta$ is itself sensitive to data quality and diversity, improving data quality can be understood as directly improving the constant $B$ or exponent $\beta$ in the scaling law, making data curation quantifiably comparable to raw dataset-size scaling in terms of its effect on the fitted loss curve.
Model compression and distillation research increasingly draws on scaling laws for distillation specifically (how a student model's achievable loss scales with teacher quality and distillation dataset size), extending the classical pretraining scaling law framework to the compressed-model regime relevant for efficient deployment.
## 12. Future Research Directions
An active direction is developing scaling laws that incorporate data quality, not just quantity, as an explicit variable — moving beyond the "tokens are tokens" assumption of classical scaling laws toward frameworks that predict how loss scales with both dataset size and a measured or inferred data-quality distribution, which matters increasingly as easily available high-quality data becomes scarcer relative to compute. Resolving the emergent-abilities debate more conclusively, potentially by developing a principled, task-agnostic way to distinguish "genuine capability thresholds" from "sharp-metric artifacts," remains an open and consequential problem for how the field communicates about and anticipates future model capabilities.
Joint scaling laws spanning training compute, inference compute, and data quality simultaneously — rather than the current, largely separate literatures on each axis — represent a natural and increasingly necessary unification, particularly as reasoning models make the training/inference compute trade-off a first-order design decision rather than an afterthought. Finally, understanding whether and how scaling laws break down or change character near the frontier of currently achievable compute (as opposed to extrapolating confidently from smaller-scale fits) is of direct, high-stakes importance for the largest training runs currently being planned across the industry.
## 13. Summary & Key Takeaways
Neural scaling laws establish that a model's pretraining loss follows remarkably predictable power-law relationships with model size, dataset size, and compute, enabling compute-optimal training decisions to be made in advance via extrapolation from smaller, cheaper experiments — a finding formalized by Kaplan et al. and substantially refined by the Chinchilla paper's more balanced compute-optimal allocation between parameters and data. The closely related but more contested phenomenon of "emergent abilities" — sudden, threshold-like jumps in certain downstream task metrics — has been shown to be substantially, though perhaps not entirely, explained by the sharp, nonlinear nature of the metrics used to measure those abilities, rather than necessarily reflecting a qualitative shift in the underlying, smoothly-improving model capability. Together, these findings have shifted large-scale model development from an ad hoc, trial-and-error process toward one grounded in extrapolatable, quantitative predictions, while also cautioning that such extrapolations carry real uncertainty and should be validated rather than trusted blindly at the frontier of achievable scale.
Keywords: neural scaling laws, Chinchilla, compute-optimal training, power law, parameter scaling, data scaling, emergent abilities, broken neural scaling laws, irreducible loss, IsoFLOP profile, scaling exponent, data-constrained scaling, exact-match accuracy, downstream scaling laws, token-to-parameter ratio, inference-cost-adjusted scaling, Kaplan scaling laws, model size, training compute, metric artifact
---
## Appendix: Practical Labs
### Lab 1: Power-Law Fit Recovers the True Scaling Exponent From Noisy Data
import numpy as np
def true_loss(compute, a, alpha):
"""Ground-truth power-law loss curve: L(C) = a * C^-alpha."""
return a * compute ** (-alpha)
def generate_noisy_losses(computes, a, alpha, noise_std, rng):
"""Simulates realistic scaling-law data: the clean power law corrupted by
multiplicative log-normal noise, matching how experimental loss
measurements scatter around the true underlying trend."""
clean = true_loss(computes, a, alpha)
noise = rng.normal(0, noise_std, size=len(computes))
return clean * np.exp(noise)
def fit_power_law(computes, losses):
"""Recovers (a, alpha) via ordinary least squares on log(L) vs log(C),
exploiting log(L) = log(a) - alpha*log(C), a linear relationship."""
log_c = np.log(computes)
log_l = np.log(losses)
design = np.vstack([log_c, np.ones_like(log_c)]).T
slope, intercept = np.linalg.lstsq(design, log_l, rcond=None)[0]
alpha_hat = -slope
a_hat = np.exp(intercept)
return a_hat, alpha_hat
def test_power_law_fit_recovers_true_scaling_exponent():
rng = np.random.RandomState(0)
true_a, true_alpha = 10.0, 0.3
computes = np.logspace(3, 9, 30) # 6 orders of magnitude, typical of real scaling studies
losses = generate_noisy_losses(computes, true_a, true_alpha, noise_std=0.05, rng=rng)
a_hat, alpha_hat = fit_power_law(computes, losses)
print(f"True alpha: {true_alpha:.4f}, Fitted alpha: {alpha_hat:.4f}")
print(f"True a: {true_a:.4f}, Fitted a: {a_hat:.4f}")
assert abs(alpha_hat - true_alpha) < 0.02, "Fitted scaling exponent should closely recover the true exponent"
assert abs(a_hat - true_a) / true_a < 0.1, "Fitted scale constant should closely recover the true constant"
# Sanity check: the fitted law should still predict a decreasing loss
# curve, matching the qualitative behavior of real scaling laws.
predicted = a_hat * computes ** (-alpha_hat)
assert np.all(np.diff(predicted) < 0), "Fitted power law should predict monotonically decreasing loss with compute"
print("Power-law scaling exponent recovery test passed.")
if __name__ == "__main__":
test_power_law_fit_recovers_true_scaling_exponent()### Lab 2: Compute-Optimal (N, D) Allocation Follows Predicted Power-Law Scaling
import numpy as np
def loss_fn(N, D, A, alpha, B, beta, E=1.5):
"""Chinchilla-style joint scaling law: loss as a function of parameter
count N and training tokens D, plus an irreducible-entropy floor E."""
return A * N ** (-alpha) + B * D ** (-beta) + E
def compute_optimal_allocation(C, A, alpha, B, beta, n_grid=4000):
"""Given a compute budget C (with C ~= 6*N*D FLOPs), searches over N
(with D implied by the compute constraint) to find the loss-minimizing
allocation -- the discrete analogue of Chinchilla's IsoFLOP profiles."""
N_candidates = np.logspace(3, 12, n_grid)
D_candidates = C / (6 * N_candidates)
valid = D_candidates > 1
N_candidates, D_candidates = N_candidates[valid], D_candidates[valid]
losses = loss_fn(N_candidates, D_candidates, A, alpha, B, beta)
best_idx = np.argmin(losses)
return N_candidates[best_idx], D_candidates[best_idx], losses[best_idx]
def test_compute_optimal_scaling_matches_theoretical_exponent_and_beats_skewed_allocations():
A, alpha = 400.0, 0.34
B, beta = 400.0, 0.28
compute_budgets = np.logspace(18, 22, 6)
optimal_Ns = []
for C in compute_budgets:
N_opt, D_opt, L_opt = compute_optimal_allocation(C, A, alpha, B, beta)
optimal_Ns.append(N_opt)
# Fit the empirical N*(C) power-law exponent and compare to the
# closed-form theoretical prediction beta / (alpha + beta).
log_C, log_N = np.log(compute_budgets), np.log(optimal_Ns)
design = np.vstack([log_C, np.ones_like(log_C)]).T
empirical_exponent, _ = np.linalg.lstsq(design, log_N, rcond=None)[0]
theoretical_exponent = beta / (alpha + beta)
print(f"Empirical N*(C) exponent: {empirical_exponent:.4f}")
print(f"Theoretical exponent beta/(alpha+beta): {theoretical_exponent:.4f}")
assert abs(empirical_exponent - theoretical_exponent) < 0.01, \
"Empirically optimal parameter scaling exponent should match the closed-form prediction"
# At a fixed compute budget, compare the optimal allocation to two skewed
# allocations (8x too parameter-heavy, 8x too data-heavy): both should be
# strictly worse (higher loss) than the compute-optimal allocation.
C_test = 1e20
N_opt, D_opt, L_opt = compute_optimal_allocation(C_test, A, alpha, B, beta)
N_param_heavy = N_opt * 8
D_param_heavy = C_test / (6 * N_param_heavy)
L_param_heavy = loss_fn(N_param_heavy, D_param_heavy, A, alpha, B, beta)
N_data_heavy = N_opt / 8
D_data_heavy = C_test / (6 * N_data_heavy)
L_data_heavy = loss_fn(N_data_heavy, D_data_heavy, A, alpha, B, beta)
print(f"Optimal loss: {L_opt:.4f} | param-heavy loss: {L_param_heavy:.4f} | data-heavy loss: {L_data_heavy:.4f}")
assert L_opt < L_param_heavy, "Compute-optimal allocation should beat an overly parameter-heavy allocation"
assert L_opt < L_data_heavy, "Compute-optimal allocation should beat an overly data-heavy allocation"
print("Compute-optimal allocation scaling test passed.")
if __name__ == "__main__":
test_compute_optimal_scaling_matches_theoretical_exponent_and_beats_skewed_allocations()### Lab 3: Sharp Metrics Manufacture the Appearance of "Emergent" Jumps
import numpy as np
def per_token_accuracy(log_scale, midpoint=0.0, width=1.0):
"""A smoothly, continuously improving per-token accuracy as a function of
log model scale (a sigmoid, standing in for the smooth loss/accuracy
improvements scaling laws actually predict)."""
return 1.0 / (1.0 + np.exp(-(log_scale - midpoint) / width))
def exact_match_accuracy(log_scale, seq_len, midpoint=0.0, width=1.0):
"""Probability that ALL seq_len output tokens are correct, assuming
per-token correctness is independent -- a sharp, nonlinear function of
the smoothly-improving per-token accuracy above."""
p = per_token_accuracy(log_scale, midpoint, width)
return p ** seq_len
def width_90_10(log_scales, metric_values):
"""The log-scale range over which a metric rises from 10% to 90% of its
maximum value -- a measure of how 'sharp' (emergent-looking) the curve
appears; a smaller width means a more sudden-looking jump."""
max_val = metric_values.max()
idx_hi = np.argmax(metric_values >= 0.9 * max_val)
idx_lo = np.argmax(metric_values >= 0.1 * max_val)
if metric_values[idx_hi] < 0.9 * max_val or metric_values[idx_lo] < 0.1 * max_val:
return None
return log_scales[idx_hi] - log_scales[idx_lo]
def test_exact_match_metric_looks_more_emergent_as_task_difficulty_increases():
log_scales = np.linspace(-6, 6, 20000)
seq_lens = [1, 5, 20, 100] # longer sequences = stricter, sharper exact-match requirement
widths = []
for seq_len in seq_lens:
acc = exact_match_accuracy(log_scales, seq_len)
w = width_90_10(log_scales, acc)
widths.append(w)
print(f"{'seq_len':>8} | {'10-90% transition width (log-scale)':>36}")
for L, w in zip(seq_lens, widths):
print(f"{L:8d} | {w:36.4f}")
# The underlying per-token accuracy (seq_len=1) has a fixed transition
# width. As seq_len grows, exact-match accuracy becomes an increasingly
# sharp (narrow-transition-width) function of the SAME smoothly improving
# underlying capability -- exactly the mechanism proposed to explain
# apparently "emergent" jumps as largely a metric artifact.
assert widths[0] > widths[1] > widths[2] > widths[3], \
"Transition width should shrink (curve should look sharper/more emergent) as sequence length increases"
# The longest sequence length tested should look dramatically sharper
# than the raw per-token metric, illustrating how misleading a sharp
# metric can be about the smoothness of the underlying capability.
assert widths[0] / widths[-1] > 1.5, \
"Exact-match on long sequences should appear substantially sharper than the underlying per-token metric"
print("Emergent-abilities-as-metric-artifact test passed.")
if __name__ == "__main__":
test_exact_match_metric_looks_more_emergent_as_task_difficulty_increases()### Lab 4: Balanced Parameter/Data Scaling Beats Skewed Scaling at Fixed Compute
import numpy as np
def loss_fn(N, D, A, alpha, B, beta, E=1.5):
return A * N ** (-alpha) + B * D ** (-beta) + E
def compute_optimal_allocation(C, A, alpha, B, beta, n_grid=4000):
N_candidates = np.logspace(3, 12, n_grid)
D_candidates = C / (6 * N_candidates)
valid = D_candidates > 1
N_candidates, D_candidates = N_candidates[valid], D_candidates[valid]
losses = loss_fn(N_candidates, D_candidates, A, alpha, B, beta)
best_idx = np.argmin(losses)
return N_candidates[best_idx], D_candidates[best_idx], losses[best_idx]
def test_balanced_scaling_outperforms_parameter_only_and_data_only_scaling():
A, alpha = 400.0, 0.34
B, beta = 400.0, 0.28
C = 5e20
N_opt, D_opt, L_opt = compute_optimal_allocation(C, A, alpha, B, beta)
# "Parameters only" strategy: fix D at a small, historically-typical
# value and pour all extra compute into N instead (mimicking early,
# pre-Chinchilla scaling practice that under-weighted data).
D_fixed_small = D_opt / 20
N_param_only = C / (6 * D_fixed_small)
L_param_only = loss_fn(N_param_only, D_fixed_small, A, alpha, B, beta)
# "Data only" strategy: fix N at a small value and pour all extra compute
# into D instead.
N_fixed_small = N_opt / 20
D_data_only = C / (6 * N_fixed_small)
L_data_only = loss_fn(N_fixed_small, D_data_only, A, alpha, B, beta)
print(f"Balanced (compute-optimal) loss: {L_opt:.4f}")
print(f"Parameter-only-scaling loss: {L_param_only:.4f}")
print(f"Data-only-scaling loss: {L_data_only:.4f}")
assert L_opt < L_param_only, "Balanced compute-optimal scaling should beat a parameters-only strategy"
assert L_opt < L_data_only, "Balanced compute-optimal scaling should beat a data-only strategy"
# The gap should be substantial, not marginal, illustrating why the
# Chinchilla-style balanced allocation was a meaningful correction to
# earlier scaling practice rather than a minor refinement.
assert L_param_only - L_opt > 0.05, "Parameter-only strategy should be meaningfully worse, not just marginally"
assert L_data_only - L_opt > 0.05, "Data-only strategy should be meaningfully worse, not just marginally"
print("Balanced vs. skewed compute allocation test passed.")
if __name__ == "__main__":
test_balanced_scaling_outperforms_parameter_only_and_data_only_scaling()