LLM Evaluation Benchmarking Contamination Measuring Progress Reliably
# LLM Evaluation, Benchmarking & Contamination: Measuring Progress Reliably
## Introduction & Motivation
As large language models have grown more capable, the task of measuring that capability accurately has become surprisingly difficult in its own right. Early language model evaluation relied heavily on a small set of standardized benchmarks with fixed, publicly available test sets, an approach inherited from earlier machine learning practice where held-out test data was assumed to be genuinely unseen by the model under evaluation. That assumption breaks down for models pretrained on enormous, web-scraped corpora, since any sufficiently popular benchmark's questions and answers are highly likely to appear, verbatim or in close paraphrase, somewhere within the trillions of tokens such a corpus contains, a problem known as benchmark contamination that can silently inflate reported scores without reflecting any genuine improvement in underlying capability.
Beyond contamination, the evaluation landscape has had to adapt to models whose most interesting capabilities, open-ended dialogue, complex multi-step reasoning, creative writing, and instruction-following across an effectively unbounded space of possible tasks, resist the simple exact-match or multiple-choice scoring that sufficed for earlier, narrower NLP benchmarks. This has driven the rise of LLM-as-judge evaluation, in which one language model scores or compares the outputs of another, and of large-scale human preference platforms such as pairwise comparison arenas, both of which introduce their own distinct measurement biases that must be understood and corrected for rather than taken as ground truth.
A rigorous approach to LLM evaluation today requires attending simultaneously to several distinct concerns: whether a benchmark's test data has leaked into a model's training corpus, whether a benchmark still meaningfully discriminates between models of differing capability or has become saturated near ceiling performance, whether automated judges (whether LLM-based or human) introduce systematic biases that distort measured preferences, and whether reported score differences reflect genuine capability gaps or merely statistical noise from an evaluation set that is too small to support the precision being claimed. Each of these concerns has become its own active area of methodology, and getting them wrong has real consequences, from misleading leaderboards to mistaken beliefs about which techniques or architectures actually advance the state of the art.
## Core Concepts & Theory
Benchmark contamination refers to the presence of test-set content, or content sufficiently similar to it, within a model's training data, which allows the model to perform well on the benchmark not because it has learned the underlying capability the benchmark was designed to measure, but because it has effectively memorized the specific test items or their answers during pretraining. Contamination is distinguished from legitimate generalization by the fact that a contaminated model's high score does not predict correspondingly strong performance on genuinely novel problems drawn from the same distribution, making contaminated benchmark results a poor guide to real-world capability, and, more insidiously, contamination can silently favor whichever model's training pipeline happened to include more of a popular benchmark's data, distorting comparisons between models rather than merely inflating a single model's absolute score.
LLM-as-judge evaluation uses one language model, typically a highly capable one, to assess or compare the outputs of other models against some rubric or in a pairwise comparison, substituting for the more expensive and slower process of collecting human judgments at scale. This approach has become widespread because it correlates reasonably well with human preference on many tasks and can be run automatically at a scale human annotation cannot match, but it inherits and sometimes amplifies known biases: judges have been repeatedly shown to favor longer responses regardless of whether the additional length reflects genuine quality, to be sensitive to which response is presented first or second in a pairwise comparison (position bias), and to favor responses stylistically similar to the judge model's own outputs, all effects that must be measured and corrected for rather than assumed absent.
Benchmark saturation describes the phenomenon in which, as models improve over time relative to a fixed benchmark's difficulty, average accuracy on that benchmark approaches its ceiling (typically close to 100 percent, or, for multiple-choice formats, close to the maximum achievable given inherent item ambiguity or noise), at which point the benchmark's ability to discriminate between models of differing genuine capability collapses, since most or all sufficiently capable models cluster near the top of the achievable score range with differences dominated by measurement noise rather than by real capability gaps. Statistical rigor in benchmark comparison, most directly the use of confidence intervals or significance testing around reported accuracy figures rather than bare point estimates, addresses the related but distinct concern that any finite evaluation set, however well designed, provides only a noisy estimate of a model's true underlying performance, and comparisons between models whose confidence intervals substantially overlap should not be reported as if one model were definitively superior to another.
## Mathematical Formulation
A common n-gram overlap approach to contamination detection computes, for each test example, whether any contiguous span of $n$ tokens (an n-gram, with $n$ commonly between 8 and 15) also appears verbatim within the training corpus. Given a training corpus tokenized into a sequence $w_1, \ldots, w_N$, the set of training n-grams is
$$ G_{ ext{train}} = \{ (w_i, w_{i+1}, \ldots, w_{i+n-1}) : 1 \le i \le N - n + 1 \} $$
and a test example with tokens $x_1, \ldots, x_m$ is flagged as contaminated if
$$ \exists\, j \in \{1, \ldots, m-n+1\} : (x_j, \ldots, x_{j+n-1}) \in G_{ ext{train}} $$
Embedding-based or fuzzy contamination detection relaxes the exact-match requirement, flagging a test example as contaminated if its similarity (for instance, cosine similarity between sentence embeddings, or a normalized edit-distance measure) to any training document exceeds a threshold $ au$, trading a higher false-positive rate for the ability to catch paraphrased or lightly modified leakage that exact n-gram matching would miss.
A simplified item-response-theory (IRT) model of benchmark performance treats each test item $j$ as having a difficulty parameter $d_j$ and each model as having a scalar ability parameter $a_i$, with the probability that model $i$ answers item $j$ correctly given by the logistic function
$$ P( ext{correct}_{ij}) = \frac{1}{1 + e^{-(a_i - d_j)}} $$
A benchmark's discriminative power for a population of models can be characterized by the variance of observed accuracy across those models, $ ext{Var}_i[\hat{P}_i]$, which this model predicts will shrink toward zero as the item difficulties $d_j$ fall increasingly far below the models' ability range $a_i$, since $P( ext{correct}_{ij}) o 1$ for essentially all models regardless of their relative ability differences, precisely the mathematical signature of benchmark saturation.
For evaluating whether an observed accuracy difference between two models is statistically meaningful, a standard approach treats each model's per-item correctness as a Bernoulli-distributed outcome and constructs a bootstrap confidence interval by resampling the observed per-item results with replacement $B$ times, computing the accuracy of each resample, and taking the resulting empirical distribution's percentiles (commonly the 2.5th and 97.5th percentiles, for a 95 percent confidence interval) as the interval bounds, providing a distribution-free estimate of the sampling uncertainty inherent in any finite-size evaluation set.
## Advanced Theory & Extensions
Canary strings, short, unique, randomly generated identifier sequences deliberately inserted into a benchmark's official release, provide a more targeted contamination detection mechanism than statistical n-gram overlap: if a canary string associated with a specific benchmark is later found within a model's training corpus (or, indirectly, if the model can be prompted to reproduce the canary string, suggesting it was memorized during training), this constitutes strong, low-false-positive evidence that the associated benchmark data was included in that training run, and canary strings have been adopted by several major benchmark releases specifically to make future contamination auditing more tractable.
Dynamic and continuously refreshed benchmarks, which periodically introduce new test items (for instance, drawn from recent real-world events or newly authored problems) and retire or de-emphasize older items presumed to have leaked into training corpora over time, directly address contamination by design rather than attempting to detect it after the fact, trading the stability and cross-time comparability of a fixed benchmark for continued measurement validity as models and their training corpora evolve. Held-out or private test sets, whose ground-truth answers are never publicly released and are instead scored via a submission and leaderboard mechanism, similarly aim to prevent contamination structurally, though this approach requires trusted infrastructure to administer and does not eliminate the risk that portions of the private set are inadvertently or adversarially exposed over time.
Pairwise preference aggregation platforms, in which human raters or an LLM judge compare two model outputs head-to-head across many models and many prompts, commonly convert the resulting win/loss records into an Elo-style rating using the same logistic pairwise-comparison model originally developed for chess ratings, $P( ext{A beats B}) = 1 / (1 + 10^{(R_B - R_A)/400})$, providing a single continuously updated scalar ranking across an open-ended and growing pool of models, though such systems inherit their own methodological subtleties around sampling bias in which models and prompts get compared how often, and around the stability of ratings as new models are continuously added to the pool.
## Computational Considerations
Exact n-gram overlap contamination detection is computationally straightforward but requires building and querying a set (or more scalably, a hash-based or suffix-array-based index) over the entire training corpus's n-grams, which for corpora spanning trillions of tokens represents a substantial one-time indexing cost, typically justified by allowing arbitrarily many downstream benchmark contamination checks to be run cheaply against the same precomputed index rather than rescanning the raw corpus for each new benchmark. Embedding-based fuzzy contamination detection trades this indexing cost for a per-comparison embedding and similarity-search cost, which, at the scale of a full pretraining corpus, generally requires an approximate nearest-neighbor search infrastructure rather than brute-force pairwise comparison, given the impracticality of computing similarity between every test item and every training document directly.
LLM-as-judge evaluation's computational cost scales with the number of items evaluated multiplied by the cost of one or more judge model inference calls per item (often two calls per pairwise comparison, to counterbalance position bias by evaluating both presentation orders), which, while far cheaper than the equivalent scale of human annotation, remains non-trivial at the scale of comprehensive multi-model, multi-benchmark evaluation suites, motivating techniques such as using a smaller or distilled judge model for coarse filtering before applying a more expensive judge only to close or contested comparisons. Bootstrap confidence interval computation requires resampling and rescoring an evaluation set many times (commonly a thousand or more resamples), which is computationally cheap relative to the original model inference cost that produced the per-item results being resampled, since it operates purely on already-computed correctness labels rather than requiring any additional model calls.
## Practical Implementation Strategies
Any benchmark comparison intended to support a genuine capability claim should be accompanied by a contamination check against the specific model's known or inferred training data composition wherever this is feasible, and, at minimum, practitioners should treat benchmarks that have been publicly available for a long period, and are highly likely to have propagated into widely used pretraining corpora, with appropriately increased skepticism relative to more recently introduced or deliberately access-controlled benchmarks. When using LLM-as-judge evaluation, systematically counterbalancing presentation order (evaluating both orderings of every pairwise comparison and averaging, or explicitly correcting for measured position bias) and controlling for response length (either by explicit length-normalization in the judge's rubric or by reporting length-controlled win rates alongside raw win rates) are now considered standard practice rather than optional refinements, given how well-documented these specific biases are.
Reporting practices should favor confidence intervals or explicit significance testing over bare point-estimate accuracy figures whenever comparing models, particularly on benchmarks with modest item counts (a few hundred items or fewer), since the resulting sampling uncertainty is frequently large enough to render an apparent ranking between two close models statistically indistinguishable from noise; this is especially important for benchmark leaderboards, where a difference of a percentage point or two between adjacent entries is often well within the noise floor of the underlying evaluation set size. Practitioners should also actively monitor a benchmark's discriminative health over time, tracking the spread of scores across the current population of evaluated models, and treat a benchmark whose leading models cluster within a narrow band near its ceiling as providing diminishing evidentiary value for further capability claims, prompting a transition toward harder or refreshed evaluation material.
## Benchmark Datasets & Evaluation
MMLU (Massive Multitask Language Understanding) and its harder successor MMLU-Pro remain widely reported broad academic-knowledge benchmarks, though MMLU in particular has become a frequently cited example of benchmark saturation among frontier models, with leading systems clustering closely near its ceiling, a dynamic that directly motivated MMLU-Pro's introduction of harder questions and a larger answer-choice set specifically to restore discriminative power. HELM (Holistic Evaluation of Language Models) takes a deliberately multi-metric approach, reporting not just accuracy but calibration, robustness, fairness, efficiency, and other axes across a standardized suite of scenarios, aiming to counteract the tendency of single-number benchmark leaderboards to obscure important differences in model behavior that aggregate accuracy alone does not capture.
Chatbot Arena and similar large-scale pairwise human-preference platforms, in which anonymous human raters compare live outputs from two randomly selected models and vote for a preferred response, converted into continuously updated Elo-style ratings, have become an influential complement to static benchmark scores specifically because they measure open-ended, realistic usage preferences that fixed-answer benchmarks cannot capture, though they introduce their own methodological considerations around rater population representativeness, prompt distribution bias (since raters self-select what to ask), and the previously discussed length and stylistic biases when LLM judges are used to scale up or substitute for human raters. Contamination-resistant benchmarks, including recently introduced private or continuously refreshed evaluation sets and benchmarks built specifically from post-training-cutoff data (such as recent competitive programming or mathematics competition problems), have grown in prominence specifically as a response to well-documented contamination findings on older, longstanding public benchmarks.
## Key Challenges & Limitations
Contamination detection itself has real limitations: exact n-gram matching misses paraphrased, translated, or lightly reformatted leakage entirely, while more permissive fuzzy or embedding-based matching risks false positives that flag genuinely novel test items sharing superficial similarity with unrelated training content, and no current detection method can rule out contamination with certainty, particularly for proprietary models whose training corpora are not disclosed, meaning contamination assessment is necessarily probabilistic and partial rather than a definitive audit. LLM-as-judge evaluation's known biases, position, length, and self-preference among them, are measurable and partially correctable, but the correction techniques themselves rest on assumptions (such as that swapping presentation order isolates position bias cleanly from any genuine order-dependent quality difference) that do not perfectly hold in every evaluation setting, meaning residual, harder-to-detect biases likely remain even in carefully designed judge-based evaluation pipelines.
Benchmark saturation creates a persistent treadmill problem: as soon as a benchmark becomes widely used and models are optimized against it, whether through direct contamination, indirect optimization pressure from a research community iterating against a visible leaderboard, or genuine capability improvement, its useful discriminative lifetime is finite, requiring continual investment in new, harder evaluation material simply to maintain the field's ability to measure genuine progress, a cost that is easy to underinvest in relative to the more immediately visible work of building new models. Single-number leaderboard rankings, even when statistically well-supported, can obscure important multidimensional differences between models, including differences in reliability across subpopulations, robustness to adversarial or out-of-distribution inputs, and calibration, which a bare accuracy or Elo ranking does not surface, motivating the more holistic but also more complex multi-metric evaluation approaches that remain less commonly adopted than simpler single-score leaderboards due to their added complexity and interpretive burden.
## Hyperparameter Tuning
The n-gram length used for exact-match contamination detection trades false-positive against false-negative rates: shorter n-grams (for instance, 5 to 8 tokens) catch more paraphrased or lightly modified leakage but risk flagging coincidental short overlaps as contamination, particularly for common phrasings or formulaic benchmark question structures, while longer n-grams (13 tokens or more, a common choice in published contamination studies) sharply reduce false positives at the cost of missing any leakage that involved even minor rewording. The similarity threshold in embedding-based fuzzy contamination detection similarly trades sensitivity against specificity, and is typically calibrated empirically against a held-out set of known-contaminated and known-clean examples rather than fixed a priori, since the appropriate threshold varies with the embedding model and domain in use.
The number of bootstrap resamples used to construct confidence intervals around benchmark accuracy affects the stability of the resulting interval estimate rather than its expected width, and values in the range of one to ten thousand resamples are standard, since the marginal precision gained from additional resamples beyond this range is small relative to the added computation. For LLM-as-judge pipelines, the choice of judge model capability relative to the models being evaluated matters substantially: judges that are themselves not meaningfully more capable than the models under evaluation tend to produce noisier and more bias-prone judgments, motivating the common practice of reserving the most capable available model specifically for the judge role even when evaluating other strong models, and periodically validating judge outputs against a sample of human ratings to confirm the judge's assessments remain well correlated with genuine human preference.
## Real-World Applications & Case Studies
Several widely publicized instances of benchmark contamination in released models, discovered through n-gram overlap audits comparing training corpus snapshots against popular benchmark test sets, have led major model developers to publish explicit contamination analyses alongside model releases and, in some cases, to deliberately filter identified benchmark content out of training corpora, or to report both an original and a decontaminated benchmark score to demonstrate the magnitude of any contamination effect. The GSM8K grade-school math benchmark's widely observed near-saturation among frontier models, alongside documented evidence of contamination given its long public availability, directly motivated the development and adoption of harder, less contaminated mathematical reasoning benchmarks explicitly designed to restore discriminative power at the frontier.
Chatbot Arena's public, continuously updated Elo leaderboard has become an influential reference point cited alongside or instead of static academic benchmark scores in model release announcements, illustrating the field's broader shift toward incorporating large-scale human preference signal directly into how model capability is publicly communicated, notwithstanding the platform's own documented susceptibility to prompt-distribution and rater-population biases relative to any specific target deployment use case. Enterprise and research evaluation pipelines increasingly combine several of the techniques discussed here into a single evaluation protocol, running contamination checks, reporting confidence intervals rather than bare scores, and using length-and-position-corrected LLM-as-judge scoring alongside targeted human evaluation for high-stakes comparisons, reflecting a broader maturation of evaluation practice as the field has become more attuned to the ways any single one of these techniques can mislead in isolation.
## Integration with Other Methods
Evaluation methodology interacts closely with hallucination detection and mitigation research, since many hallucination benchmarks face the same contamination and saturation concerns discussed here, and robust hallucination measurement similarly benefits from confidence-interval-aware reporting and from LLM-as-judge or human-verification pipelines subject to the same length and position bias corrections. Chain-of-thought and test-time compute scaling research depends heavily on reliable evaluation to demonstrate genuine reasoning improvement rather than benchmark-specific overfitting, making contamination-resistant, appropriately difficult benchmarks especially important for this line of work, where the entire premise, that additional inference-time computation genuinely improves reasoning rather than merely better matching memorized patterns, requires strong evaluation methodology to support convincingly.
Model merging and other training-free capability-combination techniques rely on benchmark evaluation to determine whether a merged model has genuinely retained multiple constituent capabilities, making the statistical rigor and contamination-awareness discussed here directly relevant to assessing merge quality claims. Red-teaming and adversarial robustness evaluation, while methodologically distinct from standard capability benchmarking, shares the core statistical concern that a robustness claim based on a small adversarial test set is subject to the same sampling-uncertainty caveats as any other benchmark claim, and increasingly incorporates confidence-interval reporting and contamination-awareness (given that some adversarial prompt datasets are themselves publicly available and therefore contamination-prone) as standard practice.
## Future Research Directions
Continued development of contamination-resistant evaluation infrastructure, including wider adoption of canary strings, dynamically refreshed benchmark content, and privately held evaluation sets with controlled, audited access, remains a priority as training corpora continue to grow and the practical difficulty of maintaining any genuinely held-out, uncontaminated public benchmark increases correspondingly. Improving the reliability of LLM-as-judge evaluation, through better-calibrated judge models, more systematic bias correction techniques beyond simple order-counterbalancing, and clearer characterization of exactly when judge-based evaluation reliably tracks human preference versus when it diverges, remains an active area given how central this evaluation paradigm has become to modern model development pipelines.
Standardized statistical reporting norms for benchmark comparisons, including default confidence-interval or significance-testing requirements for leaderboard submissions and clearer community conventions around what constitutes a meaningful, non-noise-level capability difference, would directly address the currently common practice of reporting and interpreting small point-estimate differences as meaningful without accompanying uncertainty quantification. Finally, richer, more holistic multi-dimensional evaluation frameworks that move beyond single aggregate scores, capturing calibration, robustness, fairness, and reliability across realistic deployment-relevant subpopulations and conditions, remain comparatively underdeveloped relative to simpler accuracy-style benchmarks, and closing this gap is broadly viewed as necessary for evaluation methodology to keep pace with the increasingly high-stakes, real-world deployment contexts in which large language models are now used.
## Summary & Key Takeaways
Reliable measurement of large language model capability requires addressing several distinct methodological threats simultaneously: benchmark contamination, in which test data leaks into training corpora and inflates scores without reflecting genuine capability, is detectable through techniques such as n-gram overlap analysis, canary strings, and dynamically refreshed evaluation content, though no current method can rule it out with full certainty. LLM-as-judge and human-preference-based evaluation, increasingly central given the difficulty of scoring open-ended generation with simple exact-match metrics, must be actively corrected for well-documented position, length, and self-preference biases rather than trusted as an unbiased ground truth, and benchmark saturation, in which a fixed benchmark's discriminative power collapses as models cluster near its ceiling, requires continual investment in harder or refreshed evaluation material to sustain measurement validity over time. Finally, any benchmark comparison should be accompanied by appropriate statistical uncertainty quantification, since finite evaluation sets, particularly smaller ones, frequently cannot support the precision implied by bare point-estimate leaderboard rankings, and treating close scores as meaningfully different without confidence intervals or significance testing remains one of the most common and avoidable errors in reported LLM evaluation results.
Keywords: LLM evaluation, benchmarking, benchmark contamination, LLM-as-judge, position bias, length bias, benchmark saturation, MMLU, HELM, Chatbot Arena, Elo rating, canary strings, confidence intervals, n-gram overlap, item response theory, bootstrap resampling
---
## Appendix: Practical Labs
### Lab 1: Detecting Benchmark Contamination via N-Gram Overlap
This lab implements exact n-gram overlap contamination detection, building an n-gram index from a simulated training corpus and verifying it correctly separates verbatim-leaked test examples from clean, freshly sampled ones.
import numpy as np
def build_ngram_set(corpus_text, n=13):
words = corpus_text.split()
return set(tuple(words[i:i + n]) for i in range(len(words) - n + 1))
def detect_contamination_ngram_overlap(test_examples, train_corpus_ngrams, n=13):
flags = []
for ex in test_examples:
words = ex.split()
contaminated = False
for i in range(len(words) - n + 1):
if tuple(words[i:i + n]) in train_corpus_ngrams:
contaminated = True
break
flags.append(contaminated)
return flags
def test_ngram_overlap_detects_verbatim_contamination():
rng = np.random.default_rng(0)
vocab = [f"word{i}" for i in range(200)]
def random_sentence(length):
return " ".join(rng.choice(vocab, size=length))
train_corpus = " ".join(random_sentence(20) for _ in range(200))
ngram_set = build_ngram_set(train_corpus, n=13)
# clean test examples: freshly sampled, essentially guaranteed not to
# match any 13-gram in the training corpus
clean_examples = [random_sentence(25) for _ in range(30)]
# contaminated test examples: extracted verbatim from the training
# corpus, simulating a benchmark that leaked into pretraining data
train_words = train_corpus.split()
contaminated_examples = []
for _ in range(30):
start = rng.integers(0, len(train_words) - 20)
contaminated_examples.append(" ".join(train_words[start:start + 20]))
clean_flags = detect_contamination_ngram_overlap(clean_examples, ngram_set, n=13)
contaminated_flags = detect_contamination_ngram_overlap(contaminated_examples, ngram_set, n=13)
false_positive_rate = np.mean(clean_flags)
true_positive_rate = np.mean(contaminated_flags)
print(f"false positive rate (clean flagged as contaminated): {false_positive_rate:.3f}")
print(f"true positive rate (contaminated correctly flagged): {true_positive_rate:.3f}")
assert false_positive_rate < 0.05
assert true_positive_rate > 0.95
print("N-gram contamination detection test passed.")
if __name__ == "__main__":
test_ngram_overlap_detects_verbatim_contamination()### Lab 2: LLM-as-Judge Position and Length Bias
This lab simulates an LLM judge comparing two equal-quality responses and demonstrates two well-documented systematic biases: a preference for whichever response is presented first, and a preference for longer responses, both independent of true quality.
import numpy as np
def simulate_judge_preference(quality_a, quality_b, position_bias=0.0,
length_a=0, length_b=0, length_bias=0.0, rng=None):
logit = (quality_a - quality_b) + position_bias + length_bias * (length_a - length_b)
p_prefers_a = 1 / (1 + np.exp(-logit))
return rng.random() < p_prefers_a
def test_llm_judge_position_and_length_bias_distort_preferences():
rng = np.random.default_rng(0)
n_trials = 4000
# ground truth: two responses of EQUAL quality, so an unbiased judge
# should prefer each ~50% of the time
quality_a, quality_b = 0.0, 0.0
prefers_a_unbiased = [
simulate_judge_preference(quality_a, quality_b, position_bias=0.0,
length_bias=0.0, rng=rng)
for _ in range(n_trials)
]
rate_unbiased = np.mean(prefers_a_unbiased)
prefers_a_position_biased = [
simulate_judge_preference(quality_a, quality_b, position_bias=0.8,
length_bias=0.0, rng=rng)
for _ in range(n_trials)
]
rate_position_biased = np.mean(prefers_a_position_biased)
prefers_a_length_biased = [
simulate_judge_preference(quality_a, quality_b, position_bias=0.0,
length_a=300, length_b=50, length_bias=0.01, rng=rng)
for _ in range(n_trials)
]
rate_length_biased = np.mean(prefers_a_length_biased)
print(f"prefer-A rate: unbiased={rate_unbiased:.3f} position-biased={rate_position_biased:.3f} "
f"length-biased={rate_length_biased:.3f}")
assert abs(rate_unbiased - 0.5) < 0.05
assert rate_position_biased > 0.6
assert rate_length_biased > 0.6
# swapping presentation order should flip a position-biased judge's
# preference, confirming order-sensitivity as the diagnostic for the bias
prefers_b_when_b_shown_first = [
simulate_judge_preference(quality_b, quality_a, position_bias=0.8,
length_bias=0.0, rng=rng)
for _ in range(n_trials)
]
rate_b_first = np.mean(prefers_b_when_b_shown_first)
print(f"prefer-first-shown rate when B shown first: {rate_b_first:.3f}")
assert rate_b_first > 0.6
print("LLM-as-judge bias test passed.")
if __name__ == "__main__":
test_llm_judge_position_and_length_bias_distort_preferences()### Lab 3: Benchmark Saturation Collapses Discriminative Power
This lab uses a simplified item-response-theory simulation to show that as a benchmark's items become much easier relative to model ability (saturation), the variance in observed accuracy across models collapses and rank correlation with true underlying ability degrades.
import numpy as np
def spearman_corr(x, y):
def rank(a):
order = np.argsort(a)
ranks = np.empty(len(a))
ranks[order] = np.arange(len(a))
return ranks
return np.corrcoef(rank(x), rank(y))[0, 1]
def simulate_benchmark_accuracy(abilities, item_difficulties, rng):
accs = []
for a in abilities:
p_correct = 1 / (1 + np.exp(-(a - item_difficulties)))
responses = rng.random(len(item_difficulties)) < p_correct
accs.append(responses.mean())
return np.array(accs)
def test_benchmark_saturation_collapses_discriminative_power():
rng = np.random.default_rng(0)
n_models = 30
n_items = 500
abilities = rng.normal(loc=2.0, scale=1.0, size=n_models)
# discriminating benchmark: item difficulties centered near model abilities
hard_items = rng.normal(loc=2.0, scale=1.5, size=n_items)
acc_hard = simulate_benchmark_accuracy(abilities, hard_items, rng)
# saturated benchmark: items far easier than model abilities, plus a
# smaller item count -- both realistic contributors to near-ceiling,
# noise-dominated accuracy
n_items_easy = 80
easy_items = rng.normal(loc=-4.0, scale=1.0, size=n_items_easy)
acc_easy = simulate_benchmark_accuracy(abilities, easy_items, rng)
var_hard = np.var(acc_hard)
var_easy = np.var(acc_easy)
mean_hard = np.mean(acc_hard)
mean_easy = np.mean(acc_easy)
print(f"discriminating benchmark: mean_acc={mean_hard:.3f} variance={var_hard:.6f}")
print(f"saturated benchmark: mean_acc={mean_easy:.3f} variance={var_easy:.6f}")
assert mean_easy > 0.95
assert var_easy < var_hard * 0.15
corr_hard = spearman_corr(abilities, acc_hard)
corr_easy = spearman_corr(abilities, acc_easy)
print(f"Spearman corr with true ability: discriminating={corr_hard:.3f} saturated={corr_easy:.3f}")
assert corr_hard > 0.85
assert corr_easy < corr_hard - 0.2
print("Benchmark saturation test passed.")
if __name__ == "__main__":
test_benchmark_saturation_collapses_discriminative_power()### Lab 4: Confidence Intervals Reveal When Score Differences Are Noise
This lab bootstraps confidence intervals around two models' benchmark accuracy at small and large evaluation set sizes, showing that a real accuracy gap can be statistically indistinguishable from noise on a small benchmark but clearly resolved on a larger one.
import numpy as np
def bootstrap_ci(correct, n_bootstrap=2000, rng=None):
n = len(correct)
boot_means = np.empty(n_bootstrap)
for b in range(n_bootstrap):
sample = rng.choice(correct, size=n, replace=True)
boot_means[b] = sample.mean()
lo, hi = np.percentile(boot_means, [2.5, 97.5])
return lo, hi
def test_small_benchmark_sample_size_produces_overlapping_confidence_intervals():
rng = np.random.default_rng(0)
true_acc_A = 0.62
true_acc_B = 0.58
small_overlap = large_overlap = None
small_width_A = large_width_A = None
for n_items, label in [(50, "small"), (5000, "large")]:
correct_A = (rng.random(n_items) < true_acc_A).astype(float)
correct_B = (rng.random(n_items) < true_acc_B).astype(float)
obs_acc_A = correct_A.mean()
obs_acc_B = correct_B.mean()
ci_A = bootstrap_ci(correct_A, rng=rng)
ci_B = bootstrap_ci(correct_B, rng=rng)
overlap = not (ci_A[1] < ci_B[0] or ci_B[1] < ci_A[0])
print(f"[{label} benchmark, n={n_items}] acc_A={obs_acc_A:.3f} "
f"CI={tuple(round(x, 3) for x in ci_A)} acc_B={obs_acc_B:.3f} "
f"CI={tuple(round(x, 3) for x in ci_B)} overlap={overlap}")
if label == "small":
small_overlap = overlap
small_width_A = ci_A[1] - ci_A[0]
else:
large_overlap = overlap
large_width_A = ci_A[1] - ci_A[0]
print(f"CI width shrinks with sample size: small={small_width_A:.3f} large={large_width_A:.3f}")
assert large_width_A < small_width_A * 0.3
assert not large_overlap, "with enough items, a real accuracy gap should produce non-overlapping CIs"
print("Bootstrap confidence interval test passed.")
if __name__ == "__main__":
test_small_benchmark_sample_size_produces_overlapping_confidence_intervals()