data contamination

Data contamination occurs when test data appears in training data, artificially inflating benchmark scores. **The problem**: Model memorizes test examples rather than learning generalizable skills. Scores dont reflect true capability. **How it happens**: Web scrapes include benchmark data, code repositories contain test cases, documentation quotes examples. Scale of web data makes avoidance difficult. **Detection methods**: N-gram overlap analysis, checking for exact or near-exact matches, timing analysis (correct answers faster if memorized), perplexity analysis on test examples. **High-profile concerns**: GPT-4 evaluation, HumanEval contamination in code models, MMLU leakage. **Mitigation strategies**: **Training side**: Filter training data for benchmark overlap. **Evaluation side**: Create new held-out benchmarks, use canary strings, post-hoc contamination analysis. **Reporting**: Disclose potential contamination, provide contamination analysis, test on truly held-out data. **Industry standards**: Growing expectation to report contamination analysis alongside benchmark results. Critical for trustworthy evaluation.