Benchmark contamination
Benchmark contamination is when a model has seen a benchmark's test questions or answers during training, so its score on that benchmark overstates how well it handles new problems of the same kind.
Also called: data contamination, test-set leakage
By Divya Sharma · Checked Nov 15, 2026 · How we review
Large models are trained on huge scrapes of the web, and benchmarks live on the web: in papers, GitHub repositories, forum posts and blog write-ups. Test items can end up in the training data without anyone intending it. A model that has memorised answers can then score well without having the skill the benchmark is meant to measure.
Why it matters
Sainz et al. describe the worst case as a model trained on a benchmark's test split and then evaluated on the same benchmark. They argue the extent of the problem is unknown because it is hard to measure, and that contamination leads to overestimated performance and, in research, to wrong conclusions. For a builder, it means a headline AI benchmark score may not carry over to your own tasks.
Signs and checks
Warning signs include a model reproducing a test question word for word from a partial prompt, much higher scores on an old benchmark than on a newer one of similar difficulty, and a sharp drop when questions are lightly reworded. Labs and researchers search training data for overlapping text, compare results on items published before and after a model's knowledge cutoff, and keep private held-out test sets.
How benchmarks respond
Newer benchmarks hide part of their test set, refresh questions over time, or draw tasks from material created after the models being tested. Human-preference leaderboards use fresh prompts from users, which avoids fixed test sets but brings other biases.
The practical defence is the same as always: run your own LLM evaluation on data the model could not have seen.
Related terms
- AI benchmark
- An AI benchmark is a fixed, public set of tasks with an agreed scoring method, used to compare models on the same test, for example coding issues solved, exam questions answered or retrieval quality across many datasets.
- AI training data
- AI training data is the collection of examples a model learns from, such as web text, code, images, transcripts or labelled pairs. Its size, quality, mix and licensing shape what the model can do and where it fails.
- Knowledge cutoff
- A knowledge cutoff is the date after which a model has little or no information, because its training data was collected up to around then. Events after it are unknown unless supplied in the prompt.
- Leaderboard (AI)
- An AI leaderboard is a public ranking of models by their scores on one or more benchmarks or on human preference votes, updated as new models are tested, so readers can compare models side by side.
- LLM evaluation
- LLM evaluation, often called evals, is measuring how well a model or AI application performs a task by running it on a set of test cases and scoring the outputs against defined criteria, by code, by humans or by another model.