AI benchmark
An AI benchmark is a fixed, public set of tasks with an agreed scoring method, used to compare models on the same test, for example coding issues solved, exam questions answered or retrieval quality across many datasets.
Also called: benchmark, LLM benchmark, eval suite
By Divya Sharma · Checked Nov 15, 2026 · How we review
Benchmarks give the field a shared yardstick. When a lab says a new model is better at coding, it usually means a higher score on a benchmark such as SWE-bench. Embedding models are compared on MTEB. Scores are collected on leaderboards and on our own rankings.
How to read a benchmark score
Ask what the task actually is. A multiple-choice exam, a set of real GitHub issues and a crowd vote measure very different things. Ask how it was scored: exact answer, unit tests passing, or a model acting as judge. Check the setup: how many attempts were allowed (see pass@k), which tools and prompts were used, and whether the lab ran the test itself or an independent group did. Small differences between models are often within the noise of the test.
Why scores mislead
A benchmark is only useful while models have not seen its answers. If test items leak into training data, scores rise without real improvement; this is benchmark contamination. Popular benchmarks also saturate: once top models all score near the ceiling, the test no longer separates them, and the field moves to a harder one. And a model can be tuned for a benchmark's format without getting better at the underlying skill.
The HELM project argued that a single accuracy number hides too much, and measured models across many scenarios and several metrics, including calibration and robustness.
Benchmarks vs your own evals
Use public benchmarks to build a shortlist. Then run LLM evaluation on your own tasks and data, because that is the only test that tells you how a model will do for your users.
Related terms
- Leaderboard (AI)
- An AI leaderboard is a public ranking of models by their scores on one or more benchmarks or on human preference votes, updated as new models are tested, so readers can compare models side by side.
- Benchmark contamination
- Benchmark contamination is when a model has seen a benchmark's test questions or answers during training, so its score on that benchmark overstates how well it handles new problems of the same kind.
- SWE-bench
- SWE-bench is a coding benchmark built from real GitHub issues in popular Python projects; a model or agent must edit the repository so that the issue is fixed and the project's hidden tests pass.
- MTEB
- MTEB, the Massive Text Embedding Benchmark, is a public benchmark and leaderboard that scores text embedding models across many task types, such as retrieval, clustering, classification and reranking, over many datasets and languages.
- LLM evaluation
- LLM evaluation, often called evals, is measuring how well a model or AI application performs a task by running it on a set of test cases and scoring the outputs against defined criteria, by code, by humans or by another model.
- pass@k
- pass@k is a metric for code benchmarks that gives the chance at least one of k generated solutions to a problem passes its unit tests; pass@1 is the chance that a single attempt succeeds.