HumanEval
HumanEval is a code-generation benchmark released by OpenAI in 2021: 164 hand-written Python problems, each with a function signature, docstring and unit tests, used to check whether a model writes code that actually works.
Also called: OpenAI HumanEval
By Divya Sharma · Checked Nov 15, 2026 · How we review
HumanEval came out with the Codex paper; a production version of Codex powered GitHub Copilot. Each problem gives the model a Python function signature and a docstring that describes what the function should do. The model writes the body, and the benchmark runs unit tests on it. This measures functional correctness: the code either passes the tests or it does not, regardless of how it looks.
How it is scored
Results are reported with pass@k, usually pass@1. At release, Codex solved 28.8% of the problems in one try, while GPT-3 solved none. The problems were written by hand so that they would not already be in the code that models were trained on.
Limits
HumanEval problems are short, self-contained functions, closer to interview questions than to real software work. A high score says little about fixing a bug in a large codebase, running commands or working across many files. That is what SWE-bench and Terminal-Bench try to measure. Because the problems have been public since 2021, they may also have leaked into newer training data, a problem known as benchmark contamination. Treat a HumanEval score as a basic floor for coding ability, and test a coding agent on your own tasks before choosing one.
Related terms
- pass@k
- pass@k is a metric for code benchmarks that gives the chance at least one of k generated solutions to a problem passes its unit tests; pass@1 is the chance that a single attempt succeeds.
- SWE-bench
- SWE-bench is a coding benchmark built from real GitHub issues in popular Python projects; a model or agent must edit the repository so that the issue is fixed and the project's hidden tests pass.
- Benchmark contamination
- Benchmark contamination is when a model has seen a benchmark's test questions or answers during training, so its score on that benchmark overstates how well it handles new problems of the same kind.
- AI benchmark
- An AI benchmark is a fixed, public set of tasks with an agreed scoring method, used to compare models on the same test, for example coding issues solved, exam questions answered or retrieval quality across many datasets.
- Coding agent
- A coding agent is an AI agent that works on software directly: it reads a codebase, edits files, runs commands and tests, and repeats until a change works, rather than only suggesting code for a person to paste.