pass@k
pass@k is a metric for code benchmarks that gives the chance at least one of k generated solutions to a problem passes its unit tests; pass@1 is the chance that a single attempt succeeds.
Also called: pass@1, pass@10, pass@100
By Divya Sharma · Checked Nov 15, 2026 · How we review
Code can be checked by running it. So instead of comparing a model's code with a reference answer, coding benchmarks such as HumanEval run unit tests and count a problem as solved if the code passes. pass@k asks: if the model gets k tries, how likely is it that at least one works?
How it is calculated
The metric was used by Kulal and colleagues in 2019 and made standard by OpenAI's 2021 Codex paper. Drawing exactly k samples and checking them gives a noisy number, so the paper generates n samples per problem (it used n = 200), counts the c that pass, and computes an unbiased estimate of pass@k from n, c and k. The paper ships a short numpy function for this.
Reading the numbers
pass@1 matches how most people use a coding assistant: one answer, take it or leave it. pass@10 or pass@100 shows what a model can do with retries and a test suite to pick the winner. Codex solved 28.8% of HumanEval problems at pass@1 and 70.2% with 100 samples, which shows how far the figure moves with k. Compare models only at the same k, sampling temperature and prompt.
pass@k versus pass^k
τ-bench uses a related metric written pass^k, which asks whether an agent succeeds on all k trials of the same task. pass@k rewards a single lucky success; pass^k rewards consistency. For agents that run unattended, pass^k is often the more telling number. See LLM evaluation for how these fit into a wider test plan.
Related terms
- HumanEval
- HumanEval is a code-generation benchmark released by OpenAI in 2021: 164 hand-written Python problems, each with a function signature, docstring and unit tests, used to check whether a model writes code that actually works.
- SWE-bench
- SWE-bench is a coding benchmark built from real GitHub issues in popular Python projects; a model or agent must edit the repository so that the issue is fixed and the project's hidden tests pass.
- LLM evaluation
- LLM evaluation, often called evals, is measuring how well a model or AI application performs a task by running it on a set of test cases and scoring the outputs against defined criteria, by code, by humans or by another model.
- AI benchmark
- An AI benchmark is a fixed, public set of tasks with an agreed scoring method, used to compare models on the same test, for example coding issues solved, exam questions answered or retrieval quality across many datasets.
- τ-bench
- τ-bench (tau-bench) is a benchmark that tests AI agents in simulated customer-service conversations, where a language model plays the user and the agent must use domain API tools while following written policy rules.