SWE-bench
SWE-bench is a coding benchmark built from real GitHub issues in popular Python projects; a model or agent must edit the repository so that the issue is fixed and the project's hidden tests pass.
Also called: SWE-bench Verified, SWE-bench Lite
By Divya Sharma · Checked Nov 15, 2026 · How we review
Older coding tests asked a model to write one short function. SWE-bench asks for what developers actually do: read an issue, find the right files in a large codebase, and change them. The original paper collected 2,294 problems from real issues and their pull requests across 12 Python repositories. The fix often touches several functions or files, so it rewards agents that can search, run code and iterate, not just autocomplete.
How it is scored
Each task comes with the repository at the commit before the fix and the tests that the real pull request made pass. The system under test produces a patch. The harness applies it and runs the tests. The score is the share of issues resolved. When the paper came out, the best model it tested solved under 2% of issues. Check current leaderboards for how coding agents score today.
SWE-bench Verified
SWE-bench Verified is a subset of 500 tasks from the original test set that humans checked for quality. Other variants include Lite, Multilingual and Multimodal, listed on the official leaderboards.
How to read a score
The agent scaffold matters as much as the model: the same model scores differently with different tools, prompts and step limits. Check which variant was used and whether the result came from the lab or an independent run. Because the issues are public, benchmark contamination is a real concern. mini-SWE-agent is a small agent listed as part of the SWE-bench family of projects. For terminal-heavy work, see Terminal-Bench.
Related terms
- Coding agent
- A coding agent is an AI agent that works on software directly: it reads a codebase, edits files, runs commands and tests, and repeats until a change works, rather than only suggesting code for a person to paste.
- AI benchmark
- An AI benchmark is a fixed, public set of tasks with an agreed scoring method, used to compare models on the same test, for example coding issues solved, exam questions answered or retrieval quality across many datasets.
- Terminal-Bench
- Terminal-Bench is a benchmark for AI agents that work in a command-line terminal, where each task gives the agent a real environment and is scored by tests that check whether the job was actually done.
- Benchmark contamination
- Benchmark contamination is when a model has seen a benchmark's test questions or answers during training, so its score on that benchmark overstates how well it handles new problems of the same kind.
- pass@k
- pass@k is a metric for code benchmarks that gives the chance at least one of k generated solutions to a problem passes its unit tests; pass@1 is the chance that a single attempt succeeds.