Humanity's Last Exam
Humanity's Last Exam (HLE) is a benchmark of 2,500 expert-written questions across dozens of subjects, built to be very hard for AI models, with answers that are clear and checkable but cannot be found quickly online.
Also called: HLE, HLE-Rolling
By Divya Sharma · Checked Nov 15, 2026 · How we review
HLE was released in 2025 after older tests such as MMLU stopped separating the best models; its authors note models score over 90% on such benchmarks. Subject experts around the world wrote the questions, which span mathematics, the humanities and the natural sciences. Some include images, so the test is multimodal.
Format
Questions are either multiple choice or short answer, so they can be graded automatically. Each has one known, unambiguous answer that cannot be found with a quick web search. The authors describe it as meant to be the final closed-ended academic benchmark of its kind. Alongside the public questions, they keep a private held-out set to check whether models have overfitted to the public ones.
Updates
The dataset was finalised at 2,500 questions in April 2025, and the paper was published in Nature in January 2026. The team also runs HLE-Rolling, a dynamic fork that is updated over time. Some model evaluations, including the Artificial Analysis Intelligence Index, use a subset.
What it does and does not show
A high HLE score means a model answers hard expert questions well, often with help from test-time compute. It does not show that a model can do open-ended research or long tasks, which agent benchmarks try to cover. Calibration, whether a model knows when it is likely wrong, is also reported and is often poor. See GPQA for a smaller science-only test.
Related terms
- AI benchmark
- An AI benchmark is a fixed, public set of tasks with an agreed scoring method, used to compare models on the same test, for example coding issues solved, exam questions answered or retrieval quality across many datasets.
- GPQA
- GPQA (Graduate-Level Google-Proof Q&A) is a benchmark of 448 hard multiple-choice questions in biology, physics and chemistry, written by experts so that skilled non-experts cannot answer them even with unlimited web search.
- MMLU
- MMLU (Massive Multitask Language Understanding) is a multiple-choice benchmark covering 57 subjects, from elementary maths and US history to law and computer science, that tests a language model's broad knowledge and problem solving.
- Reasoning model
- A reasoning model is a language model trained to work through a problem in a stream of intermediate thinking tokens before giving its final answer, which tends to help on maths, coding and multi-step tasks.
- Benchmark contamination
- Benchmark contamination is when a model has seen a benchmark's test questions or answers during training, so its score on that benchmark overstates how well it handles new problems of the same kind.