Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

GPQA

GPQA (Graduate-Level Google-Proof Q&A) is a benchmark of 448 hard multiple-choice questions in biology, physics and chemistry, written by experts so that skilled non-experts cannot answer them even with unlimited web search.

Also called: Graduate-Level Google-Proof Q&A, GPQA Diamond

By Divya Sharma · Checked Nov 15, 2026 · How we review

GPQA was published in 2023 by David Rein and colleagues. The questions were written and checked by people with or working towards PhDs in each field. To prove the questions are hard, the authors gave them to skilled validators from other fields, who could search the web freely.

The numbers

Domain experts reached 65% accuracy, or 74% after discounting mistakes they spotted afterwards. Non-experts reached only 34%, despite spending more than 30 minutes per question on average with full web access. That is what "Google-proof" means: you cannot look the answer up. The strongest GPT-4 baseline in the paper scored 39%.

GPQA Diamond

The paper releases three sets: an extended set of 546 questions, the main set of 448, and a Diamond set of 198. Diamond keeps only questions where both expert validators agreed with the answer and at most one of three non-experts got it right. Diamond is the version most model makers report.

Why it was built

The authors wanted a test for scalable oversight: how people can check AI answers in areas where the AI may know more than they do. For builders, GPQA is a signal of hard scientific reasoning, often reported alongside MMLU and Humanity's Last Exam. It is multiple choice and small, so a few questions move the score, and it is public, so benchmark contamination is possible. Compare it across models on the rankings rather than reading one score alone.

Related terms

AI benchmark
An AI benchmark is a fixed, public set of tasks with an agreed scoring method, used to compare models on the same test, for example coding issues solved, exam questions answered or retrieval quality across many datasets.
MMLU
MMLU (Massive Multitask Language Understanding) is a multiple-choice benchmark covering 57 subjects, from elementary maths and US history to law and computer science, that tests a language model's broad knowledge and problem solving.
Humanity's Last Exam
Humanity's Last Exam (HLE) is a benchmark of 2,500 expert-written questions across dozens of subjects, built to be very hard for AI models, with answers that are clear and checkable but cannot be found quickly online.
Reasoning model
A reasoning model is a language model trained to work through a problem in a stream of intermediate thinking tokens before giving its final answer, which tends to help on maths, coding and multi-step tasks.
Benchmark contamination
Benchmark contamination is when a model has seen a benchmark's test questions or answers during training, so its score on that benchmark overstates how well it handles new problems of the same kind.

Sources