Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

MMLU

MMLU (Massive Multitask Language Understanding) is a multiple-choice benchmark covering 57 subjects, from elementary maths and US history to law and computer science, that tests a language model's broad knowledge and problem solving.

Also called: Massive Multitask Language Understanding, MMLU-Pro

By Divya Sharma · Checked Nov 15, 2026 · How we review

MMLU was published in 2020 by Dan Hendrycks and colleagues. It asks multiple-choice questions, each with four options, across 57 tasks at levels from school to professional. A model's score is its accuracy. For several years it was the headline number in nearly every model launch, because it gave one figure for general knowledge.

Why it matters less now

Benchmarks wear out. The authors of Humanity's Last Exam note that models now score over 90% on popular benchmarks like MMLU, which leaves little room to tell strong models apart. The MMLU-Pro authors found trivial and noisy questions in it, and the test has been public long enough that benchmark contamination is a real concern. A high MMLU score today mostly says a model is not weak.

MMLU-Pro

MMLU-Pro, from 2024, is a harder version. It adds reasoning-focused questions, expands the choices from four to ten so guessing pays less, and removes trivial and noisy questions. Its authors report accuracy drops of 16% to 33% compared with MMLU, less sensitivity to prompt wording, and better results with chain-of-thought reasoning, which suggests it tests reasoning as well as recall.

Using it

Treat MMLU and MMLU-Pro as broad knowledge checks. For hard science questions look at GPQA, and for how a model ranks overall see a leaderboard or the site's rankings.

Related terms

AI benchmark
An AI benchmark is a fixed, public set of tasks with an agreed scoring method, used to compare models on the same test, for example coding issues solved, exam questions answered or retrieval quality across many datasets.
GPQA
GPQA (Graduate-Level Google-Proof Q&A) is a benchmark of 448 hard multiple-choice questions in biology, physics and chemistry, written by experts so that skilled non-experts cannot answer them even with unlimited web search.
Humanity's Last Exam
Humanity's Last Exam (HLE) is a benchmark of 2,500 expert-written questions across dozens of subjects, built to be very hard for AI models, with answers that are clear and checkable but cannot be found quickly online.
Benchmark contamination
Benchmark contamination is when a model has seen a benchmark's test questions or answers during training, so its score on that benchmark overstates how well it handles new problems of the same kind.
Leaderboard (AI)
An AI leaderboard is a public ranking of models by their scores on one or more benchmarks or on human preference votes, updated as new models are tested, so readers can compare models side by side.

Sources