LLM evaluation
LLM evaluation, often called evals, is measuring how well a model or AI application performs a task by running it on a set of test cases and scoring the outputs against defined criteria, by code, by humans or by another model.
Also called: evals, LLM evals, model evaluation
By Divya Sharma · Checked Nov 15, 2026 · How we review
Changing a prompt, swapping a model or adding a retrieval step can make an application better on one input and worse on others. Evals are how you find out before users do. Anthropic's documentation puts it simply: define success criteria first, then design evaluations to measure against them.
How it works
An eval has three parts. A dataset of inputs, ideally drawn from real use and including edge cases, often kept as a golden dataset with reference answers. A way to run the system on each input. And a grader. Graders range from exact match and code checks (does the JSON parse, does the test pass) to similarity scores, human review and LLM-as-a-judge, where a second model scores the answer against a rubric. Anthropic's guide advises structuring questions so grading can be automated, and favouring more test cases over fewer hand-graded ones.
Model evals vs application evals
Public AI benchmarks evaluate models on shared tasks, which helps you pick a shortlist. Application evals use your own data, prompts and tools, which tells you whether the system works for your users. The HELM project argued for measuring models on many metrics at once, such as accuracy, calibration and robustness, rather than accuracy alone; the same idea applies to your own evals.
In practice
Teams run evals in CI on every prompt or model change, a practice called LLM regression testing, and add failures found in production traces as new cases. Open-source frameworks include promptfoo, DeepEval and Ragas. The evaluation stage guide compares tools. Evals for safety, where the inputs are attacks, are usually called red teaming.
Related terms
- LLM-as-a-judge
- LLM-as-a-judge is an evaluation method where a strong language model grades the outputs of another model or application, using a rubric, a reference answer or a side-by-side comparison, in place of or alongside human reviewers.
- AI benchmark
- An AI benchmark is a fixed, public set of tasks with an agreed scoring method, used to compare models on the same test, for example coding issues solved, exam questions answered or retrieval quality across many datasets.
- AI red teaming
- AI red teaming is the practice of deliberately attacking a model or AI application before and after release, with human testers or automated tools, to find harmful outputs, jailbreaks, injections and other failures so they can be fixed.
- Golden dataset
- A golden dataset is a fixed, carefully checked set of example inputs paired with the answers or behaviour you expect, used as the reference when you evaluate or regression-test an AI system.
- Regression testing (LLM apps)
- Regression testing for LLM apps means re-running a fixed set of evaluation cases after every change to a prompt, model, retrieval setup or tool, to catch cases that used to work and now fail.