LLM-as-a-judge
LLM-as-a-judge is an evaluation method where a strong language model grades the outputs of another model or application, using a rubric, a reference answer or a side-by-side comparison, in place of or alongside human reviewers.
Also called: judge model, LLM judge, model-graded eval
By Divya Sharma · Checked Nov 15, 2026 · How we review
Many AI outputs cannot be checked with exact match: a summary, a support reply, a piece of advice. Human review is the gold standard but slow and expensive. A judge model can score thousands of outputs quickly, which makes it a common grader in LLM evaluation.
How it works
You give the judge the input, the output to grade and instructions. Common formats are a single score against a rubric (for example, is the answer faithful to the source, 1 to 5), pass or fail on a specific rule, or a pairwise choice between two answers. Asking the judge to explain its reasoning before giving a score makes its decisions easier to audit.
Zheng et al. named and studied the method using MT-Bench and Chatbot Arena. They reported that strong judges such as GPT-4 reached over 80% agreement with human preferences, about the level at which humans agree with each other.
Known biases
The same paper documented problems you should design around. Position bias: preferring whichever answer comes first. Verbosity bias: preferring longer answers. Self-enhancement bias: preferring answers written by the judge's own model family. Judges also struggle to grade maths and reasoning they cannot do themselves. Common fixes are swapping answer order and averaging, giving the judge reference answers, and using a different model family as judge.
Keep humans in the loop
Check a sample of judge scores against human evaluation before trusting them, and recheck when you change the judge model or prompt. A golden dataset with human labels makes this calibration repeatable. Frameworks such as DeepEval and promptfoo ship ready-made judge metrics.
Related terms
- LLM evaluation
- LLM evaluation, often called evals, is measuring how well a model or AI application performs a task by running it on a set of test cases and scoring the outputs against defined criteria, by code, by humans or by another model.
- Human evaluation
- Human evaluation is judging AI outputs with people rather than automated metrics, for example raters scoring answers against a rubric or picking the better of two anonymous responses in a blind test.
- Golden dataset
- A golden dataset is a fixed, carefully checked set of example inputs paired with the answers or behaviour you expect, used as the reference when you evaluate or regression-test an AI system.
- LMArena
- LMArena, first known as Chatbot Arena, is a public platform that ranks AI models by crowdsourced human votes: users see two anonymous models answer the same prompt and pick the better response.