Human evaluation
Human evaluation is judging AI outputs with people rather than automated metrics, for example raters scoring answers against a rubric or picking the better of two anonymous responses in a blind test.
Also called: human raters, blind test, human grading, pairwise preference
By Divya Sharma · Checked Nov 15, 2026 · How we review
Many qualities that matter, such as whether an answer is helpful, well written or appropriate, are hard to score with code. Human evaluation asks people. It is the reference that cheaper methods are checked against.
Common formats
In absolute rating, a rater scores one output on a scale or a rubric. In pairwise preference, a rater sees two outputs for the same prompt and picks the better one; this is easier to do consistently. Expert review uses domain specialists, such as doctors or lawyers, where general raters cannot judge accuracy. A blind test hides which model wrote which answer, so brand or expectation does not sway the vote.
Where it shows up
LMArena collects pairwise votes from the public and turns them into a leaderboard; its authors report that crowd votes agree well with expert raters. Model training also uses human preferences, through RLHF. In your own app, a small set of human-graded examples is the best way to check that an LLM-as-a-judge grader agrees with people before you rely on it.
Limits
Anthropic's evaluation guide calls human grading the most flexible and highest quality method, but slow and expensive. Raters disagree, tire and drift, so write clear instructions, measure agreement between raters, and mix in items with known answers from a golden dataset. Preference is also not correctness: a confident, well-formatted answer can win a vote and still be wrong.
Related terms
- LLM evaluation
- LLM evaluation, often called evals, is measuring how well a model or AI application performs a task by running it on a set of test cases and scoring the outputs against defined criteria, by code, by humans or by another model.
- LMArena
- LMArena, first known as Chatbot Arena, is a public platform that ranks AI models by crowdsourced human votes: users see two anonymous models answer the same prompt and pick the better response.
- LLM-as-a-judge
- LLM-as-a-judge is an evaluation method where a strong language model grades the outputs of another model or application, using a rubric, a reference answer or a side-by-side comparison, in place of or alongside human reviewers.
- RLHF
- RLHF (reinforcement learning from human feedback) fine-tunes a language model using human rankings of its answers: a reward model learns those preferences, then reinforcement learning pushes the model toward answers that score well.
- Golden dataset
- A golden dataset is a fixed, carefully checked set of example inputs paired with the answers or behaviour you expect, used as the reference when you evaluate or regression-test an AI system.