Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

Elo rating (model arenas)

An Elo rating, in model arenas, is a score computed from many head-to-head votes between pairs of models, so that the gap between two scores predicts how often one model's answer will be preferred over the other's.

Also called: arena score, Bradley-Terry, Elo score, Bradley-Terry model

By Divya Sharma · Checked Nov 15, 2026 · How we review

The Elo system was designed to rank chess players from game results. Model arenas borrowed it because their data looks the same: two models answer one prompt, a person picks the better reply, and the result counts as a win, loss or tie.

How it works

Each model gets a number. If a high-rated model beats a low-rated one, both scores move a little; an upset moves them more. Over thousands of votes the numbers settle so that the difference between two scores maps to an expected win rate.

Elo vs Bradley-Terry

Classic Elo updates scores one game at a time, so the order of games affects the result. That suits players whose skill changes over time, but a model version does not change. The Chatbot Arena paper explains that the leaderboard moved from Elo scores to Bradley-Terry coefficients, a statistical model of paired comparisons fitted to all votes at once, because they are better for statistical estimation. The paper describes the probability that one model beats another as a logistic function of the gap between their coefficients, and reports confidence intervals for each score. Leaderboards such as LMArena still call the result an arena score or Elo-style rating.

How to read it

The absolute number means nothing on its own; only gaps matter, and only within the same leaderboard. Look at the confidence interval: two models whose intervals overlap are not reliably ranked. Ratings also depend on who votes and what they ask, so an arena score reflects general chat preference, not performance on your task. Pair it with a task-specific AI benchmark and your own human evaluation.

Related terms

Leaderboard (AI)
An AI leaderboard is a public ranking of models by their scores on one or more benchmarks or on human preference votes, updated as new models are tested, so readers can compare models side by side.
LMArena
LMArena, first known as Chatbot Arena, is a public platform that ranks AI models by crowdsourced human votes: users see two anonymous models answer the same prompt and pick the better response.
Human evaluation
Human evaluation is judging AI outputs with people rather than automated metrics, for example raters scoring answers against a rubric or picking the better of two anonymous responses in a blind test.
AI benchmark
An AI benchmark is a fixed, public set of tasks with an agreed scoring method, used to compare models on the same test, for example coding issues solved, exam questions answered or retrieval quality across many datasets.

Sources