What it is
The LM Evaluation Harness from EleutherAI runs language models against more than 60 standard academic benchmarks, with hundreds of subtasks and variants, such as MMLU, HellaSwag and GSM8K, using fixed prompts and scoring so results can be compared.
It loads models through Hugging Face Transformers, vLLM, SGLang, GGUF and llama.cpp servers, NVIDIA NeMo and Megatron-LM, ONNX Runtime and others, and calls hosted APIs including OpenAI, Anthropic and anything behind LiteLLM. It has served as the backend for Hugging Face's Open LLM Leaderboard and appears in hundreds of papers.
Version 0.4.13 was published on 31 August 2026; a September 2026 change lets backends, metrics and filters be registered as plugins from your own package rather than a fork.
Our review
The harness measures models, and because so many papers and leaderboards use it, its numbers are among the few that travel between labs. That is its whole value. It is the tool for checking whether a fine-tune damaged general ability, or for comparing two base models on the same prompts; it does not test your prompts, retrieval pipeline or agent.
The largest practical catch is that hosted chat APIs cannot run most of the classic benchmarks as designed. The README lists OpenAI ChatCompletions, Anthropic and LiteLLM as generate-only with no logprobs, so log-likelihood tasks such as multiple-choice MMLU or perplexity need local weights or an API that returns prompt logprobs. For closed models the maintainers recommend inspecting a few outputs with --limit 10 first, because answer extraction on generative tasks can fail quietly.
The project has become lighter to adopt. Since December 2025 the base package no longer pulls in PyTorch or Transformers; you install lm_eval[hf], [vllm] or [api] as needed. Multi-GPU runs work through accelerate or vLLM, but multi-node evaluation is not supported natively.
Activity has cooled slightly: the last commit was on 14 September 2026 and there are about 1,240 open issues and pull requests. Prompt format and few-shot settings move scores, so a number from the harness is only comparable with another produced at the same version and settings. For agent and safety evaluations, labs increasingly reach for Inspect.
Where it earns its place
An ML engineer fine-tuning an 8B model on one 80 GB GPU
Check the fine-tune has not lost general ability
Run the same task set, for example MMLU, HellaSwag and GSM8K, on the base and tuned checkpoints with the vllm backend and --batch_size auto. It costs GPU hours rather than API fees, and --use_cache lets an interrupted run resume.
A research team publishing a new open-weights model
Report benchmark numbers reviewers will accept
Pin the harness version, record few-shot and chat-template settings, and push results and per-sample logs to a Hugging Face dataset with --hf_hub_log_args so others can reproduce them.
An AI platform lead choosing between two hosted models for a procurement decision
Compare vendors on reasoning and maths benchmarks
Use the openai-chat-completions or anthropic-chat backends on generative tasks such as GSM8K. Each benchmark item becomes an API call billed at the vendor's token price, and multiple-choice log-likelihood tasks are unavailable for these APIs.
Before you commit
- Setup
- pip install lm-eval (Python 3.10+) plus an extra such as lm_eval[hf] or lm_eval[vllm]; one lm_eval command with --model, --model_args and --tasks produces a first score. Local models need a GPU sized to the model; lm-eval ls tasks lists everything available.
- Integrations
- Transformers (with PEFT adapters and GPTQ), vLLM, SGLang, llama.cpp, NeMo, Megatron-LM, OpenVINO, AWS Neuron, ONNX Runtime and Windows ML; OpenAI, Anthropic, Watsonx and LiteLLM APIs; any OpenAI-compatible server. Results can go to Weights and Biases, Zeno or the Hugging Face Hub.
- Data and privacy
- Runs locally; data leaves only when you call a hosted model API or push results to the Hub with your own token. Some models need trust_remote_code, which executes model-repo code. No security advisories are published on the repository. MIT licence.
- Cost at scale
- Free software; cost is GPU time for local models or per-token API fees for hosted ones. A full suite over a large model is a substantial GPU job, so teams usually run a fixed subset per checkpoint and use --limit for smoke tests.
- Limits
- Generate-only for chat APIs, perplexity not implemented for llama.cpp servers, Cohere blocked on an API bug, and no native multi-node runs. Score drift from prompt-format changes means results should cite the version used.
What's good
- 60+ benchmarks with hundreds of subtasks
- Widely cited, so scores compare across papers
- Backends from Transformers and vLLM to NeMo, ONNX and LiteLLM
- Plugin system for custom backends and metrics without forking
What's not
- Chat APIs cannot run log-likelihood tasks such as standard MMLU
- Measures models, not your application
- No native multi-node evaluation
- About 1,240 open issues and PRs
Best for
Researchers and model trainers comparing models on standard benchmarks.
Alternatives
LM Evaluation Harness: common questions
- Is LM Evaluation Harness worth it?
- The common yardstick for model benchmarks, and the wrong instrument for judging an application built on top of one. We rate it 4.4 out of 5.
- How much does LM Evaluation Harness cost?
- Free and open source under the MIT license.
- What are the downsides of LM Evaluation Harness?
- Chat APIs cannot run log-likelihood tasks such as standard MMLU; measures models, not your application; no native multi-node evaluation; about 1,240 open issues and PRs.
- Who is LM Evaluation Harness best for?
- Researchers and model trainers comparing models on standard benchmarks.
- How much does LM Evaluation Harness cost for a team or in production?
- Free software; cost is GPU time for local models or per-token API fees for hosted ones. A full suite over a large model is a substantial GPU job, so teams usually run a fixed subset per checkpoint and use --limit for smoke tests.
- Is LM Evaluation Harness safe for company data?
- Runs locally; data leaves only when you call a hosted model API or push results to the Hub with your own token. Some models need trust_remote_code, which executes model-repo code. No security advisories are published on the repository. MIT licence.
- What are the best LM Evaluation Harness alternatives?
- The closest alternatives we have reviewed are Inspect, DeepEval and promptfoo.
Answers checked Oct 23, 2026.
Make LM Evaluation Harness? Show this rating on your site
