The best open-source LLM evaluation tools in 2026
The best open-source LLM evaluation tools right now are promptfoo, Inspect and LM Evaluation Harness, out of 5 we have reviewed. Each is ranked by our rating, with its price, our verdict and who it suits.
Updated Oct 23, 2026 · Ranked from our reviews · How we review
- 1
promptfoo turns LLM regression tests and red-team scans into a YAML file and one CI command; its new owner is the part to watch.
Best for: Teams shipping LLM apps who want regression tests and security scans in CI.
Price: CLI free and open source (MIT); Community use includes red teaming up to 10,000 probes a month. Enterprise (managed cloud) and On-Premise are priced by quote.
Read our promptfoo review4.5/5 - 2
Inspect is the shared harness of government and independent AI safety testing; product teams checking prompts in CI will find it more framework than they need.
Best for: Researchers and safety teams running agent and capability evaluations.
Price: Free and open source under the MIT license.
Read our Inspect review4.4/5 - 3
The common yardstick for model benchmarks, and the wrong instrument for judging an application built on top of one.
Best for: Researchers and model trainers comparing models on standard benchmarks.
Price: Free and open source under the MIT license.
Read our LM Evaluation Harness review4.4/5 - 4
A capable pytest-style eval library whose real cost is the judge model's bill, with a $200-a-month platform at the end of the funnel.
Best for: Python teams adding automated quality checks to LLM applications.
Price: Free and open source under the Apache 2.0 licence. Confident AI's hosted platform is a separate paid product.
Read our DeepEval review4.2/5 - 5
Its metric names became the vocabulary of RAG evaluation, but the code has been left without a maintainer.
Best for: Teams with an existing Ragas suite who can pin a version and carry their own fixes.
Price: Free and open source under the Apache 2.0 licence. You pay for the judge-model API calls each metric makes.
Read our Ragas review3.4/5