What it is
DeepEval is an open-source framework for testing LLM applications in the style of pytest: you write test cases with inputs and outputs and assert them against metrics, locally or in CI.
It ships metrics in several families: custom judges such as G-Eval and the graph-based DAG, RAG metrics (faithfulness, contextual recall and precision), agent metrics (task completion, tool correctness, plan adherence), multi-turn and voice-agent metrics, and red-teaming. Most are scored by an LLM judge, OpenAI in the quickstart, though any model can be plugged in. A TypeScript package (deepeval on npm, version 0.9.23) sits alongside the Python one.
Confident AI, the company behind it, sells a hosted platform for test results and production traces. The latest Python release on PyPI is 4.3.1, from 16 October 2026.
Our review
DeepEval is the most direct way for a Python team to put LLM checks into an existing test suite. Tests look like pytest, run with deepeval test run in CI, and fail a build when faithfulness or task completion drops below a threshold. With about 19,100 GitHub stars and releases most months, it has one of the broadest metric catalogues among open-source eval libraries, now reaching into agent planning and voice calls.
The catch is that most metrics are one LLM grading another. The quickstart sets OPENAI_API_KEY, so by default every metric on every test case is a paid API call, and a few hundred cases run on each pull request add up. Scores also move when the judge model changes. The most-discussed open issues reflect this: judges returning invalid JSON ('please use a better evaluation model'), trouble using Hugging Face models as judges, and async runs that do not complete.
Telemetry is on by default. The docs say DeepEval sends event names, metric names, an anonymous ID and your public IP address to PostHog; DEEPEVAL_TELEMETRY_OPT_OUT=1 turns it off, though an open issue titled 'Unable to disable telemetry' has 12 comments. Test data goes to Confident AI only if you log in and upload.
Confident AI is priced per organisation rather than per seat: Free with two seats and five test runs a week, Starter at $200 a month, Team at $2,000 a month with SSO and SOC 2, and a custom Enterprise tier with on-premises deployment and HIPAA. The library works fully without it, but the README and CLI point there often. Promptfoo covers similar ground through YAML config with stronger red-teaming, Ragas is narrower and RAG-only, and Inspect AI suits model-level benchmark work better than application tests.
Where it earns its place
A backend team shipping a RAG support bot on a Python stack
Block merges when answer faithfulness drops
Add faithfulness and contextual recall tests to the existing pytest suite and run them on each pull request. The library is free; the judge calls are not, so many teams run the full set nightly and a sample per PR.
An ML engineer building a tool-calling agent
Check the agent calls the right tools with the right arguments
Tool Correctness and Argument Correctness compare calls against expected ones, while Task Completion and Plan Adherence use an LLM judge. A cheaper or local judge cuts cost but should be checked against a few human labels first.
A head of AI at a 40-person software company
Decide whether to pay for a hosted eval dashboard
Confident AI Starter is $200 a month flat with unlimited seats and 5 GB-months of traces, then $1 per GB-month; SSO and SOC 2 start at Team, $2,000 a month. Keeping results in CI artefacts costs nothing but loses history and trace clustering.
Before you commit
- Setup
- pip install -U deepeval (Python 3.9 or later), set OPENAI_API_KEY or configure another judge model, write one test file with a metric and run deepeval test run.
- Integrations
- pytest and CI; tracing for LangChain, LangGraph, LlamaIndex, CrewAI, Pydantic AI, OpenAI Agents, Google ADK, AWS AgentCore, Strands, the Vercel AI SDK and Mastra; custom judge models; optional upload to Confident AI.
- Data and privacy
- Runs locally. Telemetry (event and metric names, anonymous ID, public IP) goes to PostHog unless DEEPEVAL_TELEMETRY_OPT_OUT=1. Test cases are sent to whichever judge provider you configure. Confident AI stores uploads on AWS; SSO and SOC 2 come with the Team plan.
- Cost at scale
- The library is free; judge cost scales with test cases times metrics times runs. Confident AI: Free (2 seats, 5 test runs a week), Starter $200 a month, Team $2,000 a month, Enterprise custom with on-premises deployment.
- Limits
- 775 open issues and pull requests; judges can return malformed JSON and fail tests; scores shift with the judge model; async runs are reported to hang.
What's good
- pytest-style tests that run in CI
- Broad metrics: RAG, agents, multi-turn and voice
- Any LLM can act as judge, including local models
- Python and TypeScript packages
What's not
- Most metrics cost a judge-model call per test case
- Telemetry to PostHog on by default
- Malformed judge output is a common failure
- Docs and CLI steer towards Confident AI
Best for
Python teams adding automated quality checks to LLM applications.
Alternatives
DeepEval: common questions
- Is DeepEval worth it?
- A capable pytest-style eval library whose real cost is the judge model's bill, with a $200-a-month platform at the end of the funnel. We rate it 4.2 out of 5.
- How much does DeepEval cost?
- Free and open source under the Apache 2.0 licence. Confident AI's hosted platform is a separate paid product.
- What are the downsides of DeepEval?
- Most metrics cost a judge-model call per test case; telemetry to PostHog on by default; malformed judge output is a common failure; docs and CLI steer towards Confident AI.
- Who is DeepEval best for?
- Python teams adding automated quality checks to LLM applications.
- How much does DeepEval cost for a team or in production?
- The library is free; judge cost scales with test cases times metrics times runs. Confident AI: Free (2 seats, 5 test runs a week), Starter $200 a month, Team $2,000 a month, Enterprise custom with on-premises deployment.
- Is DeepEval safe for company data?
- Runs locally. Telemetry (event and metric names, anonymous ID, public IP) goes to PostHog unless DEEPEVAL_TELEMETRY_OPT_OUT=1. Test cases are sent to whichever judge provider you configure. Confident AI stores uploads on AWS; SSO and SOC 2 come with the Team plan.
- What are the best DeepEval alternatives?
- The closest alternatives we have reviewed are promptfoo, Ragas and Inspect.
Answers checked Oct 23, 2026.
Make DeepEval? Show this rating on your site
