Testing AI apps, and letting AI test yours: the layers, the tools and what they cost

By Sheela · Nov 6, 2026

Testing now runs in two directions. If you ship an AI feature, you need tests for something that gives a slightly different answer every time. If you ship ordinary software, AI tools now offer to write and maintain your tests for you. Both markets are crowded, both are priced in ways that hide the real bill, and both have a failure mode that looks like a green build. Here is how to think about the layers, which of the 16 tools we have reviewed fits which job, and a starting setup a team lead can defend.

The map: two jobs, seven layers

Most confusion comes from comparing tools that do different jobs. A RAG metric library and a browser agent both call themselves "AI testing", and neither replaces the other. Sort the work first.

Testing an AI app or agent has four layers:

Evals: fixed inputs, scored outputs. Does the answer stay grounded, does the tool call go to the right function. This is your unit test.
Simulated conversations: a second model plays the user across several turns, because agents fail on turn four, not turn one.
Red-team regression: generated attacks (prompt injection, data leaks, harmful output) rerun on every change, so a fixed hole stays fixed.
Production scoring: the same scorers run against live traces, and bad answers from real users become new test cases.

AI testing ordinary software has three:

Record and replay for backends: capture real API calls and the database answers behind them, then replay them as tests.
Record and replay for front ends: capture real sessions and compare how the screens look before and after a change.
Browser agents: an AI that plans, writes, runs and repairs end-to-end tests, either in your repository or in a vendor's cloud.

Testing your AI app

Evals: the layer to build first

promptfoo (4.5) is the shortest route from no LLM tests to tests in CI: a YAML file of prompts, providers and assertions, and one command that shows a comparison matrix. It needs Node.js 22.22 or later. One thing to weigh: on 9 March 2026 OpenAI announced it would acquire promptfoo, and the company now says it is part of OpenAI. It stays MIT-licensed and supports other providers, but some teams will not want their grader owned by a vendor they are grading.

DeepEval (4.2) suits Python teams who want evals inside their existing test suite. Tests look like pytest, run with deepeval test run, and fail the build when faithfulness or task completion falls below a threshold. Its metric catalogue is one of the broadest in open source, reaching agent planning and voice.

Giskard (4.0) puts deterministic checks, LLM-judged checks, RAG test-set generation and a red-team scan in one Apache-2.0 library. Version 3 is a rewrite that needs Python 3.12 or later and drops the tabular-model scan v2 users relied on.

Two you may already have. Ragas (3.4) gave RAG evaluation its vocabulary (faithfulness, context precision) and still draws about 1.05 million PyPI downloads a month, but nobody has committed to it since 24 February 2026. Keep it running if it works; do not start new projects on it. Inspect (4.4), from the UK AI Security Institute, is the harness government safety institutes use, with more than 200 ready-made evals. It is the right tool for model-level benchmarks and more framework than a product team checking prompts in CI needs.

Simulated conversations

Single-prompt evals miss the agent that answers the first question well, then forgets a constraint three turns later. LangWatch Scenario (3.9) tests that by talking to your agent: one model plays the user, another judges every turn, and the whole thing runs as an ordinary pytest or vitest test. You can pin the opening message, inject a mocked tool result, assert that a particular tool was called and let the simulator carry on. Voice agents are covered too, with noise injection and p50 and p95 latency. It is a small project, about 1,000 stars with 165 open issues, and it sits best next to an eval library rather than in place of one.

Red-team regression in CI

A red-team scan run once before launch is an audit. Rerun on every pull request, it becomes a regression test. promptfoo generates attacks for prompt injection, jailbreaks and data exfiltration; by default that generation runs on its hosted service, and the free Community tier caps red teaming at 10,000 probes a month. Giskard's scan writes an adversarial suite from a plain-language description of what your agent should and should not do, mapped to the OWASP LLM Top 10. Scenario's RedTeamAgent swaps the polite simulated user for an attacker that escalates over several turns, which is closer to how real jailbreaks work than a single hostile prompt.

Production scoring

Pre-release tests only cover the inputs you thought of. Braintrust (4.3) is built around the question of whether the last change made things better: experiments, datasets and online scoring share one data model, so a bad production answer becomes a test case in a few clicks, and the same scorer runs in CI and on live traffic. Langfuse (4.6) is the open-source route, with tracing, datasets, experiments and LLM-as-judge scoring in a free self-hosted edition, at the price of running Postgres, ClickHouse, Redis and object storage yourself.

Letting AI test your software

Record and replay

Keploy (4.0) is for the backend with no tests. You run your service under keploy record, click through the flows, and it captures the API calls together with the PostgreSQL, MySQL, MongoDB, Redis, Kafka or RabbitMQ traffic they caused. keploy test replays the requests and serves the recorded dependencies as mocks, so CI needs no test database. On Linux it does this with eBPF, which means no code change but root and kernel 5.10 or later; native macOS and Windows builds only replay HTTP, MySQL and MongoDB reliably.

Meticulous (3.7) does the same for front ends. A recorder script in staging or preview builds captures sessions and the backend responses they received; on each pull request it replays them against the old and new code and groups the screens that look different. You review a list of visual changes instead of writing test code. It checks what the screen looks like, not whether the order reached your database.

Browser agents

This is the busiest corner, and the four options differ mostly in who owns the tests and who runs them.

Playwright Test Agents (4.1) ship free with Playwright. A planner explores your app and writes a Markdown test plan, a generator turns the plan into Playwright tests in your repository, and a healer repairs a failing test one at a time. They run inside the coding agent you already use (VS Code, Claude Code, Codex or opencode) and call Playwright MCP (4.7) to drive the browser. The output is ordinary Playwright code with no vendor in between.

Momentic (4.0) lets you write a test as a sentence, saves it as YAML in your repository and runs it in hosted browsers, Android emulators or iOS simulators. When a button moves, auto-heal finds it again and proposes an edit for you to review.

TestSprite (3.7) sits behind your coding agent: it reads the code or a requirements document, writes frontend and backend tests, runs them in its cloud and returns a failure bundle with screenshots, the page structure and a suggested fix.

QA Wolf (4.1) sells the job rather than the tool. Its agents write and run Playwright and Appium tests, decide whether a failure is a stale test or a real bug, and file bugs with video in Linear or Jira; on the managed plan, human QA engineers look at every failure within 24 hours. The code is exportable.

One warning about AI-generated unit tests. Qodo (3.8), the company once called CodiumAI and first known for generating tests, deprecated its test-generation command this autumn with no replacement and now sells pull request review. Coding agents will happily write unit tests on request; the risk is that a test written from the code checks what the code does, not what it should do. Have a person write or approve the assertion.

Which tool for which job

Job	Start with	Our rating	What it costs
Prompt and RAG regression tests in CI	promptfoo or DeepEval	4.5 / 4.2	Free; you pay judge and model tokens
Checks, RAG questions and red-team scan in one Python library	Giskard	4.0	Free; judge tokens, gpt-4o-mini by default
Multi-turn agent behaviour	LangWatch Scenario	3.9	Free; two LLM calls per turn per test
Model-level benchmarks and safety evals	Inspect	4.4	Free; tokens and sandbox compute
Scoring production traffic, hosted	Braintrust	4.3	Free tier; Pro $249 a month
Tracing and evals you host yourself	Langfuse	4.6	Free self-hosted; cloud from $29 a month
Backend with no tests	Keploy	4.0	Free CLI; Pro $19 a user a month
Visual regressions in a busy front end	Meticulous	3.7	Not published
End-to-end tests you own, already on Playwright	Playwright Test Agents	4.1	Free; coding-agent tokens
Plain-English end-to-end tests, web and mobile	Momentic	4.0	Free $50 of usage; Scale $500 a month
A coding agent that checks its own work in a browser	TestSprite	3.7	From $19 a month in credits
Handing end-to-end testing to someone else	QA Wolf	4.1	$1 per web run, $2 per mobile run

What it really costs

Almost every open-source eval tool is free to install and metered by tokens to run. DeepEval, Ragas and Giskard score most metrics with an LLM judge, so the bill is roughly test cases times metrics times runs. A few hundred cases on every pull request adds up, and Scenario doubles it, because a simulator and a judge both speak on every turn. Run the full suite nightly and a small smoke set per pull request.

Hosted platforms meter differently, and the meter decides who it suits. Braintrust charges for data and scores rather than seats: Pro at $249 a month covers 5 GB and 50,000 scores, then $1.50 per 1,000 scores, so judging every production request is the line that grows fastest. Langfuse's own worked example puts Pro at 5 million units at $551 a month. LangWatch's cloud is €29 a seat on Growth, and DeepEval's Confident AI jumps from Free to $200 a month.

For browser testing, the unit is the run or the step. QA Wolf's own example, 50 web tests every working day, comes to about $1,083 a month, and running the same tests on every pull request instead multiplies that. Momentic charges $0.0125 per normal step and $0.025 for a step the AI had to work out; its free $50 is about 4,000 steps, and when the balance runs out tests pause rather than overspend. TestSprite's Starter is $19 for 400 credits, but the pricing page never says what a credit buys. Keploy's Pro includes 400 test runs, then $0.22 each. Meticulous publishes no price at all; estimates on review sites range from about $2,000 a year to $600 a month.

Where these tools mislead you

Flaky judges. A judge is a model, and it fails like one. DeepEval's most-discussed issues include judges returning invalid JSON, which fails the test for a reason unrelated to your app. Scores also shift when the judge model changes, so pin the judge version and treat a judge upgrade like a dependency upgrade: rerun the baseline before comparing.
Non-determinism. Your app varies, the simulated user varies, and the judge varies. Scenario's cache_key makes the simulated user repeat itself, but a changed prompt invalidates the cache. Score pass rates over several runs and alert on a drop, rather than failing the build on a single red sample.
Self-healing that hides bugs. A healer's job is to make a red test green, and sometimes the correct result is red. Playwright's docs say the healer may decide a feature is broken and skip the test, which a busy team can mistake for a pass. Momentic proposes fixes for review rather than applying them silently, and QA Wolf's agents try to separate stale tests from real bugs. Whatever you buy, read every healed diff and count skipped tests as failures.
Recordings only know the past. Keploy and Meticulous cover only what someone did while recording, and recorded responses go stale when the schema changes. A green replay suite says nothing about the flow nobody clicked.
Your data leaves the building. TestSprite runs tests in its cloud, so your app's pages and screenshots pass through its infrastructure. Meticulous's recorder captures keystrokes and network responses; passwords are redacted by default, other personal data needs rules you write. Telemetry is on by default in promptfoo, DeepEval, Ragas and Giskard, and every judge call sends your test prompts to whichever provider you configure.

A starting setup

For a team with one AI feature and an ordinary web app, this covers most of the risk for little more than token costs:

Write 30 to 50 eval cases from real user questions in promptfoo or DeepEval, with deterministic assertions wherever the answer allows it and an LLM judge only where it does not. Run them on every pull request that touches a prompt or a model.
Add a red-team scan (promptfoo or Giskard) to the nightly build, and turn every attack that gets through into a permanent test case.
If the feature is an agent, add three to five Scenario tests for the conversations that have gone wrong before, with the simulated user cached.
Send production traces to Langfuse or Braintrust and move a handful of bad answers into the eval set each week.
For the rest of the app, cover the five flows that must never break (sign-up, checkout, the export a big customer relies on) with Playwright Test Agents if you already use Playwright, or Momentic's free tier if you do not. Have someone review every plan and every healed test.
Set the telemetry opt-outs (PROMPTFOO_DISABLE_TELEMETRY, DEEPEVAL_TELEMETRY_OPT_OUT=1, DO_NOT_TRACK=1) before the first run, not after.

The catch: every layer here makes a test suite cheaper to produce, and none of them decides what correct means. An eval set that only asks easy questions passes, a healer that rewrites assertions passes, and a replay of last month's traffic passes. The expensive part of testing has always been deciding what to check, and that work stays with your team.

Our take: treat AI in testing the way you would a fast new hire. Let it write the cases, run them and suggest fixes, and keep the list of what must be true, and the sign-off on any test it changes, with a person. Pay for a hosted service only when the bill is smaller than the time someone on your team spends keeping the free version alive.