What is agent-qa?
agent-qa, from vostride, runs user journeys written in plain English as YAML against a real browser (Playwright) or a mobile app (Appium on emulators, simulators and phones). A model reads each screen, acts, and checks the expected result, while the run records screenshots, console and network logs, the accessibility tree and, on web and Android, a video.
Its distinctive part is memidex: after each run it saves facts, cautions and repeatable procedures to an agent-qa-knowledge/ folder you commit, and feeds them to later runs as hints. A validated action cache replays steps without asking the model to plan when the screen still matches. It runs from the CLI, a local dashboard, CI, or a coding agent through an MCP server with 20 tools and resources and four skills.
The repository was created in May 2026, is written in TypeScript under the source-available Functional Source License (FSL-1.1-ALv2, which converts to Apache 2.0 later), has about 860 GitHub stars and is at v0.3.5 (11 November 2026). It needs Node.js 24 or newer.
Is agent-qa worth it?
agent-qa is worth a look for teams that want plain-English tests across web and both mobile platforms from one tool, and that will read its safety page before pointing it at anything real. Few rivals cover Android and iOS as well as the browser, and fewer document their own failure modes this plainly.
The memory is the part to judge it on. What memidex learns is written as files in your repository, so a reviewer can read, diff and revert it like code, and the docs say proposed lessons are checked before they are applied and stale ones retired. That is a sounder design than a hidden vector store. The action cache does the cost work: steps that already succeeded are replayed without a planning call while the screen still matches.
The benchmark claims need the caveats the vendor itself prints. It reports 100% pass@1 on AndroidWorld (116 of 116 tasks) and 99.6% on AppControlBench (114.5 of 115 points), with memory and self-improvement switched off. The results are self-published, not on AppControlBench's official leaderboard, and measure task completion rather than whether a QA verdict was right. Its own evaluation guide tells you to count false passes on your app instead, which is the right advice.
The defaults deserve attention. Released versions send anonymous usage events to PostHog and error reports to Sentry; error reports can quote a URL or a step's text, and both stop only when agent-qa.config.yaml sets analytics: privacy: true. There are no approval prompts, so the MCP server is as powerful as a shell in your project. The .agent-qa/ folder stores your .env values in run results and must stay out of git and CI uploads.
Against e2e, the closest rival, agent-qa offers more platforms and a reviewable memory; e2e has a permissive Apache-2.0 licence and about 14 times the stars. With three contributors and a README that says to expect bugs, this is a tool to trial on five to ten journeys before trusting it with a release gate.
Where it earns its place
A four-person startup with a React web app and an Android companion app
Catch checkout and sign-up regressions before each release without writing selectors
Write five to ten journeys as YAML with exact expected values, run them on Playwright and an Android emulator, and save the action cache on the main branch so later runs replay without planning calls. Cost is the connected model's tokens; there is no seat fee.
A developer working in Claude Code on a Next.js storefront
Have the agent prove a change works in the browser before opening a pull request
npx agent-qa@latest init --agents claude-code installs the MCP entry and skills; the agent writes a test with agent_qa_write_test, runs it and reads the evidence. Because nothing asks for confirmation, keep it on a local or staging target with test accounts.
A QA lead deciding whether to adopt an AI test runner
Measure false passes and cost on the team's own app before committing
Follow the vendor's evaluation guide: add one deliberately broken build, run each test three to five times, mark wrong verdicts with agent-qa runs mark, and read token use on the dashboard's Insights page.
Before you commit
- Setup
- Node.js 24 or newer, then npm install -D agent-qa and npx agent-qa init, which writes config and example tests, downloads Chromium and starts the dashboard; connect a model with npx agent-qa auth login or an API key. The quickstart puts it at about ten minutes. Mobile needs Appium and npx agent-qa install-mobile-drivers; hooks need Docker.
- Integrations
- Claude Code and Codex subscriptions, Anthropic, OpenAI and Gemini APIs, OpenAI- or Anthropic-compatible endpoints including local models; MCP server over stdio or HTTP with 20 tools and resources; four skills; JUnit reports for CI; plugins for BrowserStack, Hyperbrowser, Kernel and Steel.
- Data and privacy
- Tests, config and learned knowledge are files in your repo; credentials sit in ~/.agent-qa/auth.json. Declared secrets are masked as [secret:NAME] in model requests, logs and reports, but .env values, screenshots and videos are not. Anonymous PostHog events and Sentry error reports are on by default; analytics: privacy: true turns both off. The dashboard and HTTP MCP listen on 127.0.0.1 only, without tokens.
- Cost at scale
- No product fee. Spend is model tokens per run, plus up to a minute of post-run analysis per run and your CI and device infrastructure; cached steps skip planning calls. Per-run cost is not published.
- Limits
- No approval prompts on any command or MCP tool; no prompt-injection detection for app text; video only on web and Android; hooks run in Docker limited to 512 MB and one CPU; early-stage software with frequent releases.
What’s good
- Web, Android and iOS from one test format
- Learned knowledge stored as diffable files in the repo
- Validated action cache cuts repeat model calls
- Unusually frank docs on secrets, telemetry and limits
What’s not
- Usage analytics and Sentry error reports on by default
- FSL licence bars competing commercial use; not open source
- No approval prompts; MCP access equals shell access
- Early stage: v0.3, three contributors, benchmarks self-published
Best for
Small product teams that ship a web app plus native mobile apps and want plain-English regression tests they can run from a coding agent or CI on their own model.
agent-qa alternatives
agent-qa: common questions
- Is agent-qa worth it?
- agent-qa is one of the more careful agentic test runners, with honest documentation and a useful memory, but its first GitHub release came only in October 2026, it sends telemetry by default and is not open source. We rate it 3.7 out of 5.
- How much does agent-qa cost?
- $0: no paid tier, seat fee or hosted platform. You pay for the model you connect (a Claude or ChatGPT subscription, an API key or a local model) and your own CI, browsers and devices.
- What are the downsides of agent-qa?
- Usage analytics and Sentry error reports on by default; fSL licence bars competing commercial use; not open source; no approval prompts; MCP access equals shell access; early stage: v0.3, three contributors, benchmarks self-published.
- Who is agent-qa best for?
- Small product teams that ship a web app plus native mobile apps and want plain-English regression tests they can run from a coding agent or CI on their own model.
- How much does agent-qa cost for a team or in production?
- No product fee. Spend is model tokens per run, plus up to a minute of post-run analysis per run and your CI and device infrastructure; cached steps skip planning calls. Per-run cost is not published.
- Is agent-qa safe for company data?
- Tests, config and learned knowledge are files in your repo; credentials sit in ~/.agent-qa/auth.json. Declared secrets are masked as [secret:NAME] in model requests, logs and reports, but .env values, screenshots and videos are not. Anonymous PostHog events and Sentry error reports are on by default; analytics: privacy: true turns both off. The dashboard and HTTP MCP listen on 127.0.0.1 only, without tokens.
- What are the best agent-qa alternatives?
- The closest alternatives we have reviewed are e2e, Magnitude, Playwright Test Agents and Stagehand.
Answers checked Nov 13, 2026.