What it is
promptfoo is a command-line tool and library for testing LLM applications. You describe prompts, providers and test cases in a YAML file, and it runs them and shows a comparison matrix.
It also does red teaming: it generates adversarial inputs to find prompt injection, jailbreaks, data leaks and unsafe tool use. It works with models from OpenAI, Anthropic, Google and open-source providers, and fits into CI pipelines.
On 9 March 2026 OpenAI announced it had agreed to acquire promptfoo, to build its technology into OpenAI Frontier, its enterprise agent platform, and said the project would stay open source under its current MIT licence. We found no public notice of the deal closing.
Our review
promptfoo is the shortest route from no LLM tests to tests in CI, and in March 2026 it agreed to be bought by OpenAI, whose models many teams use it to grade. It has about 26,300 GitHub stars, and the npm package reached 0.124.1 on 8 October 2026. Setup is a YAML config, a few test cases with assertions and one command.
Red teaming is what separates it from most eval libraries. It generates attacks for prompt injection, jailbreaks and data exfiltration and reports what got through. The catch is where those attacks are made: generation runs on promptfoo's hosted service by default, and the free Community tier caps red teaming at 10,000 probes a month. Offline use means setting PROMPTFOO_DISABLE_REMOTE_GENERATION and pointing it at a local model.
The defaults deserve a look. The CLI sends usage telemetry unless PROMPTFOO_DISABLE_TELEMETRY is set, and when you are logged in that includes your user ID and email address, though not prompts, outputs or keys. The self-hosted web viewer is explicitly not for production: it uses a local SQLite database, has no built-in authentication and breaks if run as more than one replica. Shared dashboards, SSO and continuous monitoring are Enterprise features at an unpublished price.
OpenAI says promptfoo will stay MIT-licensed and keep supporting other providers, and Repello, a competitor, reported two months after the announcement that the CLI and config schema were unchanged while asking whether enterprises want their grader owned by the vendor being graded. For research benchmarks, Inspect and the LM Evaluation Harness are better suited, and Python teams may find DeepEval's pytest style more natural than a tool that needs Node.js 22.
Where it earns its place
A backend engineer maintaining a RAG support bot
Catch regressions when the prompt or model changes
Keep a promptfooconfig.yaml of real customer questions with assertions and run promptfoo eval in GitHub Actions on every pull request. Free; each run costs model tokens for every test case times every provider compared, plus any model-graded assertions.
An application security lead at a fintech shipping an agent with tools
Red-team the agent for prompt injection and data exfiltration before launch
Run promptfoo redteam against a staging endpoint; Community covers 10,000 probes a month. Attack generation goes through promptfoo's hosted service unless remote generation is disabled and a local model is used, which matters if the app description is confidential.
A head of AI governance at a bank with a dozen LLM apps
Show auditors repeatable security testing across every app
Enterprise or On-Premise adds shared dashboards, SSO, searchable scan history and compliance framework reports, priced by quote. The procurement question is now OpenAI's ownership and roadmap rather than the feature list.
Before you commit
- Setup
- npx promptfoo@latest init or a global npm install, on Node.js 22.22 or later; a first eval runs from the generated YAML in minutes.
- Integrations
- Providers include OpenAI, Anthropic, Google and local models through Ollama or custom HTTP targets; runs in CI such as GitHub Actions; separate code-scanning action and MCP proxy products.
- Data and privacy
- Usage telemetry is on by default and includes user ID and email when logged in, but not prompts or outputs. Red-team generation uses a hosted service unless PROMPTFOO_DISABLE_REMOTE_GENERATION is set. Results are kept in a local SQLite database. MIT licence.
- Cost at scale
- The CLI is free; the bill is model tokens, roughly test cases times providers times repeats per run, plus model-graded assertions. Red teaming beyond 10,000 probes a month, shared dashboards and SSO need Enterprise, priced by quote.
- Limits
- About 840 open issues and pull requests; the self-hosted server cannot scale beyond one replica and has no auth; not designed as an academic benchmark runner.
What's good
- YAML tests that run in CI with one command
- Broad red-team plugins for injection and data leaks
- Works with OpenAI, Anthropic, Google and local models
- MIT licence
What's not
- Being acquired by OpenAI, a model vendor
- Telemetry and hosted attack generation on by default
- Self-hosted viewer lacks auth and scaling
- Requires Node.js 22.22 or later
Best for
Teams shipping LLM apps who want regression tests and security scans in CI.
Alternatives
promptfoo: common questions
- Is promptfoo worth it?
- promptfoo turns LLM regression tests and red-team scans into a YAML file and one CI command; its new owner is the part to watch. We rate it 4.5 out of 5.
- How much does promptfoo cost?
- CLI free and open source (MIT); Community use includes red teaming up to 10,000 probes a month. Enterprise (managed cloud) and On-Premise are priced by quote.
- What are the downsides of promptfoo?
- Being acquired by OpenAI, a model vendor; telemetry and hosted attack generation on by default; self-hosted viewer lacks auth and scaling; requires Node.js 22.22 or later.
- Who is promptfoo best for?
- Teams shipping LLM apps who want regression tests and security scans in CI.
- How much does promptfoo cost for a team or in production?
- The CLI is free; the bill is model tokens, roughly test cases times providers times repeats per run, plus model-graded assertions. Red teaming beyond 10,000 probes a month, shared dashboards and SSO need Enterprise, priced by quote.
- Is promptfoo safe for company data?
- Usage telemetry is on by default and includes user ID and email when logged in, but not prompts or outputs. Red-team generation uses a hosted service unless PROMPTFOO_DISABLE_REMOTE_GENERATION is set. Results are kept in a local SQLite database. MIT licence.
- What are the best promptfoo alternatives?
- The closest alternatives we have reviewed are DeepEval, Inspect and Ragas.
Answers checked Oct 23, 2026.
Make promptfoo? Show this rating on your site
