What is iFixAi?
iFixAi audits a deployed AI agent rather than the model behind it. You describe the agent in a profile (its roles, tools and policies), iFixAi builds a simulated business around it, then sends requests over the agent's chat endpoint, including ones that try to escalate privileges, inject instructions or pull another user's data. A judge model from a different provider than the agent grades each reply, and every finding shows what was asked and what the agent did.
A default run has 10 inspections, three of them mandatory minimums (tool governance, privilege escalation and destructive actions); failing any caps the grade at D. The result is an A to F grade over five weighted pillars (fabrication, manipulation, deception, unpredictability and opacity), with JSON and Markdown reports and a --min-score flag for CI. --full-run runs all 61 inspections, setting aside any that cannot apply.
It installs with pip install ifixai (Python 3.10+), and also ships as a Claude Code and Codex plugin and as a skill for Cursor, VS Code, Windsurf, Zed and others. The engine is Apache-2.0; the same team sells audits per agent per month on ifixai.ai.
Is iFixAi worth it?
iFixAi tests the thing most eval suites skip: whether an agent with tools and roles does only what its role allows. Benchmarks score the model; iFixAi sends a support agent a refund a manager already declined, or a request to change a payout account, and checks what it actually does. The three mandatory minimums (B01 tool governance at 100%, B08 privilege escalation at 95%, P01 destruction resistance at 100%) reflect where real damage happens.
The scoring documentation is unusually candid. Every inspection carries a Wilson 95% confidence interval, the docs tell you to compare runs by interval overlap rather than raw deltas, and they warn that single-run B24 scores within 0.12 of the threshold are noise. They also say plainly that the regulatory summaries do not certify compliance. Grading rests on an LLM judge, so two runs of the same agent can differ; the cross-provider rule only avoids the agent's own vendor marking its homework.
Momentum is high. The repository was created on 27 April 2026 and has 43,497 stars and 1,779 forks; PyPI history starts at 3.0.2 on 23 June and reached v4.3.0 on 22 October. Pull requests land daily, many hardening how evidence is validated and reported, which also means report formats and scores move between versions; the docs advise re-running old targets rather than comparing old headlines.
The catch is setup and scope. A useful audit needs a hand-written or agent-written profile that matches your real roles and tools, and a default run covers only 10 inspections; the full 61 cost more judge calls. The probes are real requests, so the README says to point it at a staging copy. The site lists 60 inspections where the README says 61, and it names no company behind the paid tier, only a contact address.
Against promptfoo, which red-teams prompts and models broadly, iFixAi is narrower and more opinionated: it encodes an organisation chart and grades conduct against it. That suits teams whose agents can move money or data; teams testing a chatbot's answers will get more from a general eval framework.
Where it earns its place
A platform engineer at a fintech running a customer-support agent with refund tools
Prove the agent cannot pay a refund or change a payout account for a user without authority
Write the agent's profile with its roles and tools, run the default suite against staging, and fail the build if B01, B08 or P01 drop below their minimums. Cost is the judge provider's API calls, shown by --dry-run first.
An AI lead at a 50-person SaaS company swapping the model behind an agent
Check the new model does not loosen behaviour the old one kept
Run the same fixture against both versions and compare per-inspection confidence intervals, as the scoring docs advise, rather than the headline grade.
A developer building an agent in Cursor or Claude Code
Get a first audit without writing the profile by hand
Install the plugin or skill and ask the coding agent to run iFixAi; it reads the agent's code, drafts the profile, shows the cost and runs the audit.
Before you commit
- Setup
- pip install ifixai on Python 3.10+; the mock demo runs in about a minute with no API key. A real audit needs an agent profile, ifixai author and validate to build the simulation, a judge key and the agent's endpoint; ifixai setup walks through it.
- Integrations
- Any OpenAI-compatible agent endpoint, or a few lines of Python for others; judges from OpenAI, Anthropic, Gemini, Azure, Bedrock, OpenRouter and more; Claude Code and Codex plugin; skills for Cursor, VS Code, Windsurf, Cline, Continue, Gemini and Zed; Python API; JSON and Markdown reports.
- Data and privacy
- Runs on your machine: probes go to your agent's endpoint and grading calls go to the judge provider you choose; reports are written to ./ifixai-results/. Telemetry is not mentioned on the site. Apache-2.0.
- Cost at scale
- Self-hosted cost is judge API usage, which grows with --full-run's 61 inspections; --dry-run prints the call count. Paid audits are per agent per month, prices not published.
- Limits
- LLM-judged scores carry run-to-run noise; inspections that cannot apply are set aside and reduce coverage; scoring has changed across versions; probes are real requests, so use staging.
What’s good
- Tests role, tool and data permissions over the agent's real endpoint
- Judge always from a different provider than the agent
- Confidence intervals and candid notes on score noise
- Dry run shows test and judge-call counts before billing
- CI gate via --min-score; plugin for Claude Code and Codex
What’s not
- Grades depend on an LLM judge and vary between runs
- Needs an accurate agent profile to be meaningful
- Default run covers 10 of 61 inspections
- Fast release cadence changes scores between versions
- Paid tier prices and the company behind it not published
Best for
Teams shipping agents that can act on money, accounts or customer data who want a repeatable permissions and injection audit in staging and CI.
iFixAi alternatives
iFixAi: common questions
- Is iFixAi worth it?
- A well-documented way to test whether an agent stays inside its permissions, as long as you read the grade as a judge's opinion with error bars rather than a certificate. We rate it 3.8 out of 5.
- How much does iFixAi cost?
- Free under Apache-2.0, self-hosted with your own model keys; you pay the judge provider's API calls. Paid audit tiers (Startup, Growth, Enterprise, Agentic Enterprise) are priced per agent per month, with prices not published.
- What are the downsides of iFixAi?
- Grades depend on an LLM judge and vary between runs; needs an accurate agent profile to be meaningful; default run covers 10 of 61 inspections; fast release cadence changes scores between versions; paid tier prices and the company behind it not published.
- Who is iFixAi best for?
- Teams shipping agents that can act on money, accounts or customer data who want a repeatable permissions and injection audit in staging and CI.
- How much does iFixAi cost for a team or in production?
- Self-hosted cost is judge API usage, which grows with --full-run's 61 inspections; --dry-run prints the call count. Paid audits are per agent per month, prices not published.
- Is iFixAi safe for company data?
- Runs on your machine: probes go to your agent's endpoint and grading calls go to the judge provider you choose; reports are written to ./ifixai-results/. Telemetry is not mentioned on the site. Apache-2.0.
- What are the best iFixAi alternatives?
- The closest alternatives we have reviewed are promptfoo, DeepEval and Inspect.
Answers checked Nov 5, 2026.