What is Dokimi?
Dokimi (Greek for "test" or "proof") checks whether something you add to a coding agent's context actually helps. It runs the same questions or coding tasks twice, once without the change and once with it, grades both the same way and reports the difference with a p-value, per model. The change can be a skill, a whole skill catalogue, product documentation, an MCP documentation server, an agent persona, or a repository's AGENTS.md or CLAUDE.md.
It has three modes. Arm comparisons answer "does it help?" using an LLM judge for questions, or real Docker coding tasks run through Claude Code, Codex, Harbor or BenchFlow harnesses and graded by tests. Novelty analysis splits a skill into individual claims and asks the model about each with and without the skill, sorting them into added, redundant, degraded or still missing, and proposes a shorter version. Repository mode checks that the paths and commands your guidance names exist, measures how much guidance each model call pays for, and runs agents on the repository's own tasks with the guidance removed or kept.
It is a Python command-line tool from Intel under Apache-2.0, released on PyPI as version 0.1.0 on 19 October 2026.
Is Dokimi worth it?
Dokimi is the most rigorous open tool we have found for testing whether a skill, MCP server or AGENTS.md makes a coding agent better, and it is too new to rely on. The repository was created on 8 October 2026, has 11 GitHub stars and 9 forks, and about 390 of its 563 commits come from one Intel engineer. Activity is high (85 open issues and pull requests, a commit on the day we checked), but there is no outside adoption to point to yet.
The idea is the right one. Most harness advice is written and shared without anyone measuring it, and the context costs are real: every line of a skill or context file is paid for on every turn. Dokimi runs paired on and off arms on the same tasks and model and reports the delta with statistics, so a skill that helps a small model but does nothing for a frontier one shows up as exactly that. Its own study of 33 Intel skills found that, under lenient grading, 70% of what a skill says a frontier model already produces unprompted, and that a plausible wrong value planted in a skill was repeated in 85 to 94% of cases. Those are the project's numbers, from a prototype run outside the repository, and its docs say to treat them as evidence for the design rather than a baseline.
The catch: this is a research instrument, not a dashboard. The docs are long and precise and assume you know what a judge model, a paired test and a Harbor task are. Default models are openai/gpt-6.1-sol for answers and anthropic/claude-sonnet-5-5 as the judge, so two API keys are the usual start. Executable runs need Docker and your own graded tasks; SWE-bench images are about 4 GB each. Costs are visible but add up: triage on one 19k-token skill was 103 calls and about $1.80, the full study tier on the same skill 1,900 calls and about $14.50, and repository agent runs default to a $2.00 budget per run with no batch cap unless you set one.
Our take: Dokimi is the lab bench, not the shop floor. The free checks (dokimi repo scan, catalog check and --estimate-only) are worth an afternoon for anyone who maintains a skill pack or a long AGENTS.md; the paid experiments make sense once the project has a few more months and a few more users behind it.
Where it earns its place
A developer-relations engineer maintaining 30 agent skills for an SDK
Find out which skills, and which parts of them, a frontier model actually needs
Run dokimi skill novelty analyze at the triage tier with --estimate-only first, then for real with --max-usd set; the docs' example cost about $1.80 per 19k-token skill and returns which claims are redundant or harmful.
A tech lead whose monorepo has a 400-line AGENTS.md
Check the guidance is accurate and see what it costs on every model call
dokimi repo scan flags paths, links and commands that no longer exist, and dokimi repo budget counts the characters every call pays for at start-up. Both run without a model and cost nothing.
A platform team choosing between two MCP documentation servers
Measure whether either improves an agent's answers on their own product questions
Write a golden question set, run arms run with baseline and one mcp: arm per server on the same model, and compare judge scores and p-values. arms run has no spend cap, so size questions times arms before starting.
Before you commit
- Setup
- pip install dokimi (Python 3.11 to 3.14), set OPENAI_API_KEY and ANTHROPIC_API_KEY for the default models, or sign in to Claude Code and use the claude-cli backend for question-answer runs. Coding-task runs also need Docker.
- Integrations
- Claude Code, Codex, Harbor (Terminal-Bench) and BenchFlow harnesses for executable tasks; SWE-bench rows; Context7 and MCP servers as doc sources; model calls go through LiteLLM pricing, with OpenAI, Anthropic, Google, Vertex, OpenRouter and Bedrock routes.
- Data and privacy
- Runs locally and sends prompts, skill text and documentation only to the model providers you configure. Arms artifacts store the exact skill or doc text the judge read, so sharing a result file shares that content.
- Cost at scale
- Static checks are free. In the docs' example, triage on a 19k-token skill cost about $1.80 and the study tier about $14.50; repository agent runs default to $2.00 each, so a matrix of tasks, arms, models and repeats needs a batch cap.
- Limits
- Version 0.1.0 with fast-changing defaults (the release notes say scores differ from earlier defaults); the claude-cli backend refuses tool-using arms and sampling settings; scores from different transports cannot be pooled.
What’s good
- Paired with and without runs, per model, with p-values
- Splits a skill into claims the model already knew or needed
- Free static checks of repository guidance
- Spend caps and cost estimates before paid runs
What’s not
- Created in October 2026; 11 stars
- Mostly one maintainer
- Steep learning curve and long docs
- Headline findings come from the project's own study
Best for
Engineers who maintain a skill pack, MCP doc server or large AGENTS.md and want evidence that it changes what an agent does.
Dokimi alternatives
Dokimi: common questions
- Is Dokimi worth it?
- Dokimi asks the question every skill pack and AGENTS.md should have to answer, and asks it carefully, but it is a month old with 11 stars and needs a patient engineer to run. We rate it 3.4 out of 5.
- How much does Dokimi cost?
- Free and open source under Apache-2.0. You pay for model calls: a triage analysis of one 19k-token skill was about $1.80 in the docs' example, and several checks need no model at all.
- What are the downsides of Dokimi?
- Created in October 2026; 11 stars; mostly one maintainer; steep learning curve and long docs; headline findings come from the project's own study.
- Who is Dokimi best for?
- Engineers who maintain a skill pack, MCP doc server or large AGENTS.md and want evidence that it changes what an agent does.
- How much does Dokimi cost for a team or in production?
- Static checks are free. In the docs' example, triage on a 19k-token skill cost about $1.80 and the study tier about $14.50; repository agent runs default to $2.00 each, so a matrix of tasks, arms, models and repeats needs a batch cap.
- Is Dokimi safe for company data?
- Runs locally and sends prompts, skill text and documentation only to the model providers you configure. Arms artifacts store the exact skill or doc text the judge read, so sharing a result file shares that content.
- What are the best Dokimi alternatives?
- The closest alternatives we have reviewed are promptfoo, Inspect and SkillSpector.
Answers checked Nov 7, 2026.