What is SkillEvaluator?
SkillEvaluator is NVIDIA's open-source framework for testing agent skills, the folders of instructions and scripts defined by the Agent Skills specification. It is the tool behind the NVIDIA Verified Skills pipeline and the reference implementation of an August 2026 arXiv paper on evaluating skills rather than agents.
It works in three independent tiers. Tier 1 validates structure and frontmatter, scores quality and runs security scans with NVIDIA's SkillSpector, Gitleaks and optionally Semgrep. Tier 2 uses embeddings to flag guidance repeated within a skill or overlapping with others in a catalogue. Tier 3 runs a real agent (Codex, Claude Code or OpenCode, with Hermes experimental) in a sandbox on the Harbor framework, twice per task, once with the skill installed and once without, and reports the difference as Skill Lift.
It installs from PyPI as skillevaluator on Python 3.12 to 3.14, and skillevaluator ci init writes GitHub or GitLab workflows that check changed skills on every pull request. Version 0.9.0 shipped on 13 November 2026 under Apache 2.0, with support marked experimental: community, best-effort, no SLA.
Is SkillEvaluator worth it?
Most skill tooling checks what a skill says. SkillEvaluator measures what it does: the same task, model, inputs and grader, run with and without the skill, and the gap reported as Skill Lift. NVIDIA's own snapshot of 12 August 2026, across more than 300 verified skills for over 30 products on Claude Code and Codex, put correctness at 46 without a skill and 87 with one, and average lift at 31 points across all dimensions, or 39 leaving out security. The accompanying paper is more sober: over 947 scored paired cases covering 58 of 64 production skills on four agent harnesses, mean composite lift was 0.2134 and positive in 72.8% of cases.
Those numbers come with caveats the authors state themselves. In the blog snapshot 85% of skills got a single attempt per task and no confidence intervals are reported; a skill that loads when it is irrelevant can make the agent worse; and the efficiency effect cuts both ways, with one Jetson skill cutting tokens by 76.9% and a cuOpt install skill raising them by 120.3%. The paper also found static scores weakly correlated with LLM-judge scores (Spearman 0.14), which is the strongest case for running live tiers rather than trusting a quality grade.
The cheap path is genuinely cheap. Tier 1 runs with no API key, GPU or Docker; the bundled example skill scores 84.5 out of 100 against a default pass mark of 70, and the generated CI workflow pins a verified Docker image. Tier 3 is where cost and setup arrive: Docker or a managed sandbox (Daytona, E2B, Modal or Runloop), provider keys for both the agent and the judge, and a trial count of cases times agents times attempts times arms. The docs refuse to estimate a dollar figure, though 0.9.0 added token, judge-call and cost budgets that stop a run and keep partial results.
It is young. The first release was 0.1.0 on 5 August 2026 and 0.9.0 is the eleventh, with grading changes in 0.9.0 that the notes say make scores incomparable with earlier releases. Agent versions are pinned (Codex 0.161.0, Claude Code 2.1.292, OpenCode 1.19.5) so results stay reproducible, which also means you are not testing the CLI your team runs today. For teams publishing skills to others, that churn is worth tolerating; for a handful of personal skills, Tier 1 alone or SkillSpector covers most of the value.
Where it earns its place
A developer-relations team at a database company maintaining 25 public Claude Code and Codex skills
Block skill changes that break structure or leak secrets before they merge
Run skillevaluator ci init, accept the Tier 1 checks and choose the skills folder; each pull request then gets SkillSpector and Gitleaks scans and a quality score with a default pass mark of 70. No model key is needed for this tier, so the CI cost is runner minutes.
A platform engineer whose company's internal skill catalogue has grown past 100 entries
Find skills that repeat each other and confuse the agent's choice
Run Tier 2 with --catalog pointing at a catalogue JSON so embeddings compare guidance across skills, then merge or retire the overlapping ones. It needs an embeddings and chat provider; NVIDIA Build offers free prototyping for eligible members.
An AI lead deciding whether a vendor's skill is worth adding to 200 developers' agents
Prove the skill improves results on the team's own tasks before rolling it out
Write a small evaluation dataset from real requests, run Tier 3 with --n-attempts 1 and a cost budget on one agent, then compare Skill Lift against the no-skill baseline. Each case runs twice, so model spend roughly doubles against a single agent run.
Before you commit
- Setup
- pip install --upgrade skillevaluator, or clone and use uv run --python 3.13. skillevaluator quality-check on a skill folder gives a deterministic score with no key. Tier 3 adds the tier3 extra, provider keys and Docker; skillevaluator doctor checks readiness, not inference.
- Integrations
- Agents: Codex, Claude Code and OpenCode, Hermes experimental. Providers: NVIDIA Build, OpenAI, Anthropic and Amazon Bedrock via LiteLLM. Sandboxes: Docker, Kata, Daytona, E2B, Modal, Runloop and EC2 through Harbor 0.25.0. Scanners: SkillSpector, Gitleaks, Semgrep. Reports in terminal, HTML, JSON, Markdown and SARIF; GitHub and GitLab CI.
- Data and privacy
- Runs locally or in your CI. LLM-backed tiers send skill documentation and task traces to the model provider you configure; Harbor telemetry and .env loading are disabled in its child processes. Imported production traces are redacted and need human review before use. Apache 2.0.
- Cost at scale
- The tool is free; Tier 3 cost is eval cases times agents times attempts times two arms, each trial making several agent and judge calls, plus sandbox fees on managed backends. No dollar estimate is published; --skip-baseline roughly halves trials during dataset work but cannot produce Skill Lift.
- Limits
- Experimental support; grading changed in 0.9.0, so older scores are not comparable; pinned agent CLI versions; results vary between runs and depend on dataset quality; Claude Code on NVIDIA Build models runs through a compatibility bridge marked experimental.
What’s good
- Measures skill impact with paired with-and-without runs
- Quality checks need no API key, GPU or Docker
- Ready GitHub and GitLab CI workflows
- Published methodology and benchmark caveats
What’s not
- Scores changed meaning between releases
- Live runs need Docker, keys and a self-sized budget
- Pinned agent versions lag current CLIs
- Experimental support with no SLA
Best for
Teams that publish agent skills for others and need evidence, in CI, that each skill is safe, distinct and actually improves agent results.
SkillEvaluator alternatives
SkillEvaluator: common questions
- Is SkillEvaluator worth it?
- SkillEvaluator asks the question most skill scanners skip, whether a skill actually helps the agent, and answers it with paired live runs, at the price of a young, fast-moving tool and a model bill you size yourself. We rate it 4.0 out of 5.
- How much does SkillEvaluator cost?
- Free and open source under Apache 2.0. Tier 1 needs no model; LLM checks and live runs bill at your model provider's rates, with free prototyping on NVIDIA Build for eligible Developer Program members, plus managed-sandbox charges if you use one.
- What are the downsides of SkillEvaluator?
- Scores changed meaning between releases; live runs need Docker, keys and a self-sized budget; pinned agent versions lag current CLIs; experimental support with no SLA.
- Who is SkillEvaluator best for?
- Teams that publish agent skills for others and need evidence, in CI, that each skill is safe, distinct and actually improves agent results.
- How much does SkillEvaluator cost for a team or in production?
- The tool is free; Tier 3 cost is eval cases times agents times attempts times two arms, each trial making several agent and judge calls, plus sandbox fees on managed backends. No dollar estimate is published; --skip-baseline roughly halves trials during dataset work but cannot produce Skill Lift.
- Is SkillEvaluator safe for company data?
- Runs locally or in your CI. LLM-backed tiers send skill documentation and task traces to the model provider you configure; Harbor telemetry and .env loading are disabled in its child processes. Imported production traces are redacted and need human review before use. Apache 2.0.
- What are the best SkillEvaluator alternatives?
- The closest alternatives we have reviewed are SkillSpector, Cisco Skill Scanner, promptfoo and Inspect.
Answers checked Nov 13, 2026.
Make SkillEvaluator? Show this rating on your site
