NVIDIA/SkillEvaluator — a trending Python project

NVIDIA/SkillEvaluator — Python · 1,317 stars · 155 forks · +133 today

Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.

📺 Livestream: Ask the Experts: Evaluating Agent Skills

SkillEvaluator checks AI agent skills and measures their effect on task results: start with quality checks, then compare runs with and without a skill.

Agent skills are folders of instructions and supporting files defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline.

Three ways in, one CLI, same results.

pip install skillevaluator

skillevaluator validate \
  ./my-skill --tiers 1

skillevaluator ci init

npx skills add \
  NVIDIA/SkillEvaluator

Guides: installation, CI setup (GitHub, GitLab, report examples) and agent skill.

Both paths use bundled text-analyzer.

Install uv and Git, then get the example:

git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator

uv provisions Python 3.13 and dependencies. For standalone installation, see Installation.

uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality

No API key, scanner, GPU or Docker is needed. The example scores 86.2/100 (B), exits 0 and saves reports/first-quality/skillevaluator-quality.json. The passing threshold defaults to 70.

This checks static quality; publication evidence stays INCOMPLETE. Security needs scanners; agent impact needs live evaluation. Replace the example path with your skill directory. See the keyless walkthrough.

Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for free prototyping, subject to access terms and rate limits.

1. Sign in at build.nvidia.com and complete enrollment.
2. Open a model page, select Get API Key, and generate your key.
3. Set the provider and key in the same terminal, then run the rubric check:

export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
uv run --python 3.13 --extra llm skillevaluator rubric-eval \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-build

The command sends skill documentation to the model and saves feedback in reports/first-build/skillevaluator-rubric.json. No agent runtime or Docker is needed. Completed evaluations can still fail rubric thresholds.

The same key supports Tier 2 embeddings and supported Tier 3 agents. See the Build walkthrough or provider configuration.

Each tier is independent. Add its extra to your command:

Tier 1 fetches pinned, verified SkillSpector and Gitleaks releases when they are missing; Semgrep is optional. See scanners. --no-llm does not remove scanner requirements; a scanner that cannot run is INCOMPLETE and exits nonzero. SkillSpector provides specialized skill security scanning; Harbor, the open-source agent evaluation framework, powers sandboxed agent runs.

Tier 2 reports when inter-skill comparison did not run. Tier 3 autopilot reuses sources or creates one skill-grounded case with limited coverage. See tier guides.

After setting up the provider, scanners, and agent sandbox, run all three tiers:

uv run --python 3.13 --extra all skillevaluator doctor --env-mode docker
uv run --python 3.13 --extra all skillevaluator validate ./my-skill

doctor checks readiness, not inference success. validate runs all tiers with autopilot for skills. Use --tiers 1,2 to skip live evaluation or --tiers 1 for static checks. --no-autopilot requires an existing source. Other artifact types retain their applicable stages.

Tier 1 gates validate. Tier 2 gates unless --no-block-on-dedup is set; Tier 3 gates only with --block-on-agent-eval. Model and managed-sandbox usage can incur charges; see runtime and cost.

Standalone generation needs substantive evals/EVAL.md guidance, a --prompt file with task requests and outcomes, or matching retained traces. --force below replaces the autopilot starter; --refine refines it. Explicit --grounding skill permits skill-only starters. Review cases; templates omit unauthored negatives.

uv run --python 3.13 --extra tier3 skillevaluator tier3 create-eval-dataset ./my-skill --full --prompt eval-guidance.md --force

Use uv run --python 3.13 with the relevant extra. See Installation for extras and standalone upgrades.

Resources: paper, PDF, DOI, BibTeX, and the NVIDIA Developer blog. Cite the paper and record the SkillEvaluator release or commit; see CITATION.cff.

See the documentation for setup, tier guides, reports, CI integration, CLI reference, and performance dashboard.

Follow the installation guide to choose the full installation or a smaller per-tier setup.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Read CONTRIBUTING.md and run:

make lint && make test && make build

See project governance and the Code of Conduct.

Support level: Experimental. Community-supported, best-effort, with no SLA or NVIDIA enterprise support entitlement. Report bugs and feature requests through GitHub Issues; see SUPPORT.md.

Report suspected vulnerabilities using the private process in SECURITY.md. Do not disclose security issues in a public GitHub issue.

Release changes are recorded in CHANGELOG.md and GitHub Releases.

Apache License 2.0 — see LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md.

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Content above is the README of NVIDIA/SkillEvaluator, reproduced from GitHub’s trending list. All rights remain with the original authors.