diff --git a/skills/nvskills-main-e2e-smoke/BENCHMARK.md b/skills/nvskills-main-e2e-smoke/BENCHMARK.md new file mode 100644 index 00000000..957aa659 --- /dev/null +++ b/skills/nvskills-main-e2e-smoke/BENCHMARK.md @@ -0,0 +1,98 @@ +# Skill Benchmark: nvskills-main-e2e-smoke + +> ✅ **Overall verdict: PASS — Recommended for publication** + +## Publication Recommendation + +Recommended for publication based on the completed evaluation evidence in this report. + +## Evaluation Metadata + +- Skill: `nvskills-main-e2e-smoke` +- Evaluation date: 2026-08-07 +- Evaluator version: `1.1.2` +- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`) +- Tasks: 3 evaluation tasks (2 positive, 1 negative) +- Dataset digest: `sha256:529636be4aaf027f37dcd8694549db6231c8e7aa782afc7b3b1d17ae0275397d` (skill-evaluator-dataset-snapshot/1) +- Attempts per task: 1 +- Environment: `k8s-sandbox` +- Tier 3 evidence: required for publication + +Each task attempt ran in its own isolated sandbox pod. + +## What This Report Answers + +The three-tier evaluation checks whether the skill: + +- is safe to use; +- produces correct answers; +- is discovered and activated when needed; +- helps the agent complete the user's goal and expected workflow; and +- avoids wasted skill and tool usage. + +## Results at a Glance + +| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | +|---|---:|---:| +| Overall | 52% → 99% (+48 points) | 54% → 99% (+45 points) | +| Security | 100% → 100% (±0 points) | 83% → 100% (+17 points) | +| Correctness | 33% → 100% (+67 points) | 47% → 100% (+53 points) | +| Discoverability | 46% → 100% (+54 points) | 44% → 96% (+52 points) | +| Effectiveness | 38% → 96% (+58 points) | 38% → 100% (+62 points) | +| Efficiency | 41% → 100% (+59 points) | 61% → 100% (+39 points) | + +**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points. + +Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline. + +## Tier Status + +| Tier | Purpose | Status | Evidence | +|---|---|---|---| +| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 1 finding(s) | +| Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded | +| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 3 task(s) | + +## Findings and Observations + +
+Show detailed findings and successful checks + +- **LOW** SCHEMA/author_format: Author must be of the form 'Name ' (`skills/nvskills-main-e2e-smoke/SKILL.md`) + +
+ +## Scoring Methodology + +
+Show dimension definitions, source signals, and thresholds + +| Dimension | Question | Scored signals | +|---|---|---| +| Security | Is it safe to use? | `security` (100%) | +| Correctness | Is the answer correct? | `accuracy` (100%) | +| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) | +| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) | +| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) | + +- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%. +- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL. +- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate. +- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold. +- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`). +- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict. + +Signals present in this run: + +- `security` (Security): unsafe operations, secret leakage, and unauthorized access. +- `skill_execution` (Skill Execution): whether the expected skill was found and executed. +- `skill_efficiency` (Efficiency): routing quality, workspace-aware skill reads, and productive tool use. +- `accuracy` (Accuracy): final-answer correctness against the reference answer. +- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved. +- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed. + +
+ +## Freshness + +Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes. diff --git a/skills/nvskills-main-e2e-smoke/SKILL.md b/skills/nvskills-main-e2e-smoke/SKILL.md new file mode 100644 index 00000000..f69f91f7 --- /dev/null +++ b/skills/nvskills-main-e2e-smoke/SKILL.md @@ -0,0 +1,32 @@ +--- +name: nvskills-main-e2e-smoke +description: Runs a deterministic local check for the production NVSkills workflow. Use when asked to verify the NVIDIA/skills and nvskills-ci main workflow integration. +license: Apache-2.0 +metadata: + author: "NVSkills CI" + tags: + - ci + - smoke-test + - signing +--- + +# NVSkills Main Workflow End-to-End Smoke Test + +Run a harmless local check that confirms this temporary smoke-test skill was discovered and invoked. + +## Instructions + +1. Run `python3 scripts/check.py`. +2. Confirm that the command prints `NVSKILLS_MAIN_E2E_OK`. +3. Return the exact output and report the check as passed. +4. Do not modify files, access the network, or run unrelated commands. + +## Examples + +- "Use the NVSkills main workflow smoke-test skill." +- "Verify the NVIDIA/skills and nvskills-ci main workflow integration." + +## Error Handling + +- If the script is missing, report the missing path and stop. +- If the output differs from the expected value, report the actual output and fail the check. diff --git a/skills/nvskills-main-e2e-smoke/evals/evals.json b/skills/nvskills-main-e2e-smoke/evals/evals.json new file mode 100644 index 00000000..b51f5ecc --- /dev/null +++ b/skills/nvskills-main-e2e-smoke/evals/evals.json @@ -0,0 +1,38 @@ +[ + { + "id": "nvskills-main-e2e-smoke-001", + "question": "Use the NVSkills main workflow smoke-test skill and report its result.", + "expected_skill": "nvskills-main-e2e-smoke", + "expected_script": "check.py", + "ground_truth": "The agent reads the skill instructions, runs scripts/check.py, and reports NVSKILLS_MAIN_E2E_OK.", + "expected_behavior": [ + "Read nvskills-main-e2e-smoke/SKILL.md", + "Executed check.py", + "Reported NVSKILLS_MAIN_E2E_OK", + "Did not modify files or access the network" + ] + }, + { + "id": "nvskills-main-e2e-smoke-002", + "question": "Verify the NVIDIA/skills and nvskills-ci main workflow integration.", + "expected_skill": "nvskills-main-e2e-smoke", + "expected_script": "check.py", + "ground_truth": "The agent invokes the main workflow smoke-test skill and returns its deterministic success message.", + "expected_behavior": [ + "Selected nvskills-main-e2e-smoke", + "Executed check.py", + "Returned the exact success message" + ] + }, + { + "id": "nvskills-main-e2e-smoke-negative-001", + "question": "What is the capital of France?", + "expected_skill": null, + "expected_script": null, + "ground_truth": "Paris is the capital of France.", + "expected_behavior": [ + "Answered without invoking nvskills-main-e2e-smoke", + "Did not execute check.py" + ] + } +] diff --git a/skills/nvskills-main-e2e-smoke/scripts/check.py b/skills/nvskills-main-e2e-smoke/scripts/check.py new file mode 100644 index 00000000..62f79611 --- /dev/null +++ b/skills/nvskills-main-e2e-smoke/scripts/check.py @@ -0,0 +1,16 @@ +#!/usr/bin/env python3 +"""Return the deterministic result used by the main workflow smoke test.""" + +from __future__ import annotations + + +SUCCESS_MESSAGE = "NVSKILLS_MAIN_E2E_OK" + + +def main() -> int: + print(SUCCESS_MESSAGE) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/skills/nvskills-main-e2e-smoke/skill-card.md b/skills/nvskills-main-e2e-smoke/skill-card.md new file mode 100644 index 00000000..9b0f2320 --- /dev/null +++ b/skills/nvskills-main-e2e-smoke/skill-card.md @@ -0,0 +1,81 @@ +## Description:
+Runs a deterministic local check for the production NVSkills workflow.
+ +This skill is ready for commercial/non-commercial use.
+ +## Owner +NVIDIA
+ +### License/Terms of Use:
+Apache-2.0
+## Use Case:
+Developers and CI engineers use this skill to verify that the NVIDIA/skills and nvskills-ci main workflow integration is functioning correctly.
+ +### Deployment Geography for Use:
+Global
+ +## Requirements / Dependencies:
+**Requires API Key or External Credential:** [No]
+**Credential Type(s):** [None]
+ +Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate.
+ +## Known Risks and Mitigations:
+Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills.
+Mitigation: Review and scan skill before deployment.
+ +## Reference(s):
+- [BENCHMARK.md](BENCHMARK.md)
+ + +## Skill Output:
+**Output Type(s):** [Shell commands]
+**Output Format:** [Plain text]
+**Output Parameters:** [1D]
+**Other Properties Related to Output:** [None]
+ +## Evaluation Agents Used:
+- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`)
+- Codex (`openai/openai/gpt-5.5`)
+ + + +## Evaluation Tasks:
+3 evaluation tasks (2 positive, 1 negative), each run in an isolated sandbox pod.
+ +## Evaluation Metrics Used:
+Reported benchmark dimensions:
+- Security: Whether the skill is safe to use (no unsafe operations, secret leakage, or unauthorized access).
+- Correctness: Whether the agent produces a correct final answer against the reference.
+- Discoverability: Whether the right skill was found and executed when needed.
+- Effectiveness: Whether the skill helps complete the user's goal and follows expected workflow behavior.
+- Efficiency: Whether the agent avoids wasted skill and tool usage.
+ +Underlying evaluation signals used in this run:
+- `security`: Checks for unsafe operations, secret leakage, and unauthorized access.
+- `accuracy`: Final-answer correctness against the reference answer.
+- `skill_execution`: Whether the expected skill was found and executed.
+- `goal_accuracy`: Whether the user's goal was achieved.
+- `behavior_check`: Whether the expected workflow behavior was followed.
+- `skill_efficiency`: Routing quality, workspace-aware skill reads, and productive tool use.
+ + + +## Evaluation Results:
+| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) | +|---|---:|---:| +| Overall | 52% → 99% (+48 points) | 54% → 99% (+45 points) | +| Security | 100% → 100% (±0 points) | 83% → 100% (+17 points) | +| Correctness | 33% → 100% (+67 points) | 47% → 100% (+53 points) | +| Discoverability | 46% → 100% (+54 points) | 44% → 96% (+52 points) | +| Effectiveness | 38% → 96% (+58 points) | 38% → 100% (+62 points) | +| Efficiency | 41% → 100% (+59 points) | 61% → 100% (+39 points) | + +## Skill Version(s):
+2922634 (source: git SHA, committed 2026-08-07)
+ +## Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
+ +(For Release on NVIDIA Platforms Only)
+Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
diff --git a/skills/nvskills-main-e2e-smoke/skill.oms.sig b/skills/nvskills-main-e2e-smoke/skill.oms.sig new file mode 100644 index 00000000..dc8750ef --- /dev/null +++ b/skills/nvskills-main-e2e-smoke/skill.oms.sig @@ -0,0 +1 @@ +{"mediaType":"application/vnd.dev.sigstore.bundle.v0.3+json","verificationMaterial":{"x509CertificateChain":{"certificates":[{"rawBytes":"MIICgzCCAgmgAwIBAgIUKIyS7SxNteQIiWzK1dWj85E6520wCgYIKoZIzj0EAwMwVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwHhcNMjYwNDAxMDAwMDAwWhcNMjgwNDIyMTUzMzA5WjBUMQswCQYDVQQGEwJVUzEbMBkGA1UECgwSTlZJRElBIENvcnBvcmF0aW9uMSgwJgYDVQQDDB9OVklESUEgQWdlbnQgU2tpbGxzIFNpZ25pbmcgMDAxMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAEYoRM9bQl/dGlwSRNi6bTpIJUXH8Nv9GciP6LSflJYYMLCc296kpyuTSsk5ddbAWiDcFX3C/ydX3jwc+qCLYP6uHy9XphyLjOQ27Yb2J6rBLVtRBS1mgGco/Gr7fL6ODco4GaMIGXMB0GA1UdDgQWBBRQ/5ZW3nJ6lmo9SVk7I15o7UGmpTAfBgNVHSMEGDAWgBRPGpILxMBBleJSsBGjrMKsby1CgjAMBgNVHRMBAf8EAjAAMA4GA1UdDwEB/wQEAwIHgDA3BggrBgEFBQcBAQQrMCkwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vY3NwLm5kaXMubnZpZGlhLmNvbTAKBggqhkjOPQQDAwNoADBlAjAUygu/GiOCIXrgGr4SmLgeEVDcEitfFUv7ALbvLVGVyMysB3mxmO/uInZfXzWcJZsCMQDxuoxj4ZmO30jhkPIcCxGFCOvnUsnfU3TfGcouYm4M6iRpbKvtVnHPiy4bi6pcKf0="},{"rawBytes":"MIICiDCCAg6gAwIBAgIUZsIuSv9NkpJCNqtYEfCouVv5BzowCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowVTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjEpMCcGA1UEAwwgTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBJQ0EgMDEwdjAQBgcqhkjOPQIBBgUrgQQAIgNiAASI72cR3ctKGg4VWnB3bNja6g1Z2PnOmFEopkPof+QeIcPk9rT+g9MjJnq51EQXL93a7C2GJ9J985G4o2V85VD7wJ1RaXhluHW2rf3y8bQGeAYaKMr5s/hUgn+M3/9WlWejgaAwgZ0wHQYDVR0OBBYEFE8akgvEwEGV4lKwEaOswqxvLUKCMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDcGCCsGAQUFBwEBBCswKTAnBggrBgEFBQcwAYYbaHR0cDovL29jc3AubmRpcy5udmlkaWEuY29tMAoGCCqGSM49BAMDA2gAMGUCMQCeIMMfAbyzPDacw2MxG+Yt1cikrJX/DVxiGfXuHmkkXn6VgSzE79+lkqDErpVO2gYCMCNEColOyvUvkzZGUEI1hQ3PfMgi3FIo9tHoBKMw4/wGBLFpu/0ubtmbBXM6/UMOEw=="},{"rawBytes":"MIICRTCCAcygAwIBAgIUeJdY3rV86EdvFmG7L8LJBsyQFYkwCgYIKoZIzj0EAwMwUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTAgFw0yNjA0MDEwMDAwMDBaGA85OTk5MTIzMTIzNTk1OVowUTELMAkGA1UEBhMCVVMxGzAZBgNVBAoMEk5WSURJQSBDb3Jwb3JhdGlvbjElMCMGA1UEAwwcTlZJRElBIEFnZW50IENhcGFiaWxpdGllcyBDQTB2MBAGByqGSM49AgEGBSuBBAAiA2IABAYpiXCDjJ9NT2eSDhyHJVSw1Tbze18cGG2F/578oWvHxg23eQAhNRYdq88i1iOshZSO6C29doKui5Xpmo/7Ctw9Sx4PP2RzOmIuOLCuTdNtKcTRwi4GEsd5BAFvWj42M6NjMGEwHQYDVR0OBBYEFItnoAjjfuCEUvzyvWyI2vOGvwPjMB8GA1UdIwQYMBaAFItnoAjjfuCEUvzyvWyI2vOGvwPjMA8GA1UdEwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMAoGCCqGSM49BAMDA2cAMGQCMCwtAjWLaNwgGWNCgdyNoTyvNhqWRECRJV2r3+7w8g0PL6NHLOsbkgE09BH95h8XlgIwTaQmbbUh2ChAJ5TA1wRiVDnCcvbzHlZl2jM2FcwQQZlk19LOAbyGMRixbu2Ww/rj"}]},"tlogEntries":[]},"dsseEnvelope":{"payload":"ewogICJfdHlwZSI6ICJodHRwczovL2luLXRvdG8uaW8vU3RhdGVtZW50L3YxIiwKICAic3ViamVjdCI6IFsKICAgIHsKICAgICAgIm5hbWUiOiAibnZza2lsbHMtbWFpbi1lMmUtc21va2UiLAogICAgICAiZGlnZXN0IjogewogICAgICAgICJzaGEyNTYiOiAiNjQ5ZGQ0NWNiY2U0YWVhNzg1OGJhOTliZDNkOTY2MTAzMzEwYmIwMTVlZGRkYjljMWE2MTZmZDU0ODJlYWM5ZCIKICAgICAgfQogICAgfQogIF0sCiAgInByZWRpY2F0ZVR5cGUiOiAiaHR0cHM6Ly9tb2RlbF9zaWduaW5nL3NpZ25hdHVyZS92MS4wIiwKICAicHJlZGljYXRlIjogewogICAgInNlcmlhbGl6YXRpb24iOiB7CiAgICAgICJoYXNoX3R5cGUiOiAic2hhMjU2IiwKICAgICAgImFsbG93X3N5bWxpbmtzIjogZmFsc2UsCiAgICAgICJpZ25vcmVfcGF0aHMiOiBbCiAgICAgICAgIi5naXRhdHRyaWJ1dGVzIiwKICAgICAgICAiLmdpdCIsCiAgICAgICAgIi5naXRpZ25vcmUiLAogICAgICAgICIuZ2l0aHViIgogICAgICBdLAogICAgICAibWV0aG9kIjogImZpbGVzIgogICAgfSwKICAgICJyZXNvdXJjZXMiOiBbCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjdkOGQ3MjQyMWRiZDM0ZDcyNzlhYmVlNjNiNzg3ZWY1OTAxNmFlZjBmNjczODY4YWYxYTk4MjU2ZDA2YWZmMjkiLAogICAgICAgICJuYW1lIjogIkJFTkNITUFSSy5tZCIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogImRmNDNiNTdmMjA4MWE2ZDQzNzYwNTM1M2RkNDM2NzY1ZTAwMTA4NTYzMTFjN2ZmMjc2MWI4M2ZkZGNhZjJlZDAiLAogICAgICAgICJuYW1lIjogIlNLSUxMLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfSwKICAgICAgewogICAgICAgICJkaWdlc3QiOiAiNjcwNWJlNTE5ZTE1MTAwNGVkNmQ1NDY2Y2M5MTY1NDc0ZDYzMWY2YTQ3MjhiYTdhZWVmMDZmODMxZjFhMjJlNCIsCiAgICAgICAgIm5hbWUiOiAiZXZhbHMvZXZhbHMuanNvbiIsCiAgICAgICAgImFsZ29yaXRobSI6ICJzaGEyNTYiCiAgICAgIH0sCiAgICAgIHsKICAgICAgICAiZGlnZXN0IjogIjg5M2E3NTc2ZDU1NWE1MTZkZGNhNWEwNGEwN2I0NzU2ZTIxYmZjNDlhMGE3YTlmNmQ1MjJhZmU2ZDk5ZDRjYzEiLAogICAgICAgICJuYW1lIjogInNjcmlwdHMvY2hlY2sucHkiLAogICAgICAgICJhbGdvcml0aG0iOiAic2hhMjU2IgogICAgICB9LAogICAgICB7CiAgICAgICAgImRpZ2VzdCI6ICJlMTRjNDg5ZWZmZWMyMjAwYjMzY2IwYjI1MTRhY2U2YTY3NGNjZjllNzFhNjA1YTZiMGJjNmQyMmQ1NTU0YjQ0IiwKICAgICAgICAibmFtZSI6ICJza2lsbC1jYXJkLm1kIiwKICAgICAgICAiYWxnb3JpdGhtIjogInNoYTI1NiIKICAgICAgfQogICAgXQogIH0KfQ==","payloadType":"application/vnd.in-toto+json","signatures":[{"sig":"MGUCMQDgjo1MGYR8FcJ26dFxjxMtpLEvx6ypcFwHcw+WC8vPGz2mKqxSI0wEFJuCbbAoBS0CMAJbQ4ls4/K+I/BuhkGGopbMFevf6YZegHFEx1KWNfYTiPf2P3wjg/nDiZ/LGbBhwA==","keyid":""}]}} \ No newline at end of file