NVIDIA · mosheabr · Aug 12, 2026 · Aug 12, 2026
@@ -1,95 +1,103 @@
-# Evaluation Report
+# Skill Benchmark: omniverse-cad-to-simready
 
-Evaluation of the `omniverse-cad-to-simready` skill before publication through NVSkills-Eval.
+> ✅ **Overall verdict: PASS — Recommended for publication**
 
-This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.
+## Publication Recommendation
+
+Recommended for publication based on the completed evaluation evidence in this report.
 
-## Evaluation Summary
+## Evaluation Metadata
 
 - Skill: `omniverse-cad-to-simready`
-- Evaluation date: 2026-05-28
-- NVSkills-Eval profile: `external`
-- Overall verdict: FAIL
-- Tier 3 live agent evaluation: not available in this report
+- Evaluation date: 2026-08-12
+- Evaluator version: `1.2.4`
+- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`)
+- Tasks: 8 evaluation tasks (7 positive, 1 negative)
+- Dataset digest: `sha256:ce412af18bb5e516371ec07ea8752e4043a1d00edd007b505e30f659b8d92047` (skill-evaluator-dataset-snapshot/1)
+- Attempts per task: 1
+- Environment: `k8s-sandbox`
+- Tier 3 evidence: required for publication
 
-## Agents Used
+Each task attempt ran in its own isolated sandbox pod.
 
-- Tier 3 agent details were not available in this report.
+## What This Report Answers
 
-## Metrics Used
+The three-tier evaluation checks whether the skill:
 
-Reported benchmark dimensions:
+- is safe to use;
+- produces correct answers;
+- is discovered and activated when needed;
+- helps the agent complete the user's goal and expected workflow; and
+- avoids wasted skill and tool usage.
 
-- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
-- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
-- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
-- Effectiveness: checks whether the agent performs measurably better with the skill than without it.
-- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.
+## Results at a Glance
 
-Underlying evaluation signals used in this run:
+| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
+|---|---:|---:|
+| Overall | Not available | 48% → 82% (+33 points) |
+| Security | Not available | 50% → 81% (+31 points) |
+| Correctness | Not available | 57% → 88% (+30 points) |
+| Discoverability | Not available | 48% → 81% (+34 points) |
+| Effectiveness | Not available | 37% → 75% (+38 points) |
+| Efficiency | Not available | 49% → 83% (+34 points) |
 
-- No Tier 3 evaluation signal details were available in this report.
+**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points.
 
-## Test Tasks
+Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline.
 
-Tier 3 evaluation task details were not available in this report.
+## Tier Status
 
-## Results
+| Tier | Purpose | Status | Evidence |
+|---|---|---|---|
+| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 6 finding(s) |
+| Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded |
+| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 8 task(s) |
 
-Tier 3 dimension rollup was not available in this report.
+## Findings and Observations
 
-## Tier 1: Static Validation Summary
+<details>
+<summary>Show detailed findings and successful checks</summary>
 
-Tier 1 validation passed with observations. NVSkills-Eval ran 9 checks and found 10 total findings.
+- **MEDIUM** SCHEMA/frontmatter_field_placement: Root field 'version' is ignored; use 'metadata.version' (`skills/omniverse-cad-to-simready/SKILL.md`)
+- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/omniverse-cad-to-simready/SKILL.md`)
+- **LOW** SCHEMA/unexpected_file: Unexpected 'upstream-versions.lock.json' in skill root (`skills/omniverse-cad-to-simready/upstream-versions.lock.json`)
+- **LOW** SCHEMA/unexpected_file: Unexpected 'shared' in skill root (`skills/omniverse-cad-to-simready/shared`)
+- **LOW** SCHEMA/unexpected_file: Unexpected 'CHANGELOG.md' in skill root (`skills/omniverse-cad-to-simready/CHANGELOG.md`)
+- 1 additional finding(s) are available in the full evaluation artifacts.
 
-Top findings:
+</details>
 
-- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/omniverse-cad-to-simready/SKILL.md`)
-- LOW QUALITY/quality_correctness: No examples provided (`skills/omniverse-cad-to-simready/SKILL.md`)
-- LOW QUALITY/quality_discoverability: Description very long (422 chars, recommend 50-150) (`skills/omniverse-cad-to-simready/SKILL.md`)
-- LOW QUALITY/quality_discoverability: No '## Purpose' section (`skills/omniverse-cad-to-simready/SKILL.md`)
-- LOW SCHEMA/unexpected_file: Unexpected 'BENCHMARK.md' in skill root (`skills/omniverse-cad-to-simready/BENCHMARK.md`)
+## Scoring Methodology
 
-## Tier 2: Deduplication Summary
+<details>
+<summary>Show dimension definitions, source signals, and thresholds</summary>
 
-Tier 2 validation reported findings. NVSkills-Eval ran 2 checks and found 13 total findings.
+| Dimension | Question | Scored signals |
+|---|---|---|
+| Security | Is it safe to use? | `security` (100%) |
+| Correctness | Is the answer correct? | `accuracy` (100%) |
+| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
+| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
+| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) |
 
-Top findings:
+- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
+- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
+- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
+- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
+- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
+- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict.
 
-- HIGH DUPLICATE/duplicate: Duplicate content found across references/omni-asset-validate-geometry/scripts/run.py and references/omni-asset-validate-physics/scripts/run.py:
-  "validate()" in references/omni-asset-validate-geometry/scripts/run.py (lines 25-35)
-  vs "validate()" in references/omni-asset-validate-physics/scripts/run.py (lines 25-35) (`references/omni-asset-validate-geometry/scripts/run.py:25`)
-- HIGH DUPLICATE/duplicate: Duplicate content found within references/simready-conform-profile/scripts/run.py:
-  "_run_fet000()" in references/simready-conform-profile/scripts/run.py (lines 164-216)
-  vs "_run_fet001()" in references/simready-conform-profile/scripts/run.py (lines 219-256)
-  vs "_run_fet004()" in references/simready-conform-profile/scripts/run.py (lines 259-299)
-  vs "_run_fet005()" in references/simready-conform-profile/scripts/run.py (lines 302-367) (`references/simready-conform-profile/scripts/run.py:164`)
-- HIGH DUPLICATE/duplicate: Duplicate content found across references/content-agents/scripts/content_agent_client.py and references/content-agents/scripts/run.py and references/convert-to-usd/references/mujoco-usd-converter/scripts/run.py and references/convert-to-usd/references/urdf-usd-converter/scripts/run.py and references/convert-to-usd/references/usd-convert-cad/scripts/run.py and references/convert-to-usd/references/usd-convert-gsplat/scripts/run.py and references/identify-asset-context/scripts/run.py and references/omni-asset-validate-geometry/scripts/run.py and references/omni-asset-validate-physics/scripts/run.py and references/ovrtx-render-service/scripts/run.py and references/ovrtx-render-service/scripts/turntable.py and references/simready-conform-profile/references/FET_000_CORE/scripts/run.py and references/simready-conform-profile/references/FET_001_MINIMAL/scripts/run.py and references/simready-conform-profile/references/FET_004_SIMULATE_MULTI_BODY_PHYSICS/scripts/run.py and references/simready-conform-profile/references/FET_005_SIMULATE_GRASP_PHYSICS/scripts/author_grasp_line.py and references/simready-conform-profile/scripts/run.py and references/simready-validate/scripts/run.py and shared/simready_package.py:
-  "_emit_report()" in references/content-agents/scripts/content_agent_client.py (lines 1276-1277)
-  vs "emit()" in references/content-agents/scripts/run.py (lines 217-230)
-  vs "emit_probe()" in references/convert-to-usd/references/mujoco-usd-converter/scripts/run.py (lines 216-217)
-  vs "emit_probe()" in references/convert-to-usd/references/urdf-usd-converter/scripts/run.py (lines 214-215)
-  vs "emit_probe()" in references/convert-to-usd/references/usd-convert-cad/scripts/run.py (lines 554-555)
-  vs "emit_probe()" in references/convert-to-usd/references/usd-convert-gsplat/scripts/run.py (lines 311-312)
-  vs "_emit()" in references/identify-asset-context/scripts/run.py (lines 275-276)
-  vs "emit()" in references/omni-asset-validate-geometry/scripts/run.py (lines 38-44)
-  vs "emit()" in references/omni-asset-validate-physics/scripts/run.py (lines 38-44)
-  vs "_emit()" in references/ovrtx-render-service/scripts/run.py (lines 358-359)
-  vs "_emit()" in references/ovrtx-render-service/scripts/turntable.py (lines 227-228)
-  vs "emit()" in references/simready-conform-profile/references/FET_000_CORE/scripts/run.py (lines 213-219)
-  vs "emit()" in references/simready-conform-profile/references/FET_001_MINIMAL/scripts/run.py (lines 331-338)
-  vs "emit()" in references/simready-conform-profile/references/FET_004_SIMULATE_MULTI_BODY_PHYSICS/scripts/run.py (lines 275-288)
-  vs "write_reports()" in references/simready-conform-profile/references/FET_005_SIMULATE_GRASP_PHYSICS/scripts/author_grasp_line.py (lines 68-96)
-  vs "emit()" in references/simready-conform-profile/scripts/run.py (lines 464-475)
-  vs "emit()" in references/simready-validate/scripts/run.py (lines 700-706)
-  vs "_emit()" in shared/simready_package.py (lines 662-663) (`references/content-agents/scripts/content_agent_client.py:1276`)
-- HIGH DUPLICATE/duplicate: Duplicate content found across references/content-agents/README.md and references/content-agents/references/material-agent-client/README.md:
-  "## Rate Limits" in references/content-agents/README.md (lines 125-133)
-  vs "## Rate Limits" in references/content-agents/references/material-agent-client/README.md (lines 93-101) (`references/content-agents/README.md:125`)
-- HIGH DUPLICATE/duplicate: Duplicate content found across references/nv-core-package-sample-validation/README.md and references/nv-core-package-sample/README.md:
-  "## Upstream Reference" in references/nv-core-package-sample/README.md (lines 21-32)
-  vs "## Upstream Reference" in references/nv-core-package-sample-validation/README.md (lines 9-20) (`references/nv-core-package-sample/README.md:21`)
+Signals present in this run:
 
-## Publication Recommendation
+- `security` (Security): unsafe operations, secret leakage, and unauthorized access.
+- `skill_execution` (Skill Execution): whether the expected skill was found and executed.
+- `skill_efficiency` (Efficiency): routing quality, workspace-aware skill reads, and productive tool use.
+- `accuracy` (Accuracy): final-answer correctness against the reference answer.
+- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved.
+- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed.
+
+</details>
+
+## Freshness
 
-The skill should be reviewed before NVSkills-Eval publication. Skill owners should address the findings above and rerun NVSkills-Eval to refresh this benchmark.
+Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.
@@ -0,0 +1,46 @@
+# CAD-to-SimReady Changelog
+
+## 0.2.0 (Unreleased)
+
+This release improves the reliability, portability, dependency management, and
+security-scanner compatibility of the `omniverse-cad-to-simready` workflow.
+
+### Added
+
+- Configurable Content Agents deployment for NVIDIA, OpenAI-compatible,
+  Anthropic, and Google/Gemini model providers.
+- Deterministic upstream dependency pins and a checked-in tested-version lock
+  file for conversion, validation, rendering, and Content Agents integrations.
+- Expanded preflight reporting, runtime validation, and evaluation coverage.
+- An audited SkillSpector baseline using exact fingerprints for reviewed false
+  positives. New or changed findings continue to fail closed.
+
+### Changed
+
+- Migrated CAD conversion to the self-contained `usd-convert-cad` 0.2.0 wheel
+  in an isolated Python 3.12 environment.
+- Simplified the top-level workflow router while preserving detailed
+  stage-specific guidance in nested references.
+- Improved conversion-only routing so Content Agents deployment is skipped when
+  material and physics assignment are not requested.
+- Updated the CAD-to-SimReady skill and skill-card version to 0.2.0.
+
+### Fixed
+
+- Hardened credential environment-variable validation, forwarding, redaction,
+  and generated configuration references.
+- Improved reuse and validation of existing Content Agents endpoints, including
+  local Docker host translation.
+- Rejected missing, blank, or uniform render and thumbnail outputs instead of
+  accepting them as successful inspections.
+- Correctly counted geometry represented through USD instance proxies.
+- Improved SimReady validation, optimizer recovery, dependency checks, and
+  actionable error reporting.
+
+### Migration Notes
+
+- Use Python 3.12 for the workflow and its managed environments.
+- Install and invoke `usd-convert-cad` 0.2.0 as a wheel. The previous KAT/Kit
+  fallback is no longer supported.
+- Treat `upstream-versions.lock.json` as the tested dependency source of truth;
+  preflight blocks on incompatible direct or transitive package pins.