Add the perf-smoke performance regression gate

[Neil4561]

Adds a performance regression gate for pull requests. It benchmarks a fixed matrix of tasks on the L40S runner pool, compares each result against a rolling baseline stored on the perf-baselines branch, and posts a per-task verdict as a sticky PR comment and per-task commit statuses.

The gate is advisory. gate_config.json sets blocking: false, so it never fails a PR on its own. Making it an enforcing required check would be a separate rollout.

How a run works

perf-smoke-test.yaml runs six jobs.

config reads the Isaac Sim version from config.yaml, and validate runs static checks on tasks.json so a malformed matrix fails on a free runner in seconds rather than after an hour of GPU time. Both are skipped for draft PRs.

bench fans out one GPU job per task/backend bucket. Each job resolves and pulls the CI image, restores a Warp JIT cache keyed on the Warp version, runs perf_runtime.py inside Docker with the repo bind-mounted, retries once on failure, normalizes the output into perf_smoke_test_result.json, and posts a per-task commit status.

aggregate downloads every bench artifact, loads the matching baselines, runs the oracle, writes the summary comment, and uploads results to omni-github. It runs with always() so a crashed bucket still gets reported instead of vanishing.

baseline_update appends the run's samples to perf-baselines, and only on pushes to main, develop, or a release branch. Pull requests are strictly read-only against the baseline branch, so nothing a PR does can move the numbers it is judged against.

reseed fills buckets that came out under-filled, by calling the seeding workflow for just those tasks at the pushed commit. Without it, a bucket that resets (say, after a launch-config change) would take five separate pushes to become useful again. It is self-limiting: once a bucket is full it stops being reported as under-filled, and seeding writes to perf-baselines, which triggers no workflow.

How the verdict is decided

oracle.py produces one of PASS, WARN, BLOCK, or HARD_FAILURE per bucket, and the most severe signal wins.

A run is a HARD_FAILURE if the launch configuration did not match what was requested, if the result file is missing, or if the reported FPS is missing or not positive. A run reporting no forward progress is a dead run rather than a slow one, so it fails regardless of what the baseline says.

Otherwise the result is compared against the rolling window. The baseline contributes a median and a median absolute deviation, and the gate blocks when the measured FPS falls below median - 4.0 x MAD and the drop is at least 3 percent. The percentage floor exists so that a bucket with an unusually tight MAD cannot block on a fraction of a percent. Between median - 2.5 x MAD and the block line the verdict is WARN. There is also a noise floor so that a very tight baseline does not make the thresholds hypersensitive.

Buckets can additionally declare fixed FPS floors in tasks.json. A crossed gating floor forces a BLOCK on its own; a reporting-only floor is recorded without changing the verdict.

With no baseline, or fewer than five samples in the window, the verdict is WARN and the reason is stated as NO_BASELINE or INSUFFICIENT_WINDOW. The gate says it does not know rather than implying a pass.

What gets measured

Nine task/backend buckets:

• Isaac-Cartpole-Direct on physx and on newton
• Isaac-Velocity-Flat-G1 on physx and on newton
• IsaacContrib-Factory-GearMesh-Direct on physx
• Isaac-Reorient-Cube-Shadow-Camera-Benchmark-Direct across four physics/renderer combinations (physx and newton, each with the RTX renderer and the Newton renderer)

perf_runtime.py drives each task with random actions and no policy, discards warmup frames at the source, and reports steady-state mean FPS.

Baselines and how samples are matched

Baselines live on the orphan perf-baselines branch as append-only NDJSON, one file per GPU model, task, and backend.

A stored sample is only eligible if it matches on GPU model, task, backend, target branch, launch-config hash, baseline epoch, benchmark contract hash, and runtime contract hash, and if its commit is an ancestor of the PR's merge base. Ancestry matters because a sample from an unmerged branch describes code that may never land. The window holds between 5 and 20 samples, nearest commits first.

The hashes are the reason a verdict can be trusted: change the number of environments, the seed, the frame count, or the CUDA/driver pairing, and the fingerprint changes, so old samples stop matching instead of being silently compared against a different experiment.

Keeping the environment stable

image_era.py derives a key from the container's inputs and looks it up in a manifest on perf-baselines. When an immutable image is recorded for that key the gate pins it; otherwise it falls back to the published latest-develop tag, and builds only if that pull fails. Pinning matters because a nightly rebuild of a floating tag would otherwise shift the environment underneath a baseline that was measured in the old one.

When something goes wrong

Benchmarks retry once, since a single flaky container start should not read as a regression.

A crash always exits non-zero. Advisory mode suppresses regression failures, not execution failures, so a benchmark that never ran cannot be mistaken for one that ran fine.

Skew detection in aggregate.py handles one specific case that would otherwise blame the wrong person. The gate mounts PR source over a prebuilt image, so between a dependency pin landing and the next image publish, source can reference a symbol the installed package does not have yet. When a crash log shows a missing name from a package the image provides (newton, warp, isaacsim, mujoco, and so on), the summary reports a stale CI image and those tasks stay advisory. Isaac Lab's own source is mounted from the PR, so a missing symbol there is a real defect and still fails.

Workflows

perf-smoke-test.yaml is the gate itself, described above.

perf-smoke-seed-baselines.yaml walks a slice of a branch's history, re-runs each commit's own benchmark, and appends the results, so a new bucket can be filled without waiting for five organic pushes. It runs on manual dispatch, defaulting to a dry run, and is called by the gate's reseed job. It resolves its image through the same code path the gate uses, so seeded samples and live runs share an environment.

perf-smoke-unit-tests.yaml runs the gate's own tests on ubuntu-latest.

The Python

Under tools/perf_smoke_test/, roughly by role:

• Deciding: oracle.py, gate_types.py, gate_config.py
• Baselines: baseline_manager.py, verify_baselines.py, seed_baselines.py
• Identity and reproducibility: backend_identity.py, gpu_identity.py, launch_config.py, runtime_contract.py, contracts.py, hashing.py, image_era.py
• Running and normalizing: perf_runtime.py, build_bench_result.py, benchmark_result_adapter.py, write_launch_config.py
• Matrix and config: tasks.json, task_config.py, tasks_to_ci_matrix.py, validate_tasks.py, gate_config.json
• Reporting: aggregate.py, omni_github.py, github_gate_context.py

tools/subprocess_runner.py classifies which phase a failed subprocess died in, which is what lets a config mismatch be told apart from a crash mid-benchmark.

Tests

72 tests covering the oracle's verdicts and thresholds, baseline matching and ancestry, contract hashing, seeding and cache isolation, skew detection, and the aggregate reporting path.

They are pure Python: no GPU, no simulator, no Isaac Lab install. tools/perf_smoke_test/pyproject.toml makes pytest root there so the isaaclab-importing tools/conftest.py above it is not loaded, which is the same arrangement tools/skills/ and tools/changelog/ already use.

What to know before merging

It runs on every PR into develop from day one. The trigger list already covers pull_request: [main, develop, release/**], so that is real L40S pool time on every PR. More of a capacity question than a code question.

Baselines start empty. Every sample on perf-baselines today is stamped target_branch: perf-smoke/develop-staging, and matching requires an exact target-branch match, so none carry over. All nine buckets begin with no baseline and report NO_BASELINE or INSUFFICIENT_WINDOW until five samples accumulate, which happens on its own from develop pushes. It sorts itself out after a few, but the gate is not useful on day one.

Six of the nine buckets have inert thresholds. Every Newton bucket and three camera buckets carry a 0.0 hard floor, which can never fire. Harmless while the gate is advisory, but they need real values before it could block anything.

There is no runner-stability verdict yet. The PhysX buckets produced clean samples on July 28; the Newton buckets never ran, because the staging branch had drifted 216 commits behind and its source called newton.solvers.SolverNotifyFlags, renamed to ModelFlags in the meantime. That was the frozen branch showing its age rather than a problem with the gate, and it does not arise on develop, where source and image move together. Landing here rather than on staging is what retires that failure mode. This supersedes #6863, which carried the same fixes against staging.

Test plan

• 72 passed, 1 skipped via python3 -m pytest tools/perf_smoke_test/
• Tests collect and pass with no Isaac Lab install and no pytest flags
• All pre-commit hooks pass
• All three workflows parse as valid YAML
• No changelog fragment required; the check only demands one for packages touched under source/
• After merge: first develop push runs the gate and begins filling baselines
• Confirm no image build or seeding run is triggered unprompted

[greptile-apps]

Greptile Summary

This PR adds an advisory performance-regression gate, baseline storage and seeding machinery, benchmark identity contracts, reporting, and unit-test workflows.

• Runs a nine-bucket GPU benchmark matrix and evaluates results against rolling, ancestry-filtered baselines.
• Adds protected-branch baseline updates and targeted reseeding for under-filled buckets.
• Adds per-task statuses, a sticky PR summary, normalized artifacts, and pure-Python tests.
• The pull-request workflow currently exposes comment/status write credentials to checked-out PR code.

Confidence Score: 4/5

The PR should not merge until pull-request-controlled scripts are isolated from credentials that can forge comments and commit statuses; the reseed secret inheritance should also be narrowed.

The workflow grants issues and statuses write access at workflow scope, then checks out and executes the pull request’s Python files on the host, including one invocation with GITHUB_TOKEN explicitly present.

Files Needing Attention: .github/workflows/perf-smoke-test.yaml, .github/workflows/perf-smoke-seed-baselines.yaml

Security Review

The pull-request workflow executes PR-controlled host scripts with credentials capable of writing issue comments and commit statuses, allowing same-repository PR code to forge the gate’s visible output. The trusted reseed path also inherits more repository secrets than the called workflow needs.

Important Files Changed

Filename	Overview
.github/workflows/perf-smoke-test.yaml	Adds the main gate orchestration, but pull-request-controlled scripts execute with issue-comment and commit-status write permissions.
.github/workflows/perf-smoke-seed-baselines.yaml	Adds serialized historical baseline seeding; secret scope is broader than necessary through the caller’s secrets inheritance.
tools/perf_smoke_test/aggregate.py	Aggregates artifacts, evaluates buckets, emits reports, and identifies under-filled baselines; its execution under a write-capable PR job creates the workflow security issue.
tools/perf_smoke_test/baseline_manager.py	Implements matching, append-only updates, worktree commits, and push retries; no publishable changed-path defect was established.
tools/perf_smoke_test/oracle.py	Implements hard-failure, fixed-floor, MAD, percentage-floor, and insufficient-window verdict logic with focused tests.
tools/perf_smoke_test/seed_baselines.py	Replays historical commits in Docker and publishes baseline records with ancestry verification and cache isolation.
tools/subprocess_runner.py	Adds subprocess phase classification and timeout handling without invoking commands through a shell.

Sequence Diagram

sequenceDiagram
    participant PR as Pull request / protected push
    participant Gate as perf-smoke-test workflow
    participant GPU as GPU benchmark jobs
    participant Oracle as aggregate.py / oracle.py
    participant Base as perf-baselines branch
    participant GH as GitHub statuses/comments
    PR->>Gate: Trigger workflow
    Gate->>GPU: Fan out task/backend matrix
    GPU-->>Gate: Upload normalized results
    Gate->>Oracle: Aggregate artifacts and context
    Oracle->>Base: Read matching rolling baselines
    Base-->>Oracle: Eligible samples
    Oracle->>GH: Publish per-task verdicts and summary
    alt Protected develop push
        Oracle->>Base: Append valid samples
        Gate->>Gate: Reseed under-filled buckets
    end

Loading

_{Reviews (1): Last reviewed commit: "Fold skew detection into aggregate" | Re-trigger Greptile}

[isaaclab-review-bot]

Isaac Lab Review Bot

The advisory perf-smoke gate has a coherent producer, oracle, baseline, and reporting structure, but the proposed workflow currently contains concrete correctness and integration gaps. Seed workflow defaults make documented input modes unreachable, non-develop PRs can select the wrong fallback image, retry artifacts can be misattributed, fork reporting can fail on read-only tokens, configured window sizes are ignored, and crashed runs can still be scored from existing output. The stale-image classifier can also misclassify PR-introduced dependency API misuse.

• Design and architecture: The launch-config and runtime-contract partitioning, ancestry-aware append-only baselines, and separation of read-only aggregation from trusted baseline writes are sound. However, the stale-image exception is too broad: any missing symbol from an image-provided package is treated as environment skew, even when changed Isaac Lab source introduced an invalid package API call. That weakens the execution-health contract and should be tied to stronger evidence such as installed-versus-pinned version skew.
• API: No existing public Isaac Lab package API is changed, but the new workflow and configuration surfaces are internally inconsistent. Empty tasks and branches inputs cannot reach their documented all-task and explicit-commit behaviors because expression fallbacks replace empty values, and the advertised baseline window settings in gate_config.json are ignored in favor of module constants. These contracts should either be implemented as documented or narrowed to reflect actual behavior.
• Implementation: The benchmark-to-baseline path needs correction before relying on its results. Main and release PRs can fall back to the develop image because target-branch selection uses the synthetic event ref. Retry cleanup leaves the first attempt's timestamped runtime bundle available, while the oracle ignores nonzero exits and failure phases, allowing stale output from a failed run to be scored and appended. Aggregate status and comment writes also fail for fork PRs with read-only tokens. Fix target-branch image resolution, clear actual runtime bundles between attempts, make execution failure authoritative in the oracle, and tolerate unavailable reporting permissions.

Significant concerns. Posted 7 actionable findings inline.

Automated review; human maintainers own approval decisions.

[isaaclab-review-bot]

🔵 Suggestion · Design Architecture — Skew heuristic can excuse PR defects

detect_dependency_skew classifies any missing name in an image-provided package as a stale image. Isaac Lab source that calls a nonexistent warp/newton API raises the same ImportError/AttributeError, so a genuine defect in the change under test clears has_hard_failure and is reported as an image problem. Narrow the rule, for example to failures raised while importing the image package, or compare pinned versus installed versions first.