Skip to content

feat(runtime): persist engine lifecycle and state mutation contract - #8058

Merged
ericksoa merged 407 commits into
mainfrom
feat/persisted-engine-lifecycle
Aug 9, 2026
Merged

feat(runtime): persist engine lifecycle and state mutation contract#8058
ericksoa merged 407 commits into
mainfrom
feat/persisted-engine-lifecycle

Conversation

@ericksoa

@ericksoa ericksoa commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a dormant, provider-neutral persisted engine lifecycle and exact-runtime state-mutation
contract on top of the managed bootstrap transaction. Snapshot, clone, rebuild, backup, restore,
and recovery bind to durable engine authority. A mandatory supported-or-unsupported
stateMutation facet lets Docker, Podman, and MXC-style providers integrate without central
runtime switches.

This does not activate Podman or change default onboarding. Managed-image onboarding remains
reachable only through the hidden, undocumented --temp-managed-runtime gate. It is inherited
from PR #8261.

Related Issue

Part of #7744.

Changes

  • Define versioned lifecycle actions and durable runtime and resource authority.
  • Record restart-safe transaction phases, mutation leases, commands, receipts, and retirement
    tombstones.
  • Revalidate provider, engine, runtime handles, state digest, and lease during mutation and
    recovery.
  • Define a declarative state-mutation plan with bounded selectors and digest bindings.
  • Reject executable callbacks, unknown fields, traversal, system roots, control characters,
    duplicate selectors, inherited serialization hooks, and non-scalar Unicode.
  • Keep Docker, Kubernetes, Podman, and MXC explicitly unsupported for state mutation in this
    dormant contract slice.
  • Normalize Docker create-time attachment and absent port-binding defaults without ignoring
    active TTY, stdin, or port-binding configuration.
  • Stage the managed bootstrap request through a root-owned mode-0400 ustar stream so the stopped
    replacement authenticates the envelope independently of the host caller's numeric UID.
  • Exercise the real stopped-container create -> copy -> start path with the native managed
    bootstrap as PID 1 for every shipped managed-image agent.
  • Resolve same-repository PR identity through the GitHub REST API in the protected E2E
    maintainer procedure.
  • Repair the existing stacked merge's omitted onboarding dependencies, duplicate catalog field,
    provider surfaces, test fixtures, and source-shape inventory.
  • Add focused lifecycle, adversarial validation, Podman-dormancy, MXC-pluggability,
    Docker-normalization, bootstrap-staging, PID-1, and source-shape coverage.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference,
    runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval
    link/justification: Exact-head advisor publisher run 31288351290 reported info_only with 0 blockers, 0 warnings, and 0 suggestions.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval
    link, and follow-up issue: The exact-head Terra and Nemotron raw lanes omitted the required
    scope-risk-map analysis protocol; the publisher succeeded with an empty finding ledger, so
    there is no source finding to remediate.

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: docs-updated
  • Evidence: Reviewed .agents/skills/_shared/controlled-words.md, .agents/skills/nemoclaw-maintainer-e2e/SKILL.md, docs/security/hermes-0.19.0-dependency-review.md, docs/security/openclaw-2026.7.1-dependency-review.md, and tools/mcp-tool-discovery-runtime/dependency-review.md at commit under review 37bfd3bb3. The text matches the offline Hermes wheel overlay, offline OpenClaw package materialization, reviewed MCP bundle boundary, and PR identity checks. npm run docs completed with 0 errors and 2 Fern upgrade warnings. No blocking documentation finding remains.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in
    GitHub — commit 37bfd3bb3 is signed and GitHub reports valid verification
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr
    passed after refreshing origin/main when hooks were skipped or unavailable — commit
    37bfd3bb3 passed the local commit and pre-push hook path.
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable
    above — merge commit 37bfd3bb3: 19/19 focused tests, npm run typecheck:cli,
    npm run checks:repository, the live installer-hash verifier, commit hooks, and
    npm run docs passed.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes;
    npm run check for repo-wide validation/coverage changes — command/result: all five required
    exact-head contexts, all eight CLI shards, all three direct managed startups, and the real
    all-agent activation passed on 37bfd3bb3.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the
    style guide
    (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Commit under review: 37bfd3bb348053123980476e23af5e8d667580ef.
Exact public delta: 206 files, +12,839/-597.


Signed-off-by: Aaron Erickson aerickson@nvidia.com

Summary by CodeRabbit

  • New Features

    • Improved managed bootstrap delivery during container replacement with safer archive-based transfer.
    • Added stronger validation, leasing, recovery, and retirement support for persisted runtime lifecycle operations.
    • Added foundational planning for runtime state changes; provider execution remains unavailable pending validation.
    • Standardized managed-image runtime identities and improved cross-architecture activation support.
  • Bug Fixes

    • Prevented Docker attachment settings from triggering unnecessary launch changes.
    • Normalized empty port bindings and Docker-default tmpfs options.
    • Improved readiness checks and journal recovery during container replacement.
  • Tests

    • Expanded lifecycle, bootstrap, archive, state-plan, and activation coverage.

Advisor finding disposition

The raw second-opinion artifact identified a real offline-boundary regression. Protected run
31222691941 then exposed the decisive trusted-workflow boundary: the protected job executes its
build helper from trusted main, so candidate-only seed-overlay helper changes cannot supply the
OpenClaw archive graph before merge. Exact head removes that dependency from the candidate source
itself. The Dockerfile checksum-addresses the complete lock-selected archive set outside RUN,
selects the amd64 or arm64 native subset, verifies every archive against the committed lockfile
SHA-512 identity, and executes npm ci with networking disabled. Imported build cache remains an
optimization, not an input required for correctness; a missing or mismatched locked archive fails
closed instead of falling back to the registry.

The exact-head Terra warning to remove or connect the currently unconsumed persisted-lifecycle and
state-mutation subsystems is intentionally resolved by the binding dormant-slice boundary in
#7744, not by changing this PR into an activation slice:

  • [Epic] Support native Podman with buildless managed onboarding #7744 requires earlier provider slices to remain inert or internally complete and forbids a
    partial support claim.
  • This PR defines the mandatory supported-or-unsupported provider facet and keeps every current
    provider explicitly unsupported; no production registry or central orchestration path calls it.
  • The epic assigns real Docker and Podman stateMutation consumers, durable fencing, interruption,
    and recovery evidence to the later provider execution and activation batches.
  • Connecting a production consumer here would violate this review thesis and the activation gate;
    deleting the contract would discard the provider-neutral/MXC-pluggable seam required by the same
    epic.

The dormant source boundary is therefore intentional acceptance evidence for this slice. The later
activation batch remains blocked until named production consumers and protected recovery evidence
exist.

ericksoa added 30 commits August 1, 2026 00:40
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>

@cv cv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't add tools/mcp-tool-discovery-runtime/mcp-runtime-npm-cache-seed/*.tgz to the repo

ericksoa added a commit that referenced this pull request Aug 8, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Protected managed-image GPU qualification runs trusted workflow code
beside an exact candidate checkout, but the risk reporter resolved
`HEAD` from the trusted workspace while expecting the candidate SHA.
This binds the reporter to the explicit isolated candidate root so
protected evidence attests the code actually under test.

## Related Issue

Prerequisite for #8058.

## Changes

- Allow the risk reporter to resolve an explicitly provided absolute
tested-root path while retaining the existing workspace fallback for
other jobs.
- Bind the protected managed-image GPU job to `${{ github.workspace
}}/.candidate-runtime` and the exact candidate SHA.
- Add reporter and workflow-boundary tests proving the trusted workspace
cannot be mistaken for the candidate checkout.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: This corrects internal
protected-E2E evidence identity; it changes no user command,
configuration, default, API, or supported runtime behavior.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Maintainer-approved
prerequisite scope for #8058. The change narrows evidence identity to
the explicitly checked-out candidate, requires an absolute tested root,
retains exact-SHA equality, and adds negative workflow-contract
coverage; it does not change credentials, permissions, checkout
provenance, or runtime code.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `no-docs-needed`
- Evidence: This prerequisite and follow-up change only internal
protected E2E commit attestation, relative-root rejection, and their
workflow/source contracts. They add no user-facing command,
configuration, default, API, or supported behavior.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 9c18039 -->
<!-- docs-review-agents-blob-sha:
c4923a3 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit: Not applicable; `scripts/prepare-dgx-station-host.sh`
is unchanged.
- Station profile/scenario: Not applicable.
- Result: Not applicable.
- Supporting evidence: Not applicable.

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above —
`test/e2e-risk-signal-reporter.test.ts` and
`test/e2e/support/managed-image-protected-runtime-workflow.test.ts`:
36/36 passed; `npm run typecheck:cli`, Biome, workflow/config
validation, test-conditionals, source-shape, test-size, pre-commit,
commit-msg, and pre-push gates passed.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable; the
five-file change is limited to one protected workflow identity binding
and focused executable contract tests.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved end-to-end validation to consistently identify the correct
tested checkout and commit.
* Ensured protected runtime evidence is tied to the isolated candidate
runtime.
* Added safeguards for absolute checkout paths and expected commit
values.

* **Tests**
  * Added coverage for separated trusted and candidate runtimes.
* Added boundary tests for runtime-root and checkout-commit
configuration.
* Verified commit selection works with both manual checkout values and
the current commit.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>

# Conflicts:
#	.github/workflows/e2e.yaml
#	test/e2e-risk-signal-reporter.test.ts
#	test/e2e/support/managed-image-protected-runtime-workflow.test.ts
#	tools/e2e/managed-image-protected-runtime-workflow-boundary.mts
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
cv pushed a commit that referenced this pull request Aug 8, 2026
<!-- markdownlint-disable MD041 -->
## Summary
<!-- 1-3 plain sentences: what changes and why. Describe
before-and-after behavior when it applies. Follow the NemoClaw Writing
Guide: https://github.com/NVIDIA/NemoClaw/blob/main/WRITING.md. Do not
add unrelated prose cleanup. -->

This PR reverts the two remaining direct `main` commits made outside the
intended #8058 PR workflow. It restores only the three affected E2E
files for review and does not claim that the restored protected E2E
harness behavior is correct.

## Changes
<!-- List concrete changes. If this adds an abstraction, configuration,
fallback, migration, or compatibility path, name its current requirement
and consumer, explain why a direct change is insufficient, and identify
the test that protects it. -->

- Revert `ebbbc91c005d07cf2a52cf8673b1e4f795490e3f` (`fix(e2e): require
Ollama service cleanup`).
- Revert `5722e63901fa7187f761993e1679ff98dce44e5b` (`fix(e2e): use
installed Ollama service in protected run`).
- Restore the affected paths to their exact contents before
`5722e63901fa7187f761993e1679ff98dce44e5b`.
- Preserve unrelated commit `37f54757a680a26cc63fc0b2a2c816b15aae57c0`
and its complete diff.
- Do not duplicate the existing verified `main` revert
`fad771aa3296c8df47994602bb41ec69d9492e12`, which already reverts
`11ae5a69d2930b4092ed0e7ec6de3249806599df`.

## Type of Change

- [x] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates
<!-- Check one tests line and one docs line. Check other lines when
applicable. Add every requested justification or approval reference. -->
- [ ] Tests added or updated for changed behavior
- [x] Existing tests cover changed behavior — justification: The
semantic E2E phase check and the focused GPU E2E helper suite pass for
the restored file tree.
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: This PR restores internal E2E
harness and test files without changing a supported user surface.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Maintainer-directed
exact source reversion; the affected paths match the pre-`5722e639`
contents and add no behavior.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review
<!-- Required for code and documentation changes after the changes and
applicable validation are complete. Keep one review checkbox and one
instance of each visible or hidden field. For Evidence, list changed
documentation paths. For documentation-only changes, also state that the
writing rules and documentation style were reviewed. For other results,
explain why no documentation change is needed or why the review is
blocked. For Agent, use a consistent product and surface name, such as
Codex Desktop, Codex CLI, Claude Code, or Cursor. After committing all
review changes, put `git rev-parse --short HEAD` and `git rev-parse
--short HEAD:AGENTS.md` in the hidden metadata below. Rerun the review
and refresh that metadata after any new commit. This receipt is advisory
during the data-collection pilot. -->
- [x] Documentation writer subagent reviewed the completed changes
- Result: `no-docs-needed`
- Evidence: The diff changes only three internal E2E harness and test
files. It does not change a supported CLI, API, configuration, default,
user workflow, or documentation surface.
- Agent: Codex Desktop
<!-- docs-review-head-sha: 37ce69c -->
<!-- docs-review-agents-blob-sha: c4923a3 -->

## DGX Station Hardware Evidence
<!-- Required only when scripts/prepare-dgx-station-host.sh changes.
Maintainers must review the linked evidence before approving or merging.
This is human-reviewed evidence, not authenticated hardware provenance.
Exceptional bypasses use existing repository governance and must be
documented in the PR. -->
- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification
<!-- Check each applicable item only when supported by the requested
evidence. Run targeted tests once per relevant change set and rerun
after later behavior-affecting edits or hook autofixes. Do not rerun
hook-covered checks. -->
- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: `npm
run test:e2e-phases:check` passed for 122 tests across 79 files; `npx
vitest run --project e2e-support
test/e2e/support/gpu-e2e-helpers.test.ts` passed 24 tests.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result:
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
<!-- DCO sign-off is required in this PR description, and every commit
must appear as Verified in GitHub. Run: git config user.name && git
config user.email -->
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
  * Improved GPU end-to-end test setup and teardown reliability.
* Simplified runtime startup and cleanup behavior to reduce
test-environment failures.
* Removed obsolete coverage for deprecated service-management scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@github-actions github-actions Bot added v0.0.106 and removed v0.0.105 labels Aug 8, 2026
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@ericksoa

ericksoa commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

Exact-head handoff for 37bfd3bb348053123980476e23af5e8d667580ef:

  • Synchronized current main at 290eb94884eb29bc8aac89685e18525bb1292ed3. The two conflict resolutions preserve main's Node-tar scanner removal and retain this PR's locked-cache materializer.
  • The requested tools/mcp-tool-discovery-runtime/mcp-runtime-npm-cache-seed/*.tgz files are absent; that directory contains only .gitkeep.
  • All five required contexts are green: checks, commit-lint, dco-check, check-hash, and changes.
  • GitHub reports the commit verified and the PR mergeable. The documentation receipt passes, and all 11 review threads are resolved.
  • The advisor publisher reports 0 blockers, 0 warnings, 0 suggestions, and no follow-up needed. Its two non-required model lanes returned no analysis and failed only their output protocol.

The remaining merge-policy gate is one current human approval. reviewDecision still reflects the older changes-requested review at 054b714, whose literal cache-file request is addressed on this head. Ready for current-head human review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: architecture Architecture, design debt, major refactors, or maintainability area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery feature PR adds or expands user-visible functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants