Skip to content
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
db52f20
fix(recovery): retry blank OpenShell registration follow-up
jyaunches Jul 29, 2026
24681d5
test(e2e): prove gateway health after restart
jyaunches Jul 30, 2026
ae26a2b
test(recovery): cover repeated blank follow-ups
jyaunches Jul 30, 2026
6609e6a
fix(recovery): retry opaque registration follow-ups
jyaunches Jul 30, 2026
31c30de
fix(recovery): retry informational registration follow-ups
jyaunches Jul 30, 2026
0e531dd
fix(recovery): retry exact registration error metadata
jyaunches Jul 30, 2026
9faffab
fix(recovery): preserve supervisor during container handoff
prekshivyas Jul 30, 2026
b2b66c5
test(recovery): cover supervisor handoff lifecycle
jyaunches Jul 30, 2026
b8e273b
test(recovery): clarify supervisor handoff lifecycle
prekshivyas Jul 30, 2026
25c3d99
Merge branch 'main' into codex/fix-sandbox-survival-status
cv Jul 30, 2026
297ade9
fix(recovery): relaunch legacy supervisor in place
prekshivyas Jul 30, 2026
e45d4d8
Merge remote-tracking branch 'origin/codex/fix-sandbox-survival-statu…
prekshivyas Jul 30, 2026
ab4fca5
fix(recovery): harden supervisor launch cleanup
sandl99 Jul 30, 2026
49cf4c1
fix(recovery): stabilize restored dashboard forward
sandl99 Jul 30, 2026
5b295a5
test(recovery): keep forward retry fixture linear
sandl99 Jul 30, 2026
f1b0c95
merge(main): simplify sandbox supervisor recovery
sandl99 Jul 30, 2026
062820e
fix(recovery): explain temporary cleanup
sandl99 Jul 30, 2026
0358523
test(e2e): recover sandbox before survival checks
sandl99 Jul 30, 2026
0e1f363
test(e2e): focus sandbox survival regression
sandl99 Jul 30, 2026
14a03d5
fix(recovery): preserve supervisor during handoff
sandl99 Jul 30, 2026
2edb387
docs(recovery): clarify handoff state
sandl99 Jul 30, 2026
b06f526
fix(recovery): defer handoff commit until ready
sandl99 Jul 30, 2026
9de564f
fix(recovery): preserve sandbox state across restart
sandl99 Jul 30, 2026
3ac8315
refactor(recovery): trim handoff scaffolding
sandl99 Jul 30, 2026
ff87797
merge: resolve recovery handoff conflict
sandl99 Jul 30, 2026
1e8b1dd
merge: resolve conflicts with main
github-actions[bot] Jul 30, 2026
691389b
refactor(recovery): finish main conflict cleanup
sandl99 Jul 30, 2026
f0a952b
Merge remote-tracking branch 'origin/codex/fix-sandbox-survival-statu…
sandl99 Jul 30, 2026
eaee157
Merge remote-tracking branch 'origin/main' into codex/pr7856-linear
sandl99 Jul 30, 2026
8a60ab9
fix(recovery): stop original before replacement
sandl99 Jul 30, 2026
8bb87f9
fix(recovery): restart legacy supervisor in place
sandl99 Jul 30, 2026
0fc1c1b
docs(recovery): clarify writable state preservation
sandl99 Jul 30, 2026
0b64a9c
test(e2e): capture sandbox survival diagnostics
sandl99 Jul 31, 2026
d1a9dd2
fix(onboard): persist OpenClaw startup command
sandl99 Jul 31, 2026
ac61a67
fix(onboard): recognize default OpenClaw startup
sandl99 Jul 31, 2026
6ab61e8
test(onboard): model startup container recreation
sandl99 Jul 31, 2026
8bf8033
Merge branch 'main' into codex/fix-sandbox-survival-status
prekshivyas Jul 31, 2026
01d1b41
test(e2e): prove Docker restart delivery
prekshivyas Jul 31, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/manage-sandboxes/recover-rebuild-sandboxes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ For a local Docker-driver sandbox whose container still uses the legacy keepaliv

NemoClaw keeps the previous container available until the managed controller proves the supervisor topology, gateway health, and settle check.
Before recreation, NemoClaw backs up the state directories and files declared by the agent manifest.
It commits only after the replacement container identity and state restoration pass.
It commits only after OpenShell reports the recreated sandbox as `Ready`, the replacement container identity matches, and state restoration passes.
NemoClaw removes the temporary state backup after a successful restore or rollback.
If state restoration and rollback both fail, it retains the backup and prints host recovery guidance.
Mounted state remains available, but a committed swap does not retain other writable-layer changes.
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1167,7 +1167,7 @@ It does not use ordinary `openshell sandbox exec` or an in-sandbox manual relaun
When the root-owned managed controller attests two unchanged zero-supervisor process scans with a stable PID 1 and reports `SUPERVISOR_NOT_RUNNING`, a local Docker-driver sandbox with the legacy keepalive startup can enter a transactional container recreation.
The recreation uses a credential-free managed startup command, pins the registered container identity, and retains the previous container for rollback.
Before recreation, NemoClaw backs up the state directories and files declared by the agent manifest.
It commits only after managed gateway health, the settle check, exact replacement identity, and state restoration pass.
It commits only after managed gateway health, the settle check, OpenShell readiness, exact replacement identity, and state restoration pass.
NemoClaw removes the temporary state backup after a successful restore or rollback.
If state restoration and rollback both fail, it retains the backup and prints host recovery guidance.
Mounted state remains available, but a committed swap does not retain other writable-layer changes.
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1019,7 +1019,7 @@ The same result is inconclusive during managed settle confirmation and can be pr
NemoClaw treats `SUPERVISOR_UNAVAILABLE` as terminal because it can report unreadable or untrusted supervisor state, ambiguous discovery, or a process-identity change.
It does not retry other status or output combinations.
`SUPERVISOR_NOT_RUNNING` is a separate result that requires two zero-supervisor scans with a stable PID 1 and does not enter that retry loop.
On a supported local Docker-driver sandbox with the legacy keepalive startup, it can authorize a container-identity-pinned recreation that commits only after managed health and settle checks pass.
On a supported local Docker-driver sandbox with the legacy keepalive startup, it can authorize a container-identity-pinned recreation that commits only after managed health, settle, and OpenShell readiness checks pass.
To bypass that trusted recreation while troubleshooting, run `NEMOCLAW_DISABLE_SUPERVISOR_RELAUNCH=1 $$nemoclaw <name> recover`; NemoClaw leaves the container unchanged and returns rebuild or re-onboard guidance.
If that bounded retry is exhausted, or if `gateway restart` reports `SUPERVISOR_BUSY`, wait for the active request to finish and retry the command.
If the error mentions `SUPERVISOR_NOT_RUNNING` and trusted recreation could not proceed, `SUPERVISOR_REBUILD_REQUIRED`, a missing `nemoclaw-gateway-control` helper, or a missing managed controller, the sandbox image may predate the current lifecycle contract.
Expand Down
71 changes: 41 additions & 30 deletions src/lib/actions/sandbox/process-recovery.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1299,6 +1299,47 @@
}
return { checked: true, wasRunning: false, recovered: false, forwardRecovered: false };
}
const readinessFailureDetail = relaunch
? (() => {
const readinessOptions: RecreatedSandboxOpenShellReadyOptions = {
beforeProbe: (timeoutMs) => confirmRelaunchedManagedHealth?.(timeoutMs) ?? null,
timeoutSeconds: SANDBOX_READY_TIMEOUT_SECS,
};
const readiness =
waitForRecreatedSandboxOpenShellReadyImpl === waitForRecreatedSandboxOpenShellReady
? waitForRecreatedSandboxOpenShellReadyResult(sandboxName, readinessOptions)
: waitForRecreatedSandboxOpenShellReadyImpl(sandboxName, readinessOptions)
? ({ ready: true } as const)
: ({ failure: "openshell-readiness-failure", ready: false } as const);
return readiness.ready
? null
: recreatedSandboxOpenShellReadinessFailureDetail(
readiness.failure,
"openshellError" in readiness ? readiness.openshellError : undefined,
);
})()
: null;
if (readinessFailureDetail) {
let rolledBack = true;
Comment thread
github-code-quality[bot] marked this conversation as resolved.
Fixed
try {
rolledBack = relaunch?.finalize(false).rolledBack ?? true;
} catch {
rolledBack = false;
}
if (!rolledBack && !quiet) {
console.error(
" Automatic rollback of the previous sandbox container failed; inspect Docker state before retrying.",
);
}
return {
checked: true,
wasRunning: false,
recovered: false,
forwardRecovered: false,
forwardRecoveryFailed: true,
forwardRecoveryFailureDetail: readinessFailureDetail,
};
Comment thread
github-advanced-security[bot] marked this conversation as resolved.
Fixed
}
if (relaunch) {
try {
const completion = relaunch.finalize(true);
Expand Down Expand Up @@ -1345,36 +1386,6 @@
}
}
}
const readinessFailureDetail = relaunch
? (() => {
const readinessOptions: RecreatedSandboxOpenShellReadyOptions = {
beforeProbe: (timeoutMs) => confirmRelaunchedManagedHealth?.(timeoutMs) ?? null,
timeoutSeconds: SANDBOX_READY_TIMEOUT_SECS,
};
const readiness =
waitForRecreatedSandboxOpenShellReadyImpl === waitForRecreatedSandboxOpenShellReady
? waitForRecreatedSandboxOpenShellReadyResult(sandboxName, readinessOptions)
: waitForRecreatedSandboxOpenShellReadyImpl(sandboxName, readinessOptions)
? ({ ready: true } as const)
: ({ failure: "openshell-readiness-failure", ready: false } as const);
return readiness.ready
? null
: recreatedSandboxOpenShellReadinessFailureDetail(
readiness.failure,
"openshellError" in readiness ? readiness.openshellError : undefined,
);
})()
: null;
if (readinessFailureDetail) {
return {
checked: true,
wasRunning: false,
recovered: true,
forwardRecovered: false,
forwardRecoveryFailed: true,
forwardRecoveryFailureDetail: readinessFailureDetail,
};
}
const mcpRefusal = processRecoveryMcpReconciliationRefusal(sandboxName, false);
if (mcpRefusal) return mcpRefusal;
const forwardRecovered = ensureSandboxPortForward(sandboxName, {
Expand Down
1 change: 1 addition & 0 deletions src/lib/actions/sandbox/supervisor-relaunch.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ describe("relaunchManagedSupervisorSession", () => {
expect(options).toMatchObject({
sandboxName: "alpha",
expectedOldContainerId: "old-container-id",
keepOriginalRunningUntilFinalize: true,
waitForSupervisor: false,
});
const serialized = options?.openshellSandboxCommand.join(" ") ?? "";
Expand Down
1 change: 1 addition & 0 deletions src/lib/actions/sandbox/supervisor-relaunch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ export function relaunchManagedSupervisorSession(
sandboxName,
openshellSandboxCommand: startupCommand,
expectedOldContainerId: containerId,
keepOriginalRunningUntilFinalize: true,
waitForSupervisor: false,
});
pendingStateBackupPath = null;
Expand Down
48 changes: 48 additions & 0 deletions src/lib/onboard/docker-gpu-patch-finalize.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,25 @@ describe("finalizeDockerGpuPatchBackup", () => {
);
});

it("force-removes the running original container after replacement health is confirmed", () => {
const dockerForceRm = vi.fn((_name: string) => ({ status: 0 }));
const dockerRm = vi.fn((_name: string) => ({ status: 0 }));
const outcome = finalizeDockerGpuPatchBackup(
{
result: { ...deferredCreateResult(), backupWasRunning: true },
supervisorReady: true,
},
{ dockerForceRm, dockerRm },
);

expect(outcome).toEqual({ backupRemoved: true, rolledBack: false });
expect(dockerForceRm).toHaveBeenCalledWith(
"openshell-alpha-nemoclaw-gpu-backup-1780491860342",
expect.objectContaining({ ignoreError: true }),
);
expect(dockerRm).not.toHaveBeenCalled();
});

it("rolls back to the backup container when supervisor reconnect failed", () => {
const dockerStop = vi.fn(() => ({ status: 0 }));
const dockerRm = vi.fn((_name: string) => ({ status: 0 }));
Expand Down Expand Up @@ -65,6 +84,35 @@ describe("finalizeDockerGpuPatchBackup", () => {
).toBe(false);
});

it("rolls back to the still-running original container without restarting it", () => {
const dockerStop = vi.fn(() => ({ status: 0 }));
const dockerRm = vi.fn((_name: string) => ({ status: 0 }));
const dockerRename = vi.fn((_old: string, _next: string) => ({ status: 0 }));
const dockerStart = vi.fn(() => ({ status: 0 }));
const outcome = finalizeDockerGpuPatchBackup(
{
result: {
...deferredCreateResult(),
backupWasRunning: true,
},
supervisorReady: false,
},
{ dockerStop, dockerRm, dockerRename, dockerStart },
);

expect(outcome).toEqual({ backupRemoved: false, rolledBack: true });
expect(dockerStop).toHaveBeenCalledWith(
"new-container-id",
expect.objectContaining({ ignoreError: true }),
);
expect(dockerRename).toHaveBeenCalledWith(
"openshell-alpha-nemoclaw-gpu-backup-1780491860342",
"openshell-alpha",
expect.objectContaining({ ignoreError: true }),
);
expect(dockerStart).not.toHaveBeenCalled();
});

it("reports rolledBack=false when restoring the backup fails", () => {
const dockerStop = vi.fn(() => ({ status: 0 }));
const dockerRm = vi.fn((_name: string) => ({ status: 0 }));
Expand Down
16 changes: 13 additions & 3 deletions src/lib/onboard/docker-gpu-patch-finalize.ts
Original file line number Diff line number Diff line change
Expand Up @@ -65,14 +65,17 @@ export function finalizeDockerGpuPatchBackup(
// even if `docker rm` cannot delete it (e.g. concurrent admin action,
// daemon timeout). Reflect the actual rm status in the outcome so
// diagnostics can flag a leaked backup container.
const rmResult = resolved.dockerRm(options.result.backupContainerName, containerOpts);
const rmResult = options.result.backupWasRunning
? resolved.dockerForceRm(options.result.backupContainerName, containerOpts)
: resolved.dockerRm(options.result.backupContainerName, containerOpts);
return { backupRemoved: hasZeroDockerExitStatus(rmResult), rolledBack: false };
}
const rolledBack = rollbackToBackupContainer(
{
newContainerId: options.result.newContainerId,
backupContainerName: options.result.backupContainerName,
originalName: options.result.originalName,
backupWasRunning: options.result.backupWasRunning,
},
resolved,
);
Expand All @@ -85,7 +88,12 @@ export type SupervisorReconnectOutcome =

export function reconcileSupervisorReconnect(
execReady: boolean,
refs: { newContainerId: string; backupContainerName: string; originalName: string },
refs: {
newContainerId: string;
backupContainerName: string;
originalName: string;
backupWasRunning?: boolean;
},
deps: DockerGpuPatchDeps,
): SupervisorReconnectOutcome {
const resolved = resolveDockerGpuPatchRollbackDeps(deps);
Expand All @@ -100,7 +108,9 @@ export function reconcileSupervisorReconnect(
// leaked backup container but the user-visible sandbox is healthy.
// Surface the actual rm status so callers can fold it into diagnostics
// alongside the deferred-finalize path in `finalizeDockerGpuPatchBackup`.
const rmResult = resolved.dockerRm(refs.backupContainerName, containerOpts);
const rmResult = refs.backupWasRunning
? resolved.dockerForceRm(refs.backupContainerName, containerOpts)
: resolved.dockerRm(refs.backupContainerName, containerOpts);
return { execReady: true, backupRemoved: hasZeroDockerExitStatus(rmResult) };
}
const rolledBack = rollbackToBackupContainer(refs, resolved);
Expand Down
57 changes: 32 additions & 25 deletions src/lib/onboard/docker-gpu-patch-recreate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ import {
dockerCapture,
dockerRename,
dockerRm,
dockerRun,
dockerRunDetached,
dockerStart,
dockerStop,
Expand Down Expand Up @@ -48,7 +47,6 @@ type RecreateDeps = Required<
Pick<
DockerGpuPatchDeps,
| "dockerCapture"
| "dockerRun"
| "dockerRunDetached"
| "dockerRename"
| "dockerRm"
Expand All @@ -66,7 +64,6 @@ type RecreateDeps = Required<
function recreateDeps(deps: DockerGpuPatchDeps): RecreateDeps {
return {
dockerCapture,
dockerRun,
dockerRunDetached,
dockerRename,
dockerRm,
Expand Down Expand Up @@ -155,6 +152,7 @@ export function recreateOpenShellDockerSandboxContainer(
gpuDevice?: string | null;
timeoutSecs?: number;
waitForSupervisor?: boolean;
keepOriginalRunningUntilFinalize?: boolean;
openshellSandboxCommand?: readonly string[] | null;
requiredUlimits?: readonly import("./docker-gpu-patch-types").DockerUlimit[] | null;
expectedOldContainerId?: string | null;
Expand All @@ -171,6 +169,11 @@ export function recreateOpenShellDockerSandboxContainer(
};
try {
validateRequiredDockerUlimits(options.requiredUlimits);
if (options.keepOriginalRunningUntilFinalize && options.waitForSupervisor !== false) {
throw new Error(
"Keeping the original OpenShell supervisor running requires deferred supervisor finalization.",
);
}
const containerIds = findOpenShellDockerSandboxContainerIds(options.sandboxName, deps);
const oldContainerId = containerIds[0];
if (!oldContainerId) {
Expand Down Expand Up @@ -277,28 +280,31 @@ export function recreateOpenShellDockerSandboxContainer(
);
}
}
const cloneArgs = buildDockerGpuCloneRunArgs(inspect, selection.mode, cloneOptions);

const containerMutationOptions = {
ignoreError: true,
suppressOutput: true,
timeout: DOCKER_GPU_PATCH_TIMEOUT_MS,
};
const stopResult = d.dockerStop(oldContainerId, {
...containerMutationOptions,
timeout: DOCKER_GPU_PATCH_STOP_TIMEOUT_MS,
});
if (!hasZeroDockerExitStatus(stopResult)) {
context.rolledBack = hasZeroDockerExitStatus(
d.dockerStart(oldContainerId, containerMutationOptions),
);
throw new Error(
`Could not stop original sandbox container: ${resultText(stopResult)}; ${
context.rolledBack
? "original sandbox container confirmed running"
: "restart failed; original sandbox container may be stopped"
}`,
);
const cloneArgs = buildDockerGpuCloneRunArgs(inspect, selection.mode, cloneOptions);

const backupWasRunning = options.keepOriginalRunningUntilFinalize === true;
if (!backupWasRunning) {
const stopResult = d.dockerStop(oldContainerId, {
...containerMutationOptions,
timeout: DOCKER_GPU_PATCH_STOP_TIMEOUT_MS,
});
if (!hasZeroDockerExitStatus(stopResult)) {
context.rolledBack = hasZeroDockerExitStatus(
d.dockerStart(oldContainerId, containerMutationOptions),
);
throw new Error(
`Could not stop original sandbox container: ${resultText(stopResult)}; ${
context.rolledBack
? "original sandbox container confirmed running"
: "restart failed; original sandbox container may be stopped"
}`,
);
}
}
const renameResult = d.dockerRename(
oldContainerId,
Expand All @@ -307,9 +313,9 @@ export function recreateOpenShellDockerSandboxContainer(
);
if (!hasZeroDockerExitStatus(renameResult)) {
d.dockerRename(backupContainerName, originalName, containerMutationOptions);
const restarted = hasZeroDockerExitStatus(
d.dockerStart(oldContainerId, containerMutationOptions),
);
const restarted =
backupWasRunning ||
hasZeroDockerExitStatus(d.dockerStart(oldContainerId, containerMutationOptions));
let originalNameRestored = false;
try {
originalNameRestored =
Expand All @@ -334,7 +340,7 @@ export function recreateOpenShellDockerSandboxContainer(
});
if (!hasZeroDockerExitStatus(runResult)) {
context.rolledBack = restoreDockerGpuPatchBackupAfterRecreateFailure(
{ newContainerId: originalName, backupContainerName, originalName },
{ newContainerId: originalName, backupContainerName, originalName, backupWasRunning },
deps,
);
const containerDescription =
Expand All @@ -360,7 +366,7 @@ export function recreateOpenShellDockerSandboxContainer(
);
if (!newContainerId) {
context.rolledBack = restoreDockerGpuPatchBackupAfterRecreateFailure(
{ newContainerId: originalName, backupContainerName, originalName },
{ newContainerId: originalName, backupContainerName, originalName, backupWasRunning },
deps,
);
const containerDescription =
Expand All @@ -384,6 +390,7 @@ export function recreateOpenShellDockerSandboxContainer(
originalName,
backupContainerName,
mode: selectedMode,
backupWasRunning,
backupRemoved,
});
if (options.waitForSupervisor === false) return result(false);
Expand Down
Loading
Loading