fix(onboard): keep the pre-rollback cause when a sandbox container cannot start

docs/reference/troubleshooting.mdx

@@ -643,6 +643,12 @@ sudo ufw allow from "$SUBNET" to any port 8080 proto tcp
 $$nemoclaw onboard
 ```
 
+This reachability check uses a disposable Docker probe and does not create or replace a sandbox.
+If Docker GPU compatibility recreation fails later, follow [GPU routing or compatibility patch failed](#gpu-routing-or-compatibility-patch-failed).
+That path can restore the pre-patch sandbox.
+If its diagnostics report manual cleanup, use only the printed exact-container command.
+That command targets the failed replacement and preserves the restored sandbox.
+
 <AgentOnly variant="openclaw">
 
 ### Custom OpenClaw image creates without a gateway or dashboard
@@ -657,6 +663,19 @@ For the version-pinned plugin workflow, refer to [Install OpenClaw Plugins](../m
 
 If the sandbox is unreachable or the managed runtime paths are present, NemoClaw retains the existing generic gateway-log and host OpenShell-log guidance because the base-only failure is not proven.
 
+A custom image without the managed runtime can fail while NemoClaw starts the sandbox container.
+NemoClaw reports exit code 127 without assigning a cause unless captured logs contain the exact `env` error for missing `nemoclaw-start`.
+When that error is present, the failure output identifies the missing managed startup command and gives the same rebuild guidance.
+If NemoClaw saves pre-rollback diagnostics, the reported directory contains the captured container logs.
+If rollback succeeds, NemoClaw restores and starts the pre-patch sandbox container.
+It does not print a sandbox deletion command for the restored sandbox.
+If the failed replacement container remains, NemoClaw prints an exact-container Docker cleanup command.
+If NemoClaw cannot confirm whether the replacement remains but retains its validated exact ID, it prints the same target-safe command.
+Without a validated exact ID, it reports cleanup as unknown and prints no deletion command.
+If rollback fails, sandbox and container state can be uncertain.
+If NemoClaw reports a diagnostics directory, inspect it.
+Inspect the diagnostics before removing any container.
+
 </AgentOnly>
 
 <AgentOnly variant="openclaw,hermes">
@@ -2667,7 +2686,7 @@ Identify the matching path before applying the recovery guidance.
 | --- | --- | --- |
 | Native `--gpu` is rejected, host runtime evidence identifies GPU injection failure, or an explicit driver proof fails and host configuration confirms no GPU attachment | Ordinary Linux native attempt | The default native-only route stops. Retry with `NEMOCLAW_DOCKER_GPU_PATCH=fallback` only if you explicitly accept one bounded compatibility retry, or use `=1` to select compatibility before creation. |
 | `Cleanup could not be proven safe` | Native-to-compatibility handoff | Run the printed sandbox deletion command, verify both the gateway row and OpenShell-managed Docker containers labeled for that sandbox are absent, then rerun onboarding. |
-| The patched container exits or the compatibility attempt fails | Compatibility recreation | Inspect the saved diagnostics and the rollback outcome, then repair the NVIDIA Container Toolkit/CDI configuration. Keep the sandbox when the pre-patch container was restored; delete it only after inspection confirms restoration failed. Then rerun onboarding. |
+| The patched container exits or the compatibility attempt fails | Compatibility recreation | Inspect the saved diagnostics and the rollback outcome, then repair the NVIDIA Container Toolkit/CDI configuration. Keep the sandbox when the pre-patch container was restored. Use only an exact-container cleanup command printed after rollback. If no command was printed, inspect the sandbox and its labeled containers before removing anything. Then rerun onboarding. |
 | A recreated container inherits only a loopback DNS stub and no usable upstream | Compatibility DNS fallback | Repair the host's `systemd-resolved` upstream configuration, then rerun onboarding. |
 
 For bridge-networked compatibility recreation without an explicit container DNS setting, NemoClaw selects a usable IPv4 upstream from `systemd-resolved` and probes that exact `--dns` path before it stops the original container.
@@ -2716,9 +2735,11 @@ Use `NEMOCLAW_DOCKER_GPU_PATCH=0` only for troubleshooting because it bypasses t
 After compatibility recreation starts, onboarding keeps the pre-patch container as a rollback backup until the replacement passes the Ready, direct GPU, and applicable local-inference checks.
 If a later check fails, onboarding prints failure diagnostics and attempts to restore the pre-patch container before it exits.
 When rollback succeeds, the pre-patch sandbox remains available.
-When rollback fails, onboarding reports that the pre-patch container was not restored and prints container-cleanup guidance.
-GPU-proof diagnostics are captured before rollback and can print that guidance before the final container state is known.
-Inspect the sandbox and its labeled Docker containers before running a deletion command.
+If the failed replacement may remain and NemoClaw retains its validated exact container ID, it prints only an exact-container `docker rm -f` command.
+If replacement cleanup cannot be confirmed without a validated exact ID, onboarding reports cleanup as unknown and prints no deletion command.
+When rollback fails, onboarding reports that the sandbox and container state is uncertain and prints no deletion command.
+A diagnostic bundle captured before rollback records cleanup as pending and contains no deletion command.
+Inspect the diagnostics, the sandbox, and its labeled Docker containers before removing anything.
 
 <Note>
 Starting with NemoClaw v0.0.43, the standard installer handles the `/proc/<pid>/task/<tid>/comm` permission case during this patch path.

src/lib/onboard/docker-gpu-patch-clone.ts

@@ -299,6 +299,14 @@ export function sameContainerId(
   return left.startsWith(right) || right.startsWith(left);
 }
 
+/** Return only a complete Docker container ID that is safe for exact-ID cleanup. */
+export function fullDockerContainerId(value: string | null | undefined): string | null {
+  const normalized = String(value ?? "")
+    .trim()
+    .toLowerCase();
+  return /^[0-9a-f]{64}$/u.test(normalized) ? normalized : null;
+}
+
 function dockerNetworkAliases(
   inspect: DockerContainerInspect,
   networkMode: string | null | undefined,

src/lib/onboard/docker-gpu-patch-diagnostics-classification.test.ts

@@ -147,6 +147,41 @@ describe("Docker GPU patch diagnostics", () => {
     expect(flat).toContain("patched_create_option=--gpus all");
   });
 
+  it("explains exit code 127 when container logs prove the managed startup command is missing (#7996)", () => {
+    const result = classifyDockerGpuPatchFailure(
+      failureSnapshot("Error", { Status: "restarting", ExitCode: 127 }, "alpha   Error   1m ago"),
+      GPU_MODE,
+      { managedStartupCommandMissing: true },
+    );
+
+    expect(result.kind).toBe("patched_container_failed");
+    const hints = (result.hints ?? []).join("\n");
+    expect(hints).toContain("does not provide the NemoClaw-managed `nemoclaw-start` command");
+    expect(hints).toContain("selected agent and NemoClaw release");
+    expect(hints).not.toMatch(/OpenClaw|sandbox-base/);
+    // The prose stays out of the machine-readable on-disk summary.
+    expect(result.summaryLines.join("\n")).not.toContain("selected agent");
+  });
+
+  it("does not infer a missing startup command when its child process returns 127 (#7996)", () => {
+    const result = classify(
+      failureSnapshot("Error", { Status: "exited", ExitCode: 127 }, "alpha   Error   1m ago"),
+    );
+
+    expect(result.kind).toBe("patched_container_failed");
+    expect(result.headline).toContain("exited with code 127");
+    expect(result.hints ?? []).toEqual([]);
+  });
+
+  it("does not attach the missing-startup-command hints to other non-zero exits (#7996)", () => {
+    const result = classify(
+      failureSnapshot("Error", { Status: "exited", ExitCode: 125 }, "alpha   Error   1m ago"),
+    );
+
+    expect(result.kind).toBe("patched_container_failed");
+    expect(result.hints ?? []).toEqual([]);
+  });
+
   it("classifies an Error-phase sandbox with unknown container state as sandbox_error_phase", () => {
     const result = classify(failureSnapshot("Error", null, null));
 

src/lib/onboard/docker-gpu-patch-diagnostics-collection.test.ts

@@ -113,4 +113,145 @@ describe("Docker GPU patch diagnostics", () => {
       fs.rmSync(tmpDir, { recursive: true, force: true });
     }
   });
+
+  it("omits cleanup after rollback confirms that the replacement is absent (#7996)", () => {
+    const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-gpu-rollback-"));
+    try {
+      const diagnostics = collectDockerGpuPatchDiagnostics(
+        "alpha",
+        {
+          context: {
+            sandboxName: "alpha",
+            newContainerId: "removed-container-id",
+            rolledBack: true,
+            replacementStopConfirmed: true,
+            replacementRemovalConfirmed: true,
+            replacementPresence: "absent",
+          },
+        },
+        {
+          dockerCapture: vi.fn(() => ""),
+          dockerLogs: vi.fn(() => ""),
+          homedir: () => tmpDir,
+          now: () => new Date("2026-05-12T00:00:00Z"),
+        },
+      );
+
+      const summary = fs.readFileSync(path.join(diagnostics?.dir || "", "summary.txt"), "utf-8");
+      expect(diagnostics?.cleanupCommands).toEqual([]);
+      expect(diagnostics?.cleanupDisposition).toBe("not_required");
+      expect(summary).toContain("cleanup_required=no");
+      expect(summary).not.toContain("openshell sandbox delete");
+    } finally {
+      fs.rmSync(tmpDir, { recursive: true, force: true });
+    }
+  });
+
+  it("keeps cleanup unknown when rollback cannot confirm replacement absence (#7996)", () => {
+    const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-gpu-unknown-"));
+    try {
+      const diagnostics = collectDockerGpuPatchDiagnostics(
+        "alpha",
+        {
+          context: {
+            sandboxName: "alpha",
+            newContainerId: "unconfirmed-container-id",
+            rolledBack: true,
+            replacementStopConfirmed: false,
+            replacementRemovalConfirmed: false,
+            replacementPresence: "unknown",
+          },
+        },
+        {
+          dockerCapture: vi.fn(() => ""),
+          dockerLogs: vi.fn(() => ""),
+          homedir: () => tmpDir,
+          now: () => new Date("2026-05-12T00:00:01Z"),
+        },
+      );
+
+      const summary = fs.readFileSync(path.join(diagnostics?.dir || "", "summary.txt"), "utf-8");
+      expect(diagnostics?.cleanupCommands).toEqual([]);
+      expect(diagnostics?.cleanupDisposition).toBe("unknown");
+      expect(summary).toContain("replacement_presence=unknown");
+      expect(summary).toContain("cleanup_required=unknown");
+      expect(summary).not.toContain("openshell sandbox delete");
+    } finally {
+      fs.rmSync(tmpDir, { recursive: true, force: true });
+    }
+  });
+
+  it("keeps cleanup unknown when a present replacement lacks an exact ID (#7996)", () => {
+    const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-gpu-invalid-id-"));
+    try {
+      const diagnostics = collectDockerGpuPatchDiagnostics(
+        "alpha",
+        {
+          context: {
+            sandboxName: "alpha",
+            newContainerId: "short-container-id",
+            rolledBack: true,
+            replacementStopConfirmed: false,
+            replacementRemovalConfirmed: false,
+            replacementPresence: "present",
+          },
+        },
+        {
+          dockerCapture: vi.fn(() => ""),
+          dockerLogs: vi.fn(() => ""),
+          homedir: () => tmpDir,
+          now: () => new Date("2026-05-12T00:00:02Z"),
+        },
+      );
+
+      const summary = fs.readFileSync(path.join(diagnostics?.dir || "", "summary.txt"), "utf-8");
+      expect(diagnostics?.cleanupCommands).toEqual([]);
+      expect(diagnostics?.cleanupDisposition).toBe("unknown");
+      expect(summary).toContain("replacement_presence=present");
+      expect(summary).toContain("cleanup_required=unknown");
+    } finally {
+      fs.rmSync(tmpDir, { recursive: true, force: true });
+    }
+  });
+
+  it("uses exact-ID cleanup when post-rollback inspection finds the replacement (#7996)", () => {
+    const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-docker-gpu-inspect-"));
+    const replacementId = "b".repeat(64);
+    try {
+      const diagnostics = collectDockerGpuPatchDiagnostics(
+        "alpha",
+        {
+          context: {
+            sandboxName: "alpha",
+            newContainerId: replacementId,
+            rolledBack: true,
+            replacementStopConfirmed: false,
+            replacementRemovalConfirmed: false,
+            replacementPresence: "unknown",
+          },
+        },
+        {
+          dockerCapture: vi.fn((args: readonly string[]) =>
+            args[0] === "inspect" && args[1] === replacementId
+              ? JSON.stringify([{ Id: replacementId }])
+              : "",
+          ),
+          dockerLogs: vi.fn(() => ""),
+          homedir: () => tmpDir,
+          now: () => new Date("2026-05-12T00:00:02Z"),
+        },
+      );
+
+      const summary = fs.readFileSync(path.join(diagnostics?.dir || "", "summary.txt"), "utf-8");
+      expect(diagnostics?.cleanupCommands).toEqual([
+        `docker rm -f ${JSON.stringify(replacementId)}`,
+      ]);
+      expect(diagnostics?.cleanupDisposition).toBe("manual");
+      expect(summary).toContain("replacement_presence=present");
+      expect(summary).toContain("cleanup_required=yes");
+      expect(summary).not.toContain("openshell sandbox delete");
+    } finally {
+      fs.rmSync(tmpDir, { recursive: true, force: true });
+    }
+  });
 });

src/lib/onboard/docker-gpu-patch-diagnostics.ts

@@ -10,6 +10,7 @@ import { GATEWAY_PORT } from "../core/ports";
 import { rejectSymlinksOnPath } from "../state/config-io";
 import { nemoclawStateRoot } from "../state/state-root";
 import { createDockerGpuDiagnosticRedactor } from "./docker-gpu-diagnostic-redaction";
+import { fullDockerContainerId } from "./docker-gpu-patch-clone";
 import { DOCKER_GPU_PATCH_TIMEOUT_MS } from "./docker-gpu-patch-constants";
 import { getDockerGpuPatchFailureContext } from "./docker-gpu-patch-recreate";
 import type {
@@ -129,6 +130,14 @@ export function dockerGpuPatchCleanupCommands(sandboxName: string): string[] {
   return [`openshell sandbox delete ${JSON.stringify(sandboxName)}`];
 }
 
+function dockerGpuReplacementCleanupCommands(containerId: string): string[] {
+  return [`docker rm -f ${JSON.stringify(containerId)}`];
+}
+
+function confirmationValue(value: boolean | undefined): string {
+  return value === true ? "yes" : value === false ? "no" : "unknown";
+}
+
 export function collectDockerGpuPatchDiagnostics(
   sandboxName: string,
   options: {
@@ -140,6 +149,11 @@ export function collectDockerGpuPatchDiagnostics(
     additionalSummaryLines?: readonly string[];
     additionalSensitiveValues?: readonly string[];
     dockerTopOutput?: string | null;
+    /**
+     * The caller captured evidence before rollback and cannot yet determine
+     * whether manual cleanup will be required.
+     */
+    cleanupDisposition?: "pending-rollback";
   } = {},
   deps: DockerGpuPatchDeps = {},
 ): DockerGpuPatchDiagnostics | null {
@@ -162,6 +176,9 @@ export function collectDockerGpuPatchDiagnostics(
   }
 
   const context = options.context || getDockerGpuPatchFailureContext(options.error) || null;
+  const selectedMode = options.selectedMode || context?.selectedMode || null;
+  const snapshot = options.snapshot ?? null;
+  const classification = options.classification ?? null;
   const redactor = createDockerGpuDiagnosticRedactor(options.additionalSensitiveValues);
   let discoveredContainerIds: string[] = [];
   try {
@@ -198,17 +215,41 @@ export function collectDockerGpuPatchDiagnostics(
     writeTextFile(dir, name, JSON.stringify(redactor.redactValue(value), null, 2));
   };
 
-  const cleanupCommands = dockerGpuPatchCleanupCommands(sandboxName).map(redactor.redactText);
+  const cleanupPendingRollback = options.cleanupDisposition === "pending-rollback";
+  const prePatchRestored = context?.rolledBack === true;
+  const replacementId = fullDockerContainerId(context?.newContainerId);
+  const inspectConfirmsReplacementPresent =
+    replacementId !== null &&
+    inspectedTargets.some(({ entries }) =>
+      entries.some((entry) => fullDockerContainerId(entry.Id) === replacementId),
+    );
+  const snapshotConfirmsReplacementPresent =
+    replacementId !== null && snapshot?.patchedContainerState != null;
+  const replacementPresence =
+    snapshotConfirmsReplacementPresent || inspectConfirmsReplacementPresent
+      ? "present"
+      : (context?.replacementPresence ?? "unknown");
+  let cleanupDisposition: DockerGpuPatchDiagnostics["cleanupDisposition"];
+  let cleanupCommands: string[] = [];
+  if (cleanupPendingRollback) {
+    cleanupDisposition = "pending_rollback";
+  } else if (!prePatchRestored) {
+    cleanupDisposition = "unknown";
+  } else if (replacementPresence === "absent") {
+    cleanupDisposition = "not_required";
+  } else if (replacementId) {
+    cleanupDisposition = "manual";
+    cleanupCommands = dockerGpuReplacementCleanupCommands(replacementId).map(redactor.redactText);
+  } else {
+    cleanupDisposition = "unknown";
+  }
   const errorText = redactor.redactText(
     options.error instanceof Error
       ? options.error.message
       : options.error
         ? String(options.error)
         : "none",
   );
-  const selectedMode = options.selectedMode || context?.selectedMode || null;
-  const snapshot = options.snapshot ?? null;
-  const classification = options.classification ?? null;
   const summaryLines = [
     `created_at=${now.toISOString()}`,
     `sandbox_name=${redactor.redactText(sandboxName)}`,
@@ -218,9 +259,19 @@ export function collectDockerGpuPatchDiagnostics(
     `old_container_id=${redactor.redactText(context?.oldContainerId ?? "unknown")}`,
     `new_container_id=${redactor.redactText(context?.newContainerId ?? "unknown")}`,
     `backup_container_name=${redactor.redactText(context?.backupContainerName ?? "none")}`,
-    `rolled_back=${context?.rolledBack === true ? "yes" : context?.rolledBack === false ? "failed" : "no"}`,
-    "cleanup_commands:",
-    ...cleanupCommands.map((command) => `  ${command}`),
+    `rolled_back=${cleanupPendingRollback ? "pending" : context?.rolledBack === true ? "yes" : context?.rolledBack === false ? "failed" : "no"}`,
+    ...(context?.replacementStopConfirmed !== undefined
+      ? [`replacement_stop_confirmed=${confirmationValue(context.replacementStopConfirmed)}`]
+      : []),
+    ...(context?.replacementRemovalConfirmed !== undefined
+      ? [`replacement_removal_confirmed=${confirmationValue(context.replacementRemovalConfirmed)}`]
+      : []),
+    ...(prePatchRestored ? [`replacement_presence=${replacementPresence}`] : []),
+    `cleanup_disposition=${cleanupDisposition}`,
+    `cleanup_required=${cleanupDisposition === "manual" ? "yes" : cleanupDisposition === "not_required" ? "no" : "unknown"}`,
+    ...(cleanupCommands.length > 0
+      ? ["cleanup_commands:", ...cleanupCommands.map((command) => `  ${command}`)]
+      : []),
   ];
   if (context?.modeAttempts?.length) {
     summaryLines.push("gpu_mode_attempts:");
@@ -327,5 +378,5 @@ export function collectDockerGpuPatchDiagnostics(
     }
   }
 
-  return { dir, cleanupCommands, summaryLines };
+  return { dir, cleanupCommands, cleanupDisposition, summaryLines };
 }