Skip to content
Merged
6 changes: 6 additions & 0 deletions docs/manage-sandboxes/backup-restore.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,12 @@ Tag a snapshot with a human-readable label:
$$nemoclaw my-assistant snapshot create --name before-upgrade
```

When a directory or state file cannot be captured, `snapshot create` reports the failed items, removes the incomplete snapshot, and exits nonzero.
`snapshot list` shows no new entry, so a later restore cannot select a capture that never completed.

To keep the incomplete snapshot for diagnosis, pass `--keep-failed` or set `NEMOCLAW_KEEP_FAILED_SNAPSHOT=1`.
A kept snapshot is listed and restorable like any other, so remove it once you no longer need it.

## Restore a Snapshot

Restore the latest snapshot:
Expand Down
12 changes: 12 additions & 0 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3004,13 +3004,24 @@ $$nemoclaw my-assistant snapshot create
| Flag | Description |
|------|-------------|
| `--name <label>` | Attach a human-readable label to the snapshot so you can restore by name later |
| `--keep-failed` | Keep an incomplete snapshot on disk when creation fails, instead of removing it (also enabled by `NEMOCLAW_KEEP_FAILED_SNAPSHOT=1`) |

Names must be 1 to 63 characters from `[A-Za-z0-9._-]`, start with an alphanumeric character, and cannot look like a version selector (`v1`, `v2`, ...). Duplicate names per sandbox are rejected; pick a different name or delete the existing snapshot first.

```bash
$$nemoclaw my-assistant snapshot create --name before-upgrade
```

When a directory or state file cannot be captured, the command reports the failed items, removes the incomplete snapshot, and exits nonzero.
A removed snapshot does not appear in `snapshot list` and cannot be restored, so a later restore cannot select a capture that never completed.

Pass `--keep-failed` to keep the incomplete snapshot for diagnosis.
The kept snapshot is listed and restorable like any other, and it shifts the computed version of every snapshot created after it, so remove it once you no longer need it.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated

```bash
$$nemoclaw my-assistant snapshot create --keep-failed
```

### `$$nemoclaw <name> snapshot list`

List available snapshots for a sandbox as a table of version, name, timestamp, and path.
Expand Down Expand Up @@ -4137,6 +4148,7 @@ The following flags change defaults for commands that manage existing sandboxes.
| `NEMOCLAW_CONFIRM_LEGACY_MANAGED_RECREATE` | Exact JSON array of sandbox names | Confirms to the installer that the exact listed set of pre-fingerprint OpenClaw or Hermes sandboxes used NemoClaw-managed images, allowing recovery onto the current managed image. The normalized names must exactly match the installer's printed array. Set it only after verifying every named sandbox. Recorded custom-image evidence remains blocked. |
| `NEMOCLAW_DISABLE_INFERENCE_ROUTE_REPAIR` | `1` to enable | Skips the automatic DNS-proxy repair for stale `inference.local` routes during `$$nemoclaw <name> connect` and `$$nemoclaw <name> connect --probe-only`. Use only as a troubleshooting escape hatch. |
| `NEMOCLAW_DISABLE_SUPERVISOR_RELAUNCH` | `1` to enable | Skips the automatic trusted container recreation during `$$nemoclaw <name> recover` when two managed scans find no supervisor while PID 1 remains stable. Use only as a troubleshooting escape hatch; recovery then falls back to the rebuild or re-onboard guidance. |
| `NEMOCLAW_KEEP_FAILED_SNAPSHOT` | Exactly `1` to opt in (`true`, `yes`, `0` are not accepted) | Keeps the incomplete snapshot on disk when `$$nemoclaw <name> snapshot create` fails, instead of removing it. Equivalent to passing `--keep-failed`. A kept snapshot is listed and restorable even though its capture did not complete, and it shifts the computed version of every snapshot created after it. |
| `NEMOCLAW_SHIELDS_ACCEPT_LEGACY_BASELINE` | `1` to opt in | Allows advanced immutable-config verification to trust the current on-disk bytes for older or partial content baselines. Use only after you have rebuilt or manually inspected the sandbox state and accepted that the baseline is operator-approved. |
| `NEMOCLAW_SHIELDS_SETTLE_MS` | milliseconds (default `750`, clamped to `0` to `10000`) | Settle window NemoClaw waits after re-applying a config lockdown (during shields auto-restore and `$$nemoclaw <name> shields up` drift remediation) before re-confirming the lock still holds. Detects when an in-sandbox reconciler changes config file permissions after lockdown and re-applies the lock; if NemoClaw cannot re-confirm the lock within the retry budget, shields stay down. This narrows the window in which a reconciler can revert permissions rather than eliminating it. The best-effort `chattr +i` immutable bit remains the only fully durable lock. Raise it on hosts where the gateway settles slowly. |
| `NEMOCLAW_SKIP_UNREACHABLE_SANDBOX_BACKUP` | Exactly `1` to opt in (`true`, `yes`, `0` are not accepted) | Applies to standalone `$$nemoclaw backup-all` runs. Skips running sandboxes whose in-sandbox SSH endpoint does not answer. It does not relax the installer's strict pre-upgrade backup, which still aborts if any registered sandbox is skipped or fails. Any uncommitted state since the last successful backup is not included in the skipped backup. |
Expand Down
11 changes: 11 additions & 0 deletions src/commands/sandbox/snapshot.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,17 @@ describe("snapshot oclif commands", () => {
expect(runSandboxSnapshot).toHaveBeenCalledWith("alpha", {
kind: "create",
name: "before-upgrade",
keepFailed: false,
});
});

it("runs snapshot create asking to keep an incomplete snapshot", async () => {
await SnapshotCreateCommand.run(["alpha", "--keep-failed"], rootDir);

expect(runSandboxSnapshot).toHaveBeenCalledWith("alpha", {
kind: "create",
name: undefined,
keepFailed: true,
});
});
});
14 changes: 12 additions & 2 deletions src/commands/sandbox/snapshot/create.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,22 +15,32 @@ export default class SnapshotCreateCommand extends NemoClawCommand {
static strict = true;
static summary = "Create a snapshot of sandbox state";
static description = "Create an auto-versioned snapshot of sandbox workspace state.";
static usage = ["<name> [--name <label>]"];
static usage = ["<name> [--name <label>] [--keep-failed]"];
static examples = [
"<%= config.bin %> sandbox snapshot create alpha",
"<%= config.bin %> sandbox snapshot create alpha --name before-upgrade",
"<%= config.bin %> sandbox snapshot create alpha --keep-failed",
];
static args = {
sandboxName: sandboxNameArg,
};
static flags = {
name: Flags.string({ description: "Optional snapshot label" }),
"keep-failed": Flags.boolean({
description:
"Keep an incomplete snapshot on disk when creation fails, instead of removing it",
default: false,
}),
};

public async run(): Promise<void> {
const { args, flags } = await this.parse(SnapshotCreateCommand);
try {
await runSandboxSnapshot(args.sandboxName, { kind: "create", name: flags.name });
await runSandboxSnapshot(args.sandboxName, {
kind: "create",
name: flags.name,
keepFailed: flags["keep-failed"],
});
} catch (error) {
const snapshotError = snapshotCommandError(error);
if (snapshotError) {
Expand Down
195 changes: 195 additions & 0 deletions src/lib/actions/sandbox/snapshot-failed-create-cleanup.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";

const mocks = vi.hoisted(() => ({
backupSandboxState: vi.fn(),
captureOpenshell: vi.fn(() => ({ status: 0, output: "alpha Ready\n" })),
findBackup: vi.fn(() => ({ match: null })),
removeIncompleteSnapshot: vi.fn(
() =>
({ removed: true }) as {
removed: boolean;
error?: string;
},
),
}));

vi.mock("../../adapters/openshell/runtime", () => ({
captureOpenshell: mocks.captureOpenshell,
getOpenshellBinary: vi.fn(() => "openshell"),
runOpenshell: vi.fn(),
}));

vi.mock("../../runtime-recovery", () => ({
parseLiveSandboxNames: vi.fn(() => new Set(["alpha"])),
}));

vi.mock("../../shields", () => ({
isShieldsDown: vi.fn(() => true),
}));

vi.mock("../../shields/timer-bound-lock", () => ({
withTimerBoundShieldsMutationLock: vi.fn(
(_sandboxName: string, _command: string, operation: () => unknown) => operation(),
),
}));

vi.mock("../../state/registry", () => ({
getBaselineExclusions: vi.fn(() => []),
getSandbox: vi.fn(() => ({ name: "alpha", agent: "openclaw" })),
}));

vi.mock("../../state/sandbox", () => ({
backupSandboxState: mocks.backupSandboxState,
findBackup: mocks.findBackup,
removeIncompleteSnapshot: mocks.removeIncompleteSnapshot,
}));

vi.mock("./sandbox-gateway-routing", () => ({
probeGatewayRunning: vi.fn(() => true),
selectSandboxGatewayIfRegistered: vi.fn(() => true),
usesGatewayMetadataProbe: vi.fn(() => false),
}));

const INCOMPLETE_PATH = "/home/user/.nemoclaw/rebuild-backups/alpha/2026-08-04T06-53-38-310Z";

function failedCaptureWithPublishedSnapshot(overrides: Record<string, unknown> = {}) {
return {
success: false,
manifest: { backupPath: INCOMPLETE_PATH },
backedUpDirs: ["workspace"],
failedDirs: [],
backedUpFiles: [],
failedFiles: ["openclaw.json"],
...overrides,
};
}

describe("snapshot create cleanup after a failed capture", () => {
beforeEach(() => {
vi.clearAllMocks();
mocks.removeIncompleteSnapshot.mockReturnValue({ removed: true });
mocks.findBackup.mockReturnValue({ match: null });
vi.spyOn(console, "log").mockImplementation(() => {});
vi.spyOn(console, "error").mockImplementation(() => {});
});

afterEach(() => {
vi.restoreAllMocks();
vi.unstubAllEnvs();
});
Comment thread
laitingsheng marked this conversation as resolved.

async function createSnapshot(request: Record<string, unknown> = {}): Promise<string> {
const { runSandboxSnapshot } = await import("./snapshot");
await expect(runSandboxSnapshot("alpha", { kind: "create", ...request })).rejects.toMatchObject(
{ exitCode: 1 },
);
return vi.mocked(console.error).mock.calls.flat().join("\n");
}

it("reports the failed directories and files", async () => {
mocks.backupSandboxState.mockReturnValue(
failedCaptureWithPublishedSnapshot({
failedDirs: ["workspace", "skills"],
failedDirReasons: { workspace: "permission denied" },
}),
);

const errors = await createSnapshot();

expect(errors).toContain("Snapshot failed.");
expect(errors).toContain("Failed directories: workspace (permission denied), skills");
expect(errors).toContain("Failed files: openclaw.json");
});

it("removes the snapshot so a later restore cannot select an incomplete capture (#8201)", async () => {
mocks.backupSandboxState.mockReturnValue(failedCaptureWithPublishedSnapshot());

const errors = await createSnapshot();

expect(mocks.removeIncompleteSnapshot).toHaveBeenCalledWith(INCOMPLETE_PATH);
expect(errors).toContain("Removed the incomplete snapshot.");
});

it("keeps the snapshot when --keep-failed is passed", async () => {
mocks.backupSandboxState.mockReturnValue(failedCaptureWithPublishedSnapshot());

const errors = await createSnapshot({ keepFailed: true });

expect(mocks.removeIncompleteSnapshot).not.toHaveBeenCalled();
expect(errors).toContain(`Kept the incomplete snapshot at ${INCOMPLETE_PATH}.`);
expect(errors).toContain("can be restored even though its capture did not complete");
});

it("keeps the snapshot when NEMOCLAW_KEEP_FAILED_SNAPSHOT is exactly 1", async () => {
vi.stubEnv("NEMOCLAW_KEEP_FAILED_SNAPSHOT", "1");
mocks.backupSandboxState.mockReturnValue(failedCaptureWithPublishedSnapshot());

await createSnapshot();

expect(mocks.removeIncompleteSnapshot).not.toHaveBeenCalled();
});

it.each([
"true",
"yes",
"0",
"",
])("removes the snapshot when NEMOCLAW_KEEP_FAILED_SNAPSHOT is %j", async (value) => {
vi.stubEnv("NEMOCLAW_KEEP_FAILED_SNAPSHOT", value);
mocks.backupSandboxState.mockReturnValue(failedCaptureWithPublishedSnapshot());

await createSnapshot();

expect(mocks.removeIncompleteSnapshot).toHaveBeenCalledWith(INCOMPLETE_PATH);
});

it("names the snapshot that is still listed when removal fails", async () => {
mocks.backupSandboxState.mockReturnValue(failedCaptureWithPublishedSnapshot());
mocks.removeIncompleteSnapshot.mockReturnValue({
removed: false,
error: "EACCES: permission denied",
});

const errors = await createSnapshot();

expect(errors).toContain(
`The incomplete snapshot at '${INCOMPLETE_PATH}' could not be removed: EACCES: permission denied`,
);
expect(errors).toContain("Remove it before the next restore.");
});

it("does not attempt removal when the capture failed before publishing a snapshot", async () => {
mocks.backupSandboxState.mockReturnValue({
success: false,
backedUpDirs: [],
failedDirs: [],
backedUpFiles: [],
failedFiles: [],
error: "Snapshot name 'dup' already exists.",
});

const errors = await createSnapshot();

expect(mocks.removeIncompleteSnapshot).not.toHaveBeenCalled();
expect(errors).toContain("Snapshot name 'dup' already exists.");
});

it("does not touch a snapshot whose capture succeeded", async () => {
mocks.backupSandboxState.mockReturnValue({
success: true,
manifest: { backupPath: INCOMPLETE_PATH, timestamp: "2026-08-04T06-53-38-310Z" },
backedUpDirs: ["workspace"],
failedDirs: [],
backedUpFiles: ["openclaw.json"],
failedFiles: [],
});
const { runSandboxSnapshot } = await import("./snapshot");

await runSandboxSnapshot("alpha", { kind: "create", keepFailed: true });

expect(mocks.removeIncompleteSnapshot).not.toHaveBeenCalled();
});
});
21 changes: 0 additions & 21 deletions src/lib/actions/sandbox/snapshot.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1476,25 +1476,4 @@ describe("runSandboxSnapshot", () => {
"Warning: could not reconcile custom policy(ies): old-custom (remove failed)",
);
});

it("prints failed dirs and files when snapshot creation fails without an error", async () => {
backupSandboxStateMock.mockReturnValue({
success: false,
failedDirs: ["workspace", "skills"],
failedDirReasons: { workspace: "permission denied" },
failedFiles: ["openclaw.json"],
});
const consoleError = vi.spyOn(console, "error").mockImplementation(() => {});
vi.spyOn(console, "log").mockImplementation(() => {});
const { runSandboxSnapshot } = await import("./snapshot");

await expect(runSandboxSnapshot("alpha", { kind: "create" })).rejects.toMatchObject({
exitCode: 1,
});

const errors = consoleError.mock.calls.flat().join("\n");
expect(errors).toContain("Snapshot failed.");
expect(errors).toContain("Failed directories: workspace (permission denied), skills");
expect(errors).toContain("Failed files: openclaw.json");
});
});
39 changes: 38 additions & 1 deletion src/lib/actions/sandbox/snapshot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ const R = useColor ? "\x1b[0m" : "";

export type SnapshotRequest =
| { kind: "help" }
| { kind: "create"; name?: string }
| { kind: "create"; name?: string; keepFailed?: boolean }
Comment thread
laitingsheng marked this conversation as resolved.
Outdated
| { kind: "list" }
| {
kind: "restore";
Expand Down Expand Up @@ -659,6 +659,39 @@ function isSnapshotCreationAllowedByDcodeActivity(sandboxName: string): boolean
return false;
}

const KEEP_FAILED_SNAPSHOT_ENV = "NEMOCLAW_KEEP_FAILED_SNAPSHOT";

function keepFailedSnapshotRequested(
request: Extract<SnapshotRequest, { kind: "create" }>,
): boolean {
return request.keepFailed === true || process.env[KEEP_FAILED_SNAPSHOT_ENV] === "1";
}

function reportIncompleteSnapshot(
sandboxName: string,
backupPath: string,
keepFailed: boolean,
): void {
if (keepFailed) {
console.error(` Kept the incomplete snapshot at ${backupPath}.`);
console.error(
` It is listed by \`${CLI_NAME} ${sandboxName} snapshot list\` and can be restored even though its capture did not complete.`,
);
return;
}
const removal = sandboxState.removeIncompleteSnapshot(backupPath);
if (removal.removed) {
console.error(" Removed the incomplete snapshot.");
return;
}
console.error(
` The incomplete snapshot at '${backupPath}' could not be removed: ${removal.error}`,
);
console.error(
` It is listed by \`${CLI_NAME} ${sandboxName} snapshot list\`. Remove it before the next restore.`,
);
}

function runSnapshotCreate(
sandboxName: string,
request: Extract<SnapshotRequest, { kind: "create" }>,
Expand Down Expand Up @@ -722,6 +755,10 @@ function runSnapshotCreate(
console.error(` Failed files: ${result.failedFiles.join(", ")}`);
}
}
const incompletePath = result.manifest?.backupPath;
if (incompletePath) {
reportIncompleteSnapshot(sandboxName, incompletePath, keepFailedSnapshotRequested(request));
}
snapshotExit(1);
});
}
Expand Down
2 changes: 1 addition & 1 deletion src/lib/cli/public-display-defaults.ts
Original file line number Diff line number Diff line change
Expand Up @@ -493,7 +493,7 @@ const PUBLIC_DISPLAY_LAYOUT: Record<string, readonly PublicDisplayLayout[]> = {
{
group: "Sandbox Management",
order: 7,
flags: "[--name <name>]",
flags: "[--name <name>] [--keep-failed]",
},
],
"sandbox:snapshot:list": [
Expand Down
Loading
Loading