Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .agents/skills/_shared/controlled-words.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ other literal identifiers.
| `custom endpoint` | Technical noun | A user-supplied inference endpoint that is not one of NemoClaw's named provider choices. | compatible endpoint before compatibility is validated |
| `hosted inference` | Technical noun | Inference served by a remote provider-operated service. | local inference, cloud model |
| `inference` | Technical noun | Model execution that produces a response from an input. | AI, generation when model execution is intended |
| `inference health` | Technical noun | The complete classification from the named `inference.local` `/v1/models` status probe: `reachable` for HTTP `200` through `499`, `unhealthy` for HTTP `500` through `599`, or `unreachable` when no qualifying HTTP response arrives. | model health, successful inference, validation request |
| `inference health` | Technical noun | The classification reported for a sandbox's inference route: the `inference.local` `/v1/models` probe result, and, when that route is reachable and the sandbox records a provider and a model, the result of one inference request sent over the same route. Values are `healthy`, `unauthorized`, `reachable`, `unhealthy`, `unreachable`, and `not probed`. | model health, successful inference |
| `inference profile` | Technical noun | A blueprint selection that defines an inference provider type, provider name, endpoint, model, credential input, and route settings. | provider profile, model profile |
| `inference request` | Technical noun | One request sent through an inference route to a model. | API call when the inference purpose matters |
| `inference route reachability` | Technical noun | The `reachable` inference-health result produced when `https://inference.local/v1/models` returns HTTP `200` through `499`. It does not establish valid credentials, successful model invocation, readiness, compatibility, or support. | inference health, successful inference, validation request |
Expand Down Expand Up @@ -391,7 +391,7 @@ A result can support more than one claim only when its evidence meets each defin
| Class | Claim | Establishes | Does not establish |
|---|---|---|---|
| Operational | `inference route reachability` | The named `/v1/models` route returned HTTP `200` through `499`. | Valid credentials, successful model invocation, readiness, compatibility, or support. |
| Operational | `inference health` | The named `/v1/models` probe produced a `reachable`, `unhealthy`, or `unreachable` classification. | Valid credentials, successful model invocation, readiness, compatibility, or support. |
| Operational | `inference health` | The named `/v1/models` probe classification, plus the result of one inference request over the same route when NemoClaw sent one. | Broader API conformance, other requests or models, readiness, compatibility, or support. |
| Operational | `readiness check` | A service or resource meets named criteria to begin its intended work. | Broader reliability, compatibility, or support. |
| Operational | `validation request` | One authenticated request succeeded for the named endpoint, API family, model, and request shape. | Broader API conformance, other requests or models, reliability, or support. |
| Evidence | `verification` | Evidence confirms the stated result for the named revision and environment. | Compatibility or support unless the evidence and decision establish them. |
Expand Down
2 changes: 1 addition & 1 deletion ci/source-architecture-budget.json
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
"src/lib/actions/sandbox/connect.ts": 38,
"src/lib/actions/sandbox/destroy.ts": 29,
"src/lib/actions/sandbox/doctor.ts": 30,
"src/lib/actions/sandbox/status-snapshot.ts": 21,
"src/lib/actions/sandbox/status-snapshot.ts": 20,
"src/lib/actions/sandbox/policy-channel.ts": 29,
"src/lib/actions/sandbox/process-recovery.ts": 21,
"src/lib/actions/sandbox/rebuild-pipeline.ts": 28,
Expand Down
6 changes: 3 additions & 3 deletions docs/deployment/deploy-to-headless-server.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -235,8 +235,8 @@ $$nemoclaw headless-agent connect --probe-only
```

`$$nemoclaw headless-agent status` exits nonzero when the sandbox, gateway, local container, or authoritative inference route is not verified.
Its main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox.
HTTP `200` through `499` reports `reachable`, while HTTP `500` through `599` reports `unhealthy`.
Its main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox, then sends one inference request over the same route when that probe reports the route reachable.
The line reports `healthy` when the route served the request, `unauthorized` when the route rejected it with HTTP `401` or `403`, and `unhealthy` when the route returned HTTP `500` through `599`.
<AgentOnly variant="openclaw,hermes">

During an SSH session, `status` points to `$$nemoclaw headless-agent dashboard-url` only when the agent gateway is running and loopback dashboard access needs a port forward.
Expand All @@ -251,7 +251,7 @@ It does not restart or replace the shared host OpenShell gateway.
Readiness requires all of these results:

- The exact OpenShell sandbox row is `Ready` or `Running`.
- `$$nemoclaw headless-agent status` exits with status `0` and reports the inference route as `reachable`.
- `$$nemoclaw headless-agent status` exits with status `0` and reports the inference route as `healthy`.
- `$$nemoclaw headless-agent connect --probe-only` exits with status `0`.

## Access the Dashboard and API
Expand Down
4 changes: 2 additions & 2 deletions docs/inference/choose-local-inference-server.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,8 @@ $$nemoclaw my-assistant status
$$nemoclaw my-assistant doctor
```

Accept the result when `status` reports the inference route as `reachable` and `doctor` exits with status `0`.
Route reachability does not by itself establish successful model invocation.
Accept the result when `status` reports the inference route as `healthy` and `doctor` exits with status `0`.
The `healthy` state means the route served one inference request; it does not establish results for other requests or models.

Verify the bounded host publication:

Expand Down
2 changes: 1 addition & 1 deletion docs/inference/set-up-llama-cpp.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ $$nemoclaw my-assistant doctor
docker inspect --format '{{json .HostConfig.PortBindings}}' nemoclaw-llama-cpp
```

Accept the result when `status` reports `Managed llama.cpp: running`, the inference route is `reachable`, and `doctor` exits with status `0`.
Accept the result when `status` reports `Managed llama.cpp: running`, the inference route is `healthy`, and `doctor` exits with status `0`.
The Docker inspection output must be `{}`, confirming that the container has no Docker-published port.
The host-owned private bridge still listens on loopback.
These checks do not establish agent and model qualification.
Expand Down
3 changes: 2 additions & 1 deletion docs/inference/set-up-ollama.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,8 @@ The wizard manages the proxy lifecycle:

All proxy endpoints require the token, including `GET /api/tags`.
The host-side proxy liveness check treats any HTTP response, including `401`, as evidence that the proxy answered.
The authoritative `status` and `doctor` route probe runs inside the sandbox and reports HTTP `200` through `499` as `healthy`, HTTP `500` through `599` as `unhealthy`, and transport failures or unavailable probes as `unreachable` or `not probed`.
The `status` and `doctor` route probe runs inside the sandbox and reports HTTP `200` through `499` as reachable, HTTP `500` through `599` as `unhealthy`, and transport failures or unavailable probes as `unreachable` or `not probed`.
`status` then sends one inference request over the reachable route and reports `healthy` when the route served it, or `unauthorized` when the proxy token is no longer accepted.

If Ollama is already bound to a non-loopback address, onboarding restarts it on `127.0.0.1:11434` so the proxy becomes the only network path.

Expand Down
4 changes: 2 additions & 2 deletions docs/inference/set-up-vllm-on-two-dgx-stations.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,8 @@ $$nemoclaw my-assistant status
```

Confirm that `inference get` reports the `vllm-local` provider and the `nemotron-ultra` model.
Continue only when the `Inference` row in the status output reports `reachable`.
This result confirms that the sandbox route returned HTTP `200` through `499`; it does not prove a successful model invocation.
Continue only when the `Inference` row in the status output reports `healthy`.
This result confirms that the sandbox route served one inference request; it does not establish results for other requests or models.

## Review Reboot and Resume Behavior

Expand Down
2 changes: 1 addition & 1 deletion docs/monitoring/monitor-sandbox-activity.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ If the request fails, check these items.

1. Run `$$nemoclaw <name> status` to confirm the active provider and endpoint.
Check the main `Inference` line first.
If it shows `unhealthy`, `unreachable`, or `not probed`, inspect the labeled upstream, local-backend, and auth-proxy diagnostics to identify the failing hop.
If it shows `unauthorized`, `unhealthy`, `unreachable`, or `not probed`, inspect the labeled upstream, local-backend, and auth-proxy diagnostics to identify the failing hop.
Restart a local backend only when its own diagnostic fails.
2. Run `$$nemoclaw <name> logs --follow` to view error messages from the blueprint runner.
3. Verify that the host can reach the inference endpoint.
Expand Down
35 changes: 29 additions & 6 deletions docs/reference/commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1617,6 +1617,13 @@ Starting an already-running sandbox skips the container start and still runs the
A paused container is unpaused.
If the container was removed entirely, `start` fails and points you to `$$nemoclaw <name> rebuild`.

After the gateway and forward checks pass, `start` sends one inference request through `https://inference.local` using the sandbox's recorded provider and model.
A gateway that answers the `/v1/models` probe can still reject an inference request, so the command exits non-zero when that request fails.
It prints the probe result, including the HTTP status when the route returned one, and points you to the sandbox doctor command.
Each run sends one 16-token request through the stored provider credential, so `start` waits up to 30 seconds for it and consumes provider tokens on a hosted route.
When the sandbox records no provider or no model, `start` skips the request and exits `0`.
`doctor` still classifies an HTTP `401` or `403` route response as reachable, so correct the provider credential when `start` reports one of those statuses.

### `$$nemoclaw <name> status`

Show sandbox-scoped status, health, and inference configuration for one registered sandbox.
Expand All @@ -1643,6 +1650,19 @@ Refer to [Use Shared Gateway Routes](../inference/manage-inference/use-shared-ga
`openshellDriver` and `openshellVersion` are always strings (falling back to `"unknown"` when the registry has no value), so consumers can rely on `typeof` checks.
`agent` is always a string and reports `openclaw` when the registry records no agent for the sandbox.
`failureLayer` is `null` when no preflight failure was detected and otherwise one of `docker_unreachable`, `sandbox_container_stopped`, or `sandbox_dashboard_port_conflict`; when set, `inferenceHealth` is suppressed to `null` so automation does not see a stale remote-provider healthy status during a local outage.
`inferenceHealth.ok` reports whether the inference route served one inference request sent from inside the sandbox, not whether that route answered an HTTP request.
The route probe treats any final HTTP status from `200` through `499` as reachable, so a route with an invalidated provider credential answers HTTP `401` while the route is up.
The request uses the live gateway route's provider and model, and falls back to the recorded values when the live route is unreadable.
Each run sends one 16-token request through the stored provider credential, so `status` waits up to 30 seconds for it and consumes provider tokens on a hosted route.
When NemoClaw sends an inference request, `inferenceHealth.subprobes` reports the route probe result as the `route reachability` hop, so a failing verdict still shows that the route itself answered.
`inferenceHealth.failureLabel` reports why the inference request failed:

- `unauthorized` when the route rejected it with HTTP `401` or `403`.
- `unhealthy` when the route returned another failing HTTP status.
- `unreachable` when the request returned no HTTP status, including a probe that could not run.

A host-side upstream probe under `inferenceHealth.subprobes` stays a diagnostic and does not change `inferenceHealth.ok`, because the sandbox route is the one the agent uses.
When the route probe failed, or the sandbox records no provider or no model, NemoClaw skips the inference request and `inferenceHealth` reports the route probe result alone.
`dockerPaused` is `true` when NemoClaw detects that the Docker-driver sandbox container is paused.
In that case, text output keeps OpenShell's authoritative phase but prints a `docker unpause <container>` recovery hint instead of sending you directly to rebuild.
For terminal runtime sandboxes, the command also checks cgroup OOM kill counters.
Expand Down Expand Up @@ -1686,19 +1706,22 @@ $$nemoclaw my-assistant status --json
$$nemoclaw sandbox status my-assistant --json
```

The command probes `https://inference.local/v1/models` from inside the sandbox as the authoritative inference health check.
This check exercises the same route that agent traffic uses.
The command probes `https://inference.local/v1/models` from inside the sandbox, and when that probe reports the route reachable it sends one inference request over the same route.
That inference request is the authoritative inference health check, and both checks exercise the route that agent traffic uses.
The main `Inference` line reports one of these states:

| State | Meaning |
|-------|---------|
| `reachable` | The route returned an HTTP status from `200` through `499`. Authentication responses such as `401` and `403` confirm route reachability. |
| `unhealthy` | The route returned an HTTP status from `500` through `599`. |
| `unreachable` | The route had a transport failure, returned no final HTTP status (`000` or an interim `100` through `199`), or returned an invalid status outside `100` through `599`. |
| `healthy` | The route served the inference request. |
| `unauthorized` | The route rejected the inference request with HTTP `401` or `403`. |
| `reachable` | The route returned an HTTP status from `200` through `499` and NemoClaw did not send an inference request. |
| `unhealthy` | The route returned an HTTP status from `500` through `599`, or it rejected the inference request with another failing status. |
| `unreachable` | The route had a transport failure, returned no final HTTP status (`000` or an interim `100` through `199`), returned an invalid status outside `100` through `599`, or the inference request returned no HTTP status. |
| `not probed` | NemoClaw could not run the authoritative route probe from a reachable sandbox. |
| `not verified` | NemoClaw could not verify the sandbox or gateway state, so it skips inference probing. |

An authentication response confirms that the route is reachable, not that provider credentials are valid.
An authentication response on the route probe alone confirms that the route is reachable, not that provider credentials are valid.
`$$nemoclaw <name> doctor` sends no inference request, so it reports an HTTP `401` or `403` route response as reachable and exits `0` where `status` reports `unauthorized`.
The command can also print direct host-side provider checks such as `Inference (upstream)` and provider-specific subprobes.
For supported remote providers, this diagnostic sends an authenticated request to the configured model and accepts only a recognized Chat Completions, streaming Chat Completions, or Anthropic Messages response.
It uses a 3-second connection timeout, a 5-second total timeout, and an 8-token output limit.
Expand Down
5 changes: 3 additions & 2 deletions docs/reference/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1883,8 +1883,9 @@ Check the active provider and endpoint:
$$nemoclaw <name> status
```

The main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox, so it reflects the route the agent actually uses.
If that line shows `unhealthy`, `unreachable`, or `not probed`, inspect the labeled diagnostic lines to identify the failing hop.
The main `Inference` line probes `https://inference.local/v1/models` from inside the sandbox and then sends one inference request over the same route, so it reflects the route the agent uses.
If that line shows `unauthorized`, `unhealthy`, `unreachable`, or `not probed`, inspect the labeled diagnostic lines to identify the failing hop.
An `unauthorized` line means the route answered but rejected the request, so refresh the provider credential rather than the route.
For local Ollama and local vLLM, `Inference (ollama backend)` or the corresponding local-backend line reports the host-side service separately.
For Local Ollama, current releases can also print `Inference (auth proxy)` when a proxy token is available.
If a local backend or auth-proxy diagnostic fails, start the backend or re-run onboarding so NemoClaw can recreate the proxy token, restart the proxy, and refresh the route.
Expand Down
3 changes: 2 additions & 1 deletion docs/security/credential-rotation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,8 @@ Plan for rebuild downtime when automating messaging rotation and recreation down
## Verify the Replacement

`$$nemoclaw credentials list` confirms that a provider exists, but it does not reveal or validate the stored value.
`$$nemoclaw status` performs reachability checks without sending cloud API keys, so even an HTTP `401` or `403` can count as reachable.
`$$nemoclaw status` performs host-side reachability checks without sending cloud API keys, so even an HTTP `401` or `403` can count as reachable there.
`$$nemoclaw <name> status` also sends one inference request through the stored provider credential and reports `unauthorized` when the route rejects it with HTTP `401` or `403`.
`$$nemoclaw inference get` reports the active route and does not authenticate a model request.

Complete a real request through the rotated integration before declaring the rotation successful.
Expand Down
Loading