fix(cli): report inference health from a served request - #8731
fix(cli): report inference health from a served request#8731laitingsheng wants to merge 12 commits into
Conversation
The in-sandbox route probe reads any final HTTP 200-499 as reachable, so a rejected provider credential answered 401 while status reported healthy inference and exited 0. Status and start now send one inference request over the same route and report its result, and start exits nonzero when the request fails. Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughSandbox inference health now combines route reachability with an authenticated inference request. Startup readiness, status output, failure labels, diagnostics, tests, and documentation now distinguish healthy, unauthorized, unreachable, and unhealthy results. ChangesSandbox inference health
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant statusSnapshot
participant Gateway
participant SandboxInference
participant RouteHealth
User->>statusSnapshot: request sandbox status
statusSnapshot->>Gateway: probe inference route
Gateway-->>statusSnapshot: route reachability
statusSnapshot->>SandboxInference: send authenticated inference request
SandboxInference-->>statusSnapshot: response and HTTP status
statusSnapshot->>RouteHealth: combine route and invocation results
RouteHealth-->>User: request-aware inference health
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
laitingsheng
added
area: cli
PR Review Advisor — No blocking findings reportedAdvisor assessment: No blocking advisor findings reported Model lanes
4 terminology differences from the second opinionAdvisory only. These are normalized differences from the primary terminology receipt.
2 additional E2E selections from the second opinionAdvisory only. The primary lane did not select these E2E jobs or targets.
Second-opinion terminology and E2E selections are advisory. Live E2E does not run automatically for pull requests. 3 semantic terminology decisionsTerminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.
E2E guidanceAdvisory only. A maintainer can dispatch the default E2E suite for the commit under review. Recommended E2E: Manual-only E2E: 1 optional E2E recommendation
This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge. |
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit 2bf9536 in the TypeScript / code-coverage/cliThe overall coverage in commit 2bf9536 in the Show a code coverage summary of the most impacted files.
Updated |
|
🌿 Preview your docs: https://nvidia-preview-pr-8731.docs.buildwithfern.com/nemoclaw |
|
Maintainer decision required before readiness work continues This PR changes the product contract in ways that need explicit maintainer direction:
The sensitive-path and documentation receipts must also be refreshed for the current branch revision after the direction is settled. I am deferring implementation, approval, and merge because resolving the contract requires product and architecture judgment. |
prekshivyas
left a comment
prekshivyas
left a comment
There was a problem hiding this comment.
Blocking correctness finding on latest PR commit 4a5cbca0d.
status-snapshot.ts now takes the provider/model pair from the usable live gateway route, but still passes sb?.preferredInferenceApi from the recorded sandbox into runSandboxInferenceInvocationProbe. When live route drift changes the API family, status can send the wrong request shape/endpoint and report an otherwise usable route as unhealthy.
Derive provider, model, and API family from one route authority. If the live route is selected, use its compatible API family; otherwise use the complete recorded route contract. Add a status regression where the recorded route is Responses-based and the live route requires Chat Completions, then assert the served request uses the live route's endpoint and reports healthy.
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Addressed by a4b22d8: live and recorded inference routes now keep provider/model/API coherent; 13 focused status-inference tests and the exact-head pre-push typecheck passed.
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
|
Security review completed for the current #8731 revision. Scope:
Result: no actionable security finding. The request uses the fixed Validation on commit
Fresh repository CI is running. The prior review request is resolved by the contributor's two route-coherence commits. The follow-up harness commit repairs the module rename that caused the earlier CI shard failures. |
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
|
The failed CLI recovery check was caused by its OpenShell fixture not answering the new inference invocation. Revision 282a38b updates that fixture without changing production behavior. Five source-test files passed 84 tests, and three CLI integration files passed 29 tests. The Station pair-preparation failure was unrelated to the inference changes and passed twice in isolation, so the new revision provides a safe fresh CI run. The signed commit is Verified. |
4 participants
Summary
nemoclaw <name> statusreportedinferenceHealth.ok: trueand exited0while the in-sandbox inference route answered HTTP 401, andstartexited0on the same route. The route probe reads any final HTTP 200-499 as reachable, so a rejected provider credential passed it. Status and start now send one inference request over the same route and report its result, so a rejected credential reportsunauthorizedand exits nonzero.Related Issue
Fixes #8705
Changes
src/lib/actions/sandbox/rebuild-inference-preflight.tstoinference-invocation-probe.tsand its exports tobuildSandboxInferenceInvocationCommandandprobeSandboxInferenceInvocation. The module already sent one minimal completion throughinference.localon the sandbox's stored credential, but only Deep Agents Code rebuild preflight consumed it. Status and start are the second and third consumers, so the name now describes the behaviour rather than the first caller. The failure result addshttpStatus, and the timeout is a parameter: rebuild keeps 100 seconds, status and start use 30.src/lib/actions/sandbox/inference-route-health.ts. When the route probe reports the route reachable,statussends one inference request and its result drivesinferenceHealth.ok; HTTP 401 or 403 setsfailureLabel: "unauthorized". The route probe result becomes aroute reachabilitysubprobe so a failing verdict still shows that the route itself answered. A host-side upstream probe stays a diagnostic and does not changeok, keeping the sandbox route authoritative.startSandboxreadiness on the same request insideverifyStarted, after the existing gateway and forward checks, and exit1when it fails. A sandbox with no recorded provider or model has nothing to request, so start skips it and exits0.docs/reference/commands.mdx, including theInferencestate table, and the pages whose acceptance criteria or failing-state lists this change invalidates:docs/inference/choose-local-inference-server.mdx,docs/inference/set-up-llama-cpp.mdx,docs/inference/set-up-ollama.mdx,docs/inference/set-up-vllm-on-two-dgx-stations.mdx,docs/deployment/deploy-to-headless-server.mdx,docs/monitoring/monitor-sandbox-activity.mdx,docs/reference/troubleshooting.mdx, anddocs/security/credential-rotation.mdx.inference healthentries in.agents/skills/_shared/controlled-words.md, which defined the term as the/v1/modelsclassification alone.Contract changes maintainers should weigh
nemoclaw statusreports inference healthy on endpoint reachability, not model invocability — a green status can mask an unreachable model #6846. Two tests asserted the reported behaviour and are rewritten: the unit test expecting{ ok: true, okLabel: "reachable" }beside a failedunauthorizedsubprobe, and the CLI test named "treats an inference.local HTTP %s as healthy", which asserted exit0for 401 and 403.statusandstartnow send a 16-token request through the stored provider credential on every run, so both wait up to 30 seconds for it and consume provider tokens on a hosted route.doctoris unchanged and still classifies an HTTP 401 or 403 route response as reachable, so it exits0wherestatusnow reportsunauthorized. The difference is documented; makingdoctorconsistent needs its own change.phaseto stop reportingReadywhen inference is broken.phaseis OpenShell's own container lifecycle value, parsed insrc/lib/state/gateway.tsand shared withTERMINAL_SANDBOX_PHASES, so this change leaves it alone and reports the condition throughinferenceHealthand the exit codes instead.statusinference health can only attest a fresh exec's environment, never the serving process's — surface process-level (self-reported) health #7003 remains open. The request runs in a freshsandbox exec, so it attests the route and credential, not the long-running gateway process's own environment.Type of Change
Quality Gates
Documentation Writer Review
docs-updateddocs/reference/commands.mdx,docs/inference/choose-local-inference-server.mdx,docs/inference/set-up-llama-cpp.mdx,docs/inference/set-up-ollama.mdx,docs/inference/set-up-vllm-on-two-dgx-stations.mdx,docs/deployment/deploy-to-headless-server.mdx,docs/monitoring/monitor-sandbox-activity.mdx,docs/reference/troubleshooting.mdx,docs/security/credential-rotation.mdx,.agents/skills/_shared/controlled-words.mdDGX Station Hardware Evidence
Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run validate:prpassed after refreshingorigin/mainwhen hooks were skipped or unavailablenpx vitest run src/lib/actions/sandbox/status-snapshot-inference-health.test.ts src/lib/actions/sandbox/status-inference.test.ts src/lib/actions/sandbox/start.test.ts src/lib/actions/sandbox/inference-invocation-probe.test.ts src/lib/actions/sandbox/status-snapshot-recovery.test.ts— 5 files, 71 tests passed;npx vitest run test/cli— 35 files, 283 tests passed; the new CLI cases fail against a pre-fix build withexpected +0 to be 1; follow-up rebuild harness validation passed 2 files and 9 tests after the probe rename; the runtime-recovery compatibility test and its related CLI suites passed 3 files and 29 tests after the fixture learned the agent-request probenpm testfor broad runtime/test-harness changes;npm run checkfor repo-wide validation/coverage changes — command/result:npm run docsbuilds without warnings (doc changes only)Signed-off-by: Tinson Lai tinsonl@nvidia.com
Summary by CodeRabbit
New Features
Documentation