fix(sandbox): terminate sandbox when proxy accept loop exits unexpectedly

[politerealism]

Summary

• Add a oneshot notification channel to ProxyHandle that fires when the proxy task exits for any reason (panic, abort, or loop break)
• The sandbox main loop now races this signal alongside the entrypoint process and shutdown signals, terminating with a clear OCSF AppLifecycle error at High severity if the proxy dies
• Prevents the sandbox from continuing to report Ready with a dead proxy

Related Issue

Refs #2337

Note: This is step 2 of the fix for #2337, building on PR #2369 (step 1: accept loop retry with backoff). Step 1 prevents the proxy from dying on transient errors; step 2 catches any unexpected death as defense-in-depth.

Changes

crates/openshell-supervisor-network/src/proxy.rs

• Added exited_rx oneshot receiver field to ProxyHandle
• Created exited_tx sender inside the spawned proxy task as _proxy_exit_guard — dropped on task exit for any reason
• Added take_exit_receiver() method to extract the receiver for monitoring

crates/openshell-sandbox/src/lib.rs

• Extract proxy exit receiver before the main wait section
• Wrap it in a boxed future (std::future::pending() when no proxy exists)
• Added proxy-death arm to all 5 tokio::select! wait paths:
  • Process-enabled + sidecar control
  • Process-enabled, no sidecar
  • Network-only + sidecar control (Linux)
  • Network-only, no sidecar (Linux)
  • Non-Linux network-only

Testing

• cargo check -p openshell-supervisor-network — compiles clean, no warnings
• cargo check -p openshell-sandbox — compiles clean
• cargo test -p openshell-supervisor-network -p openshell-sandbox — all tests pass

Checklist

• Conventional commit format
• Signed-off for DCO compliance
• No secrets or credentials included
• Scoped to the issue at hand
• Follows existing sidecar control channel pattern (lib.rs:692-710)

Note: This PR depends on PR #2369 (step 1: accept loop retry with backoff) and should be merged after it.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[politerealism]

Follow-up filed: #2372 — the SSH server task is spawned fire-and-forget (run.rs:238), so if it exits unexpectedly the sandbox has no way to detect it. The same exit-notification pattern from this PR (oneshot drop-guard + tokio::select! arm) should be applied to the SSH server task to enforce the "Ready implies exec relay operational" invariant.

[politerealism]

Rebase note: This PR was branched off #2369. Once #2369 merges, this branch needs a rebase to pick up the final version of the proxy backoff code (libc constants, extracted helpers, fixed 5s cap). No code changes required — just a rebase.

[johntmyers]

gator-agent

PR Review Status

Validation: This is a concentrated sandbox/network-supervisor correctness fix for the reproducible stale-Ready failure in #2337, and no duplicate implementation was found.
Head SHA: 4e915eddb6d4c5111e95d44fff5c1b3e539f20b6

Thanks @politerealism, I checked your rebase note against PR #2369. Its current head differs from the retry commit stacked here; rebasing after #2369 lands should pick up the portable errno helper and corrected 5-second cap covered by the first two inline findings.

Review findings:

• Four actionable warnings are attached inline.
• The fail-closed proxy-exit signal is wired into every sandbox wait path and does not weaken the sandbox security boundary.

Docs: Fern docs are not needed because this restores the documented readiness invariant without changing commands, configuration, APIs, or an intentional user workflow.

Next state: gator:in-review

Please rebase after #2369 lands and address the exit-receiver contract and production-path test coverage. E2E will be required after review feedback is resolved.

[johntmyers]

gator-agent

Blocked

Head SHA: 6a1fe36ae0357d7bc0d22b91323ff8ae296b54a2

Gator is blocked because GitHub reports this PR as having merge conflicts with main (mergeStateStatus: DIRTY).

Next action: @politerealism, please rebase the branch onto the current main, resolve the conflicts, and push the updated head. Gator will review the new head and decide the required E2E label and /ok to test action after the conflict is resolved.

[politerealism]

Waiting to test and recheck this one until 2369 is approved and merges so I can rebase then go through checks.

[johntmyers]

gator-agent

PR Review Status

Validation: This remains a project-valid, concentrated fix for the stale-Ready proxy failure in #2337.
Head SHA: 00dc53e89455857e75b8dcceb2f91cd788ad228b
Base SHA: 0a3ec7a1126aad908ffb742435db25bea1c19c25
Merge base SHA: 0a3ec7a1126aad908ffb742435db25bea1c19c25
Patch ID: 287b2741b08a8605dfb9f7325b7d7e284227be8a
Gator payload: 2
Review mode: follow_up
Previous reviewed SHA: 4e915eddb6d4c5111e95d44fff5c1b3e539f20b6

Thanks @politerealism, I checked your rebase note now that #2369 has merged. The merge conflict is resolved, and the 5-second backoff cap finding (gator-inline-3618433659) is fixed. The latest author delta also replaces parts of #2369's finalized classifier, which introduces the new inline blocker below.

Blocking findings:

• GATOR-00dc53e8-01: documented transient accept errors now consume the unknown-error budget and terminate the proxy on the tenth occurrence.

Carried findings:

• gator-inline-3618433657: still open; numeric errno literals remain instead of target-aware libc constants.
• gator-inline-3618433663: still open; a second take_exit_receiver() call still manufactures an immediate false exit signal.
• gator-inline-3618433665: still open; tests still bypass the production proxy task and sandbox lifecycle selection.

Non-blocking suggestions: None.

Docs: Fern docs are not needed because this restores an existing readiness invariant without changing commands, configuration, APIs, or intentional user workflows.

Next state: gator:in-review

Please preserve #2369's portable transient-error classifier while layering the proxy-exit notification on its terminal outcome, and address the carried obligations above. E2E remains required after review feedback is resolved.

[johntmyers]

gator-agent

PR Review Status

Validation: This remains a project-valid, concentrated fix for the stale-Ready proxy failure in #2337.
Head SHA: fe8298791181db1dd5f606e4914de4f9bd406e85
Base SHA: 0a3ec7a1126aad908ffb742435db25bea1c19c25
Merge base SHA: 0a3ec7a1126aad908ffb742435db25bea1c19c25
Patch ID: 8ebd81958c1bf8492797c1af6c62754d1ebd732d
Gator payload: 2
Review mode: follow_up
Previous reviewed SHA: 00dc53e89455857e75b8dcceb2f91cd788ad228b

Thanks @politerealism, I checked your latest commit restoring the complete transient errno classifier. GATOR-00dc53e8-01 is resolved: the supported transient family now bypasses the unknown-error budget, resource-pressure errors retain capped backoff, and the added coverage exercises repeated transient classification. The Option-based receiver change also resolves the double-take contract issue.

Blocking findings:

• No blocking findings remain.

Carried findings:

• None.

Docs: Fern docs are not needed because this restores an existing readiness invariant without changing commands, configuration, APIs, or intentional user workflows.

Next state: gator:watch-pipeline

This sandbox lifecycle and network-proxy change requires test:e2e; gator will enable that gate and monitor the required checks.

[github-actions]

Label test:e2e applied, but pull-request/2370 does not exist yet. A maintainer needs to comment /ok to test fe8298791181db1dd5f606e4914de4f9bd406e85 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

[johntmyers]

/ok to test fe82987

[github-actions]

Label test:e2e applied for fe82987. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[github-actions]

Label test:e2e applied for 6262068. Open Branch E2E Checks, find the run for commit 6262068, and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

/ok to test 6262068