[Ubuntu 24.04][Inference] inference set endpoint update leaves the superseded HTTPS Pin Runtime route un-revoked with in-memory credentials

[wangericnv]

Description

When nemoclaw <sandbox> inference set --provider compatible-endpoint updates the endpoint URL of an existing compatible-endpoint that uses a DNS-backed HTTPS endpoint (routed through the HTTPS Pin Runtime adapter), the new route is committed but the superseded (old) HTTPS Pin route cannot be revoked, and its raw upstream credentials are left in the adapter's memory. NemoClaw tells the user to uninstall NemoClaw to purge them.

Each endpoint update on a DNS-backed HTTPS compatible-endpoint therefore leaks a stale Pin route plus its in-memory upstream credentials until NemoClaw is uninstalled. The endpoint-update-with-rollback path (PR #7746) does not cleanly revoke the prior route.

Platform scope: DGX Station GB300 (aarch64, Ubuntu 24.04), NemoClaw v0.0.97 + OpenShell 0.0.85; not believed platform-specific.
Regression: No — new functionality (compatible-endpoint update with rollback, PR #7746 / v0.0.97).

Steps to Reproduce

1. Onboard an OpenClaw sandbox <sandbox> (e.g. on nvidia-prod).
2. export COMPATIBLE_API_KEY=<a valid key>
3. Create a compatible-endpoint on a DNS-backed HTTPS endpoint:

   nemoclaw <sandbox> inference set --provider compatible-endpoint --endpoint-url https://<dns-host-a>/v1 \
     --model <model> --credential-env COMPATIBLE_API_KEY --inference-api openai-completions

4. Update the endpoint URL to a different DNS-backed HTTPS endpoint:

   nemoclaw <sandbox> inference set --provider compatible-endpoint --endpoint-url https://<dns-host-b>/v1 \
     --model <model> --credential-env COMPATIBLE_API_KEY --inference-api openai-completions

5. Observe the warning printed after the route is committed.

Expected Result

The update commits the new route AND revokes the superseded HTTPS Pin route, purging its in-memory upstream credentials. No residual route or credentials remain.

Actual Result

Warning: the new inference route is committed, but superseded HTTPS Pin Runtime route '<hash>' could not be
revoked: Cannot authenticate the live HTTPS Pin Runtime adapter for revocation.. The raw upstream endpoint
was not restored; uninstall NemoClaw to stop the adapter and purge its in-memory credentials if this persists.
Inference route synced for '<sandbox>': inference/<model>

The stale route + in-memory upstream credentials persist until NemoClaw is uninstalled.

[yanyunl1991]

Thanks for the report — reproduced and fixed on my side. Quick summary:

Root cause
Revoking a superseded route has to authenticate the running adapter, and the control-plane challenge proof binds the route-source policy that adapter was started with. Of the three callers that reach probeAdapterControlHealth, the two on the registration path receive that policy explicitly, which is why registering a route always worked. The revoke path supplied neither, so it proved against the 127.0.0.1/32 default while every sandbox-facing adapter is started with the discovered OpenShell bridge subnets. The digests differ, the proof does not match, and revocation throws. The caller warns and continues, so the new route commits while the superseded one stays registered in adapter memory with its upstream credential resident.

Fix
Record the route-source policy the adapter is actually running under in host recovery state, and read it back at revocation: written alongside the pid at spawn, written on every successful registration (which also covers adapter reuse, where no spawn happens), and carried forward when a sibling route is dropped. When no policy is recorded, revocation now fails closed with a distinct message rather than silently proving against the wrong default.

Re-deriving the subnets at revocation time is a smaller diff, but it re-derives the value it then authenticates with, so a drifted bridge configuration would still leave the route live. Durable provenance is what actually closes the class.

PR
#7889

Will land after CI is green and maintainer review. To try it locally:
git fetch origin fix/https-pin-revoke-source-policy-7878 && git checkout fix/https-pin-revoke-source-policy-7878, then re-run inference set --endpoint-url twice against different DNS-backed HTTPS endpoints.

If the symptom doesn't match what you saw, or you can still reproduce after this lands, reply here and I'll take another look.