feat(helm): cert-manager external issuer + OpenShift passthrough Route

[jhjaggars]

Summary

Lets cert-manager issue the gateway's server TLS certificate from a real Issuer/ClusterIssuer (for example an ACME issuer) instead of only the chart's built-in self-signed CA, and adds an OpenShift Route template with TLS passthrough so the gateway can be exposed externally with a publicly-trusted certificate while it keeps terminating its own TLS/mTLS.

Closes #2466

Changes

Dual-certificate architecture (SNI-based)

The gateway server certificate is split into two: an internal cert issued by the chart's own CA (for supervisor connections via cluster-local SANs) and an external cert issued by an operator-configured Issuer such as ACME/Let's Encrypt (for CLI and Route access via public SANs). The gateway uses SNI-based certificate selection (DualCertResolver): connections whose SNI hostname matches external_server_names receive the external cert; all others receive the internal cert. Wildcard patterns (e.g. *.example.com) are supported per RFC 6125.

Security: supervisor trust pinned to chart CA

The sandbox supervisor's gRPC client previously trusted WebPKI roots (tls-webpki-roots). This PR removes that trust and keeps supervisors pinned exclusively to the chart CA configured via OPENSHELL_TLS_CA. The UnknownCA problem (supervisor connecting to a gateway serving an ACME cert) is solved server-side: supervisors connect via internal service names, the SNI resolver presents the internal (chart CA) cert, and the TLS handshake succeeds without needing public root trust.

This is a deliberate security improvement — a user-supplied container image that ships its own root store can no longer influence the supervisor's trust decisions, closing a MITM vector where a publicly-trusted cert for an attacker-controlled name could intercept the sandbox JWT.

All four compute drivers (Docker, Podman, VM, Kubernetes) now strip OPENSHELL_GATEWAY_TLS_SERVER_NAME from the sandbox environment to prevent user-supplied overrides of the TLS server name the supervisor verifies.

Helm templates

• certManager.serverIssuerRef (new value): creates a second server Certificate from the operator's Issuer/ClusterIssuer with only externally-resolvable SANs. The internal server cert always uses the chart's own CA issuer.
• certManager.serverDnsNames entries are validated: internal-only names are rejected when an external issuer is configured.
• New templates/route.yaml: OpenShift Route with tls.termination: passthrough, gated by openshiftRoute.enabled. Includes fail guards for TLS disabled and Route host not covered by serverDnsNames.
• Empty serverDnsNames with an external issuer is rejected at install time.

Docs

• Production section in docs/kubernetes/openshift.mdx with full helm install command.
• New section in docs/kubernetes/managing-certificates.mdx documenting the dual-cert architecture.
• docs/reference/gateway-config.mdx updated with external_cert_path, external_key_path, external_server_names TOML fields.
• .agents/skills/debug-openshell-cluster/SKILL.md updated with dual-cert troubleshooting.

Testing

• mise run pre-commit passes
• Unit tests for DualCertResolver SNI selection (exact + wildcard), build_cert_resolver validation (partial config, empty names), and full TCP+TLS integration test
• Parity tests for GATEWAY_TLS_SERVER_NAME stripping across Docker, Podman, and VM drivers
• Helm lint passes all CI value variants
• Helm unittest coverage for Route template and cert-manager PKI templates

Validated end-to-end against a live OpenShift (ROSA) cluster: passthrough Route serving a real Let's Encrypt certificate issued via a Route53 DNS-01 ClusterIssuer, OIDC-authenticated CLI access via Keycloak, sandbox created and exec'd successfully with the supervisor connecting back to the gateway over the internal (chart CA) TLS path.

Checklist

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated — validated manually against a live OpenShift cluster
• Follows Conventional Commits
• Commits are signed off (DCO)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[mrunalp]

/ok to test cf02477

[mrunalp]

Blocking

1. Native roots are controlled by the sandbox image

crates/openshell-core/src/grpc_client.rs:176-180

The supervisor now calls both with_native_roots() and with_webpki_roots(). For Docker and Podman, the supervisor binary runs inside the user-selected sandbox image. The image's CA bundle is therefore not an operator-controlled trust source. rustls-native-certs also honors SSL_CERT_FILE and SSL_CERT_DIR, and both drivers pass user-provided template/spec environment variables into the supervisor.

If an attacker can also influence DNS or routing for the gateway connection, they can install their own CA, present a certificate for the gateway hostname, and receive the sandbox JWT that the supervisor sends on its RPCs. That can expose sandbox-scoped data and let the fake gateway return attacker-selected policy during supervisor startup.

There is also a compatibility regression: tonic 0.14 returns NativeCertsNotFound when the native store is empty before it considers the configured CA or WebPKI roots. A minimal BYOC image can therefore fail to start even though OPENSHELL_TLS_CA is valid.

Please remove with_native_roots() here and use the explicitly configured CA plus the compiled-in WebPKI roots. If enterprise/private server issuers need support, add an operator-mounted server CA bundle rather than trusting the sandbox image's native store. It would also be prudent to reserve or scrub SSL_CERT_FILE, SSL_CERT_DIR, and OPENSHELL_GATEWAY_TLS_SERVER_NAME from supervisor input.

2. serverIssuerRef renders a broken default client-CA configuration

deploy/helm/openshell/values.yaml:351-369
deploy/helm/openshell/templates/_gateway-workload.tpl:147-157

Setting the advertised serverIssuerRef and an external SAN while leaving the other defaults unchanged keeps clientCaFromServerTlsSecret=true. The gateway then tries to mount ca.crt from the public server certificate Secret.

ACME Secrets commonly do not contain ca.crt, leaving the gateway pod in MountVolume.SetUp failed. If an issuer does populate it, it is still not the CA that signed the chart-issued supervisor client certificate.

The chart should fail rendering when serverIssuerRef.name is combined with clientCaFromServerTlsSecret=true, or derive the correct client CA source automatically. Please add a negative Helm test for the default interaction.

3. clientIssuerRef is not independently usable as documented

deploy/helm/openshell/templates/cert-manager-pki.yaml:138-147
docs/kubernetes/managing-certificates.mdx:109-111

Changing only clientIssuerRef leaves the gateway trusting the built-in server CA while the supervisor receives the custom client issuer's CA. The gateway does not trust the new client certificate, and the supervisor no longer trusts the default server certificate. Every supervisor TLS connection consequently fails.

Either remove this option for now or model the server-verification and client-verification trust bundles separately and validate the complete configuration. kind should also be required or default to cert-manager's standard Issuer; silently defaulting to ClusterIssuer is surprising for an interface that advertises both.

4. The documented OIDC production command cannot connect

docs/kubernetes/openshift.mdx:102-135
crates/openshell-server/src/cli.rs:256-301

The Helm command never sets server.oidc.issuer, but the next step registers the CLI using --oidc-issuer. That flag configures only the CLI. Without server-side OIDC, the gateway requires a client certificate and rejects the OIDC-only CLI during the TLS handshake.

Please add server.oidc.issuer (and the expected audience) to the Helm command, or make completing the Access Control configuration an explicit prerequisite before registration.

5. The passthrough Route accepts a plaintext backend

deploy/helm/openshell/templates/route.yaml:4-26

openshiftRoute.enabled=true can currently be combined with server.disableTls=true, including by adding the Route settings to the existing OpenShift quickstart. Helm succeeds, but HAProxy forwards TLS handshake bytes to a plaintext gateway, resulting in resets or hangs.

The template should fail when passthrough routing is enabled while gateway TLS is disabled, with a test covering that combination.

[mrunalp]

Follow-up review: all five blocking items addressed

Re-reviewed at 0276980. All five blocking items from my earlier review are addressed. I verified each against the code rather than the commit message and ran the chart and crate checks.

#	Item	Fix	Verification
1	Native roots controlled by the sandbox image	.with_native_roots() removed from grpc_client.rs; tls-native-roots dropped from openshell-core/Cargo.toml; the comment explains why it should not come back	cargo check -p openshell-core --all-targets clean
2	serverIssuerRef + clientCaFromServerTlsSecret=true	fail guard in cert-manager-pki.yaml:46 plus a negative unittest	helm template with that combination errors with the expected message
3	clientIssuerRef not independently usable	Value removed from values.yaml, the template, the chart README, and the docs; serverIssuerRef.kind now defaults to Issuer	no remaining value or template references
4	Documented OIDC command cannot connect	server.oidc.issuer / server.oidc.audience added to the production Helm command, Access Control called out as a prerequisite, override table row added	both keys exist in values.yaml and render into [openshell.gateway.oidc]
5	Passthrough Route with a plaintext backend	fail guard in route.yaml:5 plus a negative unittest	helm template with openshiftRoute.enabled=true,server.disableTls=true errors as intended

mise run helm:test passes 74/74 across 7 suites, the CI values overlay renders, and the chart still creates the internal openshell-ca-tls CA that the docs point clientCaSecretName at.

Remaining

1. The env-scrubbing half of item 1 is still open

crates/openshell-core/src/grpc_client.rs:188
crates/openshell-driver-docker/src/lib.rs:2147-2152
crates/openshell-driver-kubernetes/src/driver.rs:1982

OPENSHELL_GATEWAY_TLS_SERVER_NAME still overrides the hostname the supervisor verifies. The Kubernetes driver strips it from container env, but the Docker driver merges template.environment and spec.environment straight into the supervisor's environment and never sets or removes it. Podman is the same.

Now that the WebPKI roots are trusted, a sandbox user who can also redirect the gateway hostname inside the container can satisfy verification with a publicly valid certificate for a name they control and receive the sandbox JWT. Before this change, only the configured CA was trusted, so redirection alone was not enough.

Please strip the variable in the Docker and Podman drivers to match what the Kubernetes driver already does. SSL_CERT_FILE and SSL_CERT_DIR no longer matter for this path now that rustls-native-certs is gone from it.

2. Dangling clientIssuerRef reference in the docs

docs/kubernetes/managing-certificates.mdx:102

The page still reads "unless you've also overridden clientIssuerRef". That value no longer exists.

3. A half-configured client CA still renders (optional)

deploy/helm/openshell/values.yaml:248

Setting clientCaFromServerTlsSecret=false without server.tls.clientCaSecretName falls back to the default openshell-server-client-ca, which nothing in the chart creates. The pod then lands in MountVolume.SetUp failed — the same symptom item 2 was about. The new guard's message does tell operators to set both, so this is documented but unguarded. Worth a follow-up rather than a blocker.

CI

/ok to test was granted for cf02477, not the current head — Branch Checks and Helm Lint are still waiting on the mirror, so the fix commit has not been through CI yet.

[mrunalp]

/ok to test 0276980

[mrunalp]

Follow-up review at 876959a1

All three remaining items from my previous comment are addressed and verified. One stray file needs to come out of the commit before this merges.

Verified fixed

1. OPENSHELL_GATEWAY_TLS_SERVER_NAME stripping. environment.remove(...) in crates/openshell-driver-docker/src/lib.rs:2256 and crates/openshell-driver-podman/src/container.rs:487, both placed after the template.environment / spec.environment merge, so user-supplied env cannot win. Both are single funnels — docker's build_environment delegates to build_environment_for_oci_user, and podman's build_env has one caller — so there is no bypass path. cargo test -p openshell-driver-docker -p openshell-driver-podman passes (107 + 149).

2. Dangling doc reference. The clientIssuerRef clause is gone from docs/kubernetes/managing-certificates.mdx:98-101, and no references remain anywhere in the tree.

3. Half-configured client CA. New fail guard at deploy/helm/openshell/templates/cert-manager-pki.yaml:49 plus a negative unittest. I exercised the matrix: the default install renders, clientCaFromServerTlsSecret=false with the default secret name fails with the new message, clientCaSecretName=openshell-ca-tls renders, and clientCaSecretName="" (mTLS off) renders. The guard is correctly scoped inside if .Values.certManager.enabled. mise run helm:test passes 77/77 across 7 suites.

One behavior note on that guard: it hard-fails a combination that previously rendered (certManager.enabled + clientCaFromServerTlsSecret=false + the default secret name), which is what forced the statefulset_client_ca_test.yaml change. The old expectation was wrong — with cert-manager enabled the certgen hook runs in JWT-only mode and never creates openshell-server-client-ca — so the guard is correct. But any existing release sitting on that combination will now fail its next helm upgrade, so this is worth a release note.

Please fix: a target symlink was committed

876959a1 adds a symlink at the repository root:

120000 blob e58e2c52    target -> /mnt/build-artifacts/cargo-target

.gitignore has /target/ and target/; the trailing slash makes both directory-only, so neither matches a symlink named target — that is how this slipped through. Checking the branch out materializes the symlink, and /mnt/build-artifacts does not exist outside your machine, so every clone gets a dangling redirect for all cargo builds. Please git rm target, and consider adding a bare target line to .gitignore so the pattern also catches the non-directory case.

Minor

There is no regression test for the env stripping in either driver, though crates/openshell-driver-docker/src/tests.rs already has the pattern for exactly this kind of assertion (build_environment_protects_oci_identity_metadata:574, build_environment_uses_token_file_without_raw_token_env:1422). For a security-relevant strip, a small test in both drivers would keep it from silently regressing.

CI

The head is now 876959a1; the /ok to test above was for 0276980. Branch Checks and Helm Lint still read "Waiting for /ok to test mirror".

[mrunalp]

Follow-up review at aac65a99

The branch was force-rewritten since my last pass (all commit hashes changed). The target symlink is gone, but the same rewrite also dropped one of the three fixes I confirmed last time.

Present and verified

• Env stripping. environment.remove(GATEWAY_TLS_SERVER_NAME) in crates/openshell-driver-docker/src/lib.rs:2257 and crates/openshell-driver-podman/src/container.rs:490, both after the user-env merge. cargo test -p openshell-driver-docker -p openshell-driver-podman passes (107 + 149).
• Dangling doc reference. clientIssuerRef no longer appears anywhere in the tree.
• target symlink. Removed — checking the branch out no longer materializes it.

Everything from the earlier rounds also survived: with_native_roots() is still absent (only the do-not-re-add comment remains), openshell-core's tonic features are ["channel", "tls-webpki-roots"], the route.yaml TLS guard fires, the serverIssuerRef + clientCaFromServerTlsSecret=true guard fires, and the openshift.mdx OIDC additions are intact. mise run helm:test passes 76/76.

Regressed: the default clientCaSecretName guard is gone

The fail guard that was at deploy/helm/openshell/templates/cert-manager-pki.yaml:49 and its negative unittest ("fails when clientCaFromServerTlsSecret is false but clientCaSecretName is the default") are no longer in the branch, and tests/statefulset_client_ca_test.yaml is back to unchanged from main. Only the two original guards remain in that template. The helm test count went 74 → 77 → 76, which is that one test disappearing.

The footgun is back:

helm template ... --set certManager.enabled=true --set certManager.clientCaFromServerTlsSecret=false
  ->  secretName: openshell-server-client-ca      # renders fine; nothing creates this Secret

That leaves the gateway pod in the MountVolume.SetUp failed state the original item 2 was about.

This may be deliberate — my last comment noted the guard would break helm upgrade for any release sitting on that combination, and dropping it is a defensible response. If so, nothing replaced it: the only thing steering operators now is the prose in managing-certificates.mdx plus the other guard's error message. Could you confirm whether this was an intentional revert or collateral damage from the force-push? It came out in the same rewrite that removed the stray symlink.

Still minor

No regression test for the env stripping in either driver. crates/openshell-driver-docker/src/tests.rs already has the pattern for it (build_environment_protects_oci_identity_metadata, build_environment_uses_token_file_without_raw_token_env).

CI

The head is now aac65a99. Branch Checks and Helm Lint still read "Waiting for /ok to test mirror".

[mrunalp]

/ok to test 85ec214

[github-actions]

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}