feat(gatewayapi): calico-system policy for namespaced data-plane proxies

[electricjesus]

Description

Bug fix.

Since #4690 we run a single envoy-gateway controller in calico-system with deploy.type=GatewayNamespace. The data-plane proxies now run in each Gateway's own namespace, not in calico-system.

We render one gateway policy in the calico-system tier: calico-system.envoy-gateway. It lives in calico-system and selects the controller and certgen pods. The proxies run elsewhere and match no policy in that tier. When a Gateway namespace runs a default-deny tier, the proxy there has nothing to let its traffic through. This is the same kind of miss as the conformance MetalLB pool that stayed pinned to tigera-gateway (projectcalico/calico#13095).

This adds a GlobalNetworkPolicy, calico-system.envoy-gateway-proxy, that selects the proxy pods by k8s-app == 'calico-gateway-api-proxy'. The operator stamps that Calico-owned label on the proxy pods through the EnvoyProxy pod spec, so the policy keys off a label we control rather than Envoy Gateway's own gateway.envoyproxy.io/owning-gateway-name, which we do not own and which could change upstream. A GNP covers any Gateway namespace, including ones created later, with no re-render. It allows:

• Egress to DNS, plus xDS (18000) and Wasm fetch (18002) to the controller in calico-system. The proxy dials the controller (see envoyproxy/gateway internal/infrastructure/kubernetes/proxy/resource.go and internal/xds/bootstrap). 18001 is the ratelimit path, not a proxy path, so it is left out.
• Ingress on all inbound TCP. Listener ports are user-defined, so the proxy has to accept any port to work out of the box. This also covers the 19001 metrics scrape.

The same Calico-owned labelling now covers all three envoy-gateway components (controller, certgen, proxy) from one documented place in the gateway render. That k8s-app label is not set through the standard labeller in pkg/controller/utils/component.go on purpose: the labeller keys off the object name, and the proxy pods are created by the envoy-gateway controller at runtime, so the operator never renders them to label.

Standard labels on Job pods

Rene's review asked whether the envoy components get the standard labels from that same component.go labeller. They half do, so the other half is fixed here. That is why this PR touches a shared file.

setStandardSelectorAndLabels puts the app.kubernetes.io/* labels on any object that has a CR. Its pod-template pass only handles Deployment, DaemonSet, Prometheus and Alertmanager, so Jobs fall to the default case and return early. A Job's pods end up missing the labels every other operator-rendered pod carries. This is not an envoy thing. It hits all four Jobs we render: envoy-gateway certgen, the intrusion detection Elasticsearch installer, the AWS security group setup, and the logstorage dashboards installer.

Two things to know about that fix:

• It labels the pod template only. A Job's selector belongs to the job controller, which adds controller-uid, and the field is immutable. So unlike Deployment and DaemonSet we must not fill in Spec.Selector. There is a test that asserts it stays nil, to stop someone later copying the Deployment case wholesale.
• Existing Jobs are left alone. mergeState compares Jobs on container count, container images and pod-template annotations, not labels, so this never reaches the delete-and-recreate path. No churn on upgrade. The flip side is that the labels land on Jobs created fresh, so a Job already on a cluster keeps its old pod template until something else recreates it.

How I tested it

I built this operator image and ran it on an OSS master cluster (eBPF, namespaced mode). I put a Gateway, an HTTPRoute, and an nginx backend in a normal user namespace, scoped a Calico default-deny to the proxy pod, and curled the proxy from inside the cluster.

Step	Setup	Result
Baseline	no deny	HTTP 200
Reproduce	default-deny, stock operator (no GNP)	times out, proxy fully blocked
This PR	default-deny, our operator (GNP)	HTTP 503: proxy reachable and routing, only the backend hop is denied
Backend allow	user allows proxy to backend	HTTP 200

I also tried the narrower ingress idea (allow only 19001, then Pass). Under default-deny it times out: listener traffic falls through to the user's deny and the Gateway stops serving. So allowing all inbound TCP is the right default. The cost is that an Allow is terminal in this tier, so a user cannot narrow proxy ingress with their own policy. Scaling the operator back up showed it reverts any drift on the GNP and the Gateway recovers.

I re-ran the whole repro after switching the selector to the Calico-owned k8s-app label (OSS master, eBPF, namespaced mode). The operator stamps k8s-app: calico-gateway-api-proxy on the proxy pod, the rendered GNP calico-system.envoy-gateway-proxy selects it, and the table holds: baseline 200; under a Gateway-namespace default-deny the proxy returns 503 (reachable and routing, only the backend hop denied); allowing proxy to backend gives 200. Scaling the operator down and deleting the GNP drops the proxy to a full timeout, and scaling it back up re-renders the GNP and the Gateway recovers. One thing to watch when testing this by hand: the 503 lands after Envoy's ~10s upstream timeout, so a curl timeout under 10s reads as a connect failure even though the proxy is reachable..

For the Job labelling I added two specs next to the existing daemonset and deployment ones in component_test.go. One covers a bare Job getting the labels, and asserts the selector stays nil. The other covers a Job whose pod template already carries a k8s-app from its render, which is the certgen case, and checks ours wins. I backed the component.go change out and re-ran both to confirm they fail without it.

One thing to know

The GNP covers the proxy's own needs: DNS, the control-plane link, and ingress. It does not open egress to backends, because backends are arbitrary user workloads. A user who runs default-deny in a Gateway namespace has to allow the proxy to reach their backend themselves. Until they do, the proxy is up and configured but returns 503 on the upstream. This matches how the controller policy already works, and is worth a line in the docs.

Release Note

Add a `calico-system`-tier GlobalNetworkPolicy for Envoy Gateway data-plane proxies so Gateways work under a default-deny tier when proxies run in per-Gateway namespaces (deploy.type=GatewayNamespace).
Set the standard app.kubernetes.io/* labels on the pod templates of operator-rendered Jobs, which were skipped before.

For PR author

• Tests for change.
• If changing pkg/apis/, run make gen-files (n/a, no API change).
• If changing versions, run make gen-versions (n/a).

[rene-dekker]

Could we set some calico labels on these objects and use those for policy?

Related: do the envoy related components have standard labels that we set in utils/component.go? If not,that is a gap.

[electricjesus]

did both, thanks.

The selector keys off a label we own now. Proxy pods get k8s-app: calico-gateway-api-proxy, stamped through the EnvoyProxy pod spec (ensureGatewayProxyLabel), and calico-system.envoy-gateway-proxy selects on that instead of gateway.envoyproxy.io/owning-gateway-name. Controller and certgen keep the labels they had, just set from one place now (setGatewayComponentLabel).

On your component.go question: you were right, there was a gap, and it's fixed here too.

setStandardSelectorAndLabels does set the app.kubernetes.io/* labels on any object with a CR, so the controller Deployment, the certgen Job and the EnvoyProxy all get those. But the pod-template pass below it only handles Deployment/DaemonSet/Prometheus/Alertmanager, so Jobs hit the default case and return early and their pods miss the lot. Not an envoy thing, it's all four Jobs we render. Added *batchv1.Job to that switch.

Two bits worth flagging on that one. It labels the pod template only, since a Job's selector belongs to the job controller and is immutable, so we can't do the Spec.Selector thing the Deployment case does. And existing Jobs don't get rewritten, because mergeState compares Jobs on container images and pod-template annotations rather than labels. No churn on upgrade, but it does mean a Job already on a cluster keeps its old template until something recreates it. Seemed like the better trade than recreating four Jobs on every upgrade, though shout if you'd rather it were immediate..

The proxy is the one case the labeller genuinely can't reach. It derives the label from the object name, so we'd get envoy-gateway rather than a stable calico-gateway-api-*, and the proxy pods are created by the EG controller at runtime so we never render them to label. The EnvoyProxy pod spec is the only hook we own. That's why the k8s-app labelling still sits in the gateway render, with a comment saying why.

One bit of luck: the pod-template pass only sets k8s-app when it's empty, so our explicit labels survive it.