feat: set dataPlaneHAProxyImage to decouple worker HAProxy from management cluster

[Anthony Vollmer (avollmer-redhat)]

Why

Bumping the management cluster HAProxy image (IMAGE_SHARED_INGRESS_HAPROXY) currently triggers unintended worker node rollouts across all hosted clusters because NodePools lack the hypershift.openshift.io/haproxy-image annotation. Without this annotation, worker nodes inherit the management cluster's HAProxy image, coupling data plane image versions to control plane upgrades (see incident ARO-27681 / AROSLSRE-978).

This PR enables Cluster Service to explicitly set the data plane HAProxy image on every NodePool, breaking that coupling. Once all environments have annotation coverage, AROSLSRE-994 can safely unpin the prod region digest overrides.

What

Restructures dataPlaneHAProxyImage from an empty string to a structured containerImage object (registry/repository/digest), matching the existing sharedIngressImage pattern:

• Config schema: dataPlaneHAProxyImage changed from "type": "string" to "$ref": "#/definitions/containerImage"
• Config values: Populated with the same image currently used by sharedIngressImage (safe: no image change, just enables annotation)
• values.yaml: Assembles ACR reference from structured fields (<acr.ocp.name>.azurecr.io/<repo>@<digest>)
• pipeline.yaml: Added mirror-data-plane-haproxy-image ImageMirror step to copy the image to the OCP ACR (where worker nodes pull from)
• Materialized configs and helm fixtures: Updated via make materialize

Verification

• make materialize, make validate-config-pipelines, make verify-schema all pass
• Deployed to personal dev environment (pers-usw3avol)
• Confirmed all 3 CS pod replicas have the --data-plane-ha-proxy-image=arohcpocpdev.azurecr.io/...@sha256:5e22710f... flag
• NodePool annotation will be verified in DEV (CSPR) post-merge via test cluster creation

Rollout Plan

1. This PR merges to ARO-HCP, DEV auto-deploys
2. INT/STG via sdp-pipelines config bump
3. PROD australiaeast + uksouth via sdp-pipelines with per-region digest override (sha256:4eea8f36...)
4. PROD remaining regions inherit global default
5. After full annotation coverage: AROSLSRE-994 proceeds (unpin prod regions)

Safety

The default digest (sha256:5e22710f...) matches the current sharedIngressImage digest in all non-overridden regions. Enabling the annotation does not change the actual image used — it only makes the assignment explicit, preventing future management cluster image bumps from cascading to worker nodes.

PROD uksouth and australiaeast have pinned overrides (sha256:4eea8f36...) that must be set via sdp-pipelines overlays before those regions are deployed (Phase 1.4-1.5 in the rollout plan).

Refs: ARO-27756, AROSLSRE-994

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: avollmer-redhat
Once this PR has been reviewed and has the lgtm label, please assign janboll for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• cluster-service/OWNERS
• config/OWNERS [avollmer-redhat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

This PR decouples hosted cluster worker HAProxy image selection from the management cluster by introducing a structured dataPlaneHAProxyImage configuration and wiring it through Cluster Service’s deployment so it can set the NodePool HAProxy image explicitly.

Changes:

• Converted clustersService.dataPlaneHAProxyImage from a string to a structured containerImage (registry/repository/digest) in config defaults and schema.
• Updated Cluster Service Helm values/fixtures so the deployment renders --data-plane-ha-proxy-image=<ocp-acr>/<repo>@<digest> when configured.
• Added an ImageMirror pipeline step to mirror the data plane HAProxy image into the OCP ACR prior to deployment.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
config/rendered/dev/pers/westus3.yaml	Materialized config: sets structured dataPlaneHAProxyImage fields.
config/rendered/dev/perf/westus3.yaml	Materialized config: sets structured dataPlaneHAProxyImage fields.
config/rendered/dev/dev/westus3.yaml	Materialized config: sets structured dataPlaneHAProxyImage fields.
config/rendered/dev/cspr/westus3.yaml	Materialized config: sets structured dataPlaneHAProxyImage fields.
config/rendered/dev/ci01/centralus.yaml	Materialized config: sets structured dataPlaneHAProxyImage fields.
config/rendered/dev/ci00/centralus.yaml	Materialized config: sets structured dataPlaneHAProxyImage fields.
config/config.yaml	Adds default dataPlaneHAProxyImage object (matching shared ingress image digest).
config/config.schema.json	Updates schema type for dataPlaneHAProxyImage to containerImage.
cluster-service/zz_fixture_TestHelmTemplate_dev_westus3_svc_1_cluster_service.yaml	Helm fixture: includes rendered --data-plane-ha-proxy-image=... arg.
cluster-service/values.yaml	Assembles the data plane HAProxy image ref as an OCP ACR digest reference for workers.
cluster-service/testdata/zz_fixture_TestHelmTemplate_cs_shared_arm_helper.yaml	Helm fixture: includes rendered --data-plane-ha-proxy-image=... arg.
cluster-service/testdata/zz_fixture_TestHelmTemplate_cs_distinct_arm_helper.yaml	Helm fixture: includes rendered --data-plane-ha-proxy-image=... arg.
cluster-service/testdata/zz_fixture_TestHelmTemplate_cs_containerdb.yaml	Helm fixture: includes rendered --data-plane-ha-proxy-image=... arg.
cluster-service/testdata/zz_fixture_TestHelmTemplate_cs_azuredb.yaml	Helm fixture: includes rendered --data-plane-ha-proxy-image=... arg.
cluster-service/pipeline.yaml	Adds mirror-data-plane-haproxy-image ImageMirror step and makes deploy depend on it.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

[Anthony Vollmer (avollmer-redhat)]

/retest

[Anthony Vollmer (avollmer-redhat)]

e2e-parallel failure is unrelated to this PR.

The single failing test is "Engineering should be able to retrieve kusto logs for a cluster and services" (kusto_logs_present.go:100) — it timed out after 600s with a Kusto endpoint query error:

must-gather CLI test failures:
  query-basic: command failed: exit status 1
output: ERROR: Query failed {
  "err": "failed to execute query: Op(OpQuery): Kind(KHTTPError): error from Kusto endpoint"

This is a Kusto infrastructure/connectivity flake, not related to the config/values/pipeline changes in this PR. Retesting.

[Anthony Vollmer (avollmer-redhat)]

/retest

[Anthony Vollmer (avollmer-redhat)]

e2e-parallel failure (run 2) — again unrelated to this PR.

7 test failures this run, all platform-wide infrastructure issues:

Failure	Root Cause
6x router deployment has unavailable replicas / KASLoadBalancerNotReachable	Known flake — hosted cluster router pods not ready in time
1x kusto_logs_present Kusto endpoint timeout	Same as run 1 — Kusto query infrastructure error

Dev CI is broadly broken today (Aug 24). Per CIHealth dashboard, 78% of all e2e runs are affected:

• Kusto endpoint timeout: 27% of runs (25/92) — NEW pattern
• "hosted cluster has no installed version" + router unavailable: 17% — known flake
• KASLoadBalancerNotReachable: 4% — NEW pattern
• KubePodNotReady (router): 13% — known flake

Root cause identified by Giulio Frasca (Slack thread): SEM0026 error from hcpctl caused by >10,000 unique Maestro objects flooding Kusto in 24h, causing query parameter overload. Fix in progress: #6663. Sajeel also noted a capacity issue earlier with the Dev region switched to Canada Central.

None of these failures are related to config/values/pipeline changes in this PR. Retesting.

[Anthony Vollmer (avollmer-redhat)]

e2e-parallel run 3 failed with 2 tests — both unrelated:

1. kusto_logs_present — same SEM0026 Kusto parameter overload (now hitting 34% of all runs per CIHealth). Blocked until fix(hcpctl): replace has_any with join in kusto must-gather query to avoid 10,000-argument limit #6663 merges.
2. nodepool skip minor version upgrade (4.20→4.22) — 1h timeout on the upgrade itself, not related to config changes.

The Kusto SEM0026 failure will continue to block all PRs until the Maestro object flood is resolved. /retest anyway in case the fix lands.

/retest

[Anthony Vollmer (avollmer-redhat)]

Update: #6666 is the cleaner fix for the Kusto SEM0026 issue -- renames let cluster_id → _cs_cluster_id to avoid column shadowing that caused has_any() to match every Maestro log and overflow the 10k argument limit. Giulio confirmed #6666 supersedes #6663 (Slack).

#6666 has all non-e2e checks green and e2e-parallel is currently running. Once it merges, the Kusto test should stop failing for all PRs including ours.

/retest

[Anthony Vollmer (avollmer-redhat)]

#6666 merged at 06:53 UTC — the Kusto SEM0026 cluster_id column shadowing fix is now in main. Our run 4 (failed 05:14 UTC) was still running against the pre-fix base.

Retesting now — this should be the first run with the fix included.

/retest

[openshift-ci]

Anthony Vollmer (@avollmer-redhat): The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-parallel	a660b6e	link	true	/test e2e-parallel

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[Anthony Vollmer (avollmer-redhat)]

e2e-parallel run 5 failed — 14 test failures, all platform-wide infrastructure issues unrelated to this PR:

Root Cause	Count	Details
router deployment has N unavailable replicas	11	Known mgmt cluster flake — router pods not ready within 19min deadline
cluster-version-operator has 1 unavailable replicas	1	Same class of mgmt cluster deployment availability issue
clusterServiceNodePoolStatus <no_message>	1	Dependent on cluster that failed to provision (router flake cascade)
400 status code assertion mismatch	1	Serial console test — unrelated

The Kusto SEM0026 fix (#6666) is confirmed working — zero Kusto failures this run. The sole remaining blocker is the router deployment unavailability flake on ephemeral CI management clusters, which is hitting all PRs equally.

/retest