chore(deps): bump ARO-Tools to fix EV2 retry metadata detection - #6670
chore(deps): bump ARO-Tools to fix EV2 retry metadata detection#6670Rael Garcia (raelga) wants to merge 1 commit into
Conversation
|
/test image-updater-images Retesting: failure is a clonerefs environmental issue unrelated to this PR's diff. Job pull-ci-Azure-ARO-HCP-main-image-updater-images/2092167957709852672 failed cloning refs with: ! [rejected] hcpctl-nightly -> hcpctl-nightly (would clobber existing tag) This is a git tag fetch conflict in the source clone step, before any build/test of this PR's changes ran. |
There was a problem hiding this comment.
Pull request overview
This PR bumps the github.com/Azure/ARO-Tools module set across this repo’s Go workspace modules to pick up ARO-Tools#323, which fixes EV2 retry eligibility detection by falling back from job-level finished.json to the correct step-level finished.json when needed (matching Prow sidecar metadata behavior).
Changes:
- Update
github.com/Azure/ARO-Tools/*dependencies (includingtools/prow-job-executor) tov0.0.0-20260825080653-d6d9832b1dccacross relevant modules. - Refresh transitive dependencies (notably
golang.org/x/*,github.com/go-openapi/*, and Grafana client deps) via the bump workflow. - Update corresponding
go.sumfiles to match the new module graph.
Reviewed changes
Copilot reviewed 31 out of 62 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tooling/yamlwrap/go.mod | Bump ARO-Tools/tools/yamlwrap and related indirect ARO-Tools refs. |
| tooling/yamlwrap/go.sum | Update sums for bumped ARO-Tools dependencies. |
| tooling/tenant-quota/go.mod | Refresh indirect golang.org/x/* versions after dependency bump. |
| tooling/tenant-quota/go.sum | Update sums for golang.org/x/* version refresh. |
| tooling/templatize/go.mod | Bump multiple ARO-Tools modules (pipelines, prow-job-executor, etc.) plus transitive dependency updates. |
| tooling/secret-sync/go.mod | Bump ARO-Tools/tools/secret-sync and related indirect ARO-Tools refs; refresh golang.org/x/*. |
| tooling/secret-sync/go.sum | Update sums for bumped ARO-Tools + golang.org/x/*. |
| tooling/prometheus-rules/go.mod | Refresh indirect golang.org/x/* versions (net/sys/text) after bump. |
| tooling/prometheus-rules/go.sum | Update sums for golang.org/x/* version refresh. |
| tooling/pipeline-documentation/go.mod | Bump ARO-Tools/pipelines. |
| tooling/pipeline-documentation/go.sum | Update sums for bumped ARO-Tools/pipelines. |
| tooling/olm-bundle-repkg/go.mod | Refresh golang.org/x/text and go-openapi/* indirect versions. |
| tooling/olm-bundle-repkg/go.sum | Update sums for go-openapi/* and golang.org/x/* refresh. |
| tooling/metricscache/go.mod | Refresh indirect golang.org/x/sys version. |
| tooling/metricscache/go.sum | Update sums for golang.org/x/sys refresh. |
| tooling/kustoctl/go.mod | Refresh indirect golang.org/x/* versions (crypto/net/sys/text). |
| tooling/kustoctl/go.sum | Update sums for golang.org/x/* refresh. |
| tooling/image-updater/go.mod | Refresh golang.org/x/mod and indirect golang.org/x/* versions. |
| tooling/image-updater/go.sum | Update sums for golang.org/x/* and golang.org/x/mod refresh. |
| tooling/helmtest/go.mod | Bump ARO-Tools deps used for Helm testing and refresh indirect deps. |
| tooling/hcpctl/go.mod | Bump ARO-Tools/tools/cmdutils and refresh indirect deps. |
| tooling/grafanactl/go.mod | Bump ARO-Tools/tools/grafanactl and refresh Grafana + go-openapi/* indirect deps. |
| tooling/entra-app-credentials/go.mod | Refresh indirect golang.org/x/* versions. |
| tooling/entra-app-credentials/go.sum | Update sums for golang.org/x/* refresh. |
| tooling/cleanup-sweeper/go.mod | Refresh indirect golang.org/x/* versions. |
| tooling/cleanup-sweeper/go.sum | Update sums for golang.org/x/* refresh. |
| tooling/azutils/go.mod | Refresh indirect golang.org/x/* versions (net/text). |
| tooling/azutils/go.sum | Update sums for golang.org/x/* refresh. |
| tooling/aro-hcp-exporter/go.mod | Refresh indirect golang.org/x/* versions. |
| tooling/aro-hcp-exporter/go.sum | Update sums for golang.org/x/* refresh. |
| test/go.mod | Bump ARO-Tools/config + ARO-Tools/tools/prow-job-executor and refresh transitive deps used by test tooling. |
| test-integration/go.mod | Refresh go-openapi/* and golang.org/x/* indirect deps in integration test module. |
| test-integration/go.sum | Update sums for dependency refresh in integration test module. |
| sessiongate/go.mod | Refresh go-openapi/* and golang.org/x/* indirect deps. |
| sessiongate/go.sum | Update sums for dependency refresh. |
| mgmt-agent/go.mod | Bump ARO-Tools/testutil and refresh indirect deps. |
| mgmt-agent/go.sum | Update sums for bumped ARO-Tools/testutil and refreshed deps. |
| kube-applier/go.mod | Refresh go-openapi/* and golang.org/x/* indirect deps. |
| kube-applier/go.sum | Update sums for dependency refresh. |
| internal/go.mod | Refresh go-openapi/* and golang.org/x/* indirect deps shared by services. |
| frontend/go.mod | Refresh indirect golang.org/x/* versions (notably x/term). |
| frontend/go.sum | Update sums for dependency refresh. |
| fleet/go.mod | Refresh go-openapi/* and golang.org/x/* indirect deps. |
| fleet/go.sum | Update sums for dependency refresh. |
| backend/go.mod | Refresh go-openapi/* and golang.org/x/* indirect deps. |
| backend/go.sum | Update sums for dependency refresh. |
| admin/server/go.mod | Refresh go-openapi/* and golang.org/x/* indirect deps. |
| admin/client/go.mod | Refresh golang.org/x/crypto direct requirement and golang.org/x/* indirect deps. |
| admin/client/go.sum | Update sums for golang.org/x/* refresh. |
| dev-infrastructure/scripts/postgres-access/go.mod | Refresh indirect golang.org/x/* versions for the script module. |
| dev-infrastructure/scripts/postgres-access/go.sum | Update sums for dependency refresh in script module. |
| dev-infrastructure/scripts/grafana-group-roles/go.mod | Refresh indirect golang.org/x/* versions for the script module. |
| dev-infrastructure/scripts/grafana-group-roles/go.sum | Update sums for dependency refresh in script module. |
| dev-infrastructure/scripts/acr-replication/go.mod | Refresh indirect golang.org/x/* versions for the script module. |
| dev-infrastructure/scripts/acr-replication/go.sum | Update sums for dependency refresh in script module. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
/retest All failing jobs share the identical clonerefs environmental failure, unrelated to this PR's diff (fails before any build/test step runs): ! [rejected] hcpctl-nightly -> hcpctl-nightly (would clobber existing tag) Confirmed on ci/prow/bicep-lint, config-change-detection, e2e-images, e2e-parallel, images, lint, observability-prometheus, secrets-validation, test-unit, verify (all same signature, same timestamp window ~08:31-08:34 UTC). |
|
/test test-unit |
|
Confirmed: `hcpctl-nightly` tag now points at the correct base SHA upstream (`git ls-remote --tags` matches BaseSHA a500a9d), so the tag conflict itself has resolved. The repeated identical failures are ci-operator reusing a cached `src-amd64` build image keyed to this exact base+PR SHA pair from the original transient failure window (identical log timestamps 08:31-08:34 across retests). Switching to periodic retest with backoff until the cache entry expires. |
|
/test bicep-lint Retesting after cache-expiry backoff (clonerefs hcpctl-nightly cache issue, see prior comment). |
|
/retest |
|
/test bicep-lint Retesting after cache-expiry backoff (clonerefs hcpctl-nightly cache issue, see prior comment). |
80d6445 to
24553e5
Compare
|
Force-pushed (same diff/message, amended timestamp only): the previous SHA `80d64452` had ci-operator permanently pin its src-build cache to namespace `ci-op-xjnbqgc4` from the original transient `hcpctl-nightly` tag-clobber failure. Every retest reused that poisoned cached `src-amd64` build without re-cloning (confirmed identical namespace + `Found existing build` across 3 retests over ~25 min). A new commit SHA is needed to get a fresh namespace/build. New HEAD: `24553e52`. |
|
/test image-updater-images Same clonerefs `hcpctl-nightly` tag-clobber race as before, but this is a fresh clone attempt (new SHA 24553e5, timestamps 09:04-09:06Z) and only this one job hit it this time (all others passed) — confirms a narrow, ongoing upstream tag-update race, not caused by this PR's diff. Retesting. |
|
Root cause identified and already fixed upstream: ARO-HCP#6671, merged 2026-08-25T09:01:48Z as `ba73a406`, explains the `hcpctl-nightly` tag force-move that breaks clonerefs `git fetch --tags` for every job on this repo ("accounts for ~25% of e2e-parallel failures in the last 24h" per that PR). Our base is already past that merge, but a stale in-flight hourly nightly run may still force-move the old tag once more before it fully stops. Backing off retries (~15-20 min) until it clears; not this PR's issue. |
|
/test bicep-lint Retesting after cache-expiry backoff (clonerefs hcpctl-nightly cache issue, see prior comment). |
|
/retest |
1 similar comment
|
/retest |
|
/test bicep-lint Retesting after cache-expiry backoff (clonerefs hcpctl-nightly cache issue, see prior comment). |
1 similar comment
|
/test bicep-lint Retesting after cache-expiry backoff (clonerefs hcpctl-nightly cache issue, see prior comment). |
24553e5 to
2cc8be4
Compare
|
Rebased onto main (now includes ARO-HCP#6671's fix) to get a fresh clone attempt/namespace. Confirmed the first post-fix nightly run (2026-08-25T09:15:43Z, succeeded, 5m53s) completed cleanly using the new marker-ref approach, and `hcpctl-nightly` tag is stable at `ba73a406` with no further moves. New HEAD: `2cc8be49`. |
|
Confirmed fleet-wide: other unrelated open PRs (#6672, #6673) are failing `verify`/similar jobs right now with the identical clonerefs `hcpctl-nightly` clobber signature, even though the tag itself is stable and ARO-HCP#6671 is merged. This points to Prow's shared git-mirror cache (not this PR's namespace) still holding a stale tag ref cluster-wide. Not something a retest/rebase on this PR can fix; needs the mirror to resync or a platform-side action. Backing off to a longer interval and will retest once the mirror catches up. |
|
/retest |
…LSRE-1721) Picks up Azure/ARO-Tools#323, which fixes jobAllowsEV2Retry to look at the per-step finished.json (where prow actually merges the aro-hcp-tests metadata.json) instead of only the job-level finished.json. Without this fix, the EV2 automated retry mechanism never detects retry-eligible tests for any real multi-stage ARO-HCP job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2cc8be4 to
b89541a
Compare
|
Fixed a genuine `verify-deepcopy` (go.sum tidy drift) failure: ran `make tidy`, which added missing `.../go.mod` hash lines to `tooling/prometheus-rules/go.sum` and `tooling/tenant-quota/go.sum` (and picked up equivalent additions across other modules from `go work sync`). This is the mechanical consequence of the ARO-Tools bump. New HEAD: `b89541a3`. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mmazur, raelga The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/test e2e-parallel Retesting: ci/prow/e2e-parallel (build 2092196022469005312) failed with ~28 unrelated specs (cluster creation, nodepool ops, KMS rotation, autoscaling, upgrades, etc. across multiple OCP candidate versions) all failing within a tight ~11:20-11:24 UTC window (one outlier at 11:38). That's the mass shared-CI provisioning-timeout signature, not a systemic/deterministic break. This PR only bumps ARO-Tools module versions and go.sum entries (dependency bump + tidy); it touches no e2e test code, cluster-provisioning code, or CI config. There's no plausible mechanism for it to cause 28 unrelated cluster-lifecycle specs to fail simultaneously in a narrow time window. All other required jobs (verify, test-unit, images, lint, config-change-detection, integration, etc.) are green on this SHA. Retesting as an environmental flake. |
|
Rael Garcia (@raelga): The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
What
Bumps the
github.com/Azure/ARO-Toolssubmodules to pick up ARO-Tools#323.Why
The EV2 automated retry mechanism (AROSLSRE-1721) never detected retry-eligible tests for any real multi-stage ARO-HCP job. Root cause:
jobAllowsEV2Retryonly checked the job-levelfinished.json, but prow's sidecar merges thearo-hcp-testsmetadata.jsoninto the per-stepfinished.json, never the job-level one. ARO-Tools#323 fixes this by checking the job-level file first, then falling back to locating the correct step-levelfinished.json.Ran
make bump-aro-tools(notmake ARO-Tools, which only bumps the root module reference and doesn't reliably move the individual submodules).Testing
cd test && go build ./... && go vet ./...cd tooling/templatize && go build ./...cd test && go test ./cmd/aro-hcp-tests/... ./cmd/prow-job-executor/...make fmt licensesproduced no additional diffRelated
AROSLSRE-1721