fix(infra): retry cluster ARM step on Key Vault RBAC propagation (AROSLSRE-1918) - #6708
Conversation
…SLSRE-1918) Creating the aks-etcd-encryption key via bicep is a control-plane keys/write, so the deployment identity needs a Key Vault role granting it. On a fresh region buildout that role assignment can lag propagation and the cluster ARM step fails with ForbiddenByRbac. Add that error code to the existing automatedRetry trigger list on both the svc and mgmt cluster steps so the step retries once the grant lands.
There was a problem hiding this comment.
Pull request overview
This PR updates the EV2/templatized infrastructure deployment pipelines to better tolerate transient Azure Key Vault RBAC propagation delays during fresh region buildouts. It does so by expanding the existing automated retry matchers for the cluster ARM deployment steps, allowing rollouts to self-heal once the Key Vault role assignment becomes effective.
Changes:
- Add
ForbiddenByRbactoautomatedRetry.errorContainsAnyfor the cluster ARM step in the service-cluster pipeline (svc-pipeline.yaml). - Add
ForbiddenByRbactoautomatedRetry.errorContainsAnyfor the cluster ARM step in the management-cluster pipeline (mgmt-pipeline.yaml). - Document the rationale inline (Key Vault
aks-etcd-encryptionkey creation can race RBAC propagation).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| dev-infrastructure/svc-pipeline.yaml | Adds ForbiddenByRbac to the cluster ARM step retry matcher to handle transient Key Vault RBAC propagation lag. |
| dev-infrastructure/mgmt-pipeline.yaml | Adds the same ForbiddenByRbac retry matcher to the mgmt cluster ARM step for parity and the same race condition. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mmazur, raelga The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest |
|
/test e2e-parallel Retesting: cluster-candidate-5-0 provisioning failed with: InternalServerError: [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas Same exact signature also just hit #6706 (unrelated docs/cleanup-sweeper PR) in the same time window, confirming this is a shared-CI data-plane provisioning episode, not caused by this PR's ARM-retry diff. |
|
/retest |
|
/test e2e-parallel |
|
/hold Revision abc2be2 was retested 3 times: holding |
|
/unhold |
|
/retest |
Jira: AROSLSRE-1918
What
Add
ForbiddenByRbacto the existingautomatedRetry.errorContainsAnylist on theclusterARM step in bothdev-infrastructure/svc-pipeline.yamlanddev-infrastructure/mgmt-pipeline.yaml.Why
During a prod Canary GlobalBuildout in eastus2euap, the
Service.Infracluster step failed withForbiddenByRbaconMicrosoft.KeyVault/vaults/keys/writefor theaks-etcd-encryptionkey. Creating that key via bicep is a control-plane write, so the deployment identity needs a Key Vault role that grantskeys/write. On a fresh region the role assignment can lag propagation, and the whole rollout step fails. Both cluster steps deployaks-cluster-base.bicepand create that key, so both can hit the same race. The retry lets the step recover once the assignment lands.This only covers the propagation race. A genuinely missing grant still fails after the retries, same as today, just a few minutes later.
Testing
make validate-config-pipelinespasses. This is a pipeline config change with no code path to unit test; the retry itself is exercised by EV2 at rollout time.Special notes for your reviewer
errorContainsAnyis an OR match and case-insensitive, so I matched the stable error codeForbiddenByRbacrather than a longer phrase. Rollout to int/stg/prod happens via sdp-pipelines after this merges.PR Checklist