fix: switch CI svc cluster agent pools to AMD v7 VMs for better zonal availability - #6655
fix: switch CI svc cluster agent pools to AMD v7 VMs for better zonal availability#6655Inbharaj Mani (inbharajmani) wants to merge 2 commits into
Conversation
… availability The e2e-parallel job was failing with OverconstrainedZonalAllocationRequest in westus3 because Azure lacked capacity for the Intel D*ds_v6 SKUs in zone 1. Switch all three svc cluster agent pools (system, user, infra) in ci00 and ci01 to the AMD Dads_v7 family, which is newer generation with equivalent specs and broader zonal availability.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: inbharajmani The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Pull request overview
This PR updates the AKS service cluster agent pool VM SKUs for the dev cloud’s CI environments (ci00, ci01) to move from Intel Dds_v6 to AMD Dads_v7, with the intent of improving zonal capacity/availability for CI service clusters.
Changes:
- Update
systemAgentPoolandinfraAgentPoolVM sizes fromStandard_D4ds_v6→Standard_D4ads_v7forci00/ci01service clusters. - Update
userAgentPoolVM size fromStandard_D8ds_v6→Standard_D8ads_v7and rename the pool fromu64d8dsv6→u64d8adsv7. - Commit corresponding rendered config outputs for
dev/ci00anddev/ci01incentralus.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| config/rendered/dev/ci01/centralus.yaml | Applies the new AMD v7 VM SKUs (and pool rename) to the rendered CI01 service cluster agent pools. |
| config/rendered/dev/ci00/centralus.yaml | Applies the new AMD v7 VM SKUs (and pool rename) to the rendered CI00 service cluster agent pools. |
| config/config.yaml | Updates the source template values for CI00/CI01 service cluster agent pool VM sizes (and user pool name) so future materializations are consistent. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| name: "{{ .ctx.environment }}-{{ .ctx.regionShort }}-svc" | ||
| systemAgentPool: | ||
| vmSize: Standard_D4ds_v6 | ||
| vmSize: Standard_D4ads_v7 |
| name: 'u64d8adsv7' | ||
| vmSize: Standard_D8ads_v7 | ||
| osDiskSizeGB: 64 | ||
| infraAgentPool: | ||
| vmSize: Standard_D4ds_v6 | ||
| vmSize: Standard_D4ads_v7 |
A second CI failure (PR Azure#6447) hit the same OverconstrainedZonalAllocationRequest on the mgmt cluster infra pool in westus2. Switch mgmt cluster system, user, and infra pools in ci00 and ci01 to AMD v7 equivalents for consistent zonal availability across both cluster types.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
Suppressed comments (1)
config/config.yaml:1926
- Same issue as earlier ci00 block: userAgentPool name
u64d8adsv7will be expanded to${name}nz{counter}and truncated to 12 chars in dev-infrastructure/modules/aks/pool.bicep, causing duplicate pool names when poolCount>1 and zones are empty.
vmSize: Standard_D4ads_v7
userAgentPool:
name: 'u64d8adsv7'
vmSize: Standard_D8ads_v7
| maxCount: 3 | ||
| minCount: 1 | ||
| name: u64d8dsv6 | ||
| name: u64d8adsv7 | ||
| osDiskSizeGB: 64 | ||
| poolCount: 3 |
| maxCount: 3 | ||
| minCount: 1 | ||
| name: u64d8dsv6 | ||
| name: u64d8adsv7 | ||
| osDiskSizeGB: 64 | ||
| poolCount: 3 |
| vmSize: Standard_D4ads_v7 | ||
| userAgentPool: | ||
| name: 'u64d8dsv6' | ||
| vmSize: Standard_D8ds_v6 | ||
| name: 'u64d8adsv7' | ||
| vmSize: Standard_D8ads_v7 |
|
Inbharaj Mani (@inbharajmani): The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
Inbharaj Mani (@inbharajmani) we have an ongoing evaluation for potential issues with AMD-based VMs (PERFSCALE-5456). I suggest you have a quick sync with Murali Krishnasamy (@mukrishn) to make sure there won't be latency issues when moving CI to AMD. |
|
Inbharaj Mani (@inbharajmani), During AMD_v6 testing we did not see any cluster-create performance gaps. The issues we hit came from the personal-dev env limitation, where secondaryNicCount was still set too low, I see they are set to max(7) already in CI. Moving to AMD_v7 should be fine for CI clusters. |
|
thanks for the feedback Murali Krishnasamy (@mukrishn) |
Summary
OverconstrainedZonalAllocationRequestfailures across multiple regions (westus3,westus2) where Azure lacked capacity for the v6 Intel SKUs in specific availability zonesSVC cluster changes
Standard_D4ds_v6Standard_D4ads_v7Standard_D8ds_v6Standard_D8ads_v7Standard_D4ds_v6Standard_D4ads_v7MGMT cluster changes
Standard_D8ds_v6Standard_D8ads_v7Standard_E16ds_v6Standard_E16ads_v7Standard_D8ds_v6Standard_D8ads_v7Failing jobs
Test plan
cd config && make materializepasses