Apply calico-node before Typha during upgrades to prevent cluster-wide NotReady - #5110
Apply calico-node before Typha during upgrades to prevent cluster-wide NotReady#5110AdheipSingh wants to merge 3 commits into
Conversation
During upgrades, applying Typha and calico-node in the same reconcile rolls both simultaneously. Felix may be newer than Typha but not older, so once Typha completes its rollout first, still-old calico-node pods can no longer sync, go NotReady after their readiness probe window, and the DaemonSet controller stops honoring surge limits and replaces pods cluster-wide. Apply calico-node first, then gate Typha on the DaemonSet being fully rolled out. Read the DaemonSet directly from the API server since the informer cache may still hold the pre-update object immediately after the write.
|
|
|
@AdheipSingh thanks for the PR - was there a specific issue you noticed that prompted this? We try to make the upgrade not depend on ordering regardless, so curious if we introduced an unexpected dependency in v3.32. |
| return reconcile.Result{}, err | ||
| } | ||
|
|
||
| // Apply calico-node before the remaining components so that its rollout |
There was a problem hiding this comment.
This ordering is inverted for downgrades, where node rolls to the older Felix first while Typha is still new, which is exactly the direction the comment says doesn't work.
Wonder if we need more awareness on if this is an upgrade / downgrade / or simply a configuration change.
There was a problem hiding this comment.
Correct — as written, the deferral applies to any image change, so a downgrade would also roll node first, which is the direction the skew policy doesn't support. Might consider detecting direction by comparing the resolved desired version against the deployed one and only deferring for upgrades (falling back to current behavior for downgrades and unparseable references like digests), but didn't want to add version-parsing machinery without your input. Would you prefer that in this PR, as a follow-up, or is there an existing pattern in the operator for knowing upgrade vs downgrade that we should use?
|
/sem-approve |
Hi @caseydavenport , thanks for the review this is main reason ˆˆ for this PR. We have large nodes ( 1000 + ), and when we increase maxSurge to speed up rollouts, keeping maxUnavailable as 0, we see a large disruption. This was the reason for fixing this. We saw no issue when maxSurge is set to 1, as we increase surge we see this issue. Also this is visible when we tried to upgrade 30 nodes with maxSurge 4, maxSurge 1/2 works fine on 30 nodes.....so the disruption is proportional to no of nodes and maxSurge count. A newer Felix can talk to an older Typha just fine. But an older Felix cannot talk to a newer Typha.... the connection just dies. So during any upgrade there's exactly one safe order: upgrade all the calico-nodes first, Typha last. Then every mixed pairing that ever exists during the transition is "newer Felix --> older Typha," the direction that works. The operator ignored that order. When you upgrade Calico, the tigera-operator updates Typha and calico-node in the same breath. Typha is only 3 pods it finishes upgrading in about a minute. calico-node on 200 nodes, at 4 at a time, takes half an hour. So one minute into the upgrade you have: 3 brand-new Typhas, and ~196 still-old calico-nodes — every one of them now in the forbidden pairing. They all lose their instruction feed simultaneously, all fail their health checks 90 seconds later, and all flip to NotReady at once. |
|
Unit test is generated via Claude. |
This is the part that I'm worried about - do you have log output by chance from when this happened? We try to avoid this! |
|
/sem-approve |
Let me share with you RCA document. |
RCA: cluster-wide calico-node NotReady during operator upgrade to Calico v3.32All evidence below was captured on a 30-node kubeadm cluster (Kubernetes v1.33, AWS EC2), upgrading Calico v3.31.3 → v3.32.0 with the stock, unpatched operator v1.42.0 and SummaryTypha v3.32.0 actively disconnects any sync client that does not advertise Because the operator applies the Typha Deployment and the calico-node DaemonSet in the same reconcile, Typha (3 replicas) finishes rolling ~30× faster than the DaemonSet. Every not-yet-upgraded calico-node then loses Typha permanently, fails its readiness probe after 90s (3 × 30s), and the DaemonSet controller which does not count already-unavailable pods against the rolling-update limits, replaces the entire fleet at once, ignoring Observed impact: Ready collapsed from 29/30 to 7/30 in 33 seconds on a cluster configured for Root cause: the exact point where the connection breaksReproduced in isolation (operator scaled to 0; calico-node container image set to Typha v3.32.0 server log receives the v3.31.3 hello and disconnects it, for every syncer type ( Source: Felix v3.31.3 client log — the same moment from the client side; the disconnect surfaces as EOF, and Felix loops on reconnect indefinitely: Health consequence — readiness probe on the affected pod: State during this isolation test: calico-node image The cascade: timeline of the operator-driven upgrade (unpatched v1.42.0)DaemonSet watch excerpt (columns: DESIRED CURRENT READY UP-TO-DATE AVAILABLE): 7/30 Ready. The The compatible pairing is strictly one-directional: newer Felix ↔ older Typha works; older Felix ↔ newer Typha is refused at |
4 participants
Description
Type of change: bug fix
Problem: During upgrades, the operator applies the Typha Deployment and the calico-node DaemonSet in the same reconcile, so both roll simultaneously. Typha (typically 3 replicas) finishes its rollout in about a minute, while calico-node on a large cluster takes much longer. Felix is compatible with an older Typha, but an older Felix cannot sync with a newer Typha — so once all Typha replicas are on the new version, every still-old calico-node pod loses sync, fails its readiness probe after ~90s (3 × 30s), and goes NotReady. The DaemonSet controller does not count already-unavailable pods against
maxUnavailable/maxSurge, so it then replaces pods across the whole cluster at once instead of honoring the configured rolling-update limits. On a ~200 node production cluster this presented as a cluster-wide wave of NotReady calico-node pods during a 3.31 → 3.32 upgrade.Fix:
ObservedGeneration >= Generation, all pods updated and ready) using the existing componentReady()mechanism.With this ordering, new calico-node pods connect to old Typha (the compatible direction) while rolling under the configured limits, and Typha updates last, when every Felix is already on the new version. First installs are unaffected (no DaemonSet exists, so the gate is open), and in steady state the check passes immediately because the DaemonSet generation is unchanged.
Testing: Validated on a 30-node cluster upgrading Calico v3.31.3 → v3.32.1:
nodeUpdateStrategy: {maxSurge: 4, maxUnavailable: 0}(the affected user's configuration): calico-node Ready stayed 30/30 for the entire upgrade across every DaemonSet status transition; peak 34 concurrent pods (30 + exactly 4 surge), no limit bypass; node phase ~4 minutes; Typha updated only after all 30 nodes completed.maxUnavailable: 1): Ready never dropped below 29/30 (the single pod being replaced), Typha gated for the full ~18 minute rollout and updated within seconds of the last node becoming ready.Affected components: installation controller (core_controller.go), Typha render component.
Release Note
For PR author
make gen-filesmake gen-versionsFor PR reviewers
A note for code reviewers - all pull requests must have the following:
kind/bugif this is a bugfix.kind/enhancementif this is a a new feature.enterpriseif this PR applies to Calico Enterprise only.