[release-v1.42] fix(apiserver): prevent calico-apiserver deadlock on direct 3.21-to-3.23 upgrade

[xiumozhan]

Cherry-pick of #5087 to release-v1.42 (ships Calico Enterprise 3.23.1). The commit is a clean copy of the master change; no manual edits.

Description

Bug fix.

On a direct Calico Enterprise 3.21.x → 3.23.x upgrade, the operator moves the calico-apiserver from the tigera-system namespace into calico-system and retargets the v3.projectcalico.org APIService before the apiserver's allow policy is effective. The leftover 3.21-era allow-tigera.default-deny (selector all(), calico-system namespace, allow-tigera tier) still selects the moved pod and drops its egress to the Kubernetes API, so the aggregated projectcalico.org/v3 API goes down and the operator's write of calico-system.apiserver-access returns 503 indefinitely. The result is a deadlock that does not self-heal.

This change makes the apiserver controller write a transitional policy "bridge" through the still-serving old apiserver, before the workload cutover, reproducing the intermediate state that the safe two-hop upgrade (via 3.22.x) passes through:

• rewrite allow-tigera.default-deny to exclude calico-apiserver,
• create an order-1 allow-tigera.apiserver-access in the allow-tigera tier,
• create calico-system.apiserver-access.

The bridge component is ordered before the workload component, so the controller's fail-fast apply loop requeues before any cutover if a bridge write fails. Detection uses a shared predicate — the deprecated allow-tigera tier still exists and calico-apiserver is not yet Ready in calico-system — gated on the tier watch and a no-op on the native-CRD (UseV3CRDs) path. While migrating, the existing APIServerPolicy component and the kube-controllers and tiers deletions of the transitional allow-tigera.* objects are suppressed; they resume once the apiserver is stable in calico-system, and a periodic reconcile on the tiers controller removes the emptied allow-tigera tier promptly.

Components affected: apiserver controller, installation/kube-controllers policy rendering, tiers controller, apiserver policy renderer.

Testing: unit tests (Ginkgo v2 + Gomega + controller-runtime fake client) covering the migration predicate, the bridge rendering, the controller wiring (bridge-before-cutover, failed-write requeue, and the APIServerPolicy gate), and the two cross-controller deletion gates; all affected packages pass and go vet is clean. Additionally validated end-to-end on a live Calico Enterprise 3.21.4 → 3.23.1 operator upgrade built from the v1.42.4 tag plus this change: the API server migrated to calico-system with the aggregated v3 API staying up (only the normal seconds-long cutover blip), and the transitional allow-tigera.* policies and the allow-tigera tier were cleaned up automatically, leaving a healthy 3.23.1 install.

No API/CRD changes.

Fixes EV-6821.

Release Note

Fixed a deadlock in which the Calico Enterprise API server (calico-apiserver) could become permanently unavailable during a direct upgrade from Calico Enterprise 3.21 to 3.23.

For PR author

• Tests for change.
• If changing pkg/apis/, run make gen-files — n/a, no API changes.
• If changing versions, run make gen-versions — n/a, no version changes.

For PR reviewers

• Milestone set according to targeted release.
• Appropriate labels:
  • kind/bug — this is a bug fix.
  • enterprise — the affected path (aggregated API server, allow-tigera → calico-system migration) is Calico Enterprise; the change is a no-op on the OSS native-CRD path.
• Consider docs-not-required (restores expected upgrade behaviour; no user-facing configuration or API change).