fix(apiserver): decide the API server migration from live cluster state

[xiumozhan]

Description

Cherry-pick of #5103 to release-v1.42, which ships the affected release. The load-bearing code is identical to master's; only line offsets differ.

Bug fix for EV-6821.

On a direct upgrade from the deprecated API server layout, the apiserver-controller moved the Calico API server into calico-system and repointed the v3.projectcalico.org APIService at it before the leftover allow-tigera.default-deny policy had been removed. That policy sits in the earlier-evaluated allow-tigera tier and selects all() endpoints, so it denied the moved pod at end-of-tier, the pod never became ready, and the aggregated API stayed down permanently — removing the policy requires the API server that had just been taken out of service.

This replaces the first version of this PR, which gated the move on a single cached read of that policy. The controller now decides from live state read through mgr.GetAPIReader(), so the decision cannot be made from cached contents that are no longer true: only v3.LicenseKey is excluded from the manager cache, so a cached read of a projectcalico.org/v3 object is served from the informer's last known state even when the aggregated API cannot serve.

Both inputs come from the v3.projectcalico.org APIService, which the kube-apiserver serves itself, so reading it does not depend on the thing being measured. spec.service.namespace says which layout is deployed; the Available condition is the aggregator's own most recent verdict on whether requests can be served. A migration is pending only while that APIService still points into tigera-system, so an upgrade that has already moved the API server does not wait on a policy that can no longer select the pod.

• Migration pending, aggregated API can serve, trap present — wait for the installation controller to remove it.
• Migration pending, aggregated API cannot serve — apply nothing. The trap cannot be confirmed gone and nothing can clear it while the API is down, so moving would only repoint the aggregated API onto a pod that cannot be vouched for. The 5m periodic reconcile already registered in Add() retriggers the decision.
• Anything else — reconcile normally. An unrecognised decision holds rather than proceeding, since failing open on this gate is not recoverable.

Two unsound tests are removed. The tigera-system namespace fallback treated the namespace's absence as proof the gate had passed; that does not hold on a two-hop upgrade, where the intermediate release deletes the namespace while leaving the policy in place, nor after an administrator deletes the namespace to clear a stuck upgrade. The NoMatch exemption treated a missing RESTMapper mapping as proof the policy was absent, which is not evidence about the policy at all.

On the pass that performs the migration the projectcalico.org/v3 NetworkPolicy component is applied before the workload rather than after it, removing the window in which the moved pod runs with no policy of its own. Every other pass keeps the existing order, which exists so a fresh install is not blocked on an API server that cannot come up yet.

Deliberately not changed. APIServer.Status.State remains a latch. The tiers controller gates tier creation on it through IsProjectCalicoV3Available, so making it live would stop the calico-system tier being created while the move is held, which would stop the installation controller reaching its delete of the trap — a new deadlock caused by the fix.

Components affected: pkg/controller/apiserver only.

Testing

Unit tests cover the decision table, including the two-hop state where the APIService already points at calico-system and the deny still exists, which must proceed rather than hold. Reconcile-level tests cover both hold paths, asserting that no calico-apiserver Deployment is created and the APIService is not repointed. The component ordering is pinned by a test that records real client Create order in both directions.

Live-validated twice on fresh CE 3.21.4 clusters upgrading to 3.23.1, exercising both hold branches in sequence in each run: the API server was taken down before the upgrade was applied, the gate held without applying anything while the aggregated API was unreachable, moved to waiting on the deprecated deny once it could serve again, and completed once the installation controller removed it. In both runs calico-system.apiserver-access was created before the calico-apiserver Deployment. Post-cutover unavailability while the new pod passes its readiness probe was 21 and 33 seconds, and recovered in both cases. Both runs reached the expected end state.

Release Note

Fixed a deadlock on a direct upgrade in which the Calico API server was migrated before the deprecated default-deny policy was removed, leaving the projectcalico.org/v3 API permanently unavailable.

For PR author

• Tests for change.
• If changing pkg/apis/, run make gen-files — not applicable, no API changes.
• If changing versions, run make gen-versions — not applicable, no version changes.