KAFKA-20295: Add opt-in workaround for stale controller registrations during metadata.version upgrades

[davide-armand]

This PR adds a targeted recovery path for clusters where stale controller
registrations remain in metadata after quorum reconfiguration and block
metadata.version upgrades.

This mainly affects clusters that are upgraded to a newer Kafka version via
rolling upgrade, for example 3.9 -> 4.0. In that workflow, controller IDs are
not re-used, so controllers running the old version can remain registered in
metadata even after they are no longer part of the live voter set. Those stale
registrations are still advertised as using the previous version, which prevents
a the metadata.version upgrade (kafka-features.sh).

The workaround basically consists in ignoring the stale controllers when the
kafka-features.sh is run, so that it can complete successfully.

A a proper fix that allows controllers to be unregistered in being worked on
(apache#22191), this PR provides a quickfix until
the proper fix is available.

The changes in this PR are split into three commits:

1. Ignore stale controller registrations in feature validation

   • adds the core validation change and regression coverage

2. Add opt-in stale controller check override

   • makes the behavior opt-in via a new UpdateFeatures request flag
   • when enabled, validation uses the live Raft voter set rather than all
     registered controllers
   • keeps default behavior unchanged
   • fails closed if live quorum controller IDs are unavailable when controller
     registration support exists

3. Guard AddRaftVoter with metadata.version support

   • adds an explicit safety check when re-adding a controller as a voter
   • verifies that the target controller's registered supported
     metadata.version range includes the cluster's current metadata.version
   • prevents an outdated controller from being added back into the quorum
     after having been ignored by the opt-in override

This keeps the recovery path narrow and explicit:

• default upgrade behavior is unchanged
• operators can opt in to ignoring stale non-voter controllers
• future quorum reconfiguration still enforces metadata-version compatibility

Example usage:

kafka-features.sh --bootstrap-server $BOOTSTRAP upgrade --release-version 4.0 --ignore-stale-controller-registrations

[Copilot]

Pull request overview

This PR introduces an opt-in recovery path to unblock metadata.version upgrades when stale controller registrations (no longer in the live KRaft voter set) linger in the metadata image and incorrectly fail feature validation. It also adds a safety guard to prevent re-adding outdated controllers into the quorum after using that override.

Changes:

• Adds --ignore-stale-controller-registrations to kafka-features.sh upgrade, plumbed through AdminClient UpdateFeaturesOptions into a new UpdateFeatures RPC request flag.
• Updates controller-side feature validation to optionally validate only the live voter-set controllers (and fail closed when voter IDs are unavailable while registrations are supported).
• Adds a metadata.version compatibility check when handling AddRaftVoter, and exposes Raft voter IDs via RaftClient.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tools/src/main/java/org/apache/kafka/tools/FeatureCommand.java	Adds CLI flag and forwards it into UpdateFeaturesOptions.
tools/src/test/java/org/apache/kafka/tools/FeatureCommandTest.java	Verifies the CLI flag propagates into UpdateFeaturesOptions.
clients/src/main/java/org/apache/kafka/clients/admin/UpdateFeaturesOptions.java	Adds the new AdminClient option knob for ignoring stale controller registrations.
clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java	Serializes the new request flag onto UpdateFeaturesRequestData.
clients/src/main/resources/common/message/UpdateFeaturesRequest.json	Introduces UpdateFeaturesRequest v3 with IgnoreStaleControllerRegistrations field.
clients/src/main/resources/common/message/UpdateFeaturesResponse.json	Extends response validVersions to include v3 (same schema as v2).
clients/src/test/java/org/apache/kafka/common/requests/UpdateFeaturesRequestTest.java	Adds protocol roundtrip test for v3 ignore-stale flag.
clients/src/test/java/org/apache/kafka/clients/admin/KafkaAdminClientTest.java	Ensures AdminClient sets ignore-stale flag on the request.
metadata/src/main/java/org/apache/kafka/controller/FeatureControlManager.java	Implements opt-in filtering of controller registrations using live voter IDs.
metadata/src/main/java/org/apache/kafka/controller/ConfigurationControlManager.java	Plumbs ignore-stale flag into FeatureControlManager updateFeatures.
metadata/src/main/java/org/apache/kafka/controller/ClusterFeatureSupportDescriber.java	Adds method to expose current quorum controller IDs.
metadata/src/main/java/org/apache/kafka/controller/QuorumController.java	Implements quorumControllerIds() by querying the Raft client voter set.
metadata/src/test/java/org/apache/kafka/controller/FeatureControlManagerTest.java	Adds regression tests for default vs opt-in behavior and failure-closed semantics.
core/src/main/scala/kafka/server/ControllerApis.scala	Rejects AddRaftVoter when the target controller’s metadata.version range is incompatible.
core/src/test/scala/unit/kafka/server/ControllerApisTest.scala	Tests AddRaftVoter compatibility checks and registration-required behavior.
raft/src/main/java/org/apache/kafka/raft/RaftClient.java	Adds RaftClient.voterIds() API for retrieving current voter IDs.
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java	Implements voterIds() using the control record state machine’s last voter set.
metadata/src/test/java/org/apache/kafka/metalog/LocalLogManager.java	Implements voterIds() for the test RaftClient used in metadata tests.
metadata/src/test/java/org/apache/kafka/image/publisher/SnapshotEmitterTest.java	Implements voterIds() in an RaftClient test stub.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

raft/src/main/java/org/apache/kafka/raft/RaftClient.java:140

• RaftClient is a public interface; adding voterIds() as an abstract method is source/binary-incompatible for downstream RaftClient implementations. Make this a default method (returning an empty set) so existing implementations continue to compile, while allowing KafkaRaftClient to override it.

    /**
     * Returns the current voter IDs known to this client.
     *
     * Implementations may return an empty set before initialization completes.
     *
     * This method is thread-safe and may be called from threads other than the KRaft IO thread.
     *
     * @return the current voter IDs
     */
    default Set<Integer> voterIds() {

[davide-armand]

Closing, rejected in favor of: #70