[KAFKA-20295] decommision (pre-unregister) controller patch

[HelenMel]

Goal: introduce a new operator action — decommission a controller — after which that
controller no longer participates in feature / metadata.version upgrade decisions. Physical removal
of its registration is explicitly not part of decommissioning on 4.0.x–4.2.x (D6).

Terminology (read this first)

Term	Meaning	Availability
Decommission (this task, new)	Retire a controller: it stops counting towards feature / metadata.version upgrade decisions. Its registration stays in the metadata image.	4.0.x–4.2.x fork patch, and onwards
Unregister (upstream KIP-1312)	Physically delete the controller's registration via UnregisterControllerRecord.	vanilla 4.4+ (metadata.version ≥ 4.4-IV2)

The two are not the same operation and this task does not backport unregistration. They are
sequential states of one lifecycle: a controller is decommissioned first, and once the cluster can
support it (MV ≥ 4.4-IV2) a decommissioned controller may additionally be unregistered. Per D8 the
escalation is automatic and server-side, so the operator keeps running one command.

---

1. Problem

A KRaft controller that has left the cluster keeps its RegisterControllerRecord in
__cluster_metadata forever — there is no unregister record, no fencing, no expiry. Registrations
are append-only and overwrite-by-id (ClusterControlManager.replay():679 does
controllerRegistrations.put(...), never remove). The stale entry is still consulted when
validating feature and metadata.version upgrades.

3. Requirements

3.1 Functional

• R1 An operator can decommission a controller on a cluster running 4.0.x–4.2.x whose finalized
  metadata.version is as low as 3.7-IV0, via kafka-cluster.sh decommission-controller --controller-id <id>.
• R2 Afterwards FeatureControlManager no longer consults that controller, so
  kafka-features.sh upgrade succeeds.
• R3 A controller rejoining with the same id re-registers, clears the marker, and participates
  in decisions again. Intended behaviour (D7); free, because the RPC path rebuilds Features from
  the request (ClusterControlManager.java:469-475, evidence E9).
• R4 The active controller refuses to decommission itself.
• R5 Targeting an id that has no registration at all returns CONTROLLER_ID_NOT_REGISTERED.
• R6 Decommissioning is idempotent: if the controller is already decommissioned in the
  representation this MV would write, the call succeeds and emits nothing (D11). On 4.0.x–4.2.x that
  representation is always the marker, so a repeat is always a no-op; on 4.4 a marked registration is
  escalated to a deletion rather than no-opped (R7), which is what makes C5's cleanup a plain re-run.
• R7 The operation escalates automatically where the finalized MV supports controller
  unregistration (≥ 4.4-IV2): it writes a real UnregisterControllerRecord instead of the marker
  (D8). Not implemented by this patch — that MV does not exist on 4.0.x–4.2.x, so the arm is added
  by the 4.4 forward-port (Phase 6.6). It is stated here because it defines the operation, not because
  it is in scope.

3.2 Relationship to apache#22191

• C1 — A new operator action, wire-compatible with upstream but named for what it does.
  decommission-controller is deliberately not named unregister-controller, because on
  4.0.x–4.2.x it does not unregister anything: it retires the controller from upgrade decisions and
  leaves the registration in place (§ Terminology). What it must not do is invent protocol. On the
  wire it is byte-identical to upstream's KIP-1312 RPC — apiKey 94, v0, flexible 0+, a single
  ControllerId field, ALTER on CLUSTER, error code 136 — so a patched 4.0–4.2 client reaching a
  vanilla 4.4+ controller gets precisely the escalated behaviour of R7, and vice versa. Every
  identifier we introduce is named Decommission* (D9): DecommissionControllerRequest.json,
  ApiKeys.DECOMMISSION_CONTROLLER, DecommissionControllerRequest/Response,
  DecommissionControllerOptions/Result, KafkaAdminClient#decommissionController,
  ControllerApis.handleDecommissionController. Unregister* names belong to KAFKA-20395: Support unregistering controllers apache/kafka#22191 and
  are not introduced by this patch. Compatibility here rests on the apiKey and the schema, which are
  what goes over the socket — not on local symbol names.
• C2 — One state, two representations. Below MV 4.4-IV2 a decommissioned controller is a
  marked registration; at or above it, an absent one. Both mean "does not participate in upgrade
  decisions", which is the property R2 needs. Accepted differences on the low-MV representation:
  DescribeCluster and MetadataShell still list the controller, and
  ControllerRegistrationManager's "registration removed for this node ID" branch stays dead code.
• C3 — No identifier squatting. Reserved, do not reuse: record apiKey 29, Errors 130–135, MV
  feature levels ≥ 27, and no fork-local MetadataVersion value. Taking 94 and 136 is not squatting:
  they are the numbers upstream uses for this exact wire shape (C1).
• C4 — The fork-local surface is names, never numbers or shapes. Introduced by us: the
  decommission-controller command plus the Decommission* protocol, client and handler artefacts
  listed in C1. Forbidden: new apiKey or error numbers, new or altered fields on the request schema,
  new flags or changed semantics on upstream commands (the KAFKA-20295: Add opt-in workaround for stale controller registrations during metadata.version upgrades #69 contrast, §2.3), additions to the
  public Admin interface.
• C5 — Forward-port obligation, not an operator graduation step. Because escalation is automatic
  (R7/D8) the operator has nothing to re-run, but the fork's 4.4 must keep the read-side filter
  of §4.2 so controllers decommissioned under a low MV stay decommissioned after the upgrade.
  Dropping the filter during the forward-port would silently readmit them to upgrade decisions.
  Optional cleanup once MV ≥ 4.4-IV2 is finalized: re-run decommission-controller, which then
  escalates a marked registration into a real deletion (R6 makes the re-run safe, R7 makes it
  meaningful). Vanilla, non-fork 4.4 does not honour markers — that is RK2, and it applies to
  anyone leaving the fork.

3.3 Safety

• S1 No new MetadataVersion; gate only on isControllerRegistrationSupported() (≥ 3.7-IV0).
• S2 Never write an unknown metadata record apiKey — nothing may crash an unpatched node's
  MetadataRecordSerde.
• S3 Hard error (D4, no override) when targeting an id present in
  QuorumFeatures.quorumNodeIds() (static controller.quorum.voters), because loop (3b) would
  immediately re-block upgrades with "controller N has not registered" — trading one blocker for
  another. The message must name the config and tell the operator to remove the id from it on every
  node first. Deliberate divergence from upstream, which reverted the equivalent check.
• S4 Document mixed-version behaviour: an unpatched active controller ignores the marker.

3.4 Non-goals

• Backporting IBP_4_4_IV2 or the UnregisterControllerRecord type. Decommissioning is not
  unregistration; unregistration stays a 4.4+ capability.
• Removing the registration from the metadata image (D6).
• Renaming, re-flagging or otherwise altering upstream's unregister-controller /
  kafka-metadata-quorum.sh remove-controller --unregister surfaces, which the fork inherits
  untouched when it reaches 4.4.
• Exposing decommissioning on the public Admin interface (D2).
• TTL/automatic expiry of controller registrations.
• Broker registration lifecycle; dynamic voter-set integration beyond S3 (upstream KAFKA-20865).

---

4. Design

4.1 Encoding research: a marker with no MV bump (verified 2026-08-03)

Question: can a "decommissioned controller" tag be carried without a metadata change / MV bump,
specifically as a feature entry?

Answer: yes — a feature entry needs no schema change at all and is provably inert. Evidence,
all from unmodified 4.0.x code:

#	Claim	Evidence
E1	Features is an unbounded array in v0, so an extra element is not a schema change: no .json edit, no codegen, no record-version bump, no MV bump.	RegisterControllerRecord.json:40-48 — {"name":"Features","type":"[]ControllerFeature","versions":"0+"}
E2	Feature names are never validated on the controller-registration path; the parser blindly copies every name.	ControllerRegistration.java:75-80
E3	Every name is written back verbatim, so the marker survives a snapshot round-trip even on an unpatched node.	ControllerRegistration.java:185-191 (toRecord), used by ClusterImage.write():85
E4	The marker cannot affect a real feature update: reasonNotSupported looks up only the requested name, so an extra key is never consulted.	FeatureControlManager.java:278 — getOrDefault(featureName, QuorumFeatures.DISABLED)
E5	The marker cannot be finalized by an operator: updateFeature calls Feature.featureFromName, which throws for unknown names, and reasonNotLocallySupported rejects it first anyway.	FeatureControlManager.java:230-236; Feature.java:205-211
E6	The marker cannot leak into kafka-features.sh describe / ApiVersionsResponse — those read the finalized-features image, not registrations.	ControllerServer.scala:152-158; FeatureCommand.java:229
E7	The marker cannot leak into DescribeCluster — that projection reads only listeners().	ControllerRegistrationsPublisher.java:63-77
E8	Unlike brokers, controller registrations are not validated against finalized features, so there is no processRegistrationFeature-style rejection.	ClusterControlManager.java:463-485 vs :487-509
E9	A returning controller wipes the marker automatically, because the RPC path rebuilds Features from the request. R3 is free.	ClusterControlManager.java:469-475
E10	The marker is visible in logs and MetadataShell as __decommissioned_controller: 1-1 — useful for auditing.	ControllerRegistration.toString():216-230; ClusterImageControllersNode.java:57

A tagged field would be worse. It is also technically MV-free — old readers park unknown tags in
_unknownTaggedFields and re-serialize them (MessageDataGenerator.java:526, :801) — but only
for the same message object. ClusterImage.write() builds a fresh record from
ControllerRegistration, whose builder reads only declared fields, so the tag is dropped: any
unpatched node that emits a local snapshot and later restarts loses the marker permanently. It also
costs a schema edit on three branches and risks colliding with a future upstream tag number.

Sharp edges

• getOrDefault(featureName, DISABLED) (E4) means a feature name absent from a registration is
  treated as unsupported, not unknown. A stale 3.7-era registration therefore already blocks
  upgrades of any feature it never advertised — relevant to sizing option G.
• ControllerFeatureCollection extends ImplicitLinkedHashMultiCollection
  (MessageDataGenerator.java:195), so add() permits duplicate names. The write path must be
  idempotent.
• VersionRange.of performs no validation (VersionRange.java:36-42), so any sentinel range is
  accepted and only tests will catch a mistake.

4.2 Approach: mark and exclude at one choke point

On a DecommissionControllerRequest (apiKey 94, wire-identical to upstream's, per C1), the active
controller appends a RegisterControllerRecord for the target id, copying the existing registration (IncarnationId,
ZkMigrationReady, EndPoints, real Features) and adding one extra Features entry:

Field	Value
Name	__decommissioned_controller
MinSupportedVersion	1
MaxSupportedVersion	1

The registration then stays in place. The only behavioural change is to skip marked registrations in
ClusterControlManager.controllerSupportedFeatures():832. That filter is also what the 4.4
forward-port must retain (C5), even though 4.4 writes deletions instead of markers.

That one method is the entire production surface. It is called only from
QuorumController.QuorumClusterFeatureSupportDescriber.controllerSupported()
(QuorumController.java:512-513), consumed only by FeatureControlManager.reasonNotSupported()
(FeatureControlManager.java:268). Every feature and metadata.version decision flows through it,
and nothing else does.

Payoff: patched and unpatched nodes hold byte-identical metadata images. The registration is
present everywhere; only the feature-gating read differs. There is no divergence to reason about
and no way for a node to "lose" the marker in a locally generated snapshot (E3).

Interaction with loop (3b). Skipping a marked controller in (3a) means it is never added to
foundControllers, so if its id also appears in quorumFeatures.quorumNodeIds() the next loop fails
with "controller N has not registered". S3/D4's write-time hard error is what keeps that
unreachable. Residual edge case to document: adding a marked id back into
controller.quorum.voters later re-blocks upgrades until that controller actually registers.

4.3 Option G — also write permissive feature ranges

In the same record, additionally set permissive ranges for the dead controller: metadata.version as
VersionRange.of(MetadataVersion.MINIMUM_VERSION.featureLevel(), Short.MAX_VALUE) plus an entry for
every name in Feature.FEATURES (needed because an absent name reads as unsupported, §4.1).

Under D6, "does not participate in the decision" and "always answers yes" are outcome-equivalent, so
this is a real alternative rather than a compromise. Its value is that it works using record
contents alone — no interpretation code — so it also holds on unpatched nodes and on vanilla,
non-fork 4.4. Note that C5 already obliges the fork's 4.4 to keep the filter, so G's remaining job
is narrower than it first appears: unpatched active controllers (RK1) and clusters that leave the fork.

	A (mark + exclude)	G (permissive ranges)	A + G
Satisfies the goal on patched nodes	yes	yes	yes
Satisfies the goal with an unpatched active controller	no (RK1)	yes	yes
Still correct on vanilla, non-fork 4.4	no (RK2)	yes, for features known at patch time	yes
Production code required	one method	none	one method
Records factually false ranges	no	yes	yes
Auditable reason in logs / MetadataShell	yes	no	yes

Recommendation: write both in the single record. A supplies honest, auditable semantics and the
log line explaining why a controller was skipped; G is the load-bearing part that makes the outcome
hold on readers that do not understand the marker.

Two limits to state plainly:

• G cannot cover features that do not exist yet, so a feature introduced in 4.4 could still be
  blocked on a vanilla, non-fork 4.4 cluster. Converting the marker into a real unregistration once
  MV allows (R7, Phase 5.4) remains the real fix.
• G records a lie about a node's capabilities. It is self-limiting — a returning controller
  re-registers truthfully (E9) and D4 forbids marking a static voter. The underlying hazard
  (marking a live controller lets the cluster finalize an MV it cannot support) is identical under
  A alone; G only extends the consequence to unpatched readers. The runbook precondition "target
  process is stopped" (Phase 5.3) guards it.

If reviewers reject writing false ranges, ship A alone and accept RK1 and RK2.

4.4 Fallback

If Phase 0 invalidates the feature-entry encoding, the fallback is a tagged field on
RegisterControllerRecord (taggedVersions: "0+", tag 0), accepting the snapshot-loss and
tag-collision downsides in §4.1. If that is also unacceptable, ship #69 instead (D1).

4.5 How decommissioning differs from upstream unregistration

Aspect	Upstream 4.4 unregistration	Decommissioning on 4.0–4.2	Rationale
Operator action	unregister-controller — delete the registration	decommission-controller — retire the controller from upgrade decisions	different operations, D8
Storage	UnregisterControllerRecord apiKey 29	marked RegisterControllerRecord apiKey 27	no new MV available (§2.4)
MV gate	IBP_4_4_IV2	isControllerRegistrationSupported() (≥ 3.7-IV0)	must work on the stuck cluster
Effect on the image	registration deleted	registration kept, excluded from feature gating	D6
DescribeCluster / MetadataShell	controller disappears	controller still listed, marked	D6
Static-voter guard	none (reverted; KAFKA-20865)	hard error	S3 / D4
Public Admin method	Admin#unregisterController	KafkaAdminClient#decommissionController	D2 — no public API in a patch release
Wire bytes	apiKey 94 v0, error 136	identical	C1 / D9
Type names	UnregisterController*	DecommissionController*	D9 — names describe the action

The last two rows are the crux: identical bytes, different names. Since the request carries nothing
but a controller id and the same authorization, the active controller alone decides what
"decommission" costs — a marker below MV 4.4-IV2, a deletion at or above it (R7/D8) — and a patched
4.0–4.2 client keeps interoperating with a 4.4+ controller in both directions, because apiKey 94 v0
decodes the same either way. The names never travel, so they cost nothing at runtime; they cost one
deletion at forward-port time (RK11).

---

5. Risk register

#	Risk	Impact	Mitigation
RK1	Unpatched active controller ignores the marker	upgrade re-blocked after failover	adopt option G (§4.3); otherwise require all controllers rolled before use
RK2	Vanilla, non-fork 4.4 does not honour the marker	a feature introduced after the patch could be blocked	G covers features known at patch time; automatic escalation once MV allows (R7, Phase 5.4)
RK3	Operators expect the controller to vanish from DescribeCluster and read its presence as failure	support noise	D6 is deliberate — call it out in the runbook; the marker is visible in toString() (E10)
RK4	Targeting an id still in static controller.quorum.voters	upgrade blocked by loop (3b)	S3 hard error
RK5	Upstream later adds a feature literally named __decommissioned_controller	marker misinterpreted	__ prefix; unit test asserting the name is absent from Feature.FEATURES
RK6	Some code path validates controller feature names	registration rejected / NPE	Phase 0.1 audit (E2/E5–E8 already argue against it)
RK7	Conflict with #69 on shared files	merge pain, two competing UX	D1 sequencing; avoid gratuitous edits before the gate
RK8	3 branches × RPC+CLI backport	large surface in patch releases	one canonical commit series, cherry-picked; D2 shrinks it
RK9	Tools casting Admin → KafkaAdminClient (D2)	breaks for a non-KafkaAdminClient Admin; tool tests lose MockAdminClient	clear TerseException on cast failure; cover via integration tests
RK10	On the fork's 4.4 both decommission-controller and unregister-controller exist	operators pick the wrong one, or read the two as rivals	--help text and runbook state the lifecycle (§ Terminology): decommission is the action, unregistration is what it escalates to
RK11	At the 4.4 forward-port, upstream's own UnregisterController* schema lands on apiKey 94 alongside our DecommissionController*	two .json files claiming one apiKey — duplicate ApiMessageType entries, so codegen or compilation fails	expected and mechanical: the forward-port deletes our .json, ApiKeys entry and request/response classes, and repoints KafkaAdminClient#decommissionController and the handler at upstream's types. Phase 5.5 records it; Phase 6.6 owns it
RK12	The 4.4 forward-port drops the read-side filter as "superseded by upstream"	already-decommissioned controllers silently rejoin upgrade decisions	C5 states the obligation; carry test 4.1 forward as the guard

---

[davide-armand]

This guard seems to only check the static controller.quorum.voters list via featureControl.isControllerId(controllerId), not the current live Raft voter set.
In a dynamic quorum that means a controller can still be a live voter even after being removed from static config, and this code would then decommission that still-live voter and skip it from controllerSupportedFeatures().
Can you validate against the current voter set here instead of only the static config?

[davide-armand]

The help text here says that when metadata.version supports it, decommissioning will automatically escalate into a real unregister-controller operation.
Is this already implemented in this PR?