Gate leader task assignment behind a dynamic ZooKeeper flag

[manishGoyalCode]

Gate leader task assignment behind a dynamic ZooKeeper flag

Summary

Adds a dynamic ZooKeeper-backed kill-switch for datastream task assignment. The leader Coordinator now consults a /{cluster}/assignmentEnabled znode before performing assignment, allowing operators to pause/resume assignment at runtime without a redeploy or restart.

Motivation

During incidents or maintenance, we sometimes need to temporarily stop the leader from (re)assigning datastream tasks. Previously this required code/config changes and a restart. This change introduces a runtime toggle controlled purely via ZooKeeper.

Behavior

The flag at /{cluster}/assignmentEnabled is interpreted as:

Znode state	Assignment
Absent (default)	Enabled
Value "false" (case-insensitive)	Disabled
Empty / null value	Enabled
Any other value (e.g. "true", "yes")	Enabled

When assignment is disabled, the leader blocks inside handleLeaderDoAssignment and re-checks the flag every 5s (configurable). It resumes automatically once the flag is re-enabled, the znode is deleted, the instance loses leadership, or the thread is interrupted.

Changes

Production code

• KeyBuilder.java — New assignmentEnabled(String cluster) helper returning the /{cluster}/assignmentEnabled znode path, with Javadoc describing the flag semantics.
• ZkAdapter.java — New isAssignmentEnabled() method that reads the flag from ZooKeeper. Defaults to enabled; treats only a case-insensitive "false" as disabled. Safely handles the znode being deleted between the exists() check and read (returnNullIfPathNotExists=true).
• Coordinator.java — Added blockUntilAssignmentEnabled(), invoked at the start of handleLeaderDoAssignment. It loops while the instance is leader and assignment is disabled, sleeping for the configured check period between checks, and logs when assignment is paused and resumed.
• CoordinatorConfig.java — New config brooklin.server.coordinator.assignmentEnabledCheckPeriodMs (default 5000 ms) controlling the re-check interval, with getter getAssignmentEnabledCheckPeriodMs().

Tests

• TestZkAdapter.java — testIsAssignmentEnabled() covering: absent znode (enabled), "false"/"FALSE" (disabled), "true" (enabled), unrecognized value (enabled), empty value (enabled), and znode deletion restoring the default.
• TestCoordinatorConfig.java — testAssignmentEnabledCheckPeriodConfig() validating the default (5s) and an override.

Operational usage

• Disable assignment: create/set /{cluster}/assignmentEnabled to false.
• Re-enable assignment: set the value to anything else (e.g. true) or delete the znode.

Testing Done

• Added unit tests for ZkAdapter.isAssignmentEnabled() (all flag states) and the new CoordinatorConfig property (default + override).
• mint build / mint test on datastream-server.

Risk / Compatibility

Low risk and backward compatible — the flag defaults to enabled when the znode is absent, so existing clusters behave exactly as before until an operator explicitly sets the flag.