Add observability logs for load-based partition assignment#1043
Open
dhananjay-sawner wants to merge 2 commits into
Open
Add observability logs for load-based partition assignment#1043dhananjay-sawner wants to merge 2 commits into
dhananjay-sawner wants to merge 2 commits into
Conversation
a640bab to
b1c81c6
Compare
Adds INFO logs at the critical decision points of throughput-based partition assignment so operators can verify load balancing is working (vs falling back to round-robin), after wiring in additional throughput sources for the assigner. LoadBasedPartitionAssignmentStrategy: - Log whether an assignment is initial (throughput fetched, true load-based balancing) or incremental (throughput not refetched, unassigned partitions placed round-robin). - Log throughput-provider coverage (provider class, cluster name, partition entry count) after a successful fetch. LoadBasedPartitionAssigner: - Log throughput coverage: recognized partitions (matched by an exact partition-level key vs the topic-level fallback) vs unrecognized (round-robin) partitions. - Log the resulting per-task throughput distribution (min/max/avg/total/maxMinusMin in KBps), the direct signal that byte-rate load is balanced across tasks. - When the provider returned data but some partitions matched no key, log a bounded sample (max 10) of unmatched partition names and provider keys so throughput key-format mismatches can be spotted. Logging only; no behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
b1c81c6 to
d365770
Compare
Personal snapshot coordinate so a local build can be published to the shared brooklin Artifactory repo without overwriting the canonical 5.5.23-SNAPSHOT. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds INFO logs at the critical decision points of throughput-based partition assignment to verify that load balancing is actually working — i.e. distinguish true load-based (throughput-balanced) assignment from round-robin fallback, and observe whether the resulting per-task load is balanced.
Today the assigner logs
min/max partitions across tasks, but load-based assignment intentionally makes partition counts uneven while balancing byte rate, so those counts do not prove balance. There is also no signal for whether throughput data was actually used, or whether an assignment fell back to round-robin.Changes (logging only — no behavior change)
LoadBasedPartitionAssignmentStrategyLoadBasedPartitionAssignerrecognizedpartitions split into partition-level vs topic-level fallback, vsunrecognized(round-robin) partitions, plus provider entry count.min/max/avg/total/maxMinusMinin KBps) — the direct signal that byte-rate load is balanced across tasks.Together these answer: which assignment mode ran → did the provider return data → how many partitions got real rates vs round-robin → is the resulting per-task load even.
Testing Done
./gradlew :datastream-server:compileJava :datastream-server:checkstyleMainpass../gradlew :datastream-server:test --tests "*TestLoadBasedPartitionAssigner" --tests "*TestLoadBasedPartitionAssignmentStrategy"— 15/15 pass.