Skip to content

Add observability logs for load-based partition assignment#1043

Open
dhananjay-sawner wants to merge 2 commits into
linkedin:masterfrom
dhananjay-sawner:load-based-assignment-observability-logs
Open

Add observability logs for load-based partition assignment#1043
dhananjay-sawner wants to merge 2 commits into
linkedin:masterfrom
dhananjay-sawner:load-based-assignment-observability-logs

Conversation

@dhananjay-sawner

@dhananjay-sawner dhananjay-sawner commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds INFO logs at the critical decision points of throughput-based partition assignment to verify that load balancing is actually working — i.e. distinguish true load-based (throughput-balanced) assignment from round-robin fallback, and observe whether the resulting per-task load is balanced.

Today the assigner logs min/max partitions across tasks, but load-based assignment intentionally makes partition counts uneven while balancing byte rate, so those counts do not prove balance. There is also no signal for whether throughput data was actually used, or whether an assignment fell back to round-robin.

Changes (logging only — no behavior change)

LoadBasedPartitionAssignmentStrategy

  • Log whether an assignment is initial (throughput fetched → true load-based balancing) or incremental (throughput not refetched → newly unassigned partitions placed round-robin). Throughput is only fetched when no partitions are currently assigned (STOPPED → READY, first assignment, or restart).
  • After a successful fetch, log throughput-provider coverage: provider class, cluster name, and number of partition entries returned.

LoadBasedPartitionAssigner

  • Log throughput coverage: recognized partitions split into partition-level vs topic-level fallback, vs unrecognized (round-robin) partitions, plus provider entry count.
  • Log the resulting per-task throughput distribution (min/max/avg/total/maxMinusMin in KBps) — the direct signal that byte-rate load is balanced across tasks.

Together these answer: which assignment mode ran → did the provider return data → how many partitions got real rates vs round-robin → is the resulting per-task load even.

Testing Done

  • ./gradlew :datastream-server:compileJava :datastream-server:checkstyleMain pass.
  • ./gradlew :datastream-server:test --tests "*TestLoadBasedPartitionAssigner" --tests "*TestLoadBasedPartitionAssignmentStrategy" — 15/15 pass.

@dhananjay-sawner dhananjay-sawner force-pushed the load-based-assignment-observability-logs branch from a640bab to b1c81c6 Compare July 9, 2026 14:21
Adds INFO logs at the critical decision points of throughput-based
partition assignment so operators can verify load balancing is working
(vs falling back to round-robin), after wiring in additional throughput
sources for the assigner.
LoadBasedPartitionAssignmentStrategy:
- Log whether an assignment is initial (throughput fetched, true
load-based balancing) or incremental (throughput not refetched,
unassigned partitions placed round-robin).
- Log throughput-provider coverage (provider class, cluster name,
partition entry count) after a successful fetch.
LoadBasedPartitionAssigner:
- Log throughput coverage: recognized partitions (matched by an exact
partition-level key vs the topic-level fallback) vs unrecognized
(round-robin) partitions.
- Log the resulting per-task throughput distribution
(min/max/avg/total/maxMinusMin in KBps), the direct signal that
byte-rate load is balanced across tasks.
- When the provider returned data but some partitions matched no key,
log a bounded sample (max 10) of unmatched partition names and
provider keys so throughput key-format mismatches can be spotted.
Logging only; no behavior change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@dhananjay-sawner dhananjay-sawner force-pushed the load-based-assignment-observability-logs branch from b1c81c6 to d365770 Compare July 9, 2026 14:27
Personal snapshot coordinate so a local build can be published to the
shared brooklin Artifactory repo without overwriting the canonical
5.5.23-SNAPSHOT.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant