Make rebalance colocation-group representative selection deterministic - #8789
Draft
ibrahim halatci (ihalatci) wants to merge 1 commit into
Draft
Make rebalance colocation-group representative selection deterministic#8789ibrahim halatci (ihalatci) wants to merge 1 commit into
ibrahim halatci (ihalatci) wants to merge 1 commit into
Conversation
NonColocatedDistRelationIdList() builds its list from CitusTableTypeIdList(), which scans pg_dist_partition with an unordered heap scan. The first-wins de-duplication that follows therefore picks an arbitrary table to represent each colocation group, depending on physical tuple order. That makes the rebalance plan non-reproducible for identical cluster state: the representative relation name appears in DEBUG1 output, and the shard IDs emitted in the plan are those of whichever table won. This flaked the nightly cassert run in background_rebalance_parallel_reference_tables, and background_rebalance_parallel is exposed to the same shard-ID instability. Sorting by oid makes the lowest-oid table win, which is creation order and matches what the existing expected output already asserts, so no baselines change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0789ce16-9e4e-4f49-b486-dad8d41d9b87
This was referenced Aug 20, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8789 +/- ##
==========================================
- Coverage 88.70% 88.68% -0.02%
==========================================
Files 289 289
Lines 64889 64890 +1
Branches 8181 8180 -1
==========================================
- Hits 57559 57548 -11
- Misses 4967 4976 +9
- Partials 2363 2366 +3 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a product defect, not a test flake. It was found via a nightly
background_rebalance_parallel_reference_tablesfailure, but the test was reporting the bug correctly.
Root cause
NonColocatedDistRelationIdList()(shard_rebalancer.c:2013-2058) builds its list fromCitusTableTypeIdList()and then keeps the first table seen percolocationId(the dedupcontinueat:2050).CitusTableTypeIdList()scanspg_dist_partitionwith an unorderedsystable_beginscan(
metadata_cache.c:5078-5080). So the representative table for each colocation group is whichevertuple physical order happens to yield — it can change after a HOT update, a vacuum, or any other
storage-level churn, with no schema or data change at all.
Two consequences:
DEBUG1output.The second is the serious one, and it also exposes the sibling test
background_rebalance_parallel,which is only shielded from the first because it never raises
client_min_messagestoDEBUG1. Thatruled out a test-only fix.
Reproduced on demand
Two arms on a fresh 3-node cluster with a forced HOT update on
pg_dist_partition, differing only inthe installed library (verified binary-identical to the intended build at both ends of each arm):
table1_colg1table2_colg1← bug reproducestable1_colg1table1_colg1← fixedMeasured OIDs:
17320 table1_colg1<17326 table2_colg1.Fix
Sort by OID before the de-duplication. Lowest OID is creation order, which is exactly what the
existing committed baselines already assert — so this converts an incidental property into a
guaranteed one rather than changing behaviour.
The sort is in-place on a list built fresh per call, and needs no new
#include.Validation
check-operations— 17/17 pass, with the installed library MD5-verified against the buildtree both before and after the run.
git status --porcelain -- src/test/regress/is empty; no.outandno
.sqlfile is touched by this PR. That is the actual evidence this is a real fix and nottest-shaping.
NonColocatedDistRelationIdList()consumers (:1138,:1179,:1334,:1445,:3787,:4051) — all clean; none depends on the previous arbitrary order.Deliberately not changed
CitusTableTypeIdList()inmetadata_cache.cis left unordered. Sorting at the source would imposecost on every caller; the rebalancer is the one that needs the ordering guarantee.
Refs #8776