release-14.0 is missing five bug fixes that are on main (found during 12.1.14 / 13.2 triage)

[ihalatci]

While triaging backports for the 12.1.14 patch release, I found the same four bug fixes missing from release-14.0 that were missing from release-12.1 (shipped in 12.1.14 via #8703–#8706) and from release-13.2 (#8707, PRs #8709–#8712). A fifth, #8465, is missing from all three lines.

Raising this separately because it looks like a cross-branch miss rather than a deliberate 14.x exclusion — and because two of these were explicitly labeled cherry-pick-14, closed, and then never shipped.

Missing commits

Fix	main commit	Merged to main	On main	On release-14.0
#8465 — CREATE EXTENSION IF NOT EXISTS fails for non-owner users (fixes #7091)	fde8ceace	2026-02-10	✅	❌
#8561 — crash on writable standby coordinator writing to a coordinator-local shard (fixes #8426)	8b99e078b	2026-05-06	✅	❌
#8556 — segfault in EXPLAIN with LEFT JOIN + correlated subqueries (fixes #8548)	a3d5708a6	2026-05-07	✅	❌
#8498 — type mismatch when COLLATE is used with a type cast (fixes #8469)	810e7fbb9	2026-05-22	✅	❌
#8594 — CleanupRecordExists uses a stale snapshot (shard-cleanup race)	c41586cc8	2026-06-03	✅	❌

Why this looks unintentional

• release-14.0 branched from main at 803f0ac57 (2026-01-18); all five fixes landed after that point and need explicit backporting.
• The branch is actively maintained — head 8cdb17e25 (2026-07-31), about two months after the newest of these landed on main. It is receiving backports; these five just were not picked up.
• Verified per fix with git merge-base --is-ancestor, a PR-number set-difference over the branch history, and a direct code gate confirming the vulnerable code is still present:
  • Use GetLatestSnapshot in CleanupRecordExists #8594 — shard_cleaner.c:1183 CleanupRecordExists(uint64 recordId), the old signature with no snapshot argument
  • Fix type mismatch when COLLATE is used with type cast in distributed … #8498 — multi_physical_planner.c:2882 still returns a bare CollateExpr, dropping the cast
  • Fix segfault in EXPLAIN with LEFT JOIN and correlated subqueries (#8548) #8556 — distributed_planner.c:449 PlanContainsDistributedSubPlanRTE(List *), no NULL guard
  • Fix crash on writable standby coordinator when writing to a coordinator-local shard (#8426) #8561 — no RecoveryInProgress() anywhere in local_executor.c
  • Fix CREATE EXTENSION IF NOT EXISTS for non-owner users #8465 — extension.c:185,1146,1318 never set if_not_exists

Note on methodology: a naive #\d+ grep over commit subjects gives false positives here, because Citus backport subjects use three different forms ((#8700), (Backport PR8692), (Backport PR#8651)). Matching on (?:#|PR#?)(\d{4,5}) and then confirming with a content diff is what produced the list above.

Severity

Four of the five are crash / wrong-result class:

• Fix segfault in EXPLAIN with LEFT JOIN and correlated subqueries (#8548) #8556 and Fix crash on writable standby coordinator when writing to a coordinator-local shard (#8426) #8561 are hard crashes (segfault / NULL dereference).
• Fix type mismatch when COLLATE is used with type cast in distributed … #8498 produces attribute of type record has wrong type on otherwise valid queries.
• Use GetLatestSnapshot in CleanupRecordExists #8594 can cause an erroneous drop during shard cleanup, because a stale snapshot makes an already-deleted cleanup record still look present.
• Fix CREATE EXTENSION IF NOT EXISTS for non-owner users #8465 blocks non-owner users from CREATE EXTENSION IF NOT EXISTS on an already-installed extension.

The label queue already flagged two of these for 14.x

Issue	State	Fixed on main by	On release-14.0
#8548 — segfault on EXPLAIN, reported against released Citus 14.0.0 / PG18.1	CLOSED	#8556	❌ not delivered
#8426 — error on insert on secondary worker node	CLOSED	#8561	❌ not delivered

Both carry cherry-pick-14 and can have the label cleared once the backports land. (#8470 also carries it, but its fix #8535 is still unmerged upstream, so it stays deferred.)

Backport complexity: lower than 12.1 or 13.2

release-14.0 shares main's PG16/17/18 support range and its pseudoconstant-quals error string byte-for-byte (recursive_planning.c:531-534). So neither of the adaptations the other two lines needed applies here:

• the 12.1 foreach_declared_ptr → foreach_ptr rename — not needed, foreach_declared_ptr exists on 14.0
• the 13.2 five-line rewrite of subquery_in_where_0.out — not needed, the strings already match

All five cherry-pick cleanly with zero conflicts and zero adaptation; git patch-id --stable is identical to upstream for every one. release-14.0 also has no _N numbered expected-output variants at all, so the PG-version-variant trap that bit the 13.2 backport of #8556 cannot occur here.

Suggested action

Cherry-pick all five onto release-14.0, one PR each. Note that release-14.0 is the 14.x line itself (configure.ac = 14.2.0, default_version = 14.2-1; there is no release-14.1/release-14.2 branch), so these would ship in 14.2.0.

[ihalatci]

Backport PRs are open, one per fix, all against release-14.0 and each exactly 1 commit ahead of 8cdb17e25:

Fix	PR	main commit	branch
#8594 shard-cleanup stale snapshot	#8714	c41586cc8	bp-8594-release-14.0
#8498 COLLATE + type cast	#8715	810e7fbb9	bp-8498-release-14.0
#8556 EXPLAIN segfault	#8716	a3d5708a6	bp-8556-release-14.0
#8561 writable-standby crash	#8717	8b99e078b	bp-8561-release-14.0
#8465 CREATE EXTENSION IF NOT EXISTS	#8718	fde8ceace	bp-8465-release-14.0

All five are verbatim cherry-picks — zero conflicts, zero adaptation, and git patch-id --stable identical to the upstream commit in every case.

Validated on PG16.14 with a full A/B of unmodified release-14.0 versus a stack of all five, make install before each leg and an assertion that the installed citus.so matches the worktree build. Both legs: check-multi 190 ok / 0 failed, check-multi-1 207 ok / 0 failed, with byte-identical normalized status sets. check-follower-cluster (for #8561) is 8/8 and identical to main with the same fix applied. PG17 and PG18 are covered by CI.

Once these land, the cherry-pick-14 label can be cleared from #8548 and #8426.

[ihalatci]

CI verdict on the five backport PRs: no regressions

All five finished. Raw failure counts are misleading on this branch, so here is the split by failure class.

Baseline for comparison — scheduled run 30626331970 at 8cdb17e25, which is the exact base commit of all five PRs: 8 failures of 190 jobs = 7 named jobs + 1 of 32 Test flakyness shards. release-14.0 has been red at HEAD since 2026-05-22, independently of this work.

PR	Fix	total fail	Test flakyness	named fail	extra named vs baseline
#8714	#8594	8	0 / 32	8	PG17 check-multi-1-create-citus, Coordinator N-1 PG18 check-enterprise
#8715	#8498	8	1 / 32	7	PG17 check-failure
#8716	#8556	6	0 / 32	6	none — pure subset of baseline
#8717	#8561	40	32 / 32	8	codecov/patch
#8718	#8465	8	1 / 32	7	CDC PG18 installcheck

The named failures on every PR are the baseline 7 — six N-1 mixed-version jobs pinned to lib-v14.1.0 / 14.1-1, plus PG16 check-operations — with at most one or two single-shot flakes on top.

Why #8717 shows 40 failures — root-caused, benign

test-flakyness-pre detects changed regress tests via git diff origin/main... --diff-filter=AM over src/test/regress/sql/*.sql, then test-flakyness fans them across 32 shards and runs run_test.py <test> --repeat 8 --use-whole-schedule-line on PG18.

#8717 is the only one of the five that modifies a .sql regress test (multi_follower_dml.sql). It is therefore the only PR whose flakyness matrix actually executes anything — and multi_follower_dml lives in multi_follower_schedule, which needs a follower/standby cluster the repeat-runner does not provision. So all 32 shards fail. The other four detect no changed test and pass trivially.

Decisive control — upstream #8561 on main, the identical change: 32 of 32 Test flakyness shards failed, its only other failure was codecov/patch, and it was reviewed and merged on 2026-05-06. #8717 reproduces that exactly.

The fix itself is verified correct — every properly-provisioned follower target on #8717 is green:

SUCCESS  Test Citus / PG16 - check-follower-cluster    SUCCESS  PG16 - check-add-backup-node
SUCCESS  Test Citus / PG17 - check-follower-cluster    SUCCESS  PG17 - check-add-backup-node
SUCCESS  Test Citus / PG18 - check-follower-cluster    SUCCESS  PG18 - check-add-backup-node
SUCCESS  Test Citus SQL N-1 / Worker N-1 / Lib N-1 / Coordinator N-1 - PG18 - check-follower-cluster

multi_follower_dml passes on PG16, PG17 and PG18, matching the local PG16.14 A/B (check-follower-cluster 8/8, byte-identical to the same fix applied on main).

Note for anyone repeating this analysis: Test flakyness is change-triggered, so its failures cannot be set-differenced against a baseline push that touched no test files. Split named jobs from flakyness before comparing.

Branch-health items, not caused by these backports

• Test Citus / PG16 - check-operations fails on the untouched baseline. It is a core regression job, not an N-1 variant, and my local A/B covered check-multi / check-multi-1 / check-follower-cluster but not check-operations — so it sits outside the evidence gathered here and deserves a separate look.
• The six N-1 lib-v14.1.0 / 14.1-1 failures are the same version-pinning class that makes release-13.2 red (there as lib-v13.3.0). Largest single source of red on both branches; probably worth its own infra issue.

Bottom line: none of #8714–#8718 introduces a CI regression. Merge decisions remain with the release owner.

[ihalatci]

CI closed out — final results after reruns

All checks on all five PRs have completed. Updating the earlier verdict with the post-rerun numbers.

PR	Backport of	failures	composition
#8714	#8594	6	6 × N-1 — strict subset of baseline
#8715	#8498	6	6 × N-1 — strict subset of baseline
#8716	#8556	6	6 × N-1 — strict subset of baseline
#8717	#8561	40	8 named + 32/32 change-triggered flakyness — root-caused benign (see previous comment)
#8718	#8465	6	6 × N-1 — strict subset of baseline

The remaining 6 on four of the five PRs are exactly the N-1 mixed-version jobs that fail on the untouched baseline 8cdb17e25:

Test Citus Lib N-1 / PG18 - check-multi - lib-v14.1.0
Test Citus Lib N-1 / PG18 - check-multi-1 - lib-v14.1.0
Test Citus Lib N-1 / PG18 - check-add-backup-node - lib-v14.1.0
Test Citus Coordinator N-1 / PG18 - check-multi - 14.1-1 - lib-v14.1.0
Test Citus Coordinator N-1 / PG18 - check-multi-1 - 14.1-1 - lib-v14.1.0
Test Citus Coordinator N-1 / PG18 - check-add-backup-node - 14.1-1 - lib-v14.1.0

Two items from the earlier comment are now resolved

1. Test Citus / PG16 - check-operations is flaky, not deterministically broken. It fails on the untouched baseline, but came back green on rerun on every PR. Still worth the branch owner's attention; not caused by this work.

2. Test Citus CDC / PG18 - installcheck on #8718 — cleared. It was the only per-PR failure that had no counterpart on the baseline, so it got a full investigation:

• The failure was not an assertion. t/006_cdc_schema_change_and_move.pl reported Tests: 2 Failed: 0 / All 2 subtests passed. 15 of 16 CDC tests passed. The run aborted with exit 29 because poll_query_until timed out waiting for replay_lsn ... state = 'streaming', with psql: connection to server at "localhost" port 57637 failed: Connection refused — a worker node not accepting connections.
• No code path connects it to the change. Fix CREATE EXTENSION IF NOT EXISTS for non-owner users #8465 (1a18149f2) touches exactly three files: commands/extension.c, expected/propagate_extension_commands.out, sql/propagate_extension_commands.sql. Nothing under src/test/cdc, nothing in logical decoding, replication or shard move.
• The job is historically flaky on this branch. It failed 2 of the last 8 release-14.0 branch runs with no backports present (0b4d05e9b, 152fdf425) — roughly a 25% rate.
• CDC PG16 and PG17 passed on the same commit. A real CREATE EXTENSION regression would not be PG18-exclusive.
• It came back SUCCESS on rerun with no code change.

Verdict

No backport in #8714–#8718 introduces a CI regression. Four of five are strict subsets of the baseline failure set; #8717's flakyness matrix is change-triggered and reproduces identically on upstream main #8561, which was reviewed and merged.

Note that release-14.0 is not green at HEAD independently of this work — the 6 N-1 lib-v14.1.0 jobs have been failing on the branch since 2026-05-22. That is a pre-existing version-pinning issue and a candidate for its own infra issue (it affects release-13.2 identically, as lib-v13.3.0).

Merging remains the release owner's call.