Skip to content

Stabilize single-node remote command regression output - #8752

Open
ibrahim halatci (ihalatci) with Copilot wants to merge 3 commits into
mainfrom
copilot/fix-nightly-cassert-failures-another-one
Open

Stabilize single-node remote command regression output#8752
ibrahim halatci (ihalatci) with Copilot wants to merge 3 commits into
mainfrom
copilot/fix-nightly-cassert-failures-another-one

Conversation

Copilot AI commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

DESCRIPTION: Stabilize single-node remote command regression output

single_node_enterprise asserts on the exact set of NOTICE: issuing ... lines produced by a
6-shard multi-shard SELECT. That output is racy by design.

Why

CalculateNewConnectionCount() (adaptive_executor.c:2734) decides on the executor's first cycle to
open 3 additional connections beyond the first:

  • maxNewConnectionCount = targetPoolSize - initiatedConnectionCount (:2772)
  • newConnectionsForReadyTasks = Max(0, readyTaskCount - usableConnectionCount) (:2781)
  • the slow-start cap at :2784 is inside if (ExecutorSlowStartInterval != SLOW_START_DISABLED), and
    the regression harness sets citus.executor_slow_start_interval = 0ms
    (pg_regress_multi.pl:641), so it is skipped
  • Min(4, 3) = 3 at :2794
  • the cost guard at :2843 cannot fire on the first cycle

So 3 extra connections are started on every run, including every green one. The decision is
deterministic; what is not deterministic is whether any of those connections completes TCP connect
plus authentication before the 6 very small sequential queries have already drained. When one wins,
its BEGIN is emitted into the notice stream and the diff appears.

Because there is exactly one last task, a winning connection can insert its BEGIN in exactly one
place — which is why every observed failure diff is byte-identical. That is the signature of this
race, not evidence against one.

Fix

Pin citus.max_adaptive_executor_pool_size to 1 for the duration of the block, so no extra
connection is ever started and the notice stream is fully determined. The SET LOCAL is placed
before citus.log_remote_commands is enabled, so it emits no notice of its own.

+2/−0, zero deletions. The zero-deletion count is the load-bearing verification: if pinning the
pool had altered execution in any way, existing NOTICE: issuing ... lines would have had to change.
They did not. No baseline was re-recorded to make this pass.

The test occupies its own test: line in both enterprise_minimal_schedule and
enterprise_schedule, and there is no parallel-schedule variant, so only external load can win the
race — consistent with it being seen on shared CI runners and rarely locally.

History

This has been failing intermittently for two years. #7671 reports it with the identical hunk header
@@ -465,28 +465,30 @@, and #8348 lists it as item 2 of the unstable-test tracker.

Fixes #7671
Refs #8348 (unstable test list, item 2)
Refs #8776

Co-authored-by: ihalatci <10450368+ihalatci@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix nightly cassert failures from 2026-08-12 Stabilize single-node remote command regression output Aug 12, 2026
@codecov

codecov Bot commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.70%. Comparing base (1c15171) to head (e7af831).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #8752   +/-   ##
=======================================
  Coverage   88.70%   88.70%           
=======================================
  Files         289      289           
  Lines       64888    64889    +1     
  Branches     8181     8180    -1     
=======================================
+ Hits        57558    57559    +1     
- Misses       4965     4967    +2     
+ Partials     2365     2363    -2     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ihalatci
ibrahim halatci (ihalatci) marked this pull request as ready for review August 22, 2026 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky single_node_enterprise test Nightly cassert failures (2026-08-12)

2 participants