chore(ci): flush the ClickHouse Kafka engine every 100 ms in dev and CI - #90709
Draft
jose-sequeira wants to merge 2 commits into
Draft
Conversation
Contributor
🤖 CI report🚨 Trunk lane — universal laneThis PR is assigned to the universal lane. It cannot merge in parallel with other PRs, so it can take longer to merge. Ask dev-ex if you think this is wrong. |
Tests that assert on rows landing in ClickHouse spend most of their time waiting for the Kafka engine to flush, so stream_flush_interval_ms is the floor of every such round trip. Nothing pins kafka_flush_interval_ms per table, so this profile value governs all of them. 500 ms to 100 ms takes ingestion-e2e from about 16 s to 11 s locally. The file is dev and CI only; production is unaffected.
jose-sequeira
force-pushed
the
jose-sequeira/clickhouse-test-flush-interval
branch
from
August 28, 2026 08:25
b1ef924 to
9e9874d
Compare
The Node.js change filter listed the compose files but not the ClickHouse config they mount, so a change to docker/clickhouse skipped every Jest job. The Kafka engine settings in that config set how long every ClickHouse-backed test waits, so they must run the suite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every Node.js test that waits for a row to land in ClickHouse pays the Kafka engine's flush interval on each round trip. With
stream_flush_interval_msat 500 in the dev and CI ClickHouse profile, that wait is most of the runtime of the ingestion e2e suites: instrumentingingestion-e2eshowed 85% of its time in test bodies, nearly all of it polling for that flush.posthog/clickhouse/kafka_engine.pypinskafka_flush_interval_msper table, so the profile value indocker/clickhouse/users-dev.xmlgoverns every Kafka engine table.docker-compose.dev.yml, which mounts that same file, so dev and CI share the setting.Changes
stream_flush_interval_msgoes from 500 to 100 in the dev/CI profile.ingestion-e2edrops from about 16 s to 11 s locally, and the other suites that wait on ClickHouse (sessionreplay/consumer.e2e,rerun-paginator,person-state-batch, the CDP ClickHouse tests) benefit the same way.Note
On a dev machine this means smaller, more frequent parts under sustained ingestion traffic and a little more merge load. 200 ms is the conservative fallback if that shows up; the gain scales linearly with the interval.
How did you test this code?
waitForClickHouseKafkaConsumer, so it does not hit that window.Automatic notifications
Docs update
None.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Claude Code (Claude Fable 5) under @jose-sequeira's direction, as the third layer of the stack on #90032 and #90348. Skills invoked:
/stacking-prs,/writing-pr-descriptions. The interval was identified as the per-round-trip floor after per-phase timing of the e2e harness ruled out infra setup as a cost; 100 ms was the value measured, with 200 ms noted as the conservative alternative.