fix(inkless:retention): size retention enforcement to keep up with produce - #744
Draft
jeqo wants to merge 1 commit into
Draft
fix(inkless:retention): size retention enforcement to keep up with produce#744jeqo wants to merge 1 commit into
jeqo wants to merge 1 commit into
Conversation
jeqo
force-pushed
the
jeqo/fix-retention-enforcer-defaults
branch
from
August 12, 2026 10:50
58010e4 to
c0a9979
Compare
…oduce The defaults could not keep up with a continuously written partition. Sustained deletion capacity per partition is max.batches.per.request per retention.enforcement.interval.ms, i.e. 1000 batches per 5 min = 3.3 batches/s. A partition gets roughly one batch per produce.commit.interval.ms (250ms) per broker writing to it, so 4 batches/s from a single writer already exceeded the delete rate, and in diskless any broker can accept writes for any partition. The batches table therefore grew without bound on busy partitions while data stayed past its retention. Raise capacity by ~10x: - retention.enforcement.interval.ms: 5 min -> 1 min - retention.enforcement.max.batches.per.request: 1000 -> 2000 That is 33 batches/s per partition, which covers a partition written by ~8 brokers at the default commit interval. The interval was the better of the two levers to move. Capacity is cap/interval, so it could also have come from a 10x cap, but the per-pass boundary scan is O(cap) (enforce_retention_v2 after V22) and the delete holds the log's row: fewer, heavier passes hold locks longer, which is what V21/V22 set out to reduce. More frequent, bounded passes keep the lock hold short. The cost is transaction rate: every due partition costs one transaction whether or not anything expires, so enforcement transactions grow 5x (~330/s -> ~1670/s cluster-wide at 100k partitions). consolidation.cleanup.interval.ms no longer derives its default from the retention interval, so this change does not silently make WAL pruning 5x more frequent too. Its cost profile is different (whole-file scans, not per-partition boundary scans), so it keeps 5 min explicitly. Not addressed here: enforcement cadence per broker is interval * brokerCount while producer fan-in per partition grows with brokerCount, so the shortfall is quadratic in cluster size and no fixed default covers every size. Scaling the batch budget with brokerCount (as the scheduler already does for cadence) is the follow-up.
jeqo
force-pushed
the
jeqo/fix-retention-enforcer-defaults
branch
from
August 12, 2026 16:04
c0a9979 to
924893c
Compare
jeqo
force-pushed
the
jeqo/kill-background-on-restart
branch
from
August 12, 2026 16:14
c1713c8 to
bf1bc8a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defaults could not keep up with a continuously written partition. Sustained deletion capacity per partition is max.batches.per.request per retention.enforcement.interval.ms, i.e. 1000 batches per 5 min = 3.3 batches/s. A partition gets roughly one batch per produce.commit.interval.ms (250ms) per broker writing to it, so 4 batches/s from a single writer already exceeded the delete rate, and in diskless any broker can accept writes for any partition. The batches table therefore grew without bound on busy partitions while data stayed past its retention.
Raise capacity by ~10x:
That is 33 batches/s per partition, which covers a partition written by ~8 brokers at the default commit interval.
The interval was the better of the two levers to move. Capacity is cap/interval, so it could also have come from a 10x cap, but the per-pass boundary scan is O(cap) (enforce_retention_v2 after V22) and the delete holds the log's row: fewer, heavier passes hold locks longer, which is what V21/V22 set out to reduce. More frequent, bounded passes keep the lock hold short. The cost is transaction rate: every due partition costs one transaction whether or not anything expires, so enforcement transactions grow 5x (~330/s -> ~1670/s cluster-wide at 100k partitions).
consolidation.cleanup.interval.ms no longer derives its default from the retention interval, so this change does not silently make WAL pruning 5x more frequent too. Its cost profile is different (whole-file scans, not per-partition boundary scans), so it keeps 5 min explicitly.
Not addressed here: enforcement cadence per broker is interval * brokerCount while producer fan-in per partition grows with brokerCount, so the shortfall is quadratic in cluster size and no fixed default covers every size. Scaling the batch budget with brokerCount (as the scheduler already does for cadence) is the follow-up.