Skip to content

fix(inkless:retention): size retention enforcement to keep up with produce - #744

Draft
jeqo wants to merge 1 commit into
jeqo/kill-background-on-restartfrom
jeqo/fix-retention-enforcer-defaults
Draft

fix(inkless:retention): size retention enforcement to keep up with produce#744
jeqo wants to merge 1 commit into
jeqo/kill-background-on-restartfrom
jeqo/fix-retention-enforcer-defaults

Conversation

@jeqo

@jeqo jeqo commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

The defaults could not keep up with a continuously written partition. Sustained deletion capacity per partition is max.batches.per.request per retention.enforcement.interval.ms, i.e. 1000 batches per 5 min = 3.3 batches/s. A partition gets roughly one batch per produce.commit.interval.ms (250ms) per broker writing to it, so 4 batches/s from a single writer already exceeded the delete rate, and in diskless any broker can accept writes for any partition. The batches table therefore grew without bound on busy partitions while data stayed past its retention.

Raise capacity by ~10x:

  • retention.enforcement.interval.ms: 5 min -> 1 min
  • retention.enforcement.max.batches.per.request: 1000 -> 2000

That is 33 batches/s per partition, which covers a partition written by ~8 brokers at the default commit interval.

The interval was the better of the two levers to move. Capacity is cap/interval, so it could also have come from a 10x cap, but the per-pass boundary scan is O(cap) (enforce_retention_v2 after V22) and the delete holds the log's row: fewer, heavier passes hold locks longer, which is what V21/V22 set out to reduce. More frequent, bounded passes keep the lock hold short. The cost is transaction rate: every due partition costs one transaction whether or not anything expires, so enforcement transactions grow 5x (~330/s -> ~1670/s cluster-wide at 100k partitions).

consolidation.cleanup.interval.ms no longer derives its default from the retention interval, so this change does not silently make WAL pruning 5x more frequent too. Its cost profile is different (whole-file scans, not per-partition boundary scans), so it keeps 5 min explicitly.

Not addressed here: enforcement cadence per broker is interval * brokerCount while producer fan-in per partition grows with brokerCount, so the shortfall is quadratic in cluster size and no fixed default covers every size. Scaling the batch budget with brokerCount (as the scheduler already does for cadence) is the follow-up.

@jeqo
jeqo force-pushed the jeqo/fix-retention-enforcer-defaults branch from 58010e4 to c0a9979 Compare August 12, 2026 10:50
…oduce

The defaults could not keep up with a continuously written partition. Sustained
deletion capacity per partition is max.batches.per.request per
retention.enforcement.interval.ms, i.e. 1000 batches per 5 min = 3.3 batches/s.
A partition gets roughly one batch per produce.commit.interval.ms (250ms) per broker
writing to it, so 4 batches/s from a single writer already exceeded the delete rate,
and in diskless any broker can accept writes for any partition. The batches table
therefore grew without bound on busy partitions while data stayed past its retention.

Raise capacity by ~10x:
- retention.enforcement.interval.ms: 5 min -> 1 min
- retention.enforcement.max.batches.per.request: 1000 -> 2000

That is 33 batches/s per partition, which covers a partition written by ~8 brokers at
the default commit interval.

The interval was the better of the two levers to move. Capacity is cap/interval, so it
could also have come from a 10x cap, but the per-pass boundary scan is O(cap)
(enforce_retention_v2 after V22) and the delete holds the log's row: fewer, heavier
passes hold locks longer, which is what V21/V22 set out to reduce. More frequent,
bounded passes keep the lock hold short. The cost is transaction rate: every due
partition costs one transaction whether or not anything expires, so enforcement
transactions grow 5x (~330/s -> ~1670/s cluster-wide at 100k partitions).

consolidation.cleanup.interval.ms no longer derives its default from the retention
interval, so this change does not silently make WAL pruning 5x more frequent too. Its
cost profile is different (whole-file scans, not per-partition boundary scans), so it
keeps 5 min explicitly.

Not addressed here: enforcement cadence per broker is interval * brokerCount while
producer fan-in per partition grows with brokerCount, so the shortfall is quadratic in
cluster size and no fixed default covers every size. Scaling the batch budget with
brokerCount (as the scheduler already does for cadence) is the follow-up.
@jeqo
jeqo force-pushed the jeqo/fix-retention-enforcer-defaults branch from c0a9979 to 924893c Compare August 12, 2026 16:04
@jeqo
jeqo force-pushed the jeqo/kill-background-on-restart branch from c1713c8 to bf1bc8a Compare August 12, 2026 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant