refactor(inkless): Move diskless retention/cleanup to a dedicated scheduler and stop it at shutdown [KC-417] - #739
Draft
jeqo wants to merge 2 commits into
Draft
refactor(inkless): Move diskless retention/cleanup to a dedicated scheduler and stop it at shutdown [KC-417]#739jeqo wants to merge 2 commits into
jeqo wants to merge 2 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Moves diskless retention and cleanup off Kafka’s shared scheduler to provide bounded broker shutdown.
Changes:
- Adds a dedicated stoppable background-job scheduler.
- Caps retention work per cycle and adds saturation metrics/configuration.
- Expands lifecycle, scheduling, and configuration tests.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
core/src/main/scala/kafka/server/ReplicaManager.scala |
Owns and shuts down the dedicated scheduler. |
storage/inkless/src/main/java/io/aiven/inkless/config/InklessConfig.java |
Adds the per-cycle partition cap. |
storage/inkless/src/main/java/io/aiven/inkless/control_plane/ControlPlane.java |
Clarifies retention API semantics. |
storage/inkless/src/main/java/io/aiven/inkless/delete/FileCleaner.java |
Adds cooperative shutdown and removes sleeps. |
storage/inkless/src/main/java/io/aiven/inkless/delete/RetentionEnforcer.java |
Adds bounded, stoppable partition processing. |
storage/inkless/src/main/java/io/aiven/inkless/delete/RetentionEnforcementScheduler.java |
Supports limited ready-partition polling. |
storage/inkless/src/main/java/io/aiven/inkless/delete/RetentionEnforcerMetrics.java |
Adds cycle-saturation telemetry. |
storage/inkless/src/test/java/io/aiven/inkless/config/InklessConfigTest.java |
Tests the new configuration. |
storage/inkless/src/test/java/io/aiven/inkless/delete/FileCleanerMockedTest.java |
Tests cleaner shutdown behavior. |
storage/inkless/src/test/java/io/aiven/inkless/delete/RetentionEnforcerTest.java |
Tests caps and cooperative stopping. |
storage/inkless/src/test/java/io/aiven/inkless/delete/RetentionEnforcementSchedulerTest.java |
Tests limited polling and preserved lag. |
docs/inkless/configs.rst |
Documents the new configuration. |
docs/inkless/metrics.rst |
Documents the saturation metric. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+111
to
115
| } else if (closed.get()) { | ||
| // Do not start deleting during shutdown: the worklist is re-derived from the control plane | ||
| // on the next cycle, so dropping it here costs nothing. | ||
| LOGGER.info("Skipping deletion of {} files: file cleaner closed", objectKeyPaths.size()); | ||
| } else { |
Comment on lines
+568
to
+570
| if (!bgScheduler.awaitTermination(ReplicaManager.InklessBackgroundJobShutdownGraceMs, TimeUnit.MILLISECONDS)) { | ||
| warn("Inkless background jobs did not stop within the shutdown grace period; interrupting") | ||
| bgScheduler.shutdownNow() |
| static final String RETENTION_ENFORCEMENT_CYCLE_SATURATED_RATE = "RetentionEnforcementCycleSaturatedRate"; | ||
| // Sustained increments alongside a nonzero RetentionEnforcementScheduleLagMs mean the effective | ||
| // enforcement interval is stretched by the cap, not by slow enforcement. | ||
| private static final String RETENTION_ENFORCEMENT_CYCLE_SATURATED_RATE_DOC = "Total number of retention enforcement cycles that hit retention.enforcement.max.partitions.per.cycle, leaving overdue partitions for the next cycle"; |
| ======================================== ================================================================================================================================================================== | ||
| Attribute name Description | ||
| ======================================== ================================================================================================================================================================== | ||
| RetentionEnforcementCycleSaturatedRate Total number of retention enforcement cycles that hit retention.enforcement.max.partitions.per.cycle, leaving overdue partitions for the next cycle |
Comment on lines
+147
to
+148
| LOGGER.info("Retention enforcement cycle saturated at {} partitions; overdue partitions remain queued", | ||
| maxPartitionsPerCycle); |
RetentionEnforcer and FileCleaner ran on the shared KafkaScheduler, whose shutdown drains without interrupting (awaitTermination 1 day) and is awaited early in BrokerServer.shutdown, so an in-flight cycle blocked broker restart. The shared scheduler gives no way to stop a job: it is not the jobs' owner and cannot be shut down independently of every other background task. Give the two jobs a dedicated 2-thread daemon ScheduledExecutorService, owned by ReplicaManager, so its lifecycle can be driven from ReplicaManager.shutdown before inklessSharedState.close() - the control plane and storage backend are therefore still open while a cycle finishes. The await is bounded (10s): a stalled JDBC/S3 call must not turn into a stalled shutdown. Making the jobs themselves stop at a safe point comes next; this commit only relocates them. cross-tier-log-start-reporter and the consolidated pruner stay on the shared scheduler (out of scope). Scheduling-semantics changes (deliberate): - fixed-rate -> fixed-delay. Retention is unaffected (the real cadence is wall-clock-based in RetentionEnforcementScheduler; the 500ms tick only sets poll granularity), and this drops a pre-existing catch-up burst of no-op polls after a long enforce cycle. File cleaner cadence shifts from start-to-start to end-to-start (adds the run duration; negligible at 5 min) and is safer under stall: a guaranteed gap, no back-to-back object-store hammering. - shared pool -> dedicated fixed 2-thread pool: better isolation from other background tasks, but no longer scales with background.threads (sufficient: two jobs that never self-overlap, and one must not be delayed by the other). runInklessBackgroundJob wraps each job body so an uncaught throwable cannot cancel the periodic task, which a ScheduledExecutorService would otherwise do silently - KafkaScheduler was not exposing the jobs to that.
…oppable at shutdown RetentionEnforcer and FileCleaner ran on the shared KafkaScheduler, whose shutdown drains without interrupting (awaitTermination 1 day) and is awaited early in BrokerServer.shutdown, so an in-flight cycle blocked broker restart. Both jobs are idempotent and re-derive their work from the control plane on the next start, so abandoning a cycle loses no durable progress. Move them to a dedicated 2-thread daemon ScheduledExecutorService and make them stoppable, cooperatively rather than by interrupt: shutdownNow() alone only helps a job that is between units of work, since neither job polled for interruption and a blocked JDBC/S3 call unwinds only when its own timeout fires. Interrupting mid-cycle also surfaced as ERROR logs and error metrics on an otherwise healthy restart, because both run() bodies treat any exception as a fault. close() is the stop signal: - Both jobs get a closed flag. run() after close() is a no-op (the periodic task can fire once between close() and scheduler termination), and close() is idempotent. - RetentionEnforcer checks it at every partition boundary. To have a boundary it now calls enforceRetention once per partition; that costs no extra round trips because EnforceRetentionJob already runs one transaction per partition, and the per-partition control-plane metrics are unchanged. - FileCleaner checks it between the control-plane fetch and the deletion, so shutdown never starts an object-store delete. The worklist is re-derived next cycle. - ReplicaManager.shutdownInklessBackgroundJobs: shutdown() (no new cycles) -> close() both jobs -> bounded awaitTermination -> shutdownNow() only if the grace period elapses, i.e. when a single unit of work is itself stuck. It runs before inklessSharedState.close(), so the control plane and storage stay open while a cycle finishes. On timeout shutdown proceeds anyway: a stalled call must not turn into a stalled shutdown. - runInklessBackgroundJob swallows exceptions so an uncaught throwable cannot cancel the periodic task, and does not log an InterruptedException as a fault. Scheduling-semantics changes (deliberate): - fixed-rate -> fixed-delay. Retention is unaffected (the real cadence is wall-clock-based in RetentionEnforcementScheduler; the 500ms tick only sets poll granularity), and this drops a pre-existing catch-up burst of no-op polls after a long enforce cycle. File cleaner cadence shifts from start-to-start to end-to-start (adds the run duration; negligible at 5 min) and is safer under stall: a guaranteed gap, no back-to-back object-store hammering. - shared pool -> dedicated fixed 2-thread pool: better isolation from other background tasks, but no longer scales with background.threads (sufficient: two jobs that never self-overlap). cross-tier-log-start-reporter and the consolidated pruner stay on the shared scheduler (out of scope). Also bound a run, since a stop point is only useful if it is reached in bounded time: retention.enforcement.max.partitions.per.run (default 1000, 0 = unbounded) caps the partitions one run takes on. Brokers with thousands of diskless partitions could otherwise spend a very long time in one run; it holds no locks, but it occupies its scheduler thread and can only stop between partitions. The cap is applied inside getReadyPartitions, so the excess is not polled and keeps its past-due time at the head of the queue: it is picked up by the next poll 500ms later instead of being rescheduled a full interval out, and RetentionEnforcementScheduleLagMs keeps reporting the real lag. Because the queue is ordered by due time and each broker draws its own randomized schedule, a capped run drains the most overdue first and no subset of partitions is persistently favoured. The default sits well above steady state (a broker drains ~P / (interval * brokerCount) partitions per second), so it only bites on a burst such as a restart or a previously stalled enforcer. New metric RetentionEnforcementRunCappedRate makes a cap-induced schedule lag attributable, as opposed to slow enforcement. Side note added on ControlPlane.enforceRetention: maxBatchesPerRequest is a per-partition cap, which reads ambiguously now that callers pass one partition per call; flagged for renaming to maxBatchesPerPartition in a follow-up (the config key retention.enforcement.max.batches.per.request is public API and stays).
jeqo
force-pushed
the
jeqo/kill-background-on-restart
branch
from
August 12, 2026 16:14
c1713c8 to
bf1bc8a
Compare
jeqo
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A broker restart could be held by an in-flight diskless retention or file-cleanup cycle: both jobs ran on the shared
KafkaScheduler, whose shutdown drains without interrupting (awaitTermination1 day) and is awaited early inBrokerServer.shutdown.This moves them onto a scheduler
ReplicaManagerowns, makes each job stop at a safe point when closed (rather than being interrupted mid-transaction), and bounds how much work one cycle takes on so that stop point is reached in bounded time. Both jobs are idempotent and re-derive their work from the control plane on the next start, so abandoning a cycle loses no durable progress.See the commit messages for the reasoning behind each step.
Commits
refactor(inkless): move retention/cleanup off the shared KafkaScheduler— dedicated 2-thread daemon scheduler owned byReplicaManager, fixed-delay instead of fixed-rate, uncaught-throwable guard.ReplicaManager.scalaonly.refactor(inkless): run retention/cleanup on a dedicated scheduler, stoppable at shutdown—close()becomes the stop signal (checked at partition boundaries in the enforcer, before the delete in the cleaner) withshutdownNow()demoted to a backstop; dropsFileCleaner's in-run sleeps; bounds a cycle.Config
New,
Importance.LOW:retention.enforcement.max.partitions.per.cycle10000disables the cap (previous behaviour). Partitions left over by a saturated cycle stay due and are picked up by the next poll 500ms later, so throughput is not lost.Metrics
New:
RetentionEnforcer.RetentionEnforcementCycleSaturatedRate— cycles that hit the cap, so a cap-induced schedule lag is attributable rather than being confused with slow enforcement. Read it withRetentionEnforcementScheduleLagMs. MirrorsFileCleanerCycleSaturatedRate.Operator notes
enforceRetentioncall per partition. Control-plane transaction count and per-partitionEnforceRetentionQueryTime/QueryRateare unchanged (that job was already per-partition); only the broker-side call count changes.RetentionEnforcementCycleSaturatedRatetogether with a growingRetentionEnforcementScheduleLagMsmeans raisingretention.enforcement.max.partitions.per.cycle.Testing
RetentionEnforcerTest— one call per partition; stop at a partition boundary when closed mid-cycle;run()afterclose()is a no-op;close()idempotent; saturation recorded when the ready set fills the cap, not below it, never when unbounded.RetentionEnforcementSchedulerTest— a limited poll leaves the excess queued and still overdue (lag preserved), the next poll returns it, and the two polls are disjoint.FileCleanerMockedTest— no on-thread sleep on the no-work and error paths (asserted via an unchangedMockTimeclock; fails against pre-PR code); deletion skipped when closed between fetch and delete;run()afterclose()is a no-op;close()idempotent.InklessConfigTest— new config default and explicit value.Not covered: the Scala shutdown sequence itself (close-before-await,
shutdownNowbackstop) has no convenient unit hook.Follow-ups
ControlPlane.enforceRetention'smaxBatchesPerRequestis a per-partition cap, which reads ambiguously now that callers pass one partition per call. Documented in place; renaming it tomaxBatchesPerPartitionis deferred to 1.0, where the public config keyretention.enforcement.max.batches.per.requestcan be renamed along with it.