feat: end-to-end truncation#3132
Open
williamhbaker wants to merge 10 commits into
Open
Conversation
Backfill truncation lets a capture re-backfill a binding and mark the
previously-captured data as stale. It needs new protocol surface:
capture BackfillBegin / BackfillComplete responses, materialize Flush
backfill notifications, runtime Recover.active_backfills and a Persist
active-backfill change, and shuffle Frontier latest_backfill_{begin,
complete} clocks.
Regenerate the Rust and Go bindings; no behavior change yet.
Backfill signals travel as CONTROL-flagged documents within a collection's journals: immediately-committed metadata events that must never be validated, combined, or materialized as content. Define the flag (bit 2, orthogonal to the transaction-semantics bits) and teach every reader that already skips ACKs — derive extract, the combine and derive protocols, dekaf, and the Go subscriber / shuffle / materialize paths — to skip control documents the same way.
A reader must skip the stale pre-backfill prefix of a journal once its binding is re-backfilled. Record the boundary as an `estuary.dev/truncated-at` label: a fixed-width hex Gazette clock that sorts lexically by value and only ever advances. dekaf honors it by raising its effective not_before to the max truncated-at across a collection's partitions.
A materialization that loads a stale row — one published before its binding's backfill boundary — must not reduce it into the fresh, current-generation document for the same key, or the superseded value would leak forward. Add a prior-generation flag to combiner entries. Prior-gen rows sort ahead of the current generation for a key but break the reduction group, so they drain as separate documents (in memory and through a spill file) for the caller to discard. They are marked known-valid, since a stale row need not re-validate.
Fold the begin/complete clocks of CONTROL documents into the checkpoint frontier, keyed by journal_read_suffix. Control docs are immediately-committed OUTSIDE_TXN metadata, so they never append to the log or participate in causal-hint sequencing; their clocks are instead max-reduced per binding across a checkpoint delta and surface on a terminal frontier. The slice reader also honors a journal's truncated-at label, raising each read's effective not_before so the superseded pre-backfill prefix is not shuffled.
A backfill CONTROL document carries only `_meta`, so ordinary partition mapping would route it to one default partition a constrained reader might never read. Broadcast it to every partition journal of the binding instead, under a single shared clock. Also apply the `estuary.dev/truncated-at` label to a binding's partitions, re-listing and CAS-retrying until a full pass lands without losing a race. The label only advances, so an equal or newer value is skipped and re-applies are idempotent.
Backfill state must survive transaction rotation and leader restart. Add three key families, split by task type: `AB:` holds a capture's in-progress active backfills (the clocks that drive the truncated-at journal label), while `BB:` / `BC:` hold a materialization's cumulative begin / complete boundary keyed by journal_read_suffix. The cumulative clocks ride the committed frontier and are folded back onto it during recovery, keeping the durable truncation boundary authoritative across restarts.
A capture connector may signal BackfillBegin / BackfillComplete for a binding. Each control signal must stand alone in its connector checkpoint, so isolate it in its own transaction, publish it as a CONTROL document — the begin's assigned clock is the authoritative truncated_at — and stage the active-backfill change atomically with the commit. A complete recovers its boundary from the begin clock persisted in `AB:`, so an orphaned complete is a no-op. Post-commit, only the full-key-range control producer re-applies the binding's truncated-at journal labels. A sub-range shard that inherited active backfills via a mid-backfill split must not, as it never observes the completing signal that would clear them.
Carry the per-binding truncation boundary from the frontier into the shard. The scan drops source documents older than the boundary, and a loaded row older than it is classified prior-generation so the combiner discards it instead of reducing stale state forward — its current-generation sibling is then stored as an update. Shard zero alone notifies its connector of each begin / complete, once: it is the only shard with a durable, crash-safe notified baseline, and a backfill deletion is destination-global. The leader accumulates the boundary across the session and stamps it on every Load and Persist frontier; the derive and materialize leaders discard the capture-only active_backfills that recovery now surfaces.
Regenerating protobufs reformats the checked-in bindings; run `cargo fmt --all` afterward so the diff carries only real changes.
williamhbaker
force-pushed
the
wb/truncation-v2
branch
from
91c351f to
1be6962
Compare
jgraettinger
reviewed
Jul 9, 2026
jgraettinger
left a comment
jgraettinger
left a comment
Member
There was a problem hiding this comment.
Great! We talked over VC about re-structuring the journal carrier (folding into ACKs with causal hints). A few other comments below.
| // binding's stable journal_read_suffix. | ||
| message BackfillBegin { | ||
| // Binding's journal read suffix, stable across task versions. | ||
| string journal_read_suffix = 1; |
Member
There was a problem hiding this comment.
I've been seeking thus far to push read suffixes down to the very edge, and prefer binding indices everywhere. This is why decode_recover_key_value takes binding_state_keys, for example, doing translation from suffix => binding index as it scans, and we do the reverse translation in encode_frontier. See also JournalFrontier.binding.
Is there a particular reason this needs to be a read suffix, or could it be a binding index instead?
| notified_backfill_begin: Vec<u64>, | ||
| // Backfill-complete clock the connector has been notified of per binding. | ||
| notified_backfill_complete: Vec<u64>, | ||
| // True if this is shard zero, the only shard that emits backfill |
Member
There was a problem hiding this comment.
While this is probably true, I don't think it's the runtime's job to make a determination about whether a connector instance wants a notification delivered or not: rather, we deliver the signal and the connector self-selects whether it will act on it (conditioned on its key range).
Similarly, we deliver Acknowledge to every shard, even if only shard zero will act on it.
| /// documents and is never drained: it transfers front() onto a following | ||
| /// current-generation document of its same key, and is otherwise discarded. | ||
| pub fn add_prior_gen<'s>(&'s self, binding: u16, root: HeapNode<'s>) -> Result<(), Error> { | ||
| self.add_with_flags(binding, root, true, true, true) |
Member
There was a problem hiding this comment.
Reading how this is called from runtime-next's materialize actor, we actually don't know that it's is_valid, right?
| metrics: super::Metrics, | ||
| // When Some, a deadline at which we begin a graceful session stop. | ||
| token_restart_at: Option<tokio::time::Instant>, | ||
| is_control_producer: bool, |
Member
There was a problem hiding this comment.
nit: comment on what this is? It wasn't immediately obvious to me.
(I now understand it's a guard w.r.t. whether the shard is allowed to produce backfill control messages)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
When a capture re-backfills a binding it re-snapshots the source, so the previously-captured documents in the collection become stale. This adds an end-to-end truncation boundary so downstream consumers skip or reset that superseded data instead of carrying it forward.
A capture connector signals
BackfillBegin/BackfillCompleteper binding. The capture publishes each as aCONTROLdocument into the collection's journals and stamps anestuary.dev/truncated-atlabel (the begin clock) on them. Every reader (V1 and V2 alike) treatsCONTROLdocs as metadata rather than content, and a fresh reader skips the stale pre-boundary prefix via the label. Shuffle carries the begin/complete clocks through the checkpoint frontier, and the runtime persists them so the boundary survives transaction rotation and restart. Materializations use the boundary to drop stale source documents, reset stale stored rows, and notify the connector so it can delete superseded destination rows.Scoped to single-shard, full-key-range captures; multi-shard synchronized backfills are future work.
Tested on a local stack through various scenarios with a single shard capture and materialization processing sequences of
BackfillBegin/BackfillCompletemessages.go.modto point at message: add CONTROL flag for out-of-band metadata messages gazette/core#486 pre-merge of this, or as a fast follow.Workflow steps:
(How does one use this feature, and how has it changed)
Documentation links affected:
(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)
Notes for reviewers:
There's a constant stream of merge conflicting things landing on master. I've rebased and fixed these more times than I care to count at this point. I'll do one final pass just before this is ready to merge (and the requisite re-approvals).