Use streaming DownloadBlobs RPC for batch blob downloads#6157
Draft
ndr-ds wants to merge 3 commits into
Draft
Conversation
0058d4d to
80feffe
Compare
pull Bot
pushed a commit
to ndr-ds/linera-protocol
that referenced
this pull request
May 16, 2026
…io#6201) ## Motivation The streaming `DownloadBlobs` RPC added in linera-io#5976 terminates the response stream on the first missing blob, dropping any blobs that would have been yielded after it: - gRPC (`linera-service/src/proxy/grpc.rs`): the handler yields `Err(Status::not_found(...))` for a missing blob, which terminates the gRPC response stream. Subsequent blobs queued for that batch are silently lost. - Simple transport (`linera-service/src/proxy/main.rs`): the handler interleaves `RpcMessage::Error(BlobsNotFound([single_id]))` items among `DownloadBlobResponse` items. The current consumer in `RemoteNode::download_blobs` propagates the first error via `?`, which drops the rest of the batch on the floor. A batch of N blobs where blob K is missing returns at most K successful blobs to the caller, even if the validator has all of blobs K+1..N available. The follow-up linera-io#6157 wires this RPC into the per-peer fallback path and relies on partial progress being reported correctly, which the current broken server semantics prevent. ## Proposal Both server handlers now yield only the blobs that are present, in request order, and silently skip missing blobs. The client compares the IDs it received against the IDs it asked for and re-requests any missing IDs from another peer. This preserves the existing wire format on both transports. Storage-level errors that occur before any blob has been yielded continue to surface as a top-level error on the response (gRPC) or as an early `RpcMessage::Error` (simple transport). Also addresses the review comment from afck on linera-io#6105 / linera-io#6155: the type annotation on `created_blob_ids` in `Client::process_certificates` moves from the `let` binding to the `.collect()` call (turbofish style). ## Test Plan CI. ## Release Plan - These changes should be backported to the latest `testnet` branch alongside linera-io#5976. linera-io#6156 is the WIP backport of linera-io#5976 and should be updated to include this fix before being merged.
80feffe to
fbe06d0
Compare
fbe06d0 to
6076e15
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
#5976 added a streaming
DownloadBlobsRPC endpoint, but no callers usedit:
RequestsScheduler::download_blobswas still issuing one unarydownload_blobper ID, going through per-blob request-schedulerbookkeeping and per-blob
communicate_concurrentlypeer fallback. For abatch of
Nmissing blobs that isNseparate round-trips per validator;on blob-heavy cold syncs this RPC fan-out dominates.
Proposal
Rewire
RequestsScheduler::download_blobsto issue a single streamingrequest per peer, in shuffled order. Each peer is asked for the IDs
that remain missing after previous peer attempts; partial progress is
preserved across peers when a stream errors mid-way (for example, when
the validator is missing one of the requested IDs).
Why not race peers via
communicate_concurrentlyas before: itswinner-takes-all contract discards a losing attempt's partial progress
(a peer that streamed 90% of the batch and then failed contributes
nothing), duplicates the whole batch transfer once the stagger kicks in,
and returns the first
Ok— includingOk(None)("I don't have it")from a single validator, which previously failed the whole download even
when other validators had the blob. The sequential remainder loop keeps
partial results, transfers each blob at most once per attempt round, and
only concludes
BlobsNotFoundafter every peer has been asked.The per-attempt deadline is progress-based:
blob_download_timeoutbounds the wait for the stream to open and for each subsequent item, so
a stalled peer is abandoned quickly while a large transfer that is still
making progress is never cancelled. (A whole-attempt cutoff would make
batches larger than ~1s of transfer time permanently undownloadable with
the default setting.) The flag documentation is updated to match.
The requested IDs are deduplicated up front (
BTreeSet): the clientaborts the stream if a peer echoes an ID twice, so a caller passing the
same
BlobIdmore than once — whichprocess_certificatesdoes, sinceit flat-maps
required_blob_idsacross certificates without dedup —would otherwise spuriously fail the whole batch.
A new helper
stream_blobs_from_peerdrives the stream and updatesper-peer EMA metrics inline, since
track_requestcannot expresspartial-success-then-error in its
Resultshape. An attempt that endsbecause the peer sent an unexpected or duplicate blob counts as a failed
interaction for that peer's EMA.
The now-unused per-blob client path is removed:
RemoteNode::download_blob,RequestKey::Blob, andRequestResult::Blobhad no production callersleft. The unary
ValidatorNode::download_blobRPC surface itself isunchanged (still served by validators, still used by test scaffolding).
Returns
Ok(None)if any blob is missing across all peers; returnsErronly if every peer attempt fails before yielding a single blob.This matches the previous end-state semantics.
What this optimizes (and what it does not). This reduces
client→validator RPC fan-out from
Nunary calls to one streamed callper validator. It does not reduce validator-side storage work: the
handler still performs
Nparallel single-partition point reads(
storage.read_blobs→try_join_all(read_blob)), because each bloblives under its own
root_keypartition and cross-partition reads can'tbe coalesced into a single
INquery. That storage cost is also largelyirrelevant in practice — the proxy serves these blobs ~99.8% from its
in-memory cache (testnet-conway production,
linera_read_blobcache vsdb over 24h). The win therefore scales with the number of blobs per
batch: meaningful for blob-heavy cold syncs (PM market/worker chains),
a no-op for blob-light root chains.
Test Plan
fresh-DB sync of a long testnet_conway root chain (
d45db728…, 8000blocks), old vs new binary: both ~16 blocks/s, within noise. That sync
downloaded exactly one blob (the chain's own
ChainDescription, atgenesis) — root chains carry almost no blobs, so the batching has
nothing to collapse.
market/worker chains are blob-heavy: each first-time cross-chain
message needs the sender's
ChainDescriptionblob, plus appContract/Service bytecode. Production backs this up —
linera_write_blobis 50k–230k per worker per ~2-day cold-start window. Under the old path
each of those was a separate unary RPC; they now stream. A head-to-head
blocks/s number requires PM wallet keys or a local net with a published
app + cross-chain blob traffic, so it's tracked as follow-up rather than
blocking this PR.
cargo test -p linera-core --lib requests_scheduler(27 tests) andcargo clippy --all-targetsclean;CLI.mdregenerated.Release Plan
testnetbranch(companion backport: [testnet] Use streaming DownloadBlobs RPC for batch blob downloads #6476; the prerequisite streaming endpoint Add streaming DownloadBlobs RPC endpoint #5976
is already backported via [backport] Add streaming DownloadBlobs RPC endpoint (#5976) #6156 and deployed).
Links
DownloadBlobsserver endpoint: Add streaming DownloadBlobs RPC endpoint #5976 (main), backported in [backport] Add streaming DownloadBlobs RPC endpoint (#5976) #6156 (testnet_conway)