[testnet] Use streaming DownloadBlobs RPC for batch blob downloads#6476
Draft
ndr-ds wants to merge 2 commits into
Draft
[testnet] Use streaming DownloadBlobs RPC for batch blob downloads#6476ndr-ds wants to merge 2 commits into
ndr-ds wants to merge 2 commits into
Conversation
0ecf301 to
8c17e5c
Compare
ndr-ds
added a commit
that referenced
this pull request
Jun 12, 2026
…d toolchain (#6489) ## Motivation The `lint-check-for-outdated-readme` job is now failing deterministically on every `testnet_conway` PR. `cargo install cargo-rdme --locked` resolves to the freshly released cargo-rdme 1.5.1, whose lockfile pulls `zmij 1.0.21`, which uses `hint::select_unpredictable` — stabilized in Rust 1.88. This branch's pinned toolchains (stable 1.86.0, nightly-2025-04-03) cannot build it, so the install step exits 101 before the README check even runs. The same job passed on this branch as recently as 2026-06-11 (PRs #6474, #6476) and broke on 2026-06-12 (#6488), bracketing the upstream release. `main` is unaffected (toolchain 1.95). ## Proposal Pin the tool: `cargo install cargo-rdme --locked --version 1.5.0`, with a comment explaining why. cargo-rdme 1.5.0 declares `rust-version = 1.86.0`. ## Test Plan Verified locally that `cargo install cargo-rdme --locked --version 1.5.0` builds successfully on rustc 1.86.0 (this branch's stable pin) and the binary runs. This PR's own `lint-check-for-outdated-readme` job exercises the pinned install directly, so its CI result is the end-to-end proof. ## Release Plan - CI-only change on `testnet_conway`; nothing to deploy. `main` does not need this (newer toolchain), though the pin could be ported there for reproducibility if desired.
This was referenced Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The streaming
DownloadBlobsvalidator endpoint was backported totestnet_conwayin #6156, but nothing on the client uses it yet — chainsync still downloads blobs one unary
download_blobRPC at a time. This PRbackports the caller so testnet_conway clients (notably the PM workers)
benefit from the streaming endpoint.
Proposal
Client-side only. Cherry-pick of the
mainchange (#6157):download_blobsissues one streamingDownloadBlobsrequest per validator(tried in shuffled order) carrying all still-missing blob IDs, instead of
one unary RPC per blob. Falls back to the next validator only for IDs not
yet returned, preserving partial progress.
blob_download_timeoutboundsthe wait for the stream to open and for each subsequent item, so a stalled
peer is abandoned quickly while a transfer that is still making progress is
never cancelled. (A whole-attempt cutoff would make batches larger than ~1s
of transfer time permanently undownloadable with the default setting.) Flag
docs and
CLI.mdupdated to match.BTreeSet) so a repeated ID can'tabort the stream and fail the batch —
process_certificatesdoes passduplicates, since it flat-maps
required_blob_idsacross certificates.counts as a failed interaction in that peer's EMA metrics.
Unlike #6157, this backport does not remove the now-unused per-blob
client path (
RemoteNode::download_blob,RequestKey::Blob,RequestResult::Blob) — keeping the backport surface minimal.What this optimizes (and what it does not). This reduces client→validator
RPC fan-out from
Nunary calls to one streamed call per validator. It doesnot reduce validator-side storage work: the handler still performs
Nparallel single-partition point reads, because each blob lives under its own
root_keypartition and can't be coalesced into a singleINquery. Thatstorage cost is also largely irrelevant in practice — the proxy serves these
blobs ~99.8% from its in-memory cache (testnet-conway production,
linera_read_blobcache vs db over 24h). The win therefore scales with thenumber of blobs per batch: meaningful for blob-heavy cold syncs (PM
market/worker chains), a no-op for blob-light root chains.
No protocol/storage-format change; the server side is unchanged and already
deployed on testnet_conway.
Test Plan
sync of root chain
d45db728…(8000 blocks), old vs new binary, both~16 blocks/s. That sync pulled exactly one blob (
ChainDescriptionatgenesis), so the batching has nothing to collapse — as expected for root
chains.
linera_write_blobshows PM workers writing 50k–230k blobs per ~2-daycold-start window; each was a separate unary RPC before, and now streams.
A head-to-head blocks/s number requires PM wallet keys or a local net with
a published app + cross-chain blob traffic; tracked as follow-up.
cargo test -p linera-core --lib requests_scheduler(27 tests) andcargo clippyclean;CLI.mdregenerated.Release Plan
testnet_conway(client-side only; the server endpoint [backport] Add streaming DownloadBlobs RPC endpoint (#5976) #6156 is already deployed, so no
validator hotfix is needed).
Links
main: Use streaming DownloadBlobs RPC for batch blob downloads #6157