Skip to content

Use streaming DownloadBlobs RPC for batch blob downloads#6157

Draft
ndr-ds wants to merge 3 commits into
mainfrom
ads-use-streaming-download-blobs
Draft

Use streaming DownloadBlobs RPC for batch blob downloads#6157
ndr-ds wants to merge 3 commits into
mainfrom
ads-use-streaming-download-blobs

Conversation

@ndr-ds

@ndr-ds ndr-ds commented Apr 28, 2026

Copy link
Copy Markdown
Contributor

Motivation

#5976 added a streaming DownloadBlobs RPC endpoint, but no callers used
it: RequestsScheduler::download_blobs was still issuing one unary
download_blob per ID, going through per-blob request-scheduler
bookkeeping and per-blob communicate_concurrently peer fallback. For a
batch of N missing blobs that is N separate round-trips per validator;
on blob-heavy cold syncs this RPC fan-out dominates.

Proposal

Rewire RequestsScheduler::download_blobs to issue a single streaming
request per peer, in shuffled order. Each peer is asked for the IDs
that remain missing after previous peer attempts; partial progress is
preserved across peers when a stream errors mid-way (for example, when
the validator is missing one of the requested IDs).

Why not race peers via communicate_concurrently as before: its
winner-takes-all contract discards a losing attempt's partial progress
(a peer that streamed 90% of the batch and then failed contributes
nothing), duplicates the whole batch transfer once the stagger kicks in,
and returns the first Ok — including Ok(None) ("I don't have it")
from a single validator, which previously failed the whole download even
when other validators had the blob. The sequential remainder loop keeps
partial results, transfers each blob at most once per attempt round, and
only concludes BlobsNotFound after every peer has been asked.

The per-attempt deadline is progress-based: blob_download_timeout
bounds the wait for the stream to open and for each subsequent item, so
a stalled peer is abandoned quickly while a large transfer that is still
making progress is never cancelled. (A whole-attempt cutoff would make
batches larger than ~1s of transfer time permanently undownloadable with
the default setting.) The flag documentation is updated to match.

The requested IDs are deduplicated up front (BTreeSet): the client
aborts the stream if a peer echoes an ID twice, so a caller passing the
same BlobId more than once — which process_certificates does, since
it flat-maps required_blob_ids across certificates without dedup —
would otherwise spuriously fail the whole batch.

A new helper stream_blobs_from_peer drives the stream and updates
per-peer EMA metrics inline, since track_request cannot express
partial-success-then-error in its Result shape. An attempt that ends
because the peer sent an unexpected or duplicate blob counts as a failed
interaction for that peer's EMA.

The now-unused per-blob client path is removed: RemoteNode::download_blob,
RequestKey::Blob, and RequestResult::Blob had no production callers
left. The unary ValidatorNode::download_blob RPC surface itself is
unchanged (still served by validators, still used by test scaffolding).

Returns Ok(None) if any blob is missing across all peers; returns
Err only if every peer attempt fails before yielding a single blob.
This matches the previous end-state semantics.

What this optimizes (and what it does not). This reduces
client→validator RPC fan-out from N unary calls to one streamed call
per validator. It does not reduce validator-side storage work: the
handler still performs N parallel single-partition point reads
(storage.read_blobstry_join_all(read_blob)), because each blob
lives under its own root_key partition and cross-partition reads can't
be coalesced into a single IN query. That storage cost is also largely
irrelevant in practice — the proxy serves these blobs ~99.8% from its
in-memory cache (testnet-conway production, linera_read_blob cache vs
db over 24h). The win therefore scales with the number of blobs per
batch: meaningful for blob-heavy cold syncs (PM market/worker chains),
a no-op for blob-light root chains.

Test Plan

  • No regression on blob-light chains (measured). Fresh-wallet,
    fresh-DB sync of a long testnet_conway root chain (d45db728…, 8000
    blocks), old vs new binary: both ~16 blocks/s, within noise. That sync
    downloaded exactly one blob (the chain's own ChainDescription, at
    genesis) — root chains carry almost no blobs, so the batching has
    nothing to collapse.
  • Where it should help (not yet measured end-to-end). PM
    market/worker chains are blob-heavy: each first-time cross-chain
    message needs the sender's ChainDescription blob, plus app
    Contract/Service bytecode. Production backs this up — linera_write_blob
    is 50k–230k per worker per ~2-day cold-start window. Under the old path
    each of those was a separate unary RPC; they now stream. A head-to-head
    blocks/s number requires PM wallet keys or a local net with a published
    app + cross-chain blob traffic, so it's tracked as follow-up rather than
    blocking this PR.
  • cargo test -p linera-core --lib requests_scheduler (27 tests) and
    cargo clippy --all-targets clean; CLI.md regenerated.

Release Plan

Links

@ndr-ds ndr-ds force-pushed the ads-use-streaming-download-blobs branch from 0058d4d to 80feffe Compare May 7, 2026 17:59
pull Bot pushed a commit to ndr-ds/linera-protocol that referenced this pull request May 16, 2026
…io#6201)

## Motivation

The streaming `DownloadBlobs` RPC added in linera-io#5976 terminates the response
stream on the first missing blob, dropping any blobs that would have
been yielded after it:

- gRPC (`linera-service/src/proxy/grpc.rs`): the handler yields
  `Err(Status::not_found(...))` for a missing blob, which terminates
  the gRPC response stream. Subsequent blobs queued for that batch are
  silently lost.
- Simple transport (`linera-service/src/proxy/main.rs`): the handler
  interleaves `RpcMessage::Error(BlobsNotFound([single_id]))` items
  among `DownloadBlobResponse` items. The current consumer in
  `RemoteNode::download_blobs` propagates the first error via `?`,
  which drops the rest of the batch on the floor.

A batch of N blobs where blob K is missing returns at most K successful
blobs to the caller, even if the validator has all of blobs K+1..N
available. The follow-up linera-io#6157 wires this RPC into the per-peer
fallback path and relies on partial progress being reported correctly,
which the current broken server semantics prevent.

## Proposal

Both server handlers now yield only the blobs that are present, in
request order, and silently skip missing blobs. The client compares
the IDs it received against the IDs it asked for and re-requests any
missing IDs from another peer.

This preserves the existing wire format on both transports.
Storage-level errors that occur before any blob has been yielded
continue to surface as a top-level error on the response (gRPC) or as
an early `RpcMessage::Error` (simple transport).

Also addresses the review comment from afck on linera-io#6105 / linera-io#6155: the
type annotation on `created_blob_ids` in
`Client::process_certificates` moves from the `let` binding to the
`.collect()` call (turbofish style).

## Test Plan

CI.

## Release Plan

- These changes should be backported to the latest `testnet` branch
  alongside linera-io#5976. linera-io#6156 is the WIP backport of linera-io#5976 and should be
  updated to include this fix before being merged.
@ndr-ds ndr-ds force-pushed the ads-use-streaming-download-blobs branch from 80feffe to fbe06d0 Compare June 9, 2026 20:46
@ndr-ds ndr-ds marked this pull request as ready for review June 9, 2026 20:51
@ndr-ds ndr-ds marked this pull request as draft June 9, 2026 20:54
@ndr-ds ndr-ds marked this pull request as ready for review June 9, 2026 21:20
@ndr-ds ndr-ds marked this pull request as draft June 10, 2026 03:02
@ndr-ds ndr-ds force-pushed the ads-use-streaming-download-blobs branch from fbe06d0 to 6076e15 Compare June 10, 2026 14:18
@ndr-ds ndr-ds marked this pull request as ready for review June 10, 2026 15:04
@ndr-ds ndr-ds marked this pull request as draft June 11, 2026 17:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant