IPFS peer-fetch durability layer (1.5a) - #839
Open
vilenarios wants to merge 7 commits into
Open
Conversation
Introduce `X-Ar-Io-Local-Only` — serve a CID strictly from the local Kubo blockstore, never touching public IPFS/DHT. This is the load-bearing primitive for the peer-fetch durability layer (recursion prevention) and for trustless holding measurement (a proxy 404s in local-only mode). - KuboDataSource: new `localOnly` option routed through the Kubo RPC API (:5001) with `offline=true` — raw→block/get, car→dag/export, none→cat. Confirmed against kubo v0.32.1: a local miss returns HTTP 500 with a "not found locally (offline)" body, mapped fast to IpfsNotFoundError; any other non-200 stays an error so a real Kubo fault isn't masked as a miss. Extracts shared acquireSlot / finalizeStream / setupAbort helpers; the gateway (:8080) path is behaviorally unchanged. - IpfsService: threads `localOnly`; isolated from the negative cache (no read, no write) so a probe reflects true current local state and can't poison the normal peer/public fallback; does NOT write the on-disk cache (offline UnixFS bodies lack a sniffed Content-Type) but DOES read it (a cache hit is a legitimate local hold). - routes/ipfs: parse `X-Ar-Io-Local-Only: true` (and `?local=1`); null Range under local-only; bypass payment (intra-fleet) while keeping rate limiting; echo the marker on a hit. - system.ts: pass IPFS_KUBO_API_URL to KuboDataSource. Ships as a pure passthrough — no behavior change unless a caller opts in via the header. Unit tests cover the offline RPC per-format dispatch, the 500→NotFound mapping, the negative-cache carve-out, and the route parsing/echo/payment bypass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7
…ort)
Add tier-2 fleet peer-fetch behind IPFS_PEER_FETCH_ENABLED (default false, pure
passthrough). When a gateway lacks a CID, it fetches a verifiable CAR from a peer
AR.IO gateway that holds it and imports it into local Kubo — which verifies every
block against the CID on dag/import, so a lying/tampered peer's CAR fails to
import and the next peer is tried. Named IPFS content then survives as long as any
fleet gateway holds it.
- IpfsContentSource (new interface) — shared getContent shape (incl. localOnly).
- IpfsPeerDataSource (new, tier 2): selectPeersForKey('ipfs', cid, N) →
GET {peer}/ipfs/{cid}?format=car with X-Ar-Io-Local-Only:true (no recursion) →
byte-capped stream → dag/import (form-data multipart) → verify import via the
response body (mismatch/error signature) AND status → reportSuccess/Failure.
Self-guards under local-only. Supports IPFS_PEER_FETCH_STATIC_PEERS for private
fleets / integration tests.
- SequentialIpfsSource (new composite): local (held?) → peers (acquire) → public.
Serving-vs-acquisition split: normal requests are always SERVED via the Kubo
gateway (correct sniffed Content-Type, local-first); tier 2 only ACQUIRES the
verified CAR into local Kubo, then the gateway serves it — so the offline RPC's
octet-stream default never leaks into browser-facing responses. Local-only runs
tier 1 only (genuinely offline). Blocked/abort/size/range errors do not fall
through. Touches nothing in the ingress/proxy layer (only an internal Kubo RPC
presence check + outbound peer fetches).
- KuboDataSource.isHeldLocally() — cheap offline block/stat presence gate.
- IpfsService.dataSource widened KuboDataSource → IpfsContentSource.
- system.ts wires the composite (peer source only when enabled); config.ts adds
IPFS_PEER_FETCH_{ENABLED,COUNT,TIMEOUT_MS,MAX_CAR_BYTES,STATIC_PEERS};
docker-compose plumbs them to core.
Verified against kubo v0.32.1 through the exact form-data path: good CAR → 200
{"Root":...}; byte-tampered CAR → 500 "mismatch in content integrity". Unit tests
cover happy/tamper→next-peer/local-only-guard/all-fail/no-peers (peer source) and
tier-1-only/passthrough/held-local/peer-acquire/fall-through/blocked-no-fallthrough/
abort (composite). The multi-node integration acceptance gate is the next commit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7
IpfsPeerDataSource imports `form-data` (for the dag/import multipart upload). It was only present transitively via axios, so the production image (`yarn install --production`) did not hoist it to the top level and the container crash-looped with ERR_MODULE_NOT_FOUND at runtime. Declaring it directly fixes the packaged build; the range reuses the existing lockfile entry so the lockfile is unchanged (frozen-install CI stays green). Caught by the multi-node e2e. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7
The acceptance gate for the peer-fetch layer. testcontainers spins up 2 core + 2 kubo nodes plus a request-counting malicious peer. Both kubos run `daemon --offline` (isolated from public IPFS), so a 200 on the cold node proves the content came from the fleet, not the public network. Reuses the built `core` image for the malicious peer (a garbage-CAR HTTP stub) — no extra image pull. Asserts: - core-a serves the seeded CID; core-b does not hold it yet (local-only 404). - core-b peer-fetches the CID from the fleet PAST A LYING PEER (its tampered CAR fails Kubo's dag/import verification, proven by the malicious peer's hit count incrementing) and serves correct bytes — with public IPFS isolated. - the holding signal flips: core-b's local-only probe goes 404 -> 200 after the fetch (the exact signal the observer will use). - recursion guard: a local-only miss returns 404 and never contacts a peer (the malicious peer's hit count is unchanged). Verified green against ipfs/kubo:v0.32.1. Do not enable IPFS_PEER_FETCH_ENABLED in prod until this passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7
Observability for the fleet durability layer (serve-side rate limiting already
applies to local-only via the existing ipfsRateLimiter — only payment is
bypassed, not rate limiting).
New Prometheus counters:
- ipfs_peer_fetch_total{result=success|miss} — a CID acquired from the fleet, or
a full miss (falls through to public IPFS).
- ipfs_peer_fetch_peer_attempts_total{result=success|import_verify_failed|peer_error}
— per-peer outcomes; import_verify_failed isolates a lying/tampered peer
(Kubo's block-integrity check) from transport/non-200/cap errors.
- ipfs_peer_fetch_car_bytes_total — CAR bytes imported into local Kubo.
- ipfs_local_only_serve_total{result=hit|miss} — inbound local-only serves (the
observer holding-probe / peer local-only fetches).
Wired into IpfsPeerDataSource (per-attempt + total + byte tally via the cap
stream, which now also marks block-integrity failures) and routes/ipfs.ts
(local-only hit on success, miss on the 404 branch). Unit tests assert the
success / import_verify_failed / local-only hit+miss counters increment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7
Kubo's dag/import returns HTTP 200 with an EMPTY body when pin-roots is false (the default, IPFS_PIN_ARNS_CONTENT=false) — it only echoes the Root CID when pinning. assertImportOk required "Root" in the body, so EVERY successful import with pinning off was miscounted as a failure: the request fell through to tier 3 and served from the now-local Kubo (functionally correct, but the wrong path, an extra round-trip, and peer-fetch metrics that always read failure/miss). Confirm success with an offline block/stat on the requested root instead of parsing the response. This is authoritative regardless of pin-roots AND rejects a peer that returned a VALID CAR of the WRONG content (all blocks hash, but the requested root isn't among them) — which body-parsing could not catch. Surfaced by the local soak (peer_fetch metrics showed all peer_error / car_bytes 0 despite content being served); the earlier derisk used pin-roots=true, which masked the empty-body case. New unit tests cover the empty-200 success and the wrong-content rejection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7
The IPFS subsystem (including the new peer-fetch durability layer) had no
.env.example coverage. Add a coherent IPFS section: base toggles (IPFS_ENABLED,
IPFS_KUBO_URL/API_URL, IPFS_PIN_ARNS_CONTENT) and the peer-fetch block
(IPFS_PEER_FETCH_{ENABLED,COUNT,TIMEOUT_MS,MAX_CAR_BYTES,STATIC_PEERS}) with
defaults and rationale, so operators can discover and tune the feature.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## ipfs-peer-fetch-review-base #839 +/- ##
===============================================================
+ Coverage 78.73% 79.00% +0.26%
===============================================================
Files 147 149 +2
Lines 56168 57017 +849
Branches 4273 4357 +84
===============================================================
+ Hits 44226 45046 +820
- Misses 11883 11914 +31
+ Partials 59 57 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
IPFS peer-fetch durability layer (Phase 1.5a)
Turns the gateway from a read-only proxy into a verifiable fleet serving layer: when a gateway lacks a named CID, it fetches it from a peer AR.IO gateway that holds it as a verifiable CAR, and Kubo verifies every block against the CID on
dag/import— so a lying/tampered peer is rejected and named IPFS content survives as long as any fleet gateway holds it.Ships dark: everything behind
IPFS_PEER_FETCH_ENABLED=false(a pure passthrough — zero behavior change) until enabled.What's here
localOnlyserve mode —X-Ar-Io-Local-Only: trueserves strictly from the local Kubo blockstore via the RPC API withoffline=true(raw→block/get, car→dag/export, none→cat). A local miss maps fast to 404. This is the recursion guard and the trustless holding-measurement primitive (a proxy 404s in local-only mode). Isolated from the negative cache; payment bypassed (still rate-limited).IpfsPeerDataSource(tier 2) —selectPeersForKey('ipfs', cid, N)→ CAR fetch (local-only, byte-capped) →dag/import(form-data multipart) → verified via offlineblock/stat(not response-body parsing —dag/importreturns an empty body whenpin-roots=false, andblock/statalso rejects a valid-CAR-of-wrong-content peer) → re-serve.reportSuccess/reportFailureper peer.SequentialIpfsSourcecomposite — local (held?) → peers (acquire) → public. Normal requests are always served via the Kubo gateway (correct sniffed Content-Type, local-first); tier 2 only acquires the verified CAR. Local-only runs tier 1 only. Blocked/abort/size/range errors don't fall through. Touches nothing in the ingress/proxy layer.IPFS_PEER_FETCH_{ENABLED,COUNT,TIMEOUT_MS,MAX_CAR_BYTES,STATIC_PEERS}+ docker-compose plumbing +.env.exampledocs.ipfs_peer_fetch_total,ipfs_peer_fetch_peer_attempts_total{success|import_verify_failed|peer_error},ipfs_peer_fetch_car_bytes_total,ipfs_local_only_serve_total{hit|miss}.Validation
test/end-to-end/ipfs-peer-fetch.test.ts): 2 core + 2 kubo (--offline, public IPFS isolated) + a lying-peer stub. Proves peer-fetch past a lying peer, the holding flip (local-only 404→200), and the recursion guard. Green.dag/import-body metric/routing bug (a9e2993b).sha256(bytes) == CID multihash).Not in this PR (by design)
Observer holding-probe rider (separate
ar-io-observerrepo, §7); 1.5b content routing (/ar-io/ipfs/held+ DHT-filtered discovery).🤖 Generated with Claude Code
https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7