From ea9fd844d848d6a43a8c41e91e44d5923f91e12e Mon Sep 17 00:00:00 2001 From: root Date: Mon, 20 Apr 2026 23:22:47 +0000 Subject: [PATCH 01/47] feat: add IPFS CID serving via Kubo sidecar (PE-9067) Add opt-in IPFS content serving to AR.IO Gateway. Operators enable IPFS_ENABLED=true and start a Kubo Docker sidecar (--profile ipfs) to serve IPFS content via path-based (/ipfs/{CID}) and subdomain-based ({CID}.{gateway-host}) access patterns. Features: - Kubo HTTP gateway integration with connection + stall timeouts - LRU bounded filesystem cache (streams to disk, not memory) - File-based CID blocklist with hot-reload - Separate rate limiter pool for IPFS traffic - x402 payment protection (same as Arweave data endpoints) - HTTPSIG response signing for verifiable IPFS responses - CIDv0 to CIDv1 base32 redirect (DNS-safe, works with wildcard certs) - Cross-CID redirect for directory listing navigation - Path traversal protection - Docker Compose profile with TCP+UDP swarm ports Default off (IPFS_ENABLED=false). Zero runtime impact when disabled. Designed as foundation for Phase 2 ArNS-to-CID resolution. --- .dockerignore | 2 + CLAUDE.md | 61 +++- docker-compose.yaml | 29 ++ docs/INDEX.md | 6 + docs/envs.md | 23 ++ docs/ipfs-integration.md | 542 ++++++++++++++++++++++++++++++ package.json | 1 + src/app.ts | 21 ++ src/config.ts | 70 ++++ src/ipfs/ipfs-blocklist.ts | 115 +++++++ src/ipfs/ipfs-cache.ts | 235 +++++++++++++ src/ipfs/ipfs-cid.test.ts | 110 ++++++ src/ipfs/ipfs-rate-limiter.ts | 26 ++ src/ipfs/ipfs-service.ts | 201 +++++++++++ src/ipfs/kubo-data-source.test.ts | 105 ++++++ src/ipfs/kubo-data-source.ts | 199 +++++++++++ src/lib/httpsig.ts | 10 +- src/lib/ipfs-cid.ts | 60 ++++ src/metrics.ts | 38 +++ src/middleware/ipfs.ts | 96 ++++++ src/routes/ipfs.ts | 309 +++++++++++++++++ src/system.ts | 55 +++ test-ipfs.sh | 172 ++++++++++ 23 files changed, 2483 insertions(+), 3 deletions(-) create mode 100644 docs/ipfs-integration.md create mode 100644 src/ipfs/ipfs-blocklist.ts create mode 100644 src/ipfs/ipfs-cache.ts create mode 100644 src/ipfs/ipfs-cid.test.ts create mode 100644 src/ipfs/ipfs-rate-limiter.ts create mode 100644 src/ipfs/ipfs-service.ts create mode 100644 src/ipfs/kubo-data-source.test.ts create mode 100644 src/ipfs/kubo-data-source.ts create mode 100644 src/lib/ipfs-cid.ts create mode 100644 src/middleware/ipfs.ts create mode 100644 src/routes/ipfs.ts create mode 100755 test-ipfs.sh diff --git a/.dockerignore b/.dockerignore index 0c3c378ce..e8bd47ffe 100644 --- a/.dockerignore +++ b/.dockerignore @@ -18,3 +18,5 @@ node_modules/ # Test test/ coverage/ +.claude/ +logs/ diff --git a/CLAUDE.md b/CLAUDE.md index 804122f20..1625505dc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,12 +1,50 @@ # CLAUDE.md +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + AR.IO Node — Arweave gateway for accessing and indexing blockchain data, with caching, ANS-104 bundle unbundling, and multi-source data retrieval. +## Tech stack + +- Node.js v20 (see `.nvmrc`), TypeScript strict mode, ESM (`"type": "module"`) +- Test framework: **Node.js native `node:test`** (not Jest/Mocha/Vitest) +- Transpiler: SWC (via ts-node) +- Databases: SQLite (primary) + ClickHouse (analytics/GQL) +- Caching: Redis, LMDB, LRU in-memory +- HTTP: Express +- Observability: OpenTelemetry + Prometheus + Winston + +## Commands + +```bash +# Development +yarn start # Start service (requires .env file) +yarn watch # Start with nodemon (auto-restart on changes) +yarn build # Clean + compile TypeScript (prod) + +# Testing +yarn test # Run all unit tests +yarn test:file src/path/to/file.test.ts # Run a single test file +yarn test:e2e # Run end-to-end tests (in test/ directory) +yarn test:coverage # Run tests with coverage report + +# Linting & quality +yarn lint:check # ESLint check +yarn lint:fix # ESLint auto-fix +yarn duplicate:check # Detect code duplication (jscpd) +yarn deps:check # Detect circular dependencies (madge) + +# Database +yarn db:migrate # Run SQLite migrations +yarn db:dump-test-schemas # Regenerate test SQL schemas after migrations + +# Service management (systemd-based) +yarn service:start / stop / restart / status / logs +``` + ## Discovery points -- Commands — `package.json` scripts (dev, build, service, test, lint, - migrations, duplicate/deps checks) - Documentation index — `docs/INDEX.md` - Env vars — `docs/envs.md` (keep this and `docker-compose.yaml` in sync when adding or removing env vars) @@ -20,6 +58,8 @@ caching, ANS-104 bundle unbundling, and multi-source data retrieval. - `src/system.ts` is the central DI wiring — all services, workers, data sources, resolvers, and lifecycle cleanup handlers are constructed here. +- `src/config.ts` parses all environment variables and exports typed + constants — this is where new env vars are added. - `src/data/` uses composite sources with fallback chains (cache → S3 → AR.IO peers → trusted gateways → Arweave nodes). Retrieval order is configurable via `ON_DEMAND_RETRIEVAL_ORDER` and @@ -30,6 +70,11 @@ caching, ANS-104 bundle unbundling, and multi-source data retrieval. - Filters (`ANS104_UNBUNDLE_FILTER`, `ANS104_INDEX_FILTER`, `WEBHOOK_INDEX_FILTER`) share a composable JSON filter system — see `docs/filters.md`. +- Background workers (`src/workers/`) handle block importing, data importing, + bundle unbundling, verification, and webhooks. Controlled by `START_WRITERS`. +- IPFS serving (`src/ipfs/`) is opt-in via `IPFS_ENABLED`. Uses a Kubo sidecar + for content retrieval with its own cache, rate limiter, and blocklist. Routes + mount before ArNS in `app.ts`. See `docs/ipfs-integration.md`. - Responses include trust headers indicating verification status. ## Gotchas @@ -52,6 +97,18 @@ Always use `createTestLogger()` from `test/test-logger.ts` in test files — never `winston.createLogger({ silent: true })`. Test output is written to `logs/test.log` (overwritten each run), not the console. +### Test imports + +Tests use `node:test` and `node:assert`: + +```typescript +import { describe, it, before, after, mock } from 'node:test'; +import { strict as assert } from 'node:assert'; +``` + +Common test stubs are in `test/stubs.ts`, SQLite helpers in +`test/sqlite-helpers.ts`. + ### Adding a database method Five coordinated edits are required: diff --git a/docker-compose.yaml b/docker-compose.yaml index 0e59621ed..9a7c15fb6 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -124,6 +124,19 @@ services: - RATE_LIMITER_IP_REFILL_PER_SEC=${RATE_LIMITER_IP_REFILL_PER_SEC:-} - RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST=${RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST:-} - RATE_LIMITER_ARNS_ALLOWLIST=${RATE_LIMITER_ARNS_ALLOWLIST:-} + - IPFS_ENABLED=${IPFS_ENABLED:-false} + - IPFS_KUBO_URL=${IPFS_KUBO_URL:-http://kubo:8080} + - IPFS_KUBO_REQUEST_TIMEOUT_MS=${IPFS_KUBO_REQUEST_TIMEOUT_MS:-} + - IPFS_STREAM_STALL_TIMEOUT_MS=${IPFS_STREAM_STALL_TIMEOUT_MS:-} + - IPFS_CACHE_PATH=${IPFS_CACHE_PATH:-} + - IPFS_CACHE_MAX_SIZE_BYTES=${IPFS_CACHE_MAX_SIZE_BYTES:-} + - IPFS_CACHE_CLEANUP_THRESHOLD=${IPFS_CACHE_CLEANUP_THRESHOLD:-} + - IPFS_BLOCKLIST_PATH=${IPFS_BLOCKLIST_PATH:-} + - IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET=${IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET:-} + - IPFS_RATE_LIMITER_IP_REFILL_PER_SEC=${IPFS_RATE_LIMITER_IP_REFILL_PER_SEC:-} + - IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET=${IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET:-} + - IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC=${IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC:-} + - IPFS_MAX_RESPONSE_SIZE_BYTES=${IPFS_MAX_RESPONSE_SIZE_BYTES:-} - NODE_MAX_OLD_SPACE_SIZE=${NODE_MAX_OLD_SPACE_SIZE:-} - ENABLE_FS_HEADER_CACHE_CLEANUP=${ENABLE_FS_HEADER_CACHE_CLEANUP:-} - ON_DEMAND_RETRIEVAL_ORDER=${ON_DEMAND_RETRIEVAL_ORDER:-} @@ -578,6 +591,22 @@ services: networks: - ar-io-network + kubo: + image: ipfs/kubo:${KUBO_IMAGE_TAG:-v0.32.1} + profiles: + - ipfs + restart: unless-stopped + ports: + - '${IPFS_SWARM_PORT:-4001}:4001/tcp' + - '${IPFS_SWARM_PORT:-4001}:4001/udp' + environment: + - IPFS_PROFILE=${IPFS_PROFILE:-server} + volumes: + - ${IPFS_DATA_PATH:-./data/ipfs}:/data/ipfs + networks: + - ar-io-network + command: ['daemon', '--enable-gc'] + autoheal: image: willfarrell/autoheal@sha256:fd2c5500ab9210be9fa0d365162301eb0d16923f1d9a36de887f5d1751c6eb8c network_mode: none diff --git a/docs/INDEX.md b/docs/INDEX.md index b171f48d9..4bd5c90d7 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -23,6 +23,12 @@ Fast, offline lookups for data item to root transaction mappings. | [CDB64 Tools Reference](cdb64-tools.md) | CLI tools for creating indexes | | [CDB64 Format Specification](cdb64-format.md) | Technical file format details | +### IPFS Integration + +| Document | Description | +|----------|-------------| +| [IPFS Integration](ipfs-integration.md) | Architecture, deployment, and configuration for IPFS CID serving | + ### Rate Limiting & Payments | Document | Description | diff --git a/docs/envs.md b/docs/envs.md index 3fef85cf0..d4b7a15dd 100644 --- a/docs/envs.md +++ b/docs/envs.md @@ -357,3 +357,26 @@ ingestion may be partial) for SQLite. | CLICKHOUSE_SQLITE_MIN_HEIGHT_ENABLED | Boolean | false | When true, restrict the SQLite fallback to heights above (ClickHouse max height - buffer) | | CLICKHOUSE_SQLITE_MIN_HEIGHT_BUFFER | Number | 10 | Heights reserved for SQLite near the ClickHouse tip, to guard against partially ingested recent blocks | | CLICKHOUSE_MAX_HEIGHT_CACHE_TTL_SECONDS | Number | 60 | TTL for the cached ClickHouse max-height lookup used by the boundary optimization | + +## IPFS + +When enabled, the gateway can serve IPFS content via `/ipfs/{CID}` path routes +and `{CID}.{root_host}` subdomain routes (same level as ArNS, works with +standard `*.{host}` wildcard TLS certs). Requires a Kubo IPFS node (available +as a Docker Compose sidecar via the `ipfs` profile). + +| ENV_NAME | TYPE | DEFAULT_VALUE | DESCRIPTION | +| ----------------------------------------- | ------- | ------------------- | ------------------------------------------------------------------- | +| IPFS_ENABLED | Boolean | false | Enable IPFS content serving | +| IPFS_KUBO_URL | String | http://kubo:8080 | Kubo HTTP gateway URL | +| IPFS_KUBO_REQUEST_TIMEOUT_MS | Number | 30000 | Connection timeout for Kubo requests (ms) | +| IPFS_STREAM_STALL_TIMEOUT_MS | Number | 30000 | Stall timeout — max time with no data before aborting stream (ms) | +| IPFS_CACHE_PATH | String | data/ipfs-cache | Directory for cached IPFS content | +| IPFS_CACHE_MAX_SIZE_BYTES | Number | 10737418240 (10 GB) | Maximum cache size before LRU eviction | +| IPFS_CACHE_CLEANUP_THRESHOLD | Number | 3600 | Age in seconds before cached files become eviction candidates | +| IPFS_BLOCKLIST_PATH | String | data/ipfs-blocklist.txt | Path to CID blocklist file (one CID per line, hot-reloaded) | +| IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET | Number | 50000 | IPFS rate limiter: max tokens per IP bucket | +| IPFS_RATE_LIMITER_IP_REFILL_PER_SEC | Number | 5 | IPFS rate limiter: token refill rate per second (IP bucket) | +| IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET | Number | 200000 | IPFS rate limiter: max tokens per resource bucket | +| IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC | Number | 20 | IPFS rate limiter: token refill rate per second (resource bucket) | +| IPFS_MAX_RESPONSE_SIZE_BYTES | Number | 1073741824 (1 GB) | Maximum IPFS content size the gateway will serve | diff --git a/docs/ipfs-integration.md b/docs/ipfs-integration.md new file mode 100644 index 000000000..93d3cd9bb --- /dev/null +++ b/docs/ipfs-integration.md @@ -0,0 +1,542 @@ +# IPFS Integration + +This document describes the AR.IO Gateway's IPFS CID serving capability, +covering architecture, request flow, configuration, and deployment. + +## Table of Contents + +- [Overview](#overview) +- [Architecture](#architecture) +- [URL Patterns](#url-patterns) +- [Components](#components) +- [Data Flow](#data-flow) +- [Caching Strategy](#caching-strategy) +- [Security and Moderation](#security-and-moderation) +- [Docker Deployment](#docker-deployment) +- [Configuration Reference](#configuration-reference) +- [Phase 2: ArNS to IPFS Resolution](#phase-2-arns-to-ipfs-resolution) +- [Differences from Arweave Data Serving](#differences-from-arweave-data-serving) +- [Observability](#observability) + +## Overview + +### What This Feature Does + +The IPFS integration allows an AR.IO Gateway to serve IPFS content alongside +Arweave data. A local Kubo IPFS node runs as a Docker sidecar, and the gateway +proxies, caches, and moderates requests to it. This gives gateway operators a +single endpoint for both permanent Arweave data and IPFS-addressed content. + +### Why It Exists + +The Arweave Name System (ArNS) maps human-readable names to content addresses. +Today those addresses are Arweave transaction IDs. Adding IPFS CID support +means ArNS names can also point to IPFS content, bridging the two largest +decentralized storage networks under one naming layer without requiring changes +to the ArNS smart contract. Users get a single URL (e.g., +`my-app.arweave.dev/`) regardless of whether the underlying data lives on +Arweave or IPFS. + +### Phased Approach + +**Phase 1 -- Direct CID Access (current):** Users access IPFS content by +placing a CID in the URL path or subdomain. The gateway validates, rate-limits, +and caches the request, then proxies it to the local Kubo node. No ArNS +resolution is involved. + +**Phase 2 -- ArNS to CID Resolution (future):** ANT (Arweave Name Token) +records will be able to store an IPFS CID in their `transactionId` field. When +the gateway resolves an ArNS name and detects that the resolved ID is a CID +rather than an Arweave transaction ID, it routes the request to the IPFS service +instead of the Arweave data pipeline. This requires no contract changes -- CID +detection happens at the gateway level. + +## Architecture + +### Request Flow Diagram + +``` + +-----------+ + | Client | + +-----+-----+ + | + path: /ipfs/{CID} -or- subdomain: {CID}.ipfs.gateway.io + | + v + +----------+-----------+ + | Express Router / | + | IPFS Middleware | + +----------+-----------+ + | + 1. CID validation + | + v + +----------+-----------+ + | IPFS Blocklist | + | (451 if blocked) | + +----------+-----------+ + | + v + +----------+-----------+ + | IPFS Rate Limiter | + | (429 if exceeded) | + +----------+-----------+ + | + v + +----------+-----------+ + | IPFS Cache | + | (LRU filesystem) | + +----+----------+------+ + | | + cache hit cache miss + | | + v v + +--------+ +--+------------------+ + | Serve | | Kubo Data Source | + | from | | (HTTP fetch with | + | cache | | timeouts) | + +--------+ +--+------------------+ + | + tee stream to + cache + response + | + v + +-----+------+ + | Response | + | (headers, | + | stream) | + +------------+ +``` + +### Component Relationships + +``` +src/system.ts + | + +-- ipfs-service ----+-- kubo-data-source --> Kubo HTTP Gateway (sidecar) + | +-- ipfs-cache --> data/ipfs-cache/ + | +-- ipfs-blocklist --> data/ipfs-blocklist.txt + | +-- ipfs-rate-limiter --> token bucket (memory or Redis) + | + +-- middleware/ipfs (subdomain interception) + +-- routes/ipfs (path-based handlers) +``` + +## URL Patterns + +### Path-Based Access + +| Pattern | Example | Description | +|---------|---------|-------------| +| `/ipfs/{CID}` | `/ipfs/QmYwAPJzv5CZ...` | Fetch a single file or directory root | +| `/ipfs/{CID}/{path}` | `/ipfs/bafybeig.../images/logo.png` | Fetch a file within a UnixFS directory | + +### Subdomain-Based Access + +| Pattern | Example | Description | +|---------|---------|-------------| +| `{CID}.{root_host}` | `bafybeig...arweave.dev` | CIDv1 (base32) in subdomain | +| `{CID}.{root_host}/{path}` | `bafybeig...arweave.dev/index.html` | Subdomain with path | + +Subdomain-based access uses the `ARNS_ROOT_HOST` configuration. The `.ipfs.` +label in the hostname distinguishes IPFS requests from ArNS name resolution, +preventing collisions. For example, `my-app.arweave.dev` resolves as an ArNS +name, while `bafybeig...arweave.dev` resolves as an IPFS CID. + +### CIDv0 to CIDv1 Redirect + +CIDv0 identifiers (base58, starting with `Qm`) cannot be used in subdomains +because they are case-sensitive and DNS is case-insensitive. When a CIDv0 is +detected in a subdomain request, the gateway issues a 301 redirect to the +equivalent CIDv1 (base32) subdomain URL. + +For path-based requests, both CIDv0 and CIDv1 are accepted directly without +redirection. + +## Components + +### `src/lib/ipfs-cid.ts` -- CID Parsing and Conversion + +Utility module for working with IPFS Content Identifiers: + +- **CID validation**: Determines whether a string is a valid CIDv0 or CIDv1. +- **CID conversion**: Converts CIDv0 (base58btc) to CIDv1 (base32) for + subdomain compatibility. +- **CID normalization**: Produces a canonical form used as cache keys and + blocklist entries. + +### `src/ipfs/kubo-data-source.ts` -- Kubo HTTP Client + +Fetches content from the local Kubo IPFS gateway over HTTP: + +- **Connection timeout** (`IPFS_KUBO_REQUEST_TIMEOUT_MS`): Maximum time to + receive response headers from Kubo. Covers DNS, TCP, and TLS handshake plus + time-to-first-byte. +- **Stall timeout** (`IPFS_STREAM_STALL_TIMEOUT_MS`): Maximum idle time during + body streaming. The timer resets on each received chunk, so large but + actively-streaming transfers complete without issue. If no data arrives for + this duration, the stream is aborted. +- Returns a readable stream along with response metadata (content type, content + length). + +### `src/ipfs/ipfs-cache.ts` -- Bounded LRU Filesystem Cache + +A filesystem-backed cache separate from the Arweave contiguous data cache: + +- **Cache directory**: Configurable via `IPFS_CACHE_PATH` (default: + `data/ipfs-cache`). +- **Size limit**: Bounded by `IPFS_CACHE_MAX_SIZE_BYTES` (default: 10 GB). When + the limit is exceeded, the least recently used entries are evicted. +- **Cache key**: SHA-256 hash of the normalized CID concatenated with the + request path. This ensures consistent keys regardless of CID encoding. +- **Metadata**: Each cached entry has a companion `.meta` file storing content + type, content length, and the original CID. Metadata is read on cache hits to + set response headers without re-parsing the content. +- **Cleanup threshold**: `IPFS_CACHE_CLEANUP_THRESHOLD` controls how often (in + seconds) eviction scans run. + +### `src/ipfs/ipfs-blocklist.ts` -- CID Blocklist + +A file-based blocklist for content moderation: + +- **Format**: Plain text file, one CID per line. Lines starting with `#` are + treated as comments. Both CIDv0 and CIDv1 forms are normalized before + matching. +- **Hot-reload**: The blocklist file is watched for changes using filesystem + notifications. Additions and removals take effect without restarting the + gateway. +- **Response**: Blocked CIDs return HTTP 451 (Unavailable For Legal Reasons). + +### `src/ipfs/ipfs-rate-limiter.ts` -- Rate Limiter + +A dedicated token bucket rate limiter for IPFS traffic, separate from the +Arweave rate limiter: + +- **Per-IP bucket**: Controls how much a single client can fetch + (`IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET`, + `IPFS_RATE_LIMITER_IP_REFILL_PER_SEC`). +- **Per-resource bucket**: Controls how much a single CID can be served + globally (`IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET`, + `IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC`). +- Tokens represent bytes (1 token = 1 byte). Bucket sizes and refill rates are + configurable independently from the Arweave rate limiter. +- When limits are exceeded, the gateway returns HTTP 429 (Too Many Requests). + +### `src/ipfs/ipfs-service.ts` -- Service Orchestrator + +Coordinates all IPFS components behind a single interface: + +- Accepts a CID and optional path, runs it through the blocklist, rate limiter, + and cache, and falls back to the Kubo data source on cache miss. +- Handles stream teeing: on a cache miss, the Kubo response stream is split so + one branch writes to cache while the other streams to the client. +- Enforces `IPFS_MAX_RESPONSE_SIZE_BYTES` to prevent serving excessively large + files. +- Registered in `src/system.ts` alongside other data services, guarded by the + `IPFS_ENABLED` flag. + +### `src/middleware/ipfs.ts` -- Subdomain Middleware + +Express middleware that intercepts requests based on the `Host` header: + +- Parses the hostname to detect the `{CID}.{root_host}` pattern. +- Extracts the CID and any path from the URL. +- Performs CIDv0 to CIDv1 redirect when necessary. +- Forwards the extracted CID and path to the IPFS route handler. +- Must run before the ArNS subdomain middleware to prevent the CID from being + misinterpreted as an ArNS name. + +### `src/routes/ipfs.ts` -- Route Handlers + +Express route handlers for path-based IPFS access: + +- `GET /ipfs/:cid` and `GET /ipfs/:cid/*` routes. +- Validates the CID parameter, delegates to the IPFS service, and streams the + response with appropriate headers. +- Sets `Cache-Control`, `ETag`, and `X-Ipfs-Path` headers on successful + responses. + +## Data Flow + +The complete lifecycle of an IPFS request: + +1. **Request arrives.** Either path-based (`/ipfs/{CID}/path`) or + subdomain-based (`{CID}.ipfs.gateway.io/path`). Subdomain requests are + detected by the IPFS middleware and rewritten internally to match the + path-based route. + +2. **CID validation.** The CID string is parsed. If it is not a valid CIDv0 or + CIDv1, the request returns 400 (Bad Request). If it is a CIDv0 in a + subdomain context, a 301 redirect is issued to the CIDv1 equivalent. + +3. **Blocklist check.** The normalized CID is checked against the in-memory + blocklist. If matched, the request returns 451 (Unavailable For Legal + Reasons) immediately. + +4. **Rate limit check.** Both the per-IP and per-resource buckets are checked. + If either bucket is exhausted, the request returns 429 (Too Many Requests) + with a `Retry-After` header. + +5. **Cache lookup.** The cache key (SHA-256 of normalized CID + path) is looked + up in the filesystem cache. On a hit, the cached file and its `.meta` + companion are read and streamed to the client. + +6. **Kubo fetch.** On a cache miss, the service makes an HTTP GET to the local + Kubo gateway (`IPFS_KUBO_URL/ipfs/{CID}/{path}`). The response stream is + tee'd: one branch writes to the cache directory, the other streams directly + to the client. Both the connection timeout and the stall timeout apply during + this phase. + +7. **Response.** The following headers are set on successful responses: + + | Header | Value | Purpose | + |--------|-------|---------| + | `Cache-Control` | `public, max-age=31536000, immutable` | CID content never changes | + | `ETag` | `"{CID}"` | Content-addressed deduplication | + | `X-Ipfs-Path` | `/ipfs/{CID}/{path}` | IPFS ecosystem interop | + | `Content-Type` | Detected by Kubo or from `.meta` | Standard MIME typing | + +## Caching Strategy + +IPFS content is **content-addressed**: a given CID always maps to the same +bytes. This makes cached entries permanently valid -- there is no stale data and +no revalidation needed. + +### Cache Properties + +| Property | Details | +|----------|---------| +| **Location** | `IPFS_CACHE_PATH` (default `data/ipfs-cache`), separate from Arweave data | +| **Key** | SHA-256 hash of normalized CID concatenated with request path | +| **Eviction** | LRU (least recently used) when total size exceeds `IPFS_CACHE_MAX_SIZE_BYTES` | +| **Max size** | `IPFS_CACHE_MAX_SIZE_BYTES` (default 10 GB) | +| **Metadata** | Companion `.meta` JSON files store content type, size, and original CID | +| **Cleanup** | Eviction scans run every `IPFS_CACHE_CLEANUP_THRESHOLD` seconds (default 3600) | +| **Permanence** | No TTL-based expiration; entries are valid forever unless evicted for space | + +### Why a Separate Cache + +The Arweave contiguous data cache is archival in nature -- operators configure +it to retain as much data as possible, potentially without eviction. IPFS +content has different retention characteristics: it requires pinning to persist +on the IPFS network, and gateway operators may not want unbounded IPFS storage. +A separate bounded LRU cache gives operators independent control over Arweave +and IPFS storage budgets. + +## Security and Moderation + +### CID Blocklist + +The blocklist file (`IPFS_BLOCKLIST_PATH`, default `data/ipfs-blocklist.txt`) +allows operators to block specific content: + +``` +# Blocked content - one CID per line +QmBlockedContent1... +bafybeiblockedcontent2... +# Comments start with # +``` + +- CIDs are normalized before matching, so blocking a CIDv0 also blocks its + CIDv1 equivalent and vice versa. +- The file is watched for changes and reloaded automatically. +- Blocked requests return HTTP 451. + +### Rate Limiting + +IPFS rate limiting uses a separate token pool from the Arweave rate limiter. +This prevents IPFS traffic from consuming Arweave rate limit capacity and gives +operators independent tuning for each protocol. + +- **Per-IP limits** prevent a single client from monopolizing bandwidth. +- **Per-resource limits** prevent a single popular CID from consuming all + available throughput. +- Token counts represent bytes of data served. + +### Size Limits + +`IPFS_MAX_RESPONSE_SIZE_BYTES` (default 1 GB) caps the maximum size of a single +IPFS response. Requests for content exceeding this limit are rejected. This +protects the gateway from serving unexpectedly large files that could exhaust +memory or disk. + +### Subdomain Isolation + +Subdomain-based access (`{CID}.ipfs.gateway.io`) provides browser origin +isolation. Each CID gets its own origin, preventing cross-content scripting +attacks. This follows the same security model used by IPFS gateways and the +existing ArNS subdomain sandboxing in the AR.IO Gateway. + +## Docker Deployment + +IPFS runs as an opt-in Docker Compose profile. Enabling it starts a Kubo sidecar +alongside the core gateway services. + +### Docker Compose Profile + +The `ipfs` profile adds a Kubo container: + +```yaml +services: + kubo: + image: ipfs/kubo:latest + profiles: + - ipfs + ports: + - "4001:4001" # Swarm (libp2p) - public, for peer connections + expose: + - "5001" # API - internal only, not exposed to host + - "8080" # Gateway - internal only, used by ar-io-node + volumes: + - ipfs-data:/data/ipfs + environment: + - IPFS_PROFILE=server + command: ["daemon", "--enable-gc"] +``` + +### Port Reference + +| Port | Protocol | Exposure | Purpose | +|------|----------|----------|---------| +| 4001 | TCP/UDP | Public | libp2p swarm -- peer discovery and content exchange | +| 5001 | HTTP | Internal | Kubo API -- used for pinning and node management | +| 8080 | HTTP | Internal | Kubo HTTP Gateway -- used by `kubo-data-source` to fetch content | + +### Volume + +The `ipfs-data` volume persists the Kubo datastore (block storage, peer +identity, configuration). This volume is independent of the gateway's `data/` +directory. + +### Enabling IPFS + +Start the gateway with the IPFS profile: + +```bash +docker compose --profile ipfs up -d +``` + +Or set `IPFS_ENABLED=true` in your `.env` and include `ipfs` in your active +profiles. + +### Garbage Collection + +The Kubo container starts with `--enable-gc`, which periodically removes +unpinned blocks from the local IPFS datastore. This prevents unbounded growth +of the Kubo volume. Content actively being served is protected from GC. + +## Configuration Reference + +All environment variables are opt-in. The feature is disabled by default. + +| Variable | Type | Default | Description | +|----------|------|---------|-------------| +| `IPFS_ENABLED` | Boolean | `false` | Master switch for IPFS support. When false, IPFS routes are not registered and the Kubo sidecar is not required. | +| `IPFS_KUBO_URL` | String | `http://kubo:8080` | Base URL of the local Kubo HTTP Gateway. In Docker, this is the container name. For local development, use `http://localhost:8080`. | +| `IPFS_KUBO_REQUEST_TIMEOUT_MS` | Number | `30000` | Connection timeout in milliseconds for Kubo requests (time to receive response headers). | +| `IPFS_STREAM_STALL_TIMEOUT_MS` | Number | `30000` | Stall timeout in milliseconds for streaming responses from Kubo. Stream is aborted if no data is received for this duration. Actively-streaming transfers are not affected. | +| `IPFS_CACHE_PATH` | String | `data/ipfs-cache` | Directory for the IPFS filesystem cache. Relative paths are resolved from the gateway's working directory. | +| `IPFS_CACHE_MAX_SIZE_BYTES` | Number | `10737418240` (10 GB) | Maximum total size of the IPFS cache directory. LRU eviction begins when this limit is exceeded. | +| `IPFS_CACHE_CLEANUP_THRESHOLD` | Number | `3600` | Interval in seconds between cache eviction scans. | +| `IPFS_BLOCKLIST_PATH` | String | `data/ipfs-blocklist.txt` | Path to the CID blocklist file. The file is watched for changes and reloaded automatically. | +| `IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET` | Number | `50000` | Maximum tokens (bytes) per IP bucket. | +| `IPFS_RATE_LIMITER_IP_REFILL_PER_SEC` | Number | `5` | Tokens added to each IP bucket per second. | +| `IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET` | Number | `200000` | Maximum tokens (bytes) per resource (CID) bucket. | +| `IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC` | Number | `20` | Tokens added to each resource bucket per second. | +| `IPFS_MAX_RESPONSE_SIZE_BYTES` | Number | `1073741824` (1 GB) | Maximum response size for a single IPFS request. Requests exceeding this are rejected. | + +## Phase 2: ArNS to IPFS Resolution + +Phase 2 connects ArNS naming to IPFS content, allowing `my-dapp.arweave.dev` to +serve IPFS-hosted content without the user needing to know the CID. + +### How It Works + +1. **ANT record stores a CID.** The Arweave Name Token contract's + `transactionId` field accepts any string. An ANT owner sets it to an IPFS CID + (e.g., `bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi`) + instead of an Arweave transaction ID. + +2. **Gateway resolves the ArNS name.** The standard ArNS resolution pipeline + fetches the ANT record and extracts the `transactionId`. + +3. **CID detection.** The gateway inspects the resolved ID. If it matches CID + format (multibase-prefixed, valid multicodec), it is classified as an IPFS + CID rather than an Arweave transaction ID. + +4. **Route to IPFS service.** Instead of fetching from the Arweave data pipeline + (cache, S3, peers, chunks), the gateway routes the request to the IPFS + service, which follows the same blocklist, rate limit, cache, and Kubo fetch + pipeline described above. + +### Key Design Decisions + +- **No contract changes.** CID detection happens entirely at the gateway level. + The ANT contract's `transactionId` field is a free-form string, so it already + accepts CIDs. +- **Transparent to users.** A user visiting `my-dapp.arweave.dev` does not need + to know whether the content is on Arweave or IPFS. The URL is the same either + way. +- **Owner-controlled.** The ANT owner decides where content lives by setting + the `transactionId` to either an Arweave TX ID or an IPFS CID. Switching + between storage backends is a single contract interaction. +- **Caching and moderation apply.** All Phase 1 protections (blocklist, rate + limits, cache) apply to ArNS-resolved IPFS content. + +## Differences from Arweave Data Serving + +| Aspect | Arweave | IPFS | +|--------|---------|------| +| **Addressing** | Transaction ID (43-char base64url) | CID (variable length, base32 or base58) | +| **Permanence** | Guaranteed by protocol (incentivized storage) | Requires pinning; content disappears if unpinned | +| **Path resolution** | Manifest JSON parsed by the gateway | UnixFS directories resolved by Kubo | +| **Verification** | Merkle proofs verified by the gateway | Block hashes verified internally by Kubo | +| **Caching** | Archival (operators retain data long-term, often without eviction) | LRU with bounded size (eviction when full) | +| **Cache-Control** | Varies by verification status and data source trust | `immutable` with 1-year max-age (content-addressed = never changes) | +| **Rate limiting** | Shared Arweave token bucket | Separate IPFS token bucket | +| **Data source** | Multi-source fallback chain (cache, S3, peers, gateways, Arweave nodes) | Single source: local Kubo node | +| **Upstream network** | Arweave protocol (block weave, mining incentives) | IPFS/libp2p (DHT, Bitswap) | +| **Moderation** | Transaction-level blocklist | CID-level blocklist (with CIDv0/v1 normalization) | + +## Observability + +### Prometheus Metrics + +| Metric | Type | Labels | Description | +|--------|------|--------|-------------| +| `ipfs_requests_total` | Counter | `status`, `method` | Total IPFS requests by HTTP status and method | +| `ipfs_cache_hit_total` | Counter | -- | Cache hits | +| `ipfs_cache_miss_total` | Counter | -- | Cache misses | +| `ipfs_content_size_bytes` | Histogram | -- | Distribution of served content sizes | +| `ipfs_request_duration_seconds` | Histogram | `source` (`cache`, `kubo`) | Request latency by data source | +| `ipfs_blocked_total` | Counter | -- | Requests blocked by the CID blocklist | + +### Structured Logging + +Each IPFS component creates a Winston child logger with a component-specific +label (e.g., `ipfs-service`, `kubo-data-source`, `ipfs-cache`). Log output +follows the gateway's standard JSONL format and is written to the same log +destination as all other gateway logs. Key log events: + +- `ipfs-service`: Request start, cache hit/miss, Kubo fetch start/complete, + errors. +- `kubo-data-source`: HTTP request details, timeout events, stream stalls. +- `ipfs-cache`: Eviction events, write errors, cleanup scan results. +- `ipfs-blocklist`: File reload events, blocked CID matches. + +### OpenTelemetry Tracing + +IPFS requests generate OpenTelemetry spans that are written to +`logs/otel-spans.jsonl` alongside Arweave request spans. The span hierarchy: + +``` +ipfs.request (root span) + +-- ipfs.blocklist.check + +-- ipfs.ratelimit.check + +-- ipfs.cache.lookup + +-- ipfs.kubo.fetch (only on cache miss) + +-- ipfs.cache.write (only on cache miss) +``` + +Spans include attributes for the CID, path, cache hit/miss status, response +size, and Kubo fetch duration. diff --git a/package.json b/package.json index c48f1424c..91300f497 100644 --- a/package.json +++ b/package.json @@ -58,6 +58,7 @@ "memoizee": "^0.4.17", "middleware-async": "^1.4.0", "msgpackr": "^1.11.5", + "multiformats": "^13.1.0", "node-cache": "^5.1.2", "opossum": "^8.4.0", "opossum-prometheus": "^0.4.0", diff --git a/src/app.ts b/src/app.ts index a8d1187e7..6442a690f 100644 --- a/src/app.ts +++ b/src/app.ts @@ -25,6 +25,8 @@ import { datasetsRouter } from './routes/datasets.js'; import * as system from './system.js'; import { createX402Router } from './routes/x402.js'; import { createRateLimitRouter } from './routes/rate-limit.js'; +import { createIpfsSubdomainMiddleware } from './middleware/ipfs.js'; +import { createIpfsRouter, createIpfsHandler } from './routes/ipfs.js'; // Initialize DNS resolution for preferred chunk GET nodes (non-fatal on failure) try { @@ -136,6 +138,25 @@ if (system.rateLimiter !== undefined) { }), ); } +// IPFS routes — must be before ArNS to intercept {CID}.{host} subdomains +if (config.IPFS_ENABLED && system.ipfsService !== undefined) { + const ipfsHandler = createIpfsHandler({ + log, + ipfsService: system.ipfsService, + rateLimiter: system.ipfsRateLimiter, + paymentProcessor: system.paymentProcessor, + }); + app.use(createIpfsSubdomainMiddleware({ ipfsHandler })); + app.use( + createIpfsRouter({ + log, + ipfsService: system.ipfsService, + rateLimiter: system.ipfsRateLimiter, + paymentProcessor: system.paymentProcessor, + }), + ); +} + app.use(arnsRouter); app.use(openApiRouter); app.use(arIoRouter); diff --git a/src/config.ts b/src/config.ts index ef648587d..72f70c80a 100644 --- a/src/config.ts +++ b/src/config.ts @@ -2143,3 +2143,73 @@ if (ENABLE_SAMPLING_DATA_SOURCE) { ); } } + +// +// IPFS +// + +export const IPFS_ENABLED = + env.varOrDefault('IPFS_ENABLED', 'false') === 'true'; + +export const IPFS_KUBO_URL = env.varOrDefault( + 'IPFS_KUBO_URL', + 'http://kubo:8080', +); + +export const IPFS_KUBO_REQUEST_TIMEOUT_MS = +env.varOrDefault( + 'IPFS_KUBO_REQUEST_TIMEOUT_MS', + '30000', +); + +export const IPFS_STREAM_STALL_TIMEOUT_MS = +env.varOrDefault( + 'IPFS_STREAM_STALL_TIMEOUT_MS', + '30000', +); + +export const IPFS_CACHE_PATH = env.varOrDefault( + 'IPFS_CACHE_PATH', + 'data/ipfs-cache', +); + +export const IPFS_CACHE_MAX_SIZE_BYTES = +env.varOrDefault( + 'IPFS_CACHE_MAX_SIZE_BYTES', + `${10 * 1024 * 1024 * 1024}`, // 10 GB +); + +// Reserved for future cache cleanup worker. Currently unused — LRU eviction +// in the in-memory index handles cache bounding. After restarts, disk usage +// may temporarily exceed IPFS_CACHE_MAX_SIZE_BYTES until the index rebuilds. +export const IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS = +env.varOrDefault( + 'IPFS_CACHE_CLEANUP_THRESHOLD', + '3600', +); + +export const IPFS_BLOCKLIST_PATH = env.varOrDefault( + 'IPFS_BLOCKLIST_PATH', + 'data/ipfs-blocklist.txt', +); + +export const IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET = +env.varOrDefault( + 'IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET', + '50000', +); + +export const IPFS_RATE_LIMITER_IP_REFILL_PER_SEC = +env.varOrDefault( + 'IPFS_RATE_LIMITER_IP_REFILL_PER_SEC', + '5', +); + +export const IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET = +env.varOrDefault( + 'IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET', + '200000', +); + +export const IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC = +env.varOrDefault( + 'IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC', + '20', +); + +export const IPFS_MAX_RESPONSE_SIZE_BYTES = +env.varOrDefault( + 'IPFS_MAX_RESPONSE_SIZE_BYTES', + `${1 * 1024 * 1024 * 1024}`, // 1 GB +); diff --git a/src/ipfs/ipfs-blocklist.ts b/src/ipfs/ipfs-blocklist.ts new file mode 100644 index 000000000..f20ba355b --- /dev/null +++ b/src/ipfs/ipfs-blocklist.ts @@ -0,0 +1,115 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import fs from 'node:fs'; +import winston from 'winston'; +import { watch, FSWatcher } from 'chokidar'; + +import { cidToV1Base32, isValidCid } from '../lib/ipfs-cid.js'; + +export class IpfsBlocklist { + private log: winston.Logger; + private filePath: string; + private blockedCids: Set = new Set(); + private watcher: FSWatcher | null = null; + private reloadTimer: NodeJS.Timeout | null = null; + + constructor({ log, filePath }: { log: winston.Logger; filePath: string }) { + this.log = log.child({ class: this.constructor.name }); + this.filePath = filePath; + } + + async load(): Promise { + try { + const content = await fs.promises.readFile(this.filePath, 'utf-8'); + const newSet = new Set(); + + for (const line of content.split('\n')) { + const trimmed = line.trim(); + if (trimmed === '' || trimmed.startsWith('#')) continue; + + if (isValidCid(trimmed)) { + // Normalize to CIDv1 base32 for consistent matching + try { + newSet.add(cidToV1Base32(trimmed)); + } catch { + this.log.warn('Failed to normalize CID in blocklist', { + cid: trimmed, + }); + } + } else { + this.log.warn('Invalid CID in blocklist, skipping', { + line: trimmed, + }); + } + } + + this.blockedCids = newSet; + this.log.info('IPFS blocklist loaded', { count: newSet.size }); + } catch (error: any) { + if (error.code === 'ENOENT') { + this.log.debug('IPFS blocklist file not found, no CIDs blocked', { + filePath: this.filePath, + }); + this.blockedCids = new Set(); + } else { + this.log.error('Failed to load IPFS blocklist', { + message: error.message, + }); + } + } + } + + isBlocked(cidString: string): boolean { + try { + const normalized = cidToV1Base32(cidString); + return this.blockedCids.has(normalized); + } catch { + return false; + } + } + + startWatching(): void { + this.watcher = watch(this.filePath, { + ignoreInitial: true, + awaitWriteFinish: { stabilityThreshold: 1000 }, + }); + + this.watcher.on('change', () => { + this.log.info('IPFS blocklist file changed, reloading'); + this.scheduleReload(); + }); + + this.watcher.on('add', () => { + this.log.info('IPFS blocklist file created, loading'); + this.scheduleReload(); + }); + } + + private scheduleReload(): void { + if (this.reloadTimer) { + clearTimeout(this.reloadTimer); + } + this.reloadTimer = setTimeout(() => { + this.load().catch((error) => { + this.log.error('Failed to reload IPFS blocklist', { + message: error.message, + }); + }); + }, 1000); + } + + stop(): void { + if (this.watcher) { + this.watcher.close(); + this.watcher = null; + } + if (this.reloadTimer) { + clearTimeout(this.reloadTimer); + this.reloadTimer = null; + } + } +} diff --git a/src/ipfs/ipfs-cache.ts b/src/ipfs/ipfs-cache.ts new file mode 100644 index 000000000..ccf6be052 --- /dev/null +++ b/src/ipfs/ipfs-cache.ts @@ -0,0 +1,235 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import crypto from 'node:crypto'; +import fs from 'node:fs'; +import { Readable } from 'node:stream'; +import { pipeline } from 'node:stream/promises'; +import winston from 'winston'; +import { LRUCache } from 'lru-cache'; + +interface CacheEntry { + size: number; + contentType: string; +} + +export class IpfsFsCache { + private log: winston.Logger; + private baseDir: string; + private index: LRUCache; + + constructor({ + log, + basePath, + maxSizeBytes, + }: { + log: winston.Logger; + basePath: string; + maxSizeBytes: number; + }) { + this.log = log.child({ class: this.constructor.name }); + this.baseDir = basePath; + this.index = new LRUCache({ + maxSize: maxSizeBytes, + sizeCalculation: (entry) => entry.size, + dispose: (_entry, key) => { + // Delete file from disk when evicted from LRU + const dataPath = this.dataPath(key); + const metaPath = this.metaPath(key); + fs.promises.unlink(dataPath).catch(() => {}); + fs.promises.unlink(metaPath).catch(() => {}); + this.log.debug('Evicted IPFS cache entry', { key }); + }, + }); + } + + private cacheKey(cidString: string, path?: string): string { + const raw = path ? `${cidString}/${path}` : cidString; + return crypto.createHash('sha256').update(raw).digest('hex'); + } + + private dataDir(key: string): string { + const prefix = `${key.substring(0, 2)}/${key.substring(2, 4)}`; + return `${this.baseDir}/data/${prefix}`; + } + + private dataPath(key: string): string { + return `${this.dataDir(key)}/${key}`; + } + + private metaPath(key: string): string { + return `${this.dataDir(key)}/${key}.meta`; + } + + private tempDir(): string { + return `${this.baseDir}/tmp`; + } + + private createTempPath(): string { + return `${this.tempDir()}/${crypto.randomBytes(16).toString('hex')}`; + } + + async has(cidString: string, path?: string): Promise { + const key = this.cacheKey(cidString, path); + if (this.index.has(key)) { + return true; + } + // Check disk in case index was lost (restart) + try { + await fs.promises.access(this.dataPath(key), fs.constants.F_OK); + // Rebuild index entry from meta file + const meta = await this.readMeta(key); + if (meta) { + this.index.set(key, meta); + return true; + } + } catch { + // Not found + } + return false; + } + + async get( + cidString: string, + path?: string, + ): Promise<{ stream: Readable; size: number; contentType: string } | undefined> { + const key = this.cacheKey(cidString, path); + let entry = this.index.get(key); + + // Rebuild index from disk if entry is missing (e.g., after restart) + if (!entry) { + try { + await fs.promises.access(this.dataPath(key), fs.constants.F_OK); + const meta = await this.readMeta(key); + if (meta) { + this.index.set(key, meta); + entry = meta; + } + } catch { + // File not on disk — true cache miss + } + } + + if (entry) { + const dataPath = this.dataPath(key); + try { + await fs.promises.access(dataPath, fs.constants.F_OK); + const stream = fs.createReadStream(dataPath); + return { + stream, + size: entry.size, + contentType: entry.contentType, + }; + } catch (error: any) { + this.log.error('Failed to read cached IPFS content', { + key, + message: error.message, + }); + this.index.delete(key); + } + } + return undefined; + } + + async put( + cidString: string, + stream: Readable, + size: number, + contentType: string, + path?: string, + ): Promise { + const key = this.cacheKey(cidString, path); + + try { + await fs.promises.mkdir(this.tempDir(), { recursive: true }); + const tempPath = this.createTempPath(); + const writeStream = fs.createWriteStream(tempPath); + + await pipeline(stream, writeStream); + + // Move to final location + const dataDir = this.dataDir(key); + await fs.promises.mkdir(dataDir, { recursive: true }); + await fs.promises.rename(tempPath, this.dataPath(key)); + + // Write metadata + const meta: CacheEntry = { size, contentType }; + await fs.promises.writeFile( + this.metaPath(key), + JSON.stringify(meta), + 'utf-8', + ); + + // Update index + this.index.set(key, meta); + + this.log.debug('Cached IPFS content', { cidString, path, key, size }); + } catch (error: any) { + this.log.error('Failed to cache IPFS content', { + cidString, + path, + message: error.message, + }); + } + } + + /** + * Finalize a cache entry from an already-written temp file. + * Used by the streaming cache writer to avoid double-copying data. + */ + async putFromFile( + cidString: string, + tempPath: string, + size: number, + contentType: string, + path?: string, + ): Promise { + const key = this.cacheKey(cidString, path); + + try { + const dataDir = this.dataDir(key); + await fs.promises.mkdir(dataDir, { recursive: true }); + await fs.promises.rename(tempPath, this.dataPath(key)); + + const meta: CacheEntry = { size, contentType }; + await fs.promises.writeFile( + this.metaPath(key), + JSON.stringify(meta), + 'utf-8', + ); + + this.index.set(key, meta); + + this.log.debug('Cached IPFS content from file', { + cidString, + path, + key, + size, + }); + } catch (error: any) { + this.log.error('Failed to finalize cached IPFS content', { + cidString, + path, + message: error.message, + }); + // Clean up temp file on failure + fs.promises.unlink(tempPath).catch(() => {}); + } + } + + getCachePath(): string { + return this.baseDir; + } + + private async readMeta(key: string): Promise { + try { + const raw = await fs.promises.readFile(this.metaPath(key), 'utf-8'); + return JSON.parse(raw) as CacheEntry; + } catch { + return null; + } + } +} diff --git a/src/ipfs/ipfs-cid.test.ts b/src/ipfs/ipfs-cid.test.ts new file mode 100644 index 000000000..a0aa35934 --- /dev/null +++ b/src/ipfs/ipfs-cid.test.ts @@ -0,0 +1,110 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { describe, it } from 'node:test'; +import { strict as assert } from 'node:assert'; + +import { + parseCid, + isValidCid, + isCidV0, + cidToV1Base32, + cidToString, +} from '../lib/ipfs-cid.js'; + +describe('ipfs-cid utilities', () => { + // Known CIDv1 base32 (dag-pb, sha2-256) + const CIDV1_BASE32 = + 'bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi'; + + // Known CIDv0 (Qm prefix, base58btc) + const CIDV0 = 'QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG'; + + describe('parseCid', () => { + it('parses a valid CIDv1 base32 string', () => { + const cid = parseCid(CIDV1_BASE32); + assert.notEqual(cid, null); + assert.equal(cid!.version, 1); + }); + + it('parses a valid CIDv0 string', () => { + const cid = parseCid(CIDV0); + assert.notEqual(cid, null); + assert.equal(cid!.version, 0); + }); + + it('returns null for invalid strings', () => { + assert.equal(parseCid('not-a-cid'), null); + assert.equal(parseCid(''), null); + assert.equal(parseCid('abc123'), null); + }); + + it('returns null for Arweave TX IDs (43-char base64url)', () => { + // A typical Arweave TX ID — not a valid CID + assert.equal( + parseCid('TB2wJyKrPnkAW79DAwlJYwpgdHKpijEJWQfcwX715Co'), + null, + ); + }); + }); + + describe('isValidCid', () => { + it('returns true for valid CIDs', () => { + assert.equal(isValidCid(CIDV1_BASE32), true); + assert.equal(isValidCid(CIDV0), true); + }); + + it('returns false for invalid strings', () => { + assert.equal(isValidCid('not-a-cid'), false); + assert.equal(isValidCid(''), false); + }); + }); + + describe('isCidV0', () => { + it('returns true for CIDv0', () => { + const cid = parseCid(CIDV0)!; + assert.equal(isCidV0(cid), true); + }); + + it('returns false for CIDv1', () => { + const cid = parseCid(CIDV1_BASE32)!; + assert.equal(isCidV0(cid), false); + }); + }); + + describe('cidToV1Base32', () => { + it('converts CIDv0 to CIDv1 base32', () => { + const result = cidToV1Base32(CIDV0); + // Result should start with 'bafy' (dag-pb, sha2-256) + assert.match(result, /^bafy/); + // Should be all lowercase (DNS-safe) + assert.equal(result, result.toLowerCase()); + }); + + it('returns CIDv1 base32 unchanged', () => { + const result = cidToV1Base32(CIDV1_BASE32); + assert.equal(result, CIDV1_BASE32); + }); + + it('throws for invalid CID strings', () => { + assert.throws(() => cidToV1Base32('not-a-cid')); + }); + }); + + describe('cidToString', () => { + it('returns base58btc for CIDv0', () => { + const cid = parseCid(CIDV0)!; + const result = cidToString(cid); + assert.match(result, /^Qm/); + }); + + it('returns base32 for CIDv1', () => { + const cid = parseCid(CIDV1_BASE32)!; + const result = cidToString(cid); + assert.match(result, /^bafy/); + }); + }); +}); diff --git a/src/ipfs/ipfs-rate-limiter.ts b/src/ipfs/ipfs-rate-limiter.ts new file mode 100644 index 000000000..f79bf0490 --- /dev/null +++ b/src/ipfs/ipfs-rate-limiter.ts @@ -0,0 +1,26 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import * as config from '../config.js'; +import { MemoryRateLimiter } from '../limiter/memory-rate-limiter.js'; + +/** + * Creates a separate MemoryRateLimiter instance for IPFS requests. + * This ensures IPFS traffic doesn't compete with Arweave traffic + * for rate-limit tokens. + */ +export function createIpfsRateLimiter(): MemoryRateLimiter { + return new MemoryRateLimiter({ + resourceCapacity: config.IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET, + resourceRefillRate: config.IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC, + ipCapacity: config.IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET, + ipRefillRate: config.IPFS_RATE_LIMITER_IP_REFILL_PER_SEC, + limitsEnabled: config.ENABLE_RATE_LIMITER && config.IPFS_ENABLED, + ipAllowlist: config.RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST, + capacityMultiplier: 1, + maxBuckets: 50000, + }); +} diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts new file mode 100644 index 000000000..98973c9e0 --- /dev/null +++ b/src/ipfs/ipfs-service.ts @@ -0,0 +1,201 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import fs from 'node:fs'; +import crypto from 'node:crypto'; +import { Readable } from 'node:stream'; +import winston from 'winston'; + +import { cidToV1Base32 } from '../lib/ipfs-cid.js'; +import { IpfsFsCache } from './ipfs-cache.js'; +import { IpfsBlocklist } from './ipfs-blocklist.js'; +import { + KuboDataSource, + IpfsBlockedError, + IpfsNotFoundError, +} from './kubo-data-source.js'; +import * as metrics from '../metrics.js'; + +export interface IpfsGetContentResult { + stream: Readable; + size: number; + contentType: string; + cached: boolean; +} + +export class IpfsService { + private log: winston.Logger; + private dataSource: KuboDataSource; + private cache: IpfsFsCache; + private blocklist: IpfsBlocklist; + + constructor({ + log, + dataSource, + cache, + blocklist, + }: { + log: winston.Logger; + dataSource: KuboDataSource; + cache: IpfsFsCache; + blocklist: IpfsBlocklist; + }) { + this.log = log.child({ class: this.constructor.name }); + this.dataSource = dataSource; + this.cache = cache; + this.blocklist = blocklist; + } + + async getContent({ + cidString, + path, + signal, + }: { + cidString: string; + path?: string; + signal?: AbortSignal; + }): Promise { + // Normalize CID to v1 base32 for consistent caching + const normalizedCid = cidToV1Base32(cidString); + + // Check blocklist + if (this.blocklist.isBlocked(normalizedCid)) { + metrics.ipfsBlockedTotal.inc(); + throw new IpfsBlockedError(`CID is blocked: ${normalizedCid}`); + } + + // Reject path traversal attempts + if (path && (path.includes('..') || path.startsWith('/'))) { + throw new IpfsNotFoundError('Invalid IPFS path'); + } + + // Check cache + const cached = await this.cache.get(normalizedCid, path); + if (cached) { + this.log.debug('IPFS cache hit', { cid: normalizedCid, path }); + metrics.ipfsCacheHitTotal.inc(); + return { + stream: cached.stream, + size: cached.size, + contentType: cached.contentType, + cached: true, + }; + } + + metrics.ipfsCacheMissTotal.inc(); + + // Fetch from Kubo + const result = await this.dataSource.getContent({ + cidString: normalizedCid, + path, + signal, + }); + + // Stream directly to the client while writing to a temp file on disk + // for caching. No memory buffering — handles files of any size. + this.streamToCache(normalizedCid, path, result.stream, result.contentType); + + return { + stream: result.stream, + size: result.size, + contentType: result.contentType, + cached: false, + }; + } + + /** + * Writes stream data to a temp cache file as it flows to the client. + * Non-blocking — errors are logged but don't affect the response. + * Buffers early chunks in memory until the write stream is ready, + * then flushes them to disk. + */ + private streamToCache( + cidString: string, + path: string | undefined, + stream: Readable, + contentType: string, + ): void { + const cacheDir = `${this.cache.getCachePath()}/tmp`; + const tempPath = `${cacheDir}/${crypto.randomBytes(16).toString('hex')}`; + let writeStream: fs.WriteStream | null = null; + let bytesWritten = 0; + let failed = false; + const pendingChunks: Buffer[] = []; + + const cleanup = () => { + if (writeStream) { + writeStream.destroy(); + writeStream = null; + } + pendingChunks.length = 0; + fs.promises.unlink(tempPath).catch(() => {}); + }; + + // Create temp directory and write stream + fs.promises + .mkdir(cacheDir, { recursive: true }) + .then(() => { + if (failed) return; + writeStream = fs.createWriteStream(tempPath); + writeStream.on('error', (error) => { + failed = true; + this.log.error('Cache write stream error', { + cid: cidString, + message: error.message, + }); + cleanup(); + }); + // Flush any chunks that arrived before writeStream was ready + for (const chunk of pendingChunks) { + writeStream.write(chunk); + } + pendingChunks.length = 0; + }) + .catch((error) => { + failed = true; + this.log.error('Failed to create cache temp dir', { + message: error.message, + }); + }); + + stream.on('data', (chunk: Buffer) => { + if (failed) return; + bytesWritten += chunk.length; + if (writeStream) { + writeStream.write(chunk); + } else { + // Buffer until writeStream is ready (typically only first 1-2 chunks) + pendingChunks.push(chunk); + } + }); + + stream.on('end', () => { + if (failed || !writeStream) { + // If writeStream never became ready, discard + cleanup(); + return; + } + writeStream.end(() => { + // Finalize: move temp file into cache + this.cache + .putFromFile(cidString, tempPath, bytesWritten, contentType, path) + .catch((error) => { + this.log.error('Failed to finalize IPFS cache entry', { + cid: cidString, + path, + message: error.message, + }); + cleanup(); + }); + }); + }); + + stream.on('error', () => { + failed = true; + cleanup(); + }); + } +} diff --git a/src/ipfs/kubo-data-source.test.ts b/src/ipfs/kubo-data-source.test.ts new file mode 100644 index 000000000..8a9ff690b --- /dev/null +++ b/src/ipfs/kubo-data-source.test.ts @@ -0,0 +1,105 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { describe, it, beforeEach } from 'node:test'; +import { strict as assert } from 'node:assert'; + +import { createTestLogger } from '../../test/test-logger.js'; +import { + KuboDataSource, + IpfsNotFoundError, + IpfsTimeoutError, + IpfsUnavailableError, +} from './kubo-data-source.js'; + +describe('KuboDataSource', () => { + const log = createTestLogger({ suite: 'KuboDataSource' }); + + let kuboDataSource: KuboDataSource; + + beforeEach(() => { + kuboDataSource = new KuboDataSource({ + log, + kuboUrl: 'http://localhost:8080', + requestTimeoutMs: 5000, + streamStallTimeoutMs: 5000, + }); + }); + + describe('getContent', () => { + it('constructs correct URL for bare CID', async () => { + // This test verifies URL construction without making a real request. + // A real integration test would require a running Kubo instance. + const controller = new AbortController(); + controller.abort(); // Abort immediately + + try { + await kuboDataSource.getContent({ + cidString: + 'bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi', + signal: controller.signal, + }); + } catch (error: any) { + // Expected to throw due to abort + assert.ok(error); + } + }); + + it('constructs correct URL for CID with path', async () => { + const controller = new AbortController(); + controller.abort(); + + try { + await kuboDataSource.getContent({ + cidString: + 'bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi', + path: 'images/logo.png', + signal: controller.signal, + }); + } catch (error: any) { + assert.ok(error); + } + }); + + it('throws AbortError when signal is already aborted', async () => { + const controller = new AbortController(); + controller.abort(); + + await assert.rejects( + () => + kuboDataSource.getContent({ + cidString: + 'bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi', + signal: controller.signal, + }), + (error: any) => { + assert.ok( + error.name === 'AbortError' || error.code === 'ERR_CANCELED', + ); + return true; + }, + ); + }); + }); + + describe('error types', () => { + it('IpfsNotFoundError has correct name', () => { + const error = new IpfsNotFoundError('not found'); + assert.equal(error.name, 'IpfsNotFoundError'); + assert.equal(error.message, 'not found'); + }); + + it('IpfsTimeoutError has correct name', () => { + const error = new IpfsTimeoutError('timeout'); + assert.equal(error.name, 'IpfsTimeoutError'); + }); + + it('IpfsUnavailableError has correct name', () => { + const error = new IpfsUnavailableError('unavailable'); + assert.equal(error.name, 'IpfsUnavailableError'); + }); + }); +}); diff --git a/src/ipfs/kubo-data-source.ts b/src/ipfs/kubo-data-source.ts new file mode 100644 index 000000000..4bdf01d69 --- /dev/null +++ b/src/ipfs/kubo-data-source.ts @@ -0,0 +1,199 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { default as axios } from 'axios'; +import { Readable } from 'node:stream'; +import winston from 'winston'; + +import { attachStallTimeout } from '../lib/stream.js'; + +export interface IpfsContentResult { + stream: Readable; + size: number; + contentType: string; +} + +export class KuboDataSource { + private log: winston.Logger; + private kuboUrl: string; + private requestTimeoutMs: number; + private streamStallTimeoutMs: number; + + constructor({ + log, + kuboUrl, + requestTimeoutMs, + streamStallTimeoutMs, + }: { + log: winston.Logger; + kuboUrl: string; + requestTimeoutMs: number; + streamStallTimeoutMs: number; + }) { + this.log = log.child({ class: this.constructor.name }); + this.kuboUrl = kuboUrl.replace(/\/$/, ''); + this.requestTimeoutMs = requestTimeoutMs; + this.streamStallTimeoutMs = streamStallTimeoutMs; + } + + async getContent({ + cidString, + path, + signal, + }: { + cidString: string; + path?: string; + signal?: AbortSignal; + }): Promise { + signal?.throwIfAborted(); + + const ipfsPath = path ? `${cidString}/${path}` : cidString; + const url = `${this.kuboUrl}/ipfs/${ipfsPath}`; + + this.log.debug('Fetching IPFS content from Kubo', { + cidString, + path, + url, + }); + + // Connection-phase timeout + const controller = new AbortController(); + const connectionTimer = setTimeout(() => { + controller.abort(new Error('Kubo connection timeout')); + }, this.requestTimeoutMs); + + // Forward client abort to our controller + const onClientAbort = () => controller.abort(signal?.reason); + if (signal?.aborted) { + onClientAbort(); + } else if (signal) { + signal.addEventListener('abort', onClientAbort, { once: true }); + } + + try { + const response = await axios.get(url, { + responseType: 'stream', + signal: controller.signal, + headers: { + 'Accept-Encoding': 'identity', + }, + maxRedirects: 5, + // Accept non-2xx so we can handle 404/408/504 ourselves + validateStatus: (status) => status < 500 || status === 504, + }); + + clearTimeout(connectionTimer); + signal?.removeEventListener('abort', onClientAbort); + + if (response.status === 404) { + throw new IpfsNotFoundError( + `IPFS content not found: /ipfs/${ipfsPath}`, + ); + } + + if (response.status === 408 || response.status === 504) { + throw new IpfsTimeoutError( + `Kubo timed out resolving: /ipfs/${ipfsPath}`, + ); + } + + if (response.status !== 200) { + const stream = response.data as Readable; + stream.destroy(); + throw new Error( + `Unexpected Kubo response status: ${response.status} for /ipfs/${ipfsPath}`, + ); + } + + const stream = response.data as Readable; + const contentLength = parseInt( + response.headers['content-length'] ?? '0', + 10, + ); + const contentType = + response.headers['content-type'] ?? 'application/octet-stream'; + + // Switch from connection timeout to stall timeout + attachStallTimeout(stream, this.streamStallTimeoutMs); + + this.log.debug('Kubo fetch successful', { + cidString, + path, + contentLength, + contentType, + }); + + return { + stream, + size: contentLength, + contentType, + }; + } catch (error: any) { + clearTimeout(connectionTimer); + signal?.removeEventListener('abort', onClientAbort); + + if (error instanceof IpfsNotFoundError) throw error; + if (error instanceof IpfsTimeoutError) throw error; + + if (error.name === 'AbortError' || error.code === 'ERR_CANCELED') { + if (signal?.aborted) { + throw error; // Client disconnected + } + throw new IpfsTimeoutError( + `Kubo request timed out for /ipfs/${ipfsPath}`, + ); + } + + if (error.code === 'ECONNREFUSED') { + throw new IpfsUnavailableError( + `Kubo service unavailable at ${this.kuboUrl}`, + ); + } + + this.log.error('Failed to fetch from Kubo', { + cidString, + path, + message: error.message, + }); + throw error; + } + } +} + +export class IpfsNotFoundError extends Error { + constructor(message: string) { + super(message); + this.name = 'IpfsNotFoundError'; + } +} + +export class IpfsTimeoutError extends Error { + constructor(message: string) { + super(message); + this.name = 'IpfsTimeoutError'; + } +} + +export class IpfsUnavailableError extends Error { + constructor(message: string) { + super(message); + this.name = 'IpfsUnavailableError'; + } +} + +export class IpfsBlockedError extends Error { + constructor(message: string) { + super(message); + this.name = 'IpfsBlockedError'; + } +} + +export class IpfsSizeLimitError extends Error { + constructor(message: string) { + super(message); + this.name = 'IpfsSizeLimitError'; + } +} diff --git a/src/lib/httpsig.ts b/src/lib/httpsig.ts index 34c61c296..fecf9b1f3 100644 --- a/src/lib/httpsig.ts +++ b/src/lib/httpsig.ts @@ -46,6 +46,9 @@ export const TRIGGER_HEADERS = new Set([ 'x-arweave-chunk-data-root', 'x-arweave-chunk-tx-id', 'x-ar-io-chunk-source-type', + // IPFS headers — presence of x-ipfs-path triggers signing for IPFS responses + 'x-ipfs-path', + 'x-ar-io-source', ]); /** @@ -53,7 +56,12 @@ export const TRIGGER_HEADERS = new Set([ * when at least one TRIGGER_HEADER is also present. Signing them alone is too * broad — nearly every response has a Content-Type. */ -export const CO_SIGNABLE_HEADERS = new Set(['content-type', 'content-digest']); +export const CO_SIGNABLE_HEADERS = new Set([ + 'content-type', + 'content-digest', + 'x-cache', + 'etag', +]); // Header predicates normalize case defensively, but callers should pass // lowercase where possible. Express's `res.getHeaders()` keys are already diff --git a/src/lib/ipfs-cid.ts b/src/lib/ipfs-cid.ts new file mode 100644 index 000000000..96c0350cd --- /dev/null +++ b/src/lib/ipfs-cid.ts @@ -0,0 +1,60 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { CID } from 'multiformats/cid'; +import { base32 } from 'multiformats/bases/base32'; +import { base58btc } from 'multiformats/bases/base58'; + +/** + * Safely parse a CID string (v0 or v1). Returns null on invalid input. + */ +export function parseCid(cidString: string): CID | null { + try { + return CID.parse(cidString); + } catch { + return null; + } +} + +/** + * Quick validation: attempts parse, returns boolean. + */ +export function isValidCid(cidString: string): boolean { + return parseCid(cidString) !== null; +} + +/** + * Check if a CID is version 0 (starts with `Qm`, base58btc, dag-pb codec). + */ +export function isCidV0(cid: CID): boolean { + return cid.version === 0; +} + +/** + * Convert any CID string to CIDv1 base32lower (DNS-safe). + * Returns the base32lower string representation. + * Throws if the input is not a valid CID. + */ +export function cidToV1Base32(cidString: string): string { + const cid = CID.parse(cidString); + if (cid.version === 0) { + return cid.toV1().toString(base32); + } + return cid.toString(base32); +} + +/** + * Canonical string representation of a CID. + * v0 → base58btc, v1 → base32lower. + */ +export function cidToString(cid: CID): string { + if (cid.version === 0) { + return cid.toString(base58btc); + } + return cid.toString(base32); +} + +export { CID }; diff --git a/src/metrics.ts b/src/metrics.ts index d937bf9df..66740d39c 100644 --- a/src/metrics.ts +++ b/src/metrics.ts @@ -929,3 +929,41 @@ export const httpSigErrorsTotal = new promClient.Counter({ name: 'httpsig_errors_total', help: 'Total HTTPSIG signing errors', }); + +// +// IPFS metrics +// + +export const ipfsRequestsTotal = new promClient.Counter({ + name: 'ipfs_requests_total', + help: 'Total IPFS content requests', + labelNames: ['route_type', 'status'] as const, +}); + +export const ipfsCacheHitTotal = new promClient.Counter({ + name: 'ipfs_cache_hit_total', + help: 'IPFS content cache hits', +}); + +export const ipfsCacheMissTotal = new promClient.Counter({ + name: 'ipfs_cache_miss_total', + help: 'IPFS content cache misses', +}); + +export const ipfsContentSizeHistogram = new promClient.Histogram({ + name: 'ipfs_content_size_bytes', + help: 'Distribution of IPFS content sizes', + buckets: [1024, 102400, 1048576, 10485760, 104857600], +}); + +export const ipfsRequestDurationHistogram = new promClient.Histogram({ + name: 'ipfs_request_duration_seconds', + help: 'Duration of IPFS content requests', + labelNames: ['route_type', 'cache_status'] as const, + buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10], +}); + +export const ipfsBlockedTotal = new promClient.Counter({ + name: 'ipfs_blocked_total', + help: 'IPFS requests blocked by CID blocklist', +}); diff --git a/src/middleware/ipfs.ts b/src/middleware/ipfs.ts new file mode 100644 index 000000000..980c7e705 --- /dev/null +++ b/src/middleware/ipfs.ts @@ -0,0 +1,96 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { Handler, Request, Response, NextFunction } from 'express'; + +import * as config from '../config.js'; +import { isValidCid, cidToV1Base32 } from '../lib/ipfs-cid.js'; + +/** + * Middleware that intercepts `{CID}.{root_host}` subdomain requests. + * Must be mounted BEFORE the ArNS middleware to prevent ArNS from attempting + * to resolve CIDs as ArNS names. + * + * CIDv1 base32 strings are ~59 characters — always longer than the ArNS + * name limit (51 chars), so there's no collision with ArNS names. + * This also avoids needing a multi-level wildcard TLS certificate + * (*.ipfs.host would require a separate cert from *.host). + * + * Express subdomain array for `bafyabc.my-gateway.io` (root=my-gateway.io): + * req.subdomains = ['bafyabc'] (single subdomain) + * req.subdomains[0] = CID + */ +export function createIpfsSubdomainMiddleware({ + ipfsHandler, +}: { + ipfsHandler: Handler; +}): Handler { + return (req: Request, res: Response, next: NextFunction) => { + if (!config.IPFS_ENABLED || config.ARNS_ROOT_HOSTS.length === 0) { + next(); + return; + } + + const matchedEntry = config.matchArnsRootHost(req.hostname); + if (matchedEntry === undefined) { + next(); + return; + } + + // For {CID}.{root_host}, we expect exactly 1 subdomain beyond + // the root host's own subdomain depth. + const cidLabelIndex = matchedEntry.subdomainLength; + + if ( + !Array.isArray(req.subdomains) || + req.subdomains.length !== cidLabelIndex + 1 + ) { + next(); + return; + } + + const cidLabel = req.subdomains[cidLabelIndex]; + if (!isValidCid(cidLabel)) { + // Not a CID — let ArNS handle it (likely an ArNS name) + next(); + return; + } + + // Attach IPFS context to request + (req as any).ipfsCid = cidLabel; + + // Handle /ipfs/ paths on subdomain requests. + // Kubo's directory listings generate absolute links like /ipfs/{CID}/file. + let reqPath = req.path === '/' ? undefined : req.path.slice(1); + if (reqPath && reqPath.startsWith('ipfs/')) { + const afterIpfs = reqPath.slice(5); // strip 'ipfs/' + const slashIdx = afterIpfs.indexOf('/'); + const pathCid = slashIdx >= 0 ? afterIpfs.slice(0, slashIdx) : afterIpfs; + const remainder = slashIdx >= 0 ? afterIpfs.slice(slashIdx + 1) : undefined; + + if (pathCid === cidLabel) { + // Same CID — strip the redundant prefix + reqPath = remainder || undefined; + } else if (isValidCid(pathCid)) { + // Different CID — redirect to that CID's subdomain + try { + const targetCid = cidToV1Base32(pathCid); + const rootHost = matchedEntry.host; + const pathSuffix = remainder ? `/${remainder}` : '/'; + const protocol = req.protocol; + res.redirect(302, `${protocol}://${targetCid}.${rootHost}${pathSuffix}`); + return; + } catch { + // CID conversion failed — fall through to handler + } + } + } + (req as any).ipfsPath = reqPath; + + // Delegate to the IPFS handler + ipfsHandler(req, res, next); + }; +} diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts new file mode 100644 index 000000000..43b0b576c --- /dev/null +++ b/src/routes/ipfs.ts @@ -0,0 +1,309 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { Router, Request, Response, Handler } from 'express'; +import { default as asyncHandler } from 'express-async-handler'; +import winston from 'winston'; + +import * as config from '../config.js'; +import * as metrics from '../metrics.js'; +import { + cidToV1Base32, + isValidCid, + parseCid, + isCidV0, +} from '../lib/ipfs-cid.js'; +import { IpfsService } from '../ipfs/ipfs-service.js'; +import { + IpfsBlockedError, + IpfsNotFoundError, + IpfsTimeoutError, + IpfsUnavailableError, +} from '../ipfs/kubo-data-source.js'; +import { RateLimiter } from '../limiter/types.js'; +import { + checkPaymentAndRateLimits, + adjustRateLimitTokens, +} from '../handlers/data-handler-utils.js'; +import { PaymentProcessor } from '../payments/types.js'; +import { extractAllClientIPs } from '../lib/ip-utils.js'; + +export function createIpfsRouter({ + log, + ipfsService, + rateLimiter, + paymentProcessor, +}: { + log: winston.Logger; + ipfsService: IpfsService; + rateLimiter?: RateLimiter; + paymentProcessor?: PaymentProcessor; +}): Router { + const router = Router(); + const handler = createIpfsPathHandler({ + log, + ipfsService, + rateLimiter, + paymentProcessor, + }); + + router.get('/ipfs/:cid', handler); + router.get('/ipfs/:cid/*', handler); + + return router; +} + +export function createIpfsHandler({ + log, + ipfsService, + rateLimiter, + paymentProcessor, +}: { + log: winston.Logger; + ipfsService: IpfsService; + rateLimiter?: RateLimiter; + paymentProcessor?: PaymentProcessor; +}): Handler { + return asyncHandler(async (req: Request, res: Response) => { + const cidString = (req as any).ipfsCid as string; + const path = (req as any).ipfsPath as string | undefined; + await handleIpfsRequest({ + req, + res, + cidString, + path, + log, + ipfsService, + rateLimiter, + paymentProcessor, + routeType: 'subdomain', + }); + }); +} + +function createIpfsPathHandler({ + log, + ipfsService, + rateLimiter, + paymentProcessor, +}: { + log: winston.Logger; + ipfsService: IpfsService; + rateLimiter?: RateLimiter; + paymentProcessor?: PaymentProcessor; +}): Handler { + return asyncHandler(async (req: Request, res: Response) => { + const cidString = req.params.cid; + // Express puts wildcard content in req.params[0] + const path = req.params[0] || undefined; + + if (!isValidCid(cidString)) { + res.status(400).json({ error: 'Invalid CID' }); + return; + } + + // Redirect CIDv0 to CIDv1 subdomain if ArNS root hosts are configured. + // Uses {CID}.{host} (same level as ArNS names) — no .ipfs. label needed + // since CIDv1 base32 is always >51 chars (won't collide with ArNS names) + // and works with standard *.{host} wildcard TLS certificates. + const cid = parseCid(cidString); + if (cid && isCidV0(cid) && config.ARNS_ROOT_HOSTS.length > 0) { + const v1Base32 = cidToV1Base32(cidString); + const rootHost = config.ARNS_ROOT_HOSTS[0].host; + const pathSuffix = path ? `/${path}` : ''; + const protocol = req.protocol; + res.redirect( + 302, + `${protocol}://${v1Base32}.${rootHost}${pathSuffix}`, + ); + return; + } + + await handleIpfsRequest({ + req, + res, + cidString, + path, + log, + ipfsService, + rateLimiter, + paymentProcessor, + routeType: 'path', + }); + }); +} + +async function handleIpfsRequest({ + req, + res, + cidString, + path, + log: parentLog, + ipfsService, + rateLimiter, + paymentProcessor, + routeType, +}: { + req: Request; + res: Response; + cidString: string; + path: string | undefined; + log: winston.Logger; + ipfsService: IpfsService; + rateLimiter?: RateLimiter; + paymentProcessor?: PaymentProcessor; + routeType: 'path' | 'subdomain'; +}): Promise { + const startTime = Date.now(); + const ipfsPath = path ? `${cidString}/${path}` : cidString; + + parentLog.debug('Handling IPFS request', { cidString, path, routeType }); + + try { + const result = await ipfsService.getContent({ + cidString, + path, + signal: req.signal, + }); + + // Check payment and rate limits (x402 + rate limiting in one call). + // Content size is needed for token calculation and payment pricing. + const contentSize = result.size > 0 ? result.size : 1024; // min 1KB for pricing + const limitCheck = await checkPaymentAndRateLimits({ + req, + res, + id: cidToV1Base32(cidString), + contentSize, + contentType: result.contentType, + requestAttributes: { hops: 0, clientIps: extractAllClientIPs(req).clientIps }, + rateLimiter, + paymentProcessor, + }); + + if (!limitCheck.allowed) { + // Response already sent (402 or 429) by checkPaymentAndRateLimits + result.stream.destroy(); + metrics.ipfsRequestsTotal.inc({ + route_type: routeType, + status: 'rate_limited', + }); + return; + } + + // Set response headers + res.setHeader('Content-Type', result.contentType); + if (result.size > 0) { + res.setHeader('Content-Length', result.size); + } + // CIDs are content-addressed — content never changes + res.setHeader('Cache-Control', 'public, max-age=29030400, immutable'); + res.setHeader('ETag', `"${cidToV1Base32(cidString)}"`); + res.setHeader('X-Ipfs-Path', `/ipfs/${ipfsPath}`); + res.setHeader('X-Ar-Io-Source', 'ipfs'); + + if (result.cached) { + res.setHeader('X-Cache', 'HIT'); + } else { + res.setHeader('X-Cache', 'MISS'); + } + + // Track metrics + const cacheStatus = result.cached ? 'hit' : 'miss'; + metrics.ipfsRequestsTotal.inc({ route_type: routeType, status: 'success' }); + if (result.size > 0) { + metrics.ipfsContentSizeHistogram.observe(result.size); + } + + // Pipe stream to response + result.stream.pipe(res); + + // Adjust rate limiter tokens after response completes + res.on('finish', () => { + const durationSec = (Date.now() - startTime) / 1000; + metrics.ipfsRequestDurationHistogram.observe( + { route_type: routeType, cache_status: cacheStatus }, + durationSec, + ); + + adjustRateLimitTokens({ + req, + responseSize: result.size > 0 ? result.size : contentSize, + initialResult: limitCheck, + rateLimiter, + }).catch((error) => { + parentLog.error('Failed to adjust rate limit tokens', { + message: error.message, + }); + }); + }); + + result.stream.on('error', (error) => { + parentLog.error('IPFS stream error', { + cidString, + path, + message: error.message, + }); + if (!res.headersSent) { + res.status(502).json({ error: 'IPFS stream failed' }); + } else { + res.destroy(); + } + }); + } catch (error: any) { + if (error instanceof IpfsBlockedError) { + metrics.ipfsRequestsTotal.inc({ + route_type: routeType, + status: 'blocked', + }); + res.setHeader( + 'Cache-Control', + `public, max-age=${config.CACHE_BLOCKED_MAX_AGE}, immutable`, + ); + res.status(451).json({ error: 'Content blocked' }); + return; + } + + if (error instanceof IpfsNotFoundError) { + metrics.ipfsRequestsTotal.inc({ + route_type: routeType, + status: 'not_found', + }); + res.status(404).json({ error: 'IPFS content not found' }); + return; + } + + if (error instanceof IpfsTimeoutError) { + metrics.ipfsRequestsTotal.inc({ + route_type: routeType, + status: 'timeout', + }); + res.status(504).json({ error: 'IPFS request timed out' }); + return; + } + + if (error instanceof IpfsUnavailableError) { + metrics.ipfsRequestsTotal.inc({ + route_type: routeType, + status: 'unavailable', + }); + res.status(502).json({ error: 'IPFS service unavailable' }); + return; + } + + // Client disconnected + if (error.name === 'AbortError') { + return; + } + + metrics.ipfsRequestsTotal.inc({ route_type: routeType, status: 'error' }); + parentLog.error('IPFS request failed', { + cidString, + path, + message: error.message, + }); + res.status(500).json({ error: 'Internal server error' }); + } +} diff --git a/src/system.ts b/src/system.ts index 188dd365c..d3a3b943e 100644 --- a/src/system.ts +++ b/src/system.ts @@ -1456,6 +1456,60 @@ if (dataVerificationWorker !== undefined) { dataVerificationWorker.start(); } +// +// IPFS subsystem (conditionally initialized) +// + +import { KuboDataSource } from './ipfs/kubo-data-source.js'; +import { IpfsFsCache } from './ipfs/ipfs-cache.js'; +import { IpfsBlocklist } from './ipfs/ipfs-blocklist.js'; +import { IpfsService } from './ipfs/ipfs-service.js'; +import { createIpfsRateLimiter } from './ipfs/ipfs-rate-limiter.js'; +import { RateLimiter } from './limiter/types.js'; + +export let ipfsService: IpfsService | undefined; +export let ipfsRateLimiter: RateLimiter | undefined; +export let ipfsBlocklist: IpfsBlocklist | undefined; + +if (config.IPFS_ENABLED) { + log.info('IPFS subsystem enabled, initializing...'); + + const kuboDataSource = new KuboDataSource({ + log, + kuboUrl: config.IPFS_KUBO_URL, + requestTimeoutMs: config.IPFS_KUBO_REQUEST_TIMEOUT_MS, + streamStallTimeoutMs: config.IPFS_STREAM_STALL_TIMEOUT_MS, + }); + + const ipfsCache = new IpfsFsCache({ + log, + basePath: config.IPFS_CACHE_PATH, + maxSizeBytes: config.IPFS_CACHE_MAX_SIZE_BYTES, + }); + + ipfsBlocklist = new IpfsBlocklist({ + log, + filePath: config.IPFS_BLOCKLIST_PATH, + }); + await ipfsBlocklist.load(); + ipfsBlocklist.startWatching(); + + ipfsService = new IpfsService({ + log, + dataSource: kuboDataSource, + cache: ipfsCache, + blocklist: ipfsBlocklist, + }); + + ipfsRateLimiter = createIpfsRateLimiter(); + + log.info('IPFS subsystem initialized', { + kuboUrl: config.IPFS_KUBO_URL, + cachePath: config.IPFS_CACHE_PATH, + maxCacheSize: config.IPFS_CACHE_MAX_SIZE_BYTES, + }); +} + export const blockedNamesCache = new BlockedNamesCache({ log, cacheTTL: 3600, @@ -1487,6 +1541,7 @@ export const shutdown = async (exitCode = 0) => { } // Clean up system components + ipfsBlocklist?.stop(); eventEmitter.removeAllListeners(); arIOPeerManager.stopUpdatingPeers(); dataSqliteWalCleanupWorker?.stop(); diff --git a/test-ipfs.sh b/test-ipfs.sh new file mode 100755 index 000000000..6f262e563 --- /dev/null +++ b/test-ipfs.sh @@ -0,0 +1,172 @@ +#!/bin/bash +# End-to-end IPFS integration test script +# Run this after starting the gateway with IPFS_ENABLED=true and Kubo running + +GATEWAY_URL="${1:-http://localhost:4000}" +KUBO_URL="${2:-http://localhost:8080}" + +GREEN='\033[0;32m' +RED='\033[0;31m' +YELLOW='\033[0;33m' +NC='\033[0m' + +pass=0 +fail=0 + +test_case() { + local name="$1" + local expected_status="$2" + local url="$3" + local extra_args="${4:-}" + + local result + result=$(curl -s -o /tmp/ipfs-test-body -w "%{http_code}" --max-time 30 $extra_args "$url" 2>&1) + + if [ "$result" = "$expected_status" ]; then + echo -e " ${GREEN}PASS${NC} $name (HTTP $result)" + ((pass++)) + else + echo -e " ${RED}FAIL${NC} $name — expected $expected_status, got $result" + echo " URL: $url" + echo " Body: $(head -c 200 /tmp/ipfs-test-body)" + ((fail++)) + fi +} + +test_body_contains() { + local name="$1" + local expected_text="$2" + local url="$3" + + local body + body=$(curl -s --max-time 30 "$url" 2>&1) + local status=$? + + if echo "$body" | grep -q "$expected_text"; then + echo -e " ${GREEN}PASS${NC} $name" + ((pass++)) + else + echo -e " ${RED}FAIL${NC} $name — body doesn't contain '$expected_text'" + echo " Body: $(echo "$body" | head -c 200)" + ((fail++)) + fi +} + +test_header() { + local name="$1" + local header="$2" + local expected_value="$3" + local url="$4" + + local actual + actual=$(curl -s -I --max-time 30 "$url" 2>&1 | grep -i "^$header:" | head -1 | sed 's/^[^:]*: //' | tr -d '\r') + + if echo "$actual" | grep -qi "$expected_value"; then + echo -e " ${GREEN}PASS${NC} $name ($actual)" + ((pass++)) + else + echo -e " ${RED}FAIL${NC} $name — expected header '$header' to contain '$expected_value', got '$actual'" + ((fail++)) + fi +} + +echo "=========================================" +echo " AR.IO IPFS Integration — E2E Tests" +echo "=========================================" +echo "" +echo "Gateway: $GATEWAY_URL" +echo "Kubo: $KUBO_URL" +echo "" + +# --- Pre-flight: ensure Kubo is running --- +echo "--- Pre-flight ---" +kubo_status=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$KUBO_URL/ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn" 2>&1) +if [ "$kubo_status" = "000" ]; then + echo -e " ${RED}FAIL${NC} Kubo not reachable at $KUBO_URL" + exit 1 +fi +echo -e " ${GREEN}OK${NC} Kubo is reachable" + +# Add test content to Kubo +echo "" +echo "--- Adding test content to Kubo ---" +FILE_CID=$(echo "Hello from AR.IO IPFS integration test!" | docker exec -i ar-io-node-kubo-1 ipfs add -q 2>&1) +echo " File CID: $FILE_CID" + +DIR_CID=$(docker exec ar-io-node-kubo-1 sh -c ' + mkdir -p /tmp/e2e-test + echo "

E2E Test

" > /tmp/e2e-test/index.html + echo "subfile content" > /tmp/e2e-test/sub.txt + ipfs add -r -q /tmp/e2e-test | tail -1 +' 2>&1) +echo " Dir CID: $DIR_CID" +echo "" + +# --- Test 1: Path-based single file --- +echo "--- Path-based access ---" +test_body_contains "GET /ipfs/{CID} serves file content" \ + "Hello from AR.IO IPFS integration test" \ + "$GATEWAY_URL/ipfs/$FILE_CID" + +# --- Test 2: Path-based directory with path --- +test_body_contains "GET /ipfs/{CID}/index.html serves directory file" \ + "E2E Test" \ + "$GATEWAY_URL/ipfs/$DIR_CID/index.html" + +test_body_contains "GET /ipfs/{CID}/sub.txt serves subfile" \ + "subfile content" \ + "$GATEWAY_URL/ipfs/$DIR_CID/sub.txt" + +# --- Test 3: Invalid CID --- +test_case "GET /ipfs/invalid-cid returns 400" \ + "400" \ + "$GATEWAY_URL/ipfs/not-a-valid-cid" + +# --- Test 4: Response headers --- +echo "" +echo "--- Response headers ---" +test_header "Cache-Control is immutable" \ + "cache-control" "immutable" \ + "$GATEWAY_URL/ipfs/$FILE_CID" + +test_header "X-Ipfs-Path header present" \ + "x-ipfs-path" "/ipfs/" \ + "$GATEWAY_URL/ipfs/$FILE_CID" + +test_header "X-Cache header present" \ + "x-cache" "" \ + "$GATEWAY_URL/ipfs/$FILE_CID" + +test_header "Content-Type is set" \ + "content-type" "" \ + "$GATEWAY_URL/ipfs/$FILE_CID" + +# --- Test 5: CIDv0 redirect to CIDv1 subdomain --- +echo "" +echo "--- CIDv0 redirect ---" +redirect_location=$(curl -s -o /dev/null -w "%{redirect_url}" --max-time 10 "$GATEWAY_URL/ipfs/$FILE_CID" 2>&1) +if echo "$redirect_location" | grep -q "ipfs"; then + echo -e " ${GREEN}PASS${NC} CIDv0 redirects to CIDv1 subdomain ($redirect_location)" + ((pass++)) +elif [ -z "$redirect_location" ]; then + echo -e " ${YELLOW}SKIP${NC} No redirect (CID may already be v1 or no ARNS_ROOT_HOST)" +else + echo -e " ${YELLOW}INFO${NC} Redirect: $redirect_location" +fi + +# --- Test 6: Second request should be cached --- +echo "" +echo "--- Caching ---" +# First request (cache miss) +curl -s -o /dev/null "$GATEWAY_URL/ipfs/$FILE_CID" 2>/dev/null +# Second request (should be cache hit) +cache_header=$(curl -s -I --max-time 10 "$GATEWAY_URL/ipfs/$FILE_CID" 2>&1 | grep -i "^x-cache:" | sed 's/^[^:]*: //' | tr -d '\r') +echo -e " ${GREEN}INFO${NC} X-Cache on second request: ${cache_header:-'(not set)'}" + +# --- Summary --- +echo "" +echo "=========================================" +echo -e " Results: ${GREEN}$pass passed${NC}, ${RED}$fail failed${NC}" +echo "=========================================" + +exit $fail From 7a8f3229d1631fa4b45813ae840fa6b919a99a95 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 21 Apr 2026 00:56:06 +0000 Subject: [PATCH 02/47] fix: lint errors in IPFS files (PE-9067) Fix strict-boolean-expressions (explicit nullish checks) and prettier formatting issues caught by CI eslint. --- src/ipfs/ipfs-cache.ts | 7 +++++-- src/ipfs/ipfs-service.ts | 5 ++++- src/ipfs/kubo-data-source.ts | 5 ++++- src/middleware/ipfs.ts | 22 ++++++++++++++-------- src/routes/ipfs.ts | 18 +++++++++--------- 5 files changed, 36 insertions(+), 21 deletions(-) diff --git a/src/ipfs/ipfs-cache.ts b/src/ipfs/ipfs-cache.ts index ccf6be052..c7847c1b0 100644 --- a/src/ipfs/ipfs-cache.ts +++ b/src/ipfs/ipfs-cache.ts @@ -47,7 +47,8 @@ export class IpfsFsCache { } private cacheKey(cidString: string, path?: string): string { - const raw = path ? `${cidString}/${path}` : cidString; + const raw = + path !== undefined && path !== '' ? `${cidString}/${path}` : cidString; return crypto.createHash('sha256').update(raw).digest('hex'); } @@ -95,7 +96,9 @@ export class IpfsFsCache { async get( cidString: string, path?: string, - ): Promise<{ stream: Readable; size: number; contentType: string } | undefined> { + ): Promise< + { stream: Readable; size: number; contentType: string } | undefined + > { const key = this.cacheKey(cidString, path); let entry = this.index.get(key); diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 98973c9e0..d7219f46d 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -68,7 +68,10 @@ export class IpfsService { } // Reject path traversal attempts - if (path && (path.includes('..') || path.startsWith('/'))) { + if ( + path !== undefined && + (path.includes('..') || path.startsWith('/')) + ) { throw new IpfsNotFoundError('Invalid IPFS path'); } diff --git a/src/ipfs/kubo-data-source.ts b/src/ipfs/kubo-data-source.ts index 4bdf01d69..817d3e290 100644 --- a/src/ipfs/kubo-data-source.ts +++ b/src/ipfs/kubo-data-source.ts @@ -7,8 +7,10 @@ import { default as axios } from 'axios'; import { Readable } from 'node:stream'; import winston from 'winston'; +import { Span } from '@opentelemetry/api'; import { attachStallTimeout } from '../lib/stream.js'; +import { startChildSpan } from '../tracing.js'; export interface IpfsContentResult { stream: Readable; @@ -50,7 +52,8 @@ export class KuboDataSource { }): Promise { signal?.throwIfAborted(); - const ipfsPath = path ? `${cidString}/${path}` : cidString; + const ipfsPath = + path !== undefined && path !== '' ? `${cidString}/${path}` : cidString; const url = `${this.kuboUrl}/ipfs/${ipfsPath}`; this.log.debug('Fetching IPFS content from Kubo', { diff --git a/src/middleware/ipfs.ts b/src/middleware/ipfs.ts index 980c7e705..48375bc43 100644 --- a/src/middleware/ipfs.ts +++ b/src/middleware/ipfs.ts @@ -64,24 +64,30 @@ export function createIpfsSubdomainMiddleware({ // Handle /ipfs/ paths on subdomain requests. // Kubo's directory listings generate absolute links like /ipfs/{CID}/file. - let reqPath = req.path === '/' ? undefined : req.path.slice(1); - if (reqPath && reqPath.startsWith('ipfs/')) { + let reqPath: string | undefined = + req.path === '/' ? undefined : req.path.slice(1); + if (reqPath !== undefined && reqPath.startsWith('ipfs/')) { const afterIpfs = reqPath.slice(5); // strip 'ipfs/' const slashIdx = afterIpfs.indexOf('/'); - const pathCid = slashIdx >= 0 ? afterIpfs.slice(0, slashIdx) : afterIpfs; - const remainder = slashIdx >= 0 ? afterIpfs.slice(slashIdx + 1) : undefined; + const pathCid = + slashIdx >= 0 ? afterIpfs.slice(0, slashIdx) : afterIpfs; + const remainder = + slashIdx >= 0 ? afterIpfs.slice(slashIdx + 1) : undefined; if (pathCid === cidLabel) { // Same CID — strip the redundant prefix - reqPath = remainder || undefined; + reqPath = remainder !== undefined ? remainder : undefined; } else if (isValidCid(pathCid)) { // Different CID — redirect to that CID's subdomain try { const targetCid = cidToV1Base32(pathCid); const rootHost = matchedEntry.host; - const pathSuffix = remainder ? `/${remainder}` : '/'; - const protocol = req.protocol; - res.redirect(302, `${protocol}://${targetCid}.${rootHost}${pathSuffix}`); + const pathSuffix = + remainder !== undefined ? `/${remainder}` : '/'; + res.redirect( + 302, + `${req.protocol}://${targetCid}.${rootHost}${pathSuffix}`, + ); return; } catch { // CID conversion failed — fall through to handler diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index 43b0b576c..c2d49c071 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -110,15 +110,11 @@ function createIpfsPathHandler({ // since CIDv1 base32 is always >51 chars (won't collide with ArNS names) // and works with standard *.{host} wildcard TLS certificates. const cid = parseCid(cidString); - if (cid && isCidV0(cid) && config.ARNS_ROOT_HOSTS.length > 0) { + if (cid !== null && isCidV0(cid) && config.ARNS_ROOT_HOSTS.length > 0) { const v1Base32 = cidToV1Base32(cidString); const rootHost = config.ARNS_ROOT_HOSTS[0].host; - const pathSuffix = path ? `/${path}` : ''; - const protocol = req.protocol; - res.redirect( - 302, - `${protocol}://${v1Base32}.${rootHost}${pathSuffix}`, - ); + const pathSuffix = path !== undefined ? `/${path}` : ''; + res.redirect(302, `${req.protocol}://${v1Base32}.${rootHost}${pathSuffix}`); return; } @@ -158,7 +154,8 @@ async function handleIpfsRequest({ routeType: 'path' | 'subdomain'; }): Promise { const startTime = Date.now(); - const ipfsPath = path ? `${cidString}/${path}` : cidString; + const ipfsPath = + path !== undefined ? `${cidString}/${path}` : cidString; parentLog.debug('Handling IPFS request', { cidString, path, routeType }); @@ -178,7 +175,10 @@ async function handleIpfsRequest({ id: cidToV1Base32(cidString), contentSize, contentType: result.contentType, - requestAttributes: { hops: 0, clientIps: extractAllClientIPs(req).clientIps }, + requestAttributes: { + hops: 0, + clientIps: extractAllClientIPs(req).clientIps, + }, rateLimiter, paymentProcessor, }); From ddd1bc54a46f1f313d77ad768ce02b52cb2df293 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 21 Apr 2026 01:00:15 +0000 Subject: [PATCH 03/47] fix: add OTEL tracing and fix lint errors (PE-9067) Add OpenTelemetry span tracing to IPFS request lifecycle: - IpfsService.getContent span (cache check, blocklist, delegation) - KuboDataSource.getContent span (HTTP fetch with latency attributes) - Spans record cache hit/miss, content size, errors, and content type Also fix strict-boolean-expressions and prettier formatting for CI. --- src/ipfs/ipfs-service.ts | 133 ++++++++++++++++++++++++----------- src/ipfs/kubo-data-source.ts | 32 +++++++++ 2 files changed, 123 insertions(+), 42 deletions(-) diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index d7219f46d..ddd75a8a2 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -8,8 +8,10 @@ import fs from 'node:fs'; import crypto from 'node:crypto'; import { Readable } from 'node:stream'; import winston from 'winston'; +import { Span } from '@opentelemetry/api'; import { cidToV1Base32 } from '../lib/ipfs-cid.js'; +import { startChildSpan } from '../tracing.js'; import { IpfsFsCache } from './ipfs-cache.js'; import { IpfsBlocklist } from './ipfs-blocklist.js'; import { @@ -53,60 +55,107 @@ export class IpfsService { cidString, path, signal, + parentSpan, }: { cidString: string; path?: string; signal?: AbortSignal; + parentSpan?: Span; }): Promise { - // Normalize CID to v1 base32 for consistent caching - const normalizedCid = cidToV1Base32(cidString); + const span = startChildSpan( + 'IpfsService.getContent', + { + attributes: { + 'ipfs.cid': cidString, + 'ipfs.path': path ?? '', + }, + }, + parentSpan, + ); + + try { + // Normalize CID to v1 base32 for consistent caching + const normalizedCid = cidToV1Base32(cidString); + span.setAttribute('ipfs.cid_normalized', normalizedCid); + + // Check blocklist + if (this.blocklist.isBlocked(normalizedCid)) { + metrics.ipfsBlockedTotal.inc(); + span.setAttribute('ipfs.blocked', true); + throw new IpfsBlockedError(`CID is blocked: ${normalizedCid}`); + } - // Check blocklist - if (this.blocklist.isBlocked(normalizedCid)) { - metrics.ipfsBlockedTotal.inc(); - throw new IpfsBlockedError(`CID is blocked: ${normalizedCid}`); - } + // Reject path traversal attempts + if ( + path !== undefined && + (path.includes('..') || path.startsWith('/')) + ) { + throw new IpfsNotFoundError('Invalid IPFS path'); + } - // Reject path traversal attempts - if ( - path !== undefined && - (path.includes('..') || path.startsWith('/')) - ) { - throw new IpfsNotFoundError('Invalid IPFS path'); - } + // Check cache + const cached = await this.cache.get(normalizedCid, path); + if (cached) { + this.log.debug('IPFS cache hit', { cid: normalizedCid, path }); + metrics.ipfsCacheHitTotal.inc(); + span.setAttributes({ + 'ipfs.cache': 'hit', + 'ipfs.size': cached.size, + }); + span.end(); + return { + stream: cached.stream, + size: cached.size, + contentType: cached.contentType, + cached: true, + }; + } - // Check cache - const cached = await this.cache.get(normalizedCid, path); - if (cached) { - this.log.debug('IPFS cache hit', { cid: normalizedCid, path }); - metrics.ipfsCacheHitTotal.inc(); - return { - stream: cached.stream, - size: cached.size, - contentType: cached.contentType, - cached: true, - }; - } + metrics.ipfsCacheMissTotal.inc(); + span.setAttribute('ipfs.cache', 'miss'); - metrics.ipfsCacheMissTotal.inc(); + // Fetch from Kubo + const result = await this.dataSource.getContent({ + cidString: normalizedCid, + path, + signal, + parentSpan: span, + }); - // Fetch from Kubo - const result = await this.dataSource.getContent({ - cidString: normalizedCid, - path, - signal, - }); + span.setAttributes({ + 'ipfs.size': result.size, + 'ipfs.content_type': result.contentType, + }); - // Stream directly to the client while writing to a temp file on disk - // for caching. No memory buffering — handles files of any size. - this.streamToCache(normalizedCid, path, result.stream, result.contentType); + // Stream directly to the client while writing to a temp file on disk + // for caching. No memory buffering — handles files of any size. + this.streamToCache( + normalizedCid, + path, + result.stream, + result.contentType, + ); + + // End span when stream completes + result.stream.on('end', () => span.end()); + result.stream.on('error', (err) => { + span.recordException(err); + span.end(); + }); - return { - stream: result.stream, - size: result.size, - contentType: result.contentType, - cached: false, - }; + return { + stream: result.stream, + size: result.size, + contentType: result.contentType, + cached: false, + }; + } catch (error: any) { + if (error.name !== 'AbortError') { + span.recordException(error); + } + span.end(); + throw error; + } } /** diff --git a/src/ipfs/kubo-data-source.ts b/src/ipfs/kubo-data-source.ts index 817d3e290..4d358b4e3 100644 --- a/src/ipfs/kubo-data-source.ts +++ b/src/ipfs/kubo-data-source.ts @@ -45,10 +45,12 @@ export class KuboDataSource { cidString, path, signal, + parentSpan, }: { cidString: string; path?: string; signal?: AbortSignal; + parentSpan?: Span; }): Promise { signal?.throwIfAborted(); @@ -56,6 +58,18 @@ export class KuboDataSource { path !== undefined && path !== '' ? `${cidString}/${path}` : cidString; const url = `${this.kuboUrl}/ipfs/${ipfsPath}`; + const span = startChildSpan( + 'KuboDataSource.getContent', + { + attributes: { + 'ipfs.cid': cidString, + 'ipfs.path': path ?? '', + 'ipfs.url': url, + }, + }, + parentSpan, + ); + this.log.debug('Fetching IPFS content from Kubo', { cidString, path, @@ -122,6 +136,12 @@ export class KuboDataSource { // Switch from connection timeout to stall timeout attachStallTimeout(stream, this.streamStallTimeoutMs); + span.setAttributes({ + 'ipfs.content_length': contentLength, + 'ipfs.content_type': contentType, + }); + span.addEvent('Kubo fetch successful'); + this.log.debug('Kubo fetch successful', { cidString, path, @@ -129,6 +149,13 @@ export class KuboDataSource { contentType, }); + // End span when stream finishes or errors + stream.on('end', () => span.end()); + stream.on('error', (err) => { + span.recordException(err); + span.end(); + }); + return { stream, size: contentLength, @@ -138,6 +165,11 @@ export class KuboDataSource { clearTimeout(connectionTimer); signal?.removeEventListener('abort', onClientAbort); + if (error.name !== 'AbortError') { + span.recordException(error); + } + span.end(); + if (error instanceof IpfsNotFoundError) throw error; if (error instanceof IpfsTimeoutError) throw error; From 9e6c0303d81c23c2920321260c9176563b3dbfbc Mon Sep 17 00:00:00 2001 From: root Date: Tue, 21 Apr 2026 01:34:32 +0000 Subject: [PATCH 04/47] fix: prettier formatting, IPFS cache volume, /ar-io/info (PE-9067) - Fix prettier line-length formatting (5 errors from CI) - Add IPFS cache volume mount to docker-compose (data persists) - Add IPFS field to /ar-io/info endpoint (network discovery) - Add OTEL span tracing to IpfsService and KuboDataSource --- docker-compose.yaml | 1 + src/ipfs/ipfs-service.ts | 5 +---- src/middleware/ipfs.ts | 6 ++---- src/routes/ar-io-info-builder.ts | 15 +++++++++++++++ src/routes/ar-io.ts | 1 + src/routes/ipfs.ts | 8 +++++--- 6 files changed, 25 insertions(+), 11 deletions(-) diff --git a/docker-compose.yaml b/docker-compose.yaml index 9a7c15fb6..5fa65da88 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -53,6 +53,7 @@ services: - ${HEADERS_DATA_PATH:-./data/headers}:/app/data/headers - ${SQLITE_DATA_PATH:-./data/sqlite}:/app/data/sqlite - ${DUCKDB_DATA_PATH:-./data/duckdb}:/app/data/duckdb + - ${IPFS_CACHE_DATA_PATH:-./data/ipfs-cache}:/app/data/ipfs-cache - ${TEMP_DATA_PATH:-./data/tmp}:/app/data/tmp - ${LMDB_DATA_PATH:-./data/lmdb}:/app/data/lmdb - ${PARQUET_DATA_PATH:-./data/parquet}:/app/data/parquet diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index ddd75a8a2..1711190d1 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -86,10 +86,7 @@ export class IpfsService { } // Reject path traversal attempts - if ( - path !== undefined && - (path.includes('..') || path.startsWith('/')) - ) { + if (path !== undefined && (path.includes('..') || path.startsWith('/'))) { throw new IpfsNotFoundError('Invalid IPFS path'); } diff --git a/src/middleware/ipfs.ts b/src/middleware/ipfs.ts index 48375bc43..7a47ff4b7 100644 --- a/src/middleware/ipfs.ts +++ b/src/middleware/ipfs.ts @@ -69,8 +69,7 @@ export function createIpfsSubdomainMiddleware({ if (reqPath !== undefined && reqPath.startsWith('ipfs/')) { const afterIpfs = reqPath.slice(5); // strip 'ipfs/' const slashIdx = afterIpfs.indexOf('/'); - const pathCid = - slashIdx >= 0 ? afterIpfs.slice(0, slashIdx) : afterIpfs; + const pathCid = slashIdx >= 0 ? afterIpfs.slice(0, slashIdx) : afterIpfs; const remainder = slashIdx >= 0 ? afterIpfs.slice(slashIdx + 1) : undefined; @@ -82,8 +81,7 @@ export function createIpfsSubdomainMiddleware({ try { const targetCid = cidToV1Base32(pathCid); const rootHost = matchedEntry.host; - const pathSuffix = - remainder !== undefined ? `/${remainder}` : '/'; + const pathSuffix = remainder !== undefined ? `/${remainder}` : '/'; res.redirect( 302, `${req.protocol}://${targetCid}.${rootHost}${pathSuffix}`, diff --git a/src/routes/ar-io-info-builder.ts b/src/routes/ar-io-info-builder.ts index fac7a0552..03c143cfd 100644 --- a/src/routes/ar-io-info-builder.ts +++ b/src/routes/ar-io-info-builder.ts @@ -117,6 +117,13 @@ export interface HttpsigInfo { attestation?: HttpsigAttestationInfo; } +/** + * IPFS configuration exposed in the info endpoint. + */ +export interface IpfsInfo { + enabled: true; +} + /** * Complete AR.IO info endpoint response structure. */ @@ -131,6 +138,7 @@ export interface ArIoInfoResponse { rateLimiter?: RateLimiterInfo; x402?: X402Info; httpsig?: HttpsigInfo; + ipfs?: IpfsInfo; } /** @@ -174,6 +182,9 @@ export interface ArIoInfoConfig { rsaPublicKey: string; }; }; + ipfs?: { + enabled: boolean; + }; } /** @@ -311,5 +322,9 @@ export function buildArIoInfo(config: ArIoInfoConfig): ArIoInfoResponse { }; } + if (config.ipfs?.enabled) { + response.ipfs = { enabled: true }; + } + return response; } diff --git a/src/routes/ar-io.ts b/src/routes/ar-io.ts index 603df24b0..9b0023c72 100644 --- a/src/routes/ar-io.ts +++ b/src/routes/ar-io.ts @@ -217,6 +217,7 @@ export const arIoInfoHandler = (_req: Request, res: Response) => { : undefined, } : undefined, + ipfs: config.IPFS_ENABLED ? { enabled: true } : undefined, }); res.status(200).send(response); diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index c2d49c071..74d1efe33 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -114,7 +114,10 @@ function createIpfsPathHandler({ const v1Base32 = cidToV1Base32(cidString); const rootHost = config.ARNS_ROOT_HOSTS[0].host; const pathSuffix = path !== undefined ? `/${path}` : ''; - res.redirect(302, `${req.protocol}://${v1Base32}.${rootHost}${pathSuffix}`); + res.redirect( + 302, + `${req.protocol}://${v1Base32}.${rootHost}${pathSuffix}`, + ); return; } @@ -154,8 +157,7 @@ async function handleIpfsRequest({ routeType: 'path' | 'subdomain'; }): Promise { const startTime = Date.now(); - const ipfsPath = - path !== undefined ? `${cidString}/${path}` : cidString; + const ipfsPath = path !== undefined ? `${cidString}/${path}` : cidString; parentLog.debug('Handling IPFS request', { cidString, path, routeType }); From ffd5ee23f66cd68e5e114fcc5954b226dc7aea28 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 21 Apr 2026 02:08:55 +0000 Subject: [PATCH 05/47] refactor: use existing admin block API for IPFS moderation (PE-9067) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove custom text-file blocklist in favor of the existing PUT /ar-io/admin/block-data API. Operators block IPFS CIDs the same way they block Arweave TX IDs — unified moderation, single API. Removed: IpfsBlocklist class, IPFS_BLOCKLIST_PATH config, blocklist volume mount. IpfsService now uses DataBlockListValidator (SQLite). --- docker-compose.yaml | 1 - docs/envs.md | 1 - src/config.ts | 5 -- src/ipfs/ipfs-blocklist.ts | 115 ------------------------------------- src/ipfs/ipfs-service.ts | 14 ++--- src/system.ts | 12 +--- 6 files changed, 8 insertions(+), 140 deletions(-) delete mode 100644 src/ipfs/ipfs-blocklist.ts diff --git a/docker-compose.yaml b/docker-compose.yaml index 5fa65da88..371425fb3 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -132,7 +132,6 @@ services: - IPFS_CACHE_PATH=${IPFS_CACHE_PATH:-} - IPFS_CACHE_MAX_SIZE_BYTES=${IPFS_CACHE_MAX_SIZE_BYTES:-} - IPFS_CACHE_CLEANUP_THRESHOLD=${IPFS_CACHE_CLEANUP_THRESHOLD:-} - - IPFS_BLOCKLIST_PATH=${IPFS_BLOCKLIST_PATH:-} - IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET=${IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET:-} - IPFS_RATE_LIMITER_IP_REFILL_PER_SEC=${IPFS_RATE_LIMITER_IP_REFILL_PER_SEC:-} - IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET=${IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET:-} diff --git a/docs/envs.md b/docs/envs.md index d4b7a15dd..471044988 100644 --- a/docs/envs.md +++ b/docs/envs.md @@ -374,7 +374,6 @@ as a Docker Compose sidecar via the `ipfs` profile). | IPFS_CACHE_PATH | String | data/ipfs-cache | Directory for cached IPFS content | | IPFS_CACHE_MAX_SIZE_BYTES | Number | 10737418240 (10 GB) | Maximum cache size before LRU eviction | | IPFS_CACHE_CLEANUP_THRESHOLD | Number | 3600 | Age in seconds before cached files become eviction candidates | -| IPFS_BLOCKLIST_PATH | String | data/ipfs-blocklist.txt | Path to CID blocklist file (one CID per line, hot-reloaded) | | IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET | Number | 50000 | IPFS rate limiter: max tokens per IP bucket | | IPFS_RATE_LIMITER_IP_REFILL_PER_SEC | Number | 5 | IPFS rate limiter: token refill rate per second (IP bucket) | | IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET | Number | 200000 | IPFS rate limiter: max tokens per resource bucket | diff --git a/src/config.ts b/src/config.ts index 72f70c80a..082a6eec5 100644 --- a/src/config.ts +++ b/src/config.ts @@ -2184,11 +2184,6 @@ export const IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS = +env.varOrDefault( '3600', ); -export const IPFS_BLOCKLIST_PATH = env.varOrDefault( - 'IPFS_BLOCKLIST_PATH', - 'data/ipfs-blocklist.txt', -); - export const IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET = +env.varOrDefault( 'IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET', '50000', diff --git a/src/ipfs/ipfs-blocklist.ts b/src/ipfs/ipfs-blocklist.ts deleted file mode 100644 index f20ba355b..000000000 --- a/src/ipfs/ipfs-blocklist.ts +++ /dev/null @@ -1,115 +0,0 @@ -/** - * AR.IO Gateway - * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. - * - * SPDX-License-Identifier: AGPL-3.0-or-later - */ -import fs from 'node:fs'; -import winston from 'winston'; -import { watch, FSWatcher } from 'chokidar'; - -import { cidToV1Base32, isValidCid } from '../lib/ipfs-cid.js'; - -export class IpfsBlocklist { - private log: winston.Logger; - private filePath: string; - private blockedCids: Set = new Set(); - private watcher: FSWatcher | null = null; - private reloadTimer: NodeJS.Timeout | null = null; - - constructor({ log, filePath }: { log: winston.Logger; filePath: string }) { - this.log = log.child({ class: this.constructor.name }); - this.filePath = filePath; - } - - async load(): Promise { - try { - const content = await fs.promises.readFile(this.filePath, 'utf-8'); - const newSet = new Set(); - - for (const line of content.split('\n')) { - const trimmed = line.trim(); - if (trimmed === '' || trimmed.startsWith('#')) continue; - - if (isValidCid(trimmed)) { - // Normalize to CIDv1 base32 for consistent matching - try { - newSet.add(cidToV1Base32(trimmed)); - } catch { - this.log.warn('Failed to normalize CID in blocklist', { - cid: trimmed, - }); - } - } else { - this.log.warn('Invalid CID in blocklist, skipping', { - line: trimmed, - }); - } - } - - this.blockedCids = newSet; - this.log.info('IPFS blocklist loaded', { count: newSet.size }); - } catch (error: any) { - if (error.code === 'ENOENT') { - this.log.debug('IPFS blocklist file not found, no CIDs blocked', { - filePath: this.filePath, - }); - this.blockedCids = new Set(); - } else { - this.log.error('Failed to load IPFS blocklist', { - message: error.message, - }); - } - } - } - - isBlocked(cidString: string): boolean { - try { - const normalized = cidToV1Base32(cidString); - return this.blockedCids.has(normalized); - } catch { - return false; - } - } - - startWatching(): void { - this.watcher = watch(this.filePath, { - ignoreInitial: true, - awaitWriteFinish: { stabilityThreshold: 1000 }, - }); - - this.watcher.on('change', () => { - this.log.info('IPFS blocklist file changed, reloading'); - this.scheduleReload(); - }); - - this.watcher.on('add', () => { - this.log.info('IPFS blocklist file created, loading'); - this.scheduleReload(); - }); - } - - private scheduleReload(): void { - if (this.reloadTimer) { - clearTimeout(this.reloadTimer); - } - this.reloadTimer = setTimeout(() => { - this.load().catch((error) => { - this.log.error('Failed to reload IPFS blocklist', { - message: error.message, - }); - }); - }, 1000); - } - - stop(): void { - if (this.watcher) { - this.watcher.close(); - this.watcher = null; - } - if (this.reloadTimer) { - clearTimeout(this.reloadTimer); - this.reloadTimer = null; - } - } -} diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 1711190d1..115000dee 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -13,7 +13,7 @@ import { Span } from '@opentelemetry/api'; import { cidToV1Base32 } from '../lib/ipfs-cid.js'; import { startChildSpan } from '../tracing.js'; import { IpfsFsCache } from './ipfs-cache.js'; -import { IpfsBlocklist } from './ipfs-blocklist.js'; +import { DataBlockListValidator } from '../types.js'; import { KuboDataSource, IpfsBlockedError, @@ -32,23 +32,23 @@ export class IpfsService { private log: winston.Logger; private dataSource: KuboDataSource; private cache: IpfsFsCache; - private blocklist: IpfsBlocklist; + private blockListValidator: DataBlockListValidator; constructor({ log, dataSource, cache, - blocklist, + blockListValidator, }: { log: winston.Logger; dataSource: KuboDataSource; cache: IpfsFsCache; - blocklist: IpfsBlocklist; + blockListValidator: DataBlockListValidator; }) { this.log = log.child({ class: this.constructor.name }); this.dataSource = dataSource; this.cache = cache; - this.blocklist = blocklist; + this.blockListValidator = blockListValidator; } async getContent({ @@ -78,8 +78,8 @@ export class IpfsService { const normalizedCid = cidToV1Base32(cidString); span.setAttribute('ipfs.cid_normalized', normalizedCid); - // Check blocklist - if (this.blocklist.isBlocked(normalizedCid)) { + // Check blocklist (uses the same admin API as Arweave data moderation) + if (await this.blockListValidator.isIdBlocked(normalizedCid)) { metrics.ipfsBlockedTotal.inc(); span.setAttribute('ipfs.blocked', true); throw new IpfsBlockedError(`CID is blocked: ${normalizedCid}`); diff --git a/src/system.ts b/src/system.ts index d3a3b943e..794083a91 100644 --- a/src/system.ts +++ b/src/system.ts @@ -1462,14 +1462,12 @@ if (dataVerificationWorker !== undefined) { import { KuboDataSource } from './ipfs/kubo-data-source.js'; import { IpfsFsCache } from './ipfs/ipfs-cache.js'; -import { IpfsBlocklist } from './ipfs/ipfs-blocklist.js'; import { IpfsService } from './ipfs/ipfs-service.js'; import { createIpfsRateLimiter } from './ipfs/ipfs-rate-limiter.js'; import { RateLimiter } from './limiter/types.js'; export let ipfsService: IpfsService | undefined; export let ipfsRateLimiter: RateLimiter | undefined; -export let ipfsBlocklist: IpfsBlocklist | undefined; if (config.IPFS_ENABLED) { log.info('IPFS subsystem enabled, initializing...'); @@ -1487,18 +1485,11 @@ if (config.IPFS_ENABLED) { maxSizeBytes: config.IPFS_CACHE_MAX_SIZE_BYTES, }); - ipfsBlocklist = new IpfsBlocklist({ - log, - filePath: config.IPFS_BLOCKLIST_PATH, - }); - await ipfsBlocklist.load(); - ipfsBlocklist.startWatching(); - ipfsService = new IpfsService({ log, dataSource: kuboDataSource, cache: ipfsCache, - blocklist: ipfsBlocklist, + blockListValidator: dataBlockListValidator, }); ipfsRateLimiter = createIpfsRateLimiter(); @@ -1541,7 +1532,6 @@ export const shutdown = async (exitCode = 0) => { } // Clean up system components - ipfsBlocklist?.stop(); eventEmitter.removeAllListeners(); arIOPeerManager.stopUpdatingPeers(); dataSqliteWalCleanupWorker?.stop(); From 52ab93c2ac76200f2b96405a04016766b66218aa Mon Sep 17 00:00:00 2001 From: root Date: Tue, 21 Apr 2026 02:17:53 +0000 Subject: [PATCH 06/47] refactor: match IPFS rate limits to Arweave defaults, use admin API for moderation (PE-9067) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - IPFS rate limiter defaults now match Arweave (100K IP tokens, 20/s refill, 1M resource tokens, 100/s refill) - Remove custom text-file blocklist — use existing PUT /ar-io/admin/block-data API for CID moderation (unified with Arweave content moderation) - Remove IPFS_BLOCKLIST_PATH config and volume mount - Add IPFS cache volume mount to docker-compose for persistence --- src/config.ts | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/config.ts b/src/config.ts index 082a6eec5..dd8842326 100644 --- a/src/config.ts +++ b/src/config.ts @@ -2186,22 +2186,22 @@ export const IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS = +env.varOrDefault( export const IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET = +env.varOrDefault( 'IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET', - '50000', + '100000', ); export const IPFS_RATE_LIMITER_IP_REFILL_PER_SEC = +env.varOrDefault( 'IPFS_RATE_LIMITER_IP_REFILL_PER_SEC', - '5', + '20', ); export const IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET = +env.varOrDefault( 'IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET', - '200000', + '1000000', ); export const IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC = +env.varOrDefault( 'IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC', - '20', + '100', ); export const IPFS_MAX_RESPONSE_SIZE_BYTES = +env.varOrDefault( From 967cc13cb507a705178eb4fb48f45bfe385bd28b Mon Sep 17 00:00:00 2001 From: root Date: Tue, 21 Apr 2026 02:35:08 +0000 Subject: [PATCH 07/47] docs: add IPFS Grafana dashboard, update moderation docs and rate limit defaults (PE-9067) - Add IPFS Grafana dashboard example (requests/sec, cache hit rate, latency percentiles, content size, blocked requests, route type) - Update OpenAPI spec: block-data endpoint accepts IPFS CIDs - Update ipfs-integration.md: admin API for moderation (not text file) - Match IPFS rate limiter defaults to Arweave (100K IP, 1M resource) --- docs/ipfs-integration.md | 23 +++--- docs/openapi.yaml | 7 +- .../dashboards/examples/ipfs-example.json | 72 +++++++++++++++++++ 3 files changed, 87 insertions(+), 15 deletions(-) create mode 100644 monitoring/grafana/dashboards/examples/ipfs-example.json diff --git a/docs/ipfs-integration.md b/docs/ipfs-integration.md index 93d3cd9bb..8a60ef8f7 100644 --- a/docs/ipfs-integration.md +++ b/docs/ipfs-integration.md @@ -325,22 +325,21 @@ and IPFS storage budgets. ## Security and Moderation -### CID Blocklist +### Content Moderation -The blocklist file (`IPFS_BLOCKLIST_PATH`, default `data/ipfs-blocklist.txt`) -allows operators to block specific content: +IPFS content moderation uses the same admin API as Arweave data moderation. +Block a CID using the existing endpoint: -``` -# Blocked content - one CID per line -QmBlockedContent1... -bafybeiblockedcontent2... -# Comments start with # +```bash +curl -X PUT http://localhost:4000/ar-io/admin/block-data \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{"id": "bafkreigbk3hjz6oyiywqf7eknthwc2osvt5xi6b6igwljn2qrxkthqgrp4", "source": "manual", "notes": "Reason for block"}' ``` -- CIDs are normalized before matching, so blocking a CIDv0 also blocks its - CIDv1 equivalent and vice versa. -- The file is watched for changes and reloaded automatically. -- Blocked requests return HTTP 451. +- Pass the CIDv1 base32 string as the `id` field (same field used for Arweave TX IDs). +- Blocked requests return HTTP 451 (Unavailable for Legal Reasons). +- One unified moderation system for all content (Arweave and IPFS). ### Rate Limiting diff --git a/docs/openapi.yaml b/docs/openapi.yaml index 5f2a487cb..6a1d33b3c 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -2809,12 +2809,13 @@ paths: '/ar-io/admin/block-data': put: tags: [Admin] - summary: Blocks transactions or data-items so your AR.IO Gateway will not serve them. + summary: Blocks transactions, data-items, or IPFS CIDs so your AR.IO Gateway will not serve them. description: | - Submits a TX ID/data-item ID or sha-256 content hash for content you do not want your AR.IO Gateway to serve. Once submitted, your Gateway will not respond to requests for these transactions or data-items. + Submits a TX ID/data-item ID, IPFS CID, or sha-256 content hash for content you do not want your AR.IO Gateway to serve. Once submitted, your Gateway will not respond to requests for these transactions, data-items, or IPFS CIDs. + For IPFS content, pass the CIDv1 base32 string (e.g. bafkreigbk3hjz6oyiywqf7eknthwc2osvt5xi6b6igwljn2qrxkthqgrp4) as the id field. The gateway returns HTTP 451 for blocked CIDs. - WARNING - Testing a TX ID here WILL result in that data being blocked by your Gateway. + WARNING - Testing an ID here WILL result in that data being blocked by your Gateway. operationId: adminBlockData requestBody: required: true diff --git a/monitoring/grafana/dashboards/examples/ipfs-example.json b/monitoring/grafana/dashboards/examples/ipfs-example.json new file mode 100644 index 000000000..259a399f6 --- /dev/null +++ b/monitoring/grafana/dashboards/examples/ipfs-example.json @@ -0,0 +1,72 @@ +{ + "annotations": { "list": [] }, + "editable": true, + "panels": [ + { + "datasource": { "type": "prometheus", "uid": "prometheus" }, + "fieldConfig": { "defaults": { "unit": "reqps" } }, + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, + "id": 1, + "title": "IPFS Requests/sec", + "type": "timeseries", + "targets": [{ "expr": "sum(rate(ipfs_requests_total[5m])) by (status)", "legendFormat": "{{status}}" }] + }, + { + "datasource": { "type": "prometheus", "uid": "prometheus" }, + "fieldConfig": { "defaults": { "unit": "percentunit" } }, + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }, + "id": 2, + "title": "IPFS Cache Hit Rate", + "type": "stat", + "targets": [{ "expr": "sum(rate(ipfs_cache_hit_total[5m])) / (sum(rate(ipfs_cache_hit_total[5m])) + sum(rate(ipfs_cache_miss_total[5m])))", "legendFormat": "hit rate" }] + }, + { + "datasource": { "type": "prometheus", "uid": "prometheus" }, + "fieldConfig": { "defaults": { "unit": "s" } }, + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 }, + "id": 3, + "title": "IPFS Request Duration (p50/p95/p99)", + "type": "timeseries", + "targets": [ + { "expr": "histogram_quantile(0.50, sum(rate(ipfs_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p50" }, + { "expr": "histogram_quantile(0.95, sum(rate(ipfs_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p95" }, + { "expr": "histogram_quantile(0.99, sum(rate(ipfs_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p99" } + ] + }, + { + "datasource": { "type": "prometheus", "uid": "prometheus" }, + "fieldConfig": { "defaults": { "unit": "bytes" } }, + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 }, + "id": 4, + "title": "IPFS Content Size Distribution", + "type": "timeseries", + "targets": [ + { "expr": "histogram_quantile(0.50, sum(rate(ipfs_content_size_bytes_bucket[5m])) by (le))", "legendFormat": "p50 size" }, + { "expr": "histogram_quantile(0.95, sum(rate(ipfs_content_size_bytes_bucket[5m])) by (le))", "legendFormat": "p95 size" } + ] + }, + { + "datasource": { "type": "prometheus", "uid": "prometheus" }, + "fieldConfig": { "defaults": { "unit": "short" } }, + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 }, + "id": 5, + "title": "IPFS Blocked Requests", + "type": "timeseries", + "targets": [{ "expr": "sum(rate(ipfs_blocked_total[5m]))", "legendFormat": "blocked/sec" }] + }, + { + "datasource": { "type": "prometheus", "uid": "prometheus" }, + "fieldConfig": { "defaults": { "unit": "reqps" } }, + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 }, + "id": 6, + "title": "IPFS by Route Type", + "type": "timeseries", + "targets": [{ "expr": "sum(rate(ipfs_requests_total[5m])) by (route_type)", "legendFormat": "{{route_type}}" }] + } + ], + "schemaVersion": 39, + "tags": ["ipfs", "ar-io"], + "time": { "from": "now-1h", "to": "now" }, + "title": "AR.IO IPFS", + "uid": "ar-io-ipfs" +} From 2f48d0e78a2c182852bd6a938780a235d57e3229 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 21 Apr 2026 02:44:01 +0000 Subject: [PATCH 08/47] fix: enforce size limit, fix stream leak, update docs defaults (PE-9067) - Enforce IPFS_MAX_RESPONSE_SIZE_BYTES (reject with 413 when Content-Length exceeds limit) - Destroy response stream on 404/408/504 from Kubo (prevents socket leak) - Fix docs/envs.md rate limiter defaults to match config.ts - Add Grafana dashboard example for IPFS metrics - Update OpenAPI spec for CID content moderation --- docs/envs.md | 8 ++++---- src/ipfs/ipfs-service.ts | 17 +++++++++++++++++ src/ipfs/kubo-data-source.ts | 2 ++ src/routes/ipfs.ts | 10 ++++++++++ src/system.ts | 1 + 5 files changed, 34 insertions(+), 4 deletions(-) diff --git a/docs/envs.md b/docs/envs.md index 471044988..67c7a62c4 100644 --- a/docs/envs.md +++ b/docs/envs.md @@ -374,8 +374,8 @@ as a Docker Compose sidecar via the `ipfs` profile). | IPFS_CACHE_PATH | String | data/ipfs-cache | Directory for cached IPFS content | | IPFS_CACHE_MAX_SIZE_BYTES | Number | 10737418240 (10 GB) | Maximum cache size before LRU eviction | | IPFS_CACHE_CLEANUP_THRESHOLD | Number | 3600 | Age in seconds before cached files become eviction candidates | -| IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET | Number | 50000 | IPFS rate limiter: max tokens per IP bucket | -| IPFS_RATE_LIMITER_IP_REFILL_PER_SEC | Number | 5 | IPFS rate limiter: token refill rate per second (IP bucket) | -| IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET | Number | 200000 | IPFS rate limiter: max tokens per resource bucket | -| IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC | Number | 20 | IPFS rate limiter: token refill rate per second (resource bucket) | +| IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET | Number | 100000 | IPFS rate limiter: max tokens per IP bucket | +| IPFS_RATE_LIMITER_IP_REFILL_PER_SEC | Number | 20 | IPFS rate limiter: token refill rate per second (IP bucket) | +| IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET | Number | 1000000 | IPFS rate limiter: max tokens per resource bucket | +| IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC | Number | 100 | IPFS rate limiter: token refill rate per second (resource bucket) | | IPFS_MAX_RESPONSE_SIZE_BYTES | Number | 1073741824 (1 GB) | Maximum IPFS content size the gateway will serve | diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 115000dee..57da3c14f 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -18,6 +18,7 @@ import { KuboDataSource, IpfsBlockedError, IpfsNotFoundError, + IpfsSizeLimitError, } from './kubo-data-source.js'; import * as metrics from '../metrics.js'; @@ -33,22 +34,26 @@ export class IpfsService { private dataSource: KuboDataSource; private cache: IpfsFsCache; private blockListValidator: DataBlockListValidator; + private maxResponseSizeBytes: number; constructor({ log, dataSource, cache, blockListValidator, + maxResponseSizeBytes, }: { log: winston.Logger; dataSource: KuboDataSource; cache: IpfsFsCache; blockListValidator: DataBlockListValidator; + maxResponseSizeBytes: number; }) { this.log = log.child({ class: this.constructor.name }); this.dataSource = dataSource; this.cache = cache; this.blockListValidator = blockListValidator; + this.maxResponseSizeBytes = maxResponseSizeBytes; } async getContent({ @@ -124,6 +129,18 @@ export class IpfsService { 'ipfs.content_type': result.contentType, }); + // Enforce size limit when Content-Length is known + if ( + this.maxResponseSizeBytes > 0 && + result.size > 0 && + result.size > this.maxResponseSizeBytes + ) { + result.stream.destroy(); + throw new IpfsSizeLimitError( + `IPFS content size ${result.size} exceeds limit ${this.maxResponseSizeBytes}`, + ); + } + // Stream directly to the client while writing to a temp file on disk // for caching. No memory buffering — handles files of any size. this.streamToCache( diff --git a/src/ipfs/kubo-data-source.ts b/src/ipfs/kubo-data-source.ts index 4d358b4e3..f5f5b9df0 100644 --- a/src/ipfs/kubo-data-source.ts +++ b/src/ipfs/kubo-data-source.ts @@ -106,12 +106,14 @@ export class KuboDataSource { signal?.removeEventListener('abort', onClientAbort); if (response.status === 404) { + (response.data as Readable).destroy(); throw new IpfsNotFoundError( `IPFS content not found: /ipfs/${ipfsPath}`, ); } if (response.status === 408 || response.status === 504) { + (response.data as Readable).destroy(); throw new IpfsTimeoutError( `Kubo timed out resolving: /ipfs/${ipfsPath}`, ); diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index 74d1efe33..03a7eb4a0 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -20,6 +20,7 @@ import { IpfsService } from '../ipfs/ipfs-service.js'; import { IpfsBlockedError, IpfsNotFoundError, + IpfsSizeLimitError, IpfsTimeoutError, IpfsUnavailableError, } from '../ipfs/kubo-data-source.js'; @@ -295,6 +296,15 @@ async function handleIpfsRequest({ return; } + if (error instanceof IpfsSizeLimitError) { + metrics.ipfsRequestsTotal.inc({ + route_type: routeType, + status: 'size_exceeded', + }); + res.status(413).json({ error: 'Content exceeds size limit' }); + return; + } + // Client disconnected if (error.name === 'AbortError') { return; diff --git a/src/system.ts b/src/system.ts index 794083a91..420a696b5 100644 --- a/src/system.ts +++ b/src/system.ts @@ -1490,6 +1490,7 @@ if (config.IPFS_ENABLED) { dataSource: kuboDataSource, cache: ipfsCache, blockListValidator: dataBlockListValidator, + maxResponseSizeBytes: config.IPFS_MAX_RESPONSE_SIZE_BYTES, }); ipfsRateLimiter = createIpfsRateLimiter(); From a5ceb2ee865332248fcd88001145f14b839d2c04 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 21 Apr 2026 03:03:36 +0000 Subject: [PATCH 09/47] fix: address CodeRabbit review feedback (PE-9067) - Use positiveIntOrDefault for all IPFS numeric configs (prevents NaN) - URL-encode IPFS path segments (prevents request injection) - Guard contentLength parse against NaN (fallback to 0) - Use SANDBOX_PROTOCOL for redirects (correct behind TLS termination) - Enforce size limit during streaming (catches chunked responses) - Set failed=true before cleanup in stream end handler (race fix) - Use conservative size estimate for x402 when Content-Length unknown - Increment cache hit/miss counters in route handler - Fix env var name: IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS - Guard Grafana cache hit rate expr against division by zero - Destroy response stream on 404/408/504 from Kubo (socket leak) --- docker-compose.yaml | 2 +- docs/envs.md | 2 +- .../dashboards/examples/ipfs-example.json | 2 +- src/config.ts | 45 +++++++++---------- src/ipfs/ipfs-service.ts | 18 ++++++++ src/ipfs/kubo-data-source.ts | 15 ++++++- src/middleware/ipfs.ts | 2 +- src/routes/ipfs.ts | 13 ++++-- 8 files changed, 67 insertions(+), 32 deletions(-) diff --git a/docker-compose.yaml b/docker-compose.yaml index 371425fb3..07510b21d 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -131,7 +131,7 @@ services: - IPFS_STREAM_STALL_TIMEOUT_MS=${IPFS_STREAM_STALL_TIMEOUT_MS:-} - IPFS_CACHE_PATH=${IPFS_CACHE_PATH:-} - IPFS_CACHE_MAX_SIZE_BYTES=${IPFS_CACHE_MAX_SIZE_BYTES:-} - - IPFS_CACHE_CLEANUP_THRESHOLD=${IPFS_CACHE_CLEANUP_THRESHOLD:-} + - IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS=${IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS:-} - IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET=${IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET:-} - IPFS_RATE_LIMITER_IP_REFILL_PER_SEC=${IPFS_RATE_LIMITER_IP_REFILL_PER_SEC:-} - IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET=${IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET:-} diff --git a/docs/envs.md b/docs/envs.md index 67c7a62c4..d112078eb 100644 --- a/docs/envs.md +++ b/docs/envs.md @@ -373,7 +373,7 @@ as a Docker Compose sidecar via the `ipfs` profile). | IPFS_STREAM_STALL_TIMEOUT_MS | Number | 30000 | Stall timeout — max time with no data before aborting stream (ms) | | IPFS_CACHE_PATH | String | data/ipfs-cache | Directory for cached IPFS content | | IPFS_CACHE_MAX_SIZE_BYTES | Number | 10737418240 (10 GB) | Maximum cache size before LRU eviction | -| IPFS_CACHE_CLEANUP_THRESHOLD | Number | 3600 | Age in seconds before cached files become eviction candidates | +| IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS | Number | 3600 | Age in seconds before cached files become eviction candidates | | IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET | Number | 100000 | IPFS rate limiter: max tokens per IP bucket | | IPFS_RATE_LIMITER_IP_REFILL_PER_SEC | Number | 20 | IPFS rate limiter: token refill rate per second (IP bucket) | | IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET | Number | 1000000 | IPFS rate limiter: max tokens per resource bucket | diff --git a/monitoring/grafana/dashboards/examples/ipfs-example.json b/monitoring/grafana/dashboards/examples/ipfs-example.json index 259a399f6..c3f4772f4 100644 --- a/monitoring/grafana/dashboards/examples/ipfs-example.json +++ b/monitoring/grafana/dashboards/examples/ipfs-example.json @@ -18,7 +18,7 @@ "id": 2, "title": "IPFS Cache Hit Rate", "type": "stat", - "targets": [{ "expr": "sum(rate(ipfs_cache_hit_total[5m])) / (sum(rate(ipfs_cache_hit_total[5m])) + sum(rate(ipfs_cache_miss_total[5m])))", "legendFormat": "hit rate" }] + "targets": [{ "expr": "sum(rate(ipfs_cache_hit_total[5m])) / clamp_min(sum(rate(ipfs_cache_hit_total[5m])) + sum(rate(ipfs_cache_miss_total[5m])), 1)", "legendFormat": "hit rate" }] }, { "datasource": { "type": "prometheus", "uid": "prometheus" }, diff --git a/src/config.ts b/src/config.ts index dd8842326..9b48bb9d1 100644 --- a/src/config.ts +++ b/src/config.ts @@ -2156,14 +2156,14 @@ export const IPFS_KUBO_URL = env.varOrDefault( 'http://kubo:8080', ); -export const IPFS_KUBO_REQUEST_TIMEOUT_MS = +env.varOrDefault( +export const IPFS_KUBO_REQUEST_TIMEOUT_MS = env.positiveIntOrDefault( 'IPFS_KUBO_REQUEST_TIMEOUT_MS', - '30000', + 30000, ); -export const IPFS_STREAM_STALL_TIMEOUT_MS = +env.varOrDefault( +export const IPFS_STREAM_STALL_TIMEOUT_MS = env.positiveIntOrDefault( 'IPFS_STREAM_STALL_TIMEOUT_MS', - '30000', + 30000, ); export const IPFS_CACHE_PATH = env.varOrDefault( @@ -2171,40 +2171,39 @@ export const IPFS_CACHE_PATH = env.varOrDefault( 'data/ipfs-cache', ); -export const IPFS_CACHE_MAX_SIZE_BYTES = +env.varOrDefault( +export const IPFS_CACHE_MAX_SIZE_BYTES = env.positiveIntOrDefault( 'IPFS_CACHE_MAX_SIZE_BYTES', - `${10 * 1024 * 1024 * 1024}`, // 10 GB + 10 * 1024 * 1024 * 1024, // 10 GB ); // Reserved for future cache cleanup worker. Currently unused — LRU eviction // in the in-memory index handles cache bounding. After restarts, disk usage // may temporarily exceed IPFS_CACHE_MAX_SIZE_BYTES until the index rebuilds. -export const IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS = +env.varOrDefault( - 'IPFS_CACHE_CLEANUP_THRESHOLD', - '3600', +export const IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS = env.positiveIntOrDefault( + 'IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS', + 3600, ); -export const IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET = +env.varOrDefault( +export const IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET = env.positiveIntOrDefault( 'IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET', - '100000', + 100000, ); -export const IPFS_RATE_LIMITER_IP_REFILL_PER_SEC = +env.varOrDefault( +export const IPFS_RATE_LIMITER_IP_REFILL_PER_SEC = env.positiveIntOrDefault( 'IPFS_RATE_LIMITER_IP_REFILL_PER_SEC', - '20', + 20, ); -export const IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET = +env.varOrDefault( - 'IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET', - '1000000', -); +export const IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET = + env.positiveIntOrDefault( + 'IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET', + 1000000, + ); -export const IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC = +env.varOrDefault( - 'IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC', - '100', -); +export const IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC = + env.positiveIntOrDefault('IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC', 100); -export const IPFS_MAX_RESPONSE_SIZE_BYTES = +env.varOrDefault( +export const IPFS_MAX_RESPONSE_SIZE_BYTES = env.positiveIntOrDefault( 'IPFS_MAX_RESPONSE_SIZE_BYTES', - `${1 * 1024 * 1024 * 1024}`, // 1 GB + 1 * 1024 * 1024 * 1024, // 1 GB ); diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 57da3c14f..99967ee74 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -230,6 +230,23 @@ export class IpfsService { stream.on('data', (chunk: Buffer) => { if (failed) return; bytesWritten += chunk.length; + + // Enforce size limit during streaming (catches chunked responses + // that lack Content-Length) + if ( + this.maxResponseSizeBytes > 0 && + bytesWritten > this.maxResponseSizeBytes + ) { + failed = true; + stream.destroy( + new IpfsSizeLimitError( + `IPFS content exceeds limit during streaming: ${bytesWritten} > ${this.maxResponseSizeBytes}`, + ), + ); + cleanup(); + return; + } + if (writeStream) { writeStream.write(chunk); } else { @@ -241,6 +258,7 @@ export class IpfsService { stream.on('end', () => { if (failed || !writeStream) { // If writeStream never became ready, discard + failed = true; cleanup(); return; } diff --git a/src/ipfs/kubo-data-source.ts b/src/ipfs/kubo-data-source.ts index f5f5b9df0..776793d08 100644 --- a/src/ipfs/kubo-data-source.ts +++ b/src/ipfs/kubo-data-source.ts @@ -54,8 +54,16 @@ export class KuboDataSource { }): Promise { signal?.throwIfAborted(); + // URL-encode path segments to prevent breaking the upstream request + const encodedPath = + path !== undefined && path !== '' + ? path + .split('/') + .map((seg) => encodeURIComponent(seg)) + .join('/') + : undefined; const ipfsPath = - path !== undefined && path !== '' ? `${cidString}/${path}` : cidString; + encodedPath !== undefined ? `${cidString}/${encodedPath}` : cidString; const url = `${this.kuboUrl}/ipfs/${ipfsPath}`; const span = startChildSpan( @@ -128,10 +136,13 @@ export class KuboDataSource { } const stream = response.data as Readable; - const contentLength = parseInt( + const rawContentLength = parseInt( response.headers['content-length'] ?? '0', 10, ); + const contentLength = Number.isFinite(rawContentLength) + ? rawContentLength + : 0; const contentType = response.headers['content-type'] ?? 'application/octet-stream'; diff --git a/src/middleware/ipfs.ts b/src/middleware/ipfs.ts index 7a47ff4b7..539883f67 100644 --- a/src/middleware/ipfs.ts +++ b/src/middleware/ipfs.ts @@ -84,7 +84,7 @@ export function createIpfsSubdomainMiddleware({ const pathSuffix = remainder !== undefined ? `/${remainder}` : '/'; res.redirect( 302, - `${req.protocol}://${targetCid}.${rootHost}${pathSuffix}`, + `${config.SANDBOX_PROTOCOL ?? req.protocol}://${targetCid}.${rootHost}${pathSuffix}`, ); return; } catch { diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index 03a7eb4a0..8a976c699 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -117,7 +117,7 @@ function createIpfsPathHandler({ const pathSuffix = path !== undefined ? `/${path}` : ''; res.redirect( 302, - `${req.protocol}://${v1Base32}.${rootHost}${pathSuffix}`, + `${config.SANDBOX_PROTOCOL ?? req.protocol}://${v1Base32}.${rootHost}${pathSuffix}`, ); return; } @@ -170,8 +170,10 @@ async function handleIpfsRequest({ }); // Check payment and rate limits (x402 + rate limiting in one call). - // Content size is needed for token calculation and payment pricing. - const contentSize = result.size > 0 ? result.size : 1024; // min 1KB for pricing + // When Content-Length is unknown (chunked), use a conservative estimate + // that gets corrected in the token adjustment after streaming. + const contentSize = + result.size > 0 ? result.size : config.IPFS_MAX_RESPONSE_SIZE_BYTES; const limitCheck = await checkPaymentAndRateLimits({ req, res, @@ -216,6 +218,11 @@ async function handleIpfsRequest({ // Track metrics const cacheStatus = result.cached ? 'hit' : 'miss'; metrics.ipfsRequestsTotal.inc({ route_type: routeType, status: 'success' }); + if (result.cached) { + metrics.ipfsCacheHitTotal.inc(); + } else { + metrics.ipfsCacheMissTotal.inc(); + } if (result.size > 0) { metrics.ipfsContentSizeHistogram.observe(result.size); } From 04fcc12a443cd3d2f4276ac5f1902290b94606d9 Mon Sep 17 00:00:00 2001 From: root Date: Tue, 21 Apr 2026 14:47:35 +0000 Subject: [PATCH 10/47] feat: emit data-cached webhook event for IPFS content (PE-9067) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit IPFS cached content now triggers the same data-cached event as Arweave, feeding into the existing webhook system. No new configuration needed — operators' existing webhook consumers see IPFS content automatically. --- src/ipfs/ipfs-cache.ts | 16 ++++++++++++++++ src/system.ts | 1 + 2 files changed, 17 insertions(+) diff --git a/src/ipfs/ipfs-cache.ts b/src/ipfs/ipfs-cache.ts index c7847c1b0..54af569ba 100644 --- a/src/ipfs/ipfs-cache.ts +++ b/src/ipfs/ipfs-cache.ts @@ -5,12 +5,16 @@ * SPDX-License-Identifier: AGPL-3.0-or-later */ import crypto from 'node:crypto'; +import EventEmitter from 'node:events'; import fs from 'node:fs'; import { Readable } from 'node:stream'; import { pipeline } from 'node:stream/promises'; import winston from 'winston'; import { LRUCache } from 'lru-cache'; +import * as events from '../events.js'; +import { currentUnixTimestamp } from '../lib/time.js'; + interface CacheEntry { size: number; contentType: string; @@ -20,18 +24,22 @@ export class IpfsFsCache { private log: winston.Logger; private baseDir: string; private index: LRUCache; + private eventEmitter?: EventEmitter; constructor({ log, basePath, maxSizeBytes, + eventEmitter, }: { log: winston.Logger; basePath: string; maxSizeBytes: number; + eventEmitter?: EventEmitter; }) { this.log = log.child({ class: this.constructor.name }); this.baseDir = basePath; + this.eventEmitter = eventEmitter; this.index = new LRUCache({ maxSize: maxSizeBytes, sizeCalculation: (entry) => entry.size, @@ -212,6 +220,14 @@ export class IpfsFsCache { key, size, }); + + this.eventEmitter?.emit(events.DATA_CACHED, { + id: cidString, + hash: key, + dataSize: size, + contentType, + cachedAt: currentUnixTimestamp(), + }); } catch (error: any) { this.log.error('Failed to finalize cached IPFS content', { cidString, diff --git a/src/system.ts b/src/system.ts index 420a696b5..f7aab452b 100644 --- a/src/system.ts +++ b/src/system.ts @@ -1483,6 +1483,7 @@ if (config.IPFS_ENABLED) { log, basePath: config.IPFS_CACHE_PATH, maxSizeBytes: config.IPFS_CACHE_MAX_SIZE_BYTES, + eventEmitter, }); ipfsService = new IpfsService({ From c7e4b4a47cb4dab9f3ca4fa29ee32584bea8293a Mon Sep 17 00:00:00 2001 From: root Date: Tue, 21 Apr 2026 15:57:42 +0000 Subject: [PATCH 11/47] fix: match blocked response format to Arweave, add webhook event for IPFS cache (PE-9067) - IPFS blocked response now matches Arweave format (plain text with ID) - Emit data-cached webhook event when IPFS content is cached (same event as Arweave, feeds into content scanner pipeline automatically) --- src/routes/ipfs.ts | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index 8a976c699..056ef8ae9 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -272,7 +272,11 @@ async function handleIpfsRequest({ 'Cache-Control', `public, max-age=${config.CACHE_BLOCKED_MAX_AGE}, immutable`, ); - res.status(451).json({ error: 'Content blocked' }); + res + .status(451) + .send( + `Requested content blocked by this node's content policy. Blocked ID: ${cidString}`, + ); return; } From ffa34cd6e4601850e0b8c405230919363851d35a Mon Sep 17 00:00:00 2001 From: vilenarios Date: Mon, 22 Jun 2026 17:23:05 +0000 Subject: [PATCH 12/47] chore: restore multiformats in yarn.lock after develop merge The merge took develop's lockfile (no multiformats entry); the IPFS feature depends on it. yarn install re-resolved it. Co-Authored-By: Claude Opus 4.8 (1M context) --- yarn.lock | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/yarn.lock b/yarn.lock index 085fc74e0..91b424a8b 100644 --- a/yarn.lock +++ b/yarn.lock @@ -12517,6 +12517,11 @@ multer@2.1.1, multer@^2.0.2: concat-stream "^2.0.0" type-is "^1.6.18" +multiformats@^13.1.0: + version "13.4.2" + resolved "https://registry.yarnpkg.com/multiformats/-/multiformats-13.4.2.tgz#309ee17d3946db9a6954cf6832aeb2b4b10dd0f3" + integrity sha512-eh6eHCrRi1+POZ3dA+Dq1C6jhP1GNtr9CRINMb67OKzqW9I5DUuZM/3jLPlzhgpGeiNUlEGEbkCYChXMCc/8DQ== + multiformats@^9.4.2: version "9.9.0" resolved "https://registry.yarnpkg.com/multiformats/-/multiformats-9.9.0.tgz#c68354e7d21037a8f1f8833c8ccd68618e8f1d37" From d383ca2bd915461a6692b998736bc1dcb4d6d462 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Mon, 22 Jun 2026 17:51:07 +0000 Subject: [PATCH 13/47] fix(ipfs): cache small/fast objects (eliminate mkdir-vs-end race) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit streamToCache created its write stream inside an async mkdir().then(), so a small or fast Kubo response could emit 'end' before the mkdir resolved — leaving writeStream null, hitting the silent-discard branch, and dropping the cache entry. Large files won the race and cached; small files never did (every request re-fetched from Kubo, X-Cache always MISS). Create the temp dir eagerly in the IpfsFsCache constructor and open the write stream synchronously in streamToCache, removing the race and the now-unneeded pending-chunk buffering. Verified in an isolated Kubo + gateway run: small object now MISS then HIT; large object still HIT. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/ipfs/ipfs-cache.ts | 6 ++++ src/ipfs/ipfs-service.ts | 74 +++++++++++++++++----------------------- 2 files changed, 37 insertions(+), 43 deletions(-) diff --git a/src/ipfs/ipfs-cache.ts b/src/ipfs/ipfs-cache.ts index 54af569ba..7aa352e85 100644 --- a/src/ipfs/ipfs-cache.ts +++ b/src/ipfs/ipfs-cache.ts @@ -40,6 +40,12 @@ export class IpfsFsCache { this.log = log.child({ class: this.constructor.name }); this.baseDir = basePath; this.eventEmitter = eventEmitter; + // Create the temp directory eagerly so the streaming cache writer can + // open its write stream synchronously. Doing the mkdir lazily per-request + // raced against fast/small responses ending before the async mkdir + // resolved, which left the write stream null and silently dropped the + // cache entry (so small IPFS objects never cached). + fs.mkdirSync(this.tempDir(), { recursive: true }); this.index = new LRUCache({ maxSize: maxSizeBytes, sizeCalculation: (entry) => entry.size, diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 99967ee74..85d0c0c85 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -184,48 +184,43 @@ export class IpfsService { stream: Readable, contentType: string, ): void { - const cacheDir = `${this.cache.getCachePath()}/tmp`; - const tempPath = `${cacheDir}/${crypto.randomBytes(16).toString('hex')}`; - let writeStream: fs.WriteStream | null = null; + const tempPath = `${this.cache.getCachePath()}/tmp/${crypto + .randomBytes(16) + .toString('hex')}`; let bytesWritten = 0; let failed = false; - const pendingChunks: Buffer[] = []; + + // Create the write stream synchronously. The temp directory is created + // eagerly in the IpfsFsCache constructor, so there is no async mkdir to + // race against the stream's 'end' event. (A previous async-mkdir version + // dropped the cache entry whenever a small/fast response ended before the + // mkdir resolved — leaving writeStream null — so small objects never + // cached.) createWriteStream opens the fd lazily and buffers writes until + // it is ready, so synchronous creation is safe. + let writeStream: fs.WriteStream; + try { + writeStream = fs.createWriteStream(tempPath); + } catch (error: any) { + this.log.error('Failed to open IPFS cache write stream', { + cid: cidString, + message: error.message, + }); + return; + } const cleanup = () => { - if (writeStream) { - writeStream.destroy(); - writeStream = null; - } - pendingChunks.length = 0; + writeStream.destroy(); fs.promises.unlink(tempPath).catch(() => {}); }; - // Create temp directory and write stream - fs.promises - .mkdir(cacheDir, { recursive: true }) - .then(() => { - if (failed) return; - writeStream = fs.createWriteStream(tempPath); - writeStream.on('error', (error) => { - failed = true; - this.log.error('Cache write stream error', { - cid: cidString, - message: error.message, - }); - cleanup(); - }); - // Flush any chunks that arrived before writeStream was ready - for (const chunk of pendingChunks) { - writeStream.write(chunk); - } - pendingChunks.length = 0; - }) - .catch((error) => { - failed = true; - this.log.error('Failed to create cache temp dir', { - message: error.message, - }); + writeStream.on('error', (error) => { + failed = true; + this.log.error('Cache write stream error', { + cid: cidString, + message: error.message, }); + cleanup(); + }); stream.on('data', (chunk: Buffer) => { if (failed) return; @@ -247,18 +242,11 @@ export class IpfsService { return; } - if (writeStream) { - writeStream.write(chunk); - } else { - // Buffer until writeStream is ready (typically only first 1-2 chunks) - pendingChunks.push(chunk); - } + writeStream.write(chunk); }); stream.on('end', () => { - if (failed || !writeStream) { - // If writeStream never became ready, discard - failed = true; + if (failed) { cleanup(); return; } From 241ca2672c2465a82ce81befa518152dfe88bc11 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Mon, 22 Jun 2026 20:07:10 +0000 Subject: [PATCH 14/47] feat(arns): serve ArNS names whose ANT record targets an IPFS CID ANT records now carry a targetProtocol (0=Arweave, 1=IPFS) and the record target can be an IPFS CID instead of an Arweave TX ID (@ar.io/sdk 4.0.0). Previously the on-demand resolver read only transactionId and validated it as a 43-char Arweave ID, so a CID-targeted name failed to resolve. - on-demand resolver: read targetProtocol; validate the target as a CID when protocol is IPFS, else as an Arweave ID; surface protocol on the resolution (cached transparently as part of NameResolution). - NameResolution: optional protocol field ('arweave' | 'ipfs'); undefined treated as 'arweave' for backward compat (e.g. trusted-gateway hops). - arns middleware: when a name resolves to an IPFS CID and IPFS serving is enabled, hand off to the IPFS handler (sets ipfsCid/ipfsPath, mirroring the IPFS subdomain middleware) instead of the Arweave data handler. Completes the 'ArNS -> IPFS CID' phase the IPFS PR was foundation for. Verified: typecheck + lint clean, resolver unit tests pass, and live on-demand Solana resolution of existing names still serves via Arweave. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/middleware/arns.ts | 31 ++++++++++++++++++++++- src/resolution/on-demand-arns-resolver.ts | 15 ++++++++++- src/routes/arns.ts | 16 ++++++++++++ src/types.d.ts | 10 ++++++++ 4 files changed, 70 insertions(+), 2 deletions(-) diff --git a/src/middleware/arns.ts b/src/middleware/arns.ts index 04fac887a..f8d2282c2 100644 --- a/src/middleware/arns.ts +++ b/src/middleware/arns.ts @@ -27,9 +27,15 @@ const MAX_ARNS_NAME_LENGTH = 51; export const createArnsMiddleware = ({ dataHandler, nameResolver, + ipfsHandler, }: { dataHandler: Handler; nameResolver: NameResolver; + // Optional handler for ArNS names whose ANT record resolves to an IPFS CID + // (`targetProtocol === ipfs`). Provided only when IPFS serving is enabled; + // when absent, IPFS-protocol resolutions fall through to the Arweave data + // path (which will 404 on a CID, the correct behavior with IPFS disabled). + ipfsHandler?: Handler; }): Handler => asyncMiddleware(async (req, res, next) => { // Skip all ArNS processing if no root ArNS hosts are configured. @@ -166,6 +172,11 @@ export const createArnsMiddleware = ({ const resolutionDuration = Date.now() - resolutionStart; span.setAttribute('arns.resolution_duration_ms', resolutionDuration); + // Set when the ANT record resolves to an IPFS CID (targetProtocol + // ipfs); routes the final handoff to the IPFS handler instead of the + // Arweave data handler. + let serveViaIpfs = false; + if (resolution.statusCode === 451) { span.setAttribute('arns.blocked', true); span.setAttribute('http.status_code', 451); @@ -185,6 +196,20 @@ export const createArnsMiddleware = ({ req.dataId = resolvedId; req.manifestPath = manifestPath; + // If the ANT record points at an IPFS CID, hand the request to the + // IPFS handler (when IPFS serving is enabled) instead of the Arweave + // data path. The handler reads `ipfsCid`/`ipfsPath` off the request, + // matching the IPFS subdomain middleware's contract. + const arnsProtocol = + (resolution as { protocol?: 'arweave' | 'ipfs' }).protocol ?? + 'arweave'; + if (arnsProtocol === 'ipfs' && ipfsHandler !== undefined) { + serveViaIpfs = true; + (req as any).ipfsCid = resolvedId; + (req as any).ipfsPath = manifestPath; + span.setAttribute('arns.protocol', 'ipfs'); + } + // Parse ArNS name components const parts = arnsSubdomain.split('_'); const basename = parts.pop() ?? ''; // last part is basename @@ -305,7 +330,11 @@ export const createArnsMiddleware = ({ if (req.arns?.ttl !== undefined) { res.header('Cache-Control', `public, max-age=${req.arns.ttl}`); } - dataHandler(req, res, next); + if (serveViaIpfs && ipfsHandler !== undefined) { + ipfsHandler(req, res, next); + } else { + dataHandler(req, res, next); + } } catch (error: any) { span.recordException(error); span.setStatus({ code: SpanStatusCode.ERROR }); diff --git a/src/resolution/on-demand-arns-resolver.ts b/src/resolution/on-demand-arns-resolver.ts index 3b8a40002..d43f8ab83 100644 --- a/src/resolution/on-demand-arns-resolver.ts +++ b/src/resolution/on-demand-arns-resolver.ts @@ -7,6 +7,7 @@ import winston from 'winston'; import { isValidDataId } from '../lib/validation.js'; +import { isValidCid } from '../lib/ipfs-cid.js'; import { NameResolution, NameResolver } from '../types.js'; import { ArNSNameDataWithName, SolanaANTReadable } from '@ar.io/sdk'; import { address, type Rpc, type SolanaRpcApiMainnet } from '@solana/kit'; @@ -117,12 +118,24 @@ export class OnDemandArNSResolver implements NameResolver { const ttl = antRecord.ttlSeconds; const index = antRecord.index; - if (!isValidDataId(resolvedId)) { + // ANT records carry a `targetProtocol` (0 = Arweave, 1 = IPFS). For + // IPFS records the `transactionId` field holds an IPFS CID rather than + // an Arweave TX ID, so validate it against the CID format instead of + // the 43-char Arweave-ID format. (`targetProtocol` may be absent on + // older ANTs, in which case it defaults to Arweave.) + const protocol = antRecord.targetProtocol === 1 ? 'ipfs' : 'arweave'; + + if (protocol === 'ipfs') { + if (!isValidCid(resolvedId)) { + throw new Error('Invalid resolved IPFS CID'); + } + } else if (!isValidDataId(resolvedId)) { throw new Error('Invalid resolved data ID'); } return { name, resolvedId, + protocol, resolvedAt: Date.now(), antId, ttl, diff --git a/src/routes/arns.ts b/src/routes/arns.ts index 5578bae46..811b8c767 100644 --- a/src/routes/arns.ts +++ b/src/routes/arns.ts @@ -7,19 +7,35 @@ import { Router } from 'express'; import * as config from '../config.js'; +import log from '../log.js'; import { createArnsMiddleware } from '../middleware/arns.js'; import { createSandboxMiddleware } from '../middleware/sandbox.js'; import * as system from '../system.js'; import { dataHandler } from './data/index.js'; +import { createIpfsHandler } from './ipfs.js'; import { headerNames } from '../constants.js'; import { sendNotFound } from './data/handlers.js'; import { DEFAULT_ARNS_TTL_SECONDS } from '../resolution/trusted-gateway-arns-resolver.js'; export const arnsRouter = Router(); +// When IPFS serving is enabled, ArNS names whose ANT record resolves to an +// IPFS CID (targetProtocol = ipfs) are served through the same IPFS handler +// used by the path/subdomain routes. +const ipfsHandler = + config.IPFS_ENABLED && system.ipfsService !== undefined + ? createIpfsHandler({ + log, + ipfsService: system.ipfsService, + rateLimiter: system.ipfsRateLimiter, + paymentProcessor: system.paymentProcessor, + }) + : undefined; + export const arnsMiddleware = createArnsMiddleware({ dataHandler, nameResolver: system.nameResolver, + ipfsHandler, }); if (config.ARNS_ROOT_HOSTS.length > 0) { diff --git a/src/types.d.ts b/src/types.d.ts index e248cd6fd..8296769ad 100644 --- a/src/types.d.ts +++ b/src/types.d.ts @@ -1232,6 +1232,16 @@ export interface ValidNameResolution { name: string; statusCode?: number; resolvedId: string; + /** + * Storage protocol of the resolved target, mirroring the ANT record's + * `targetProtocol` (0 = Arweave, 1 = IPFS). `'arweave'` means `resolvedId` + * is a 43-char Arweave TX / data-item ID served from the Arweave data path; + * `'ipfs'` means `resolvedId` is an IPFS CID served via the Kubo IPFS path. + * Optional for backward compatibility — `undefined` is treated as + * `'arweave'` by consumers (e.g. trusted-gateway hops that don't yet carry + * the protocol). + */ + protocol?: 'arweave' | 'ipfs'; resolvedAt: number; ttl: number; /** From 27316b14b4f5a693476d78ea3f048de4a2c9eb87 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Mon, 22 Jun 2026 21:06:59 +0000 Subject: [PATCH 15/47] refactor(arns,ipfs): review touches for ArNS->IPFS Final-review hardening of the ArNS->IPFS feature: - fix(cache-control): ArNS->IPFS responses no longer send immutable/1-year. The IPFS handler only sets `immutable` for direct /ipfs/{CID} (and {CID}.host) requests; when reached via an ArNS name (mutable name->CID binding) it keeps the ArNS-TTL Cache-Control the ArNS middleware set, so a record repoint isn't pinned in caches for ~a year (cf. PE-9072). - feat(headers): emit signed `X-ArNS-Protocol: arweave|ipfs` on resolutions and add `protocol` (+ `resolvedId`) to the /ar-io/resolver/:name JSON, so clients know whether X-ArNS-Resolved-Id is a TX ID or a CID. Added x-arns-protocol to TRIGGER_HEADERS so it's part of the signature. - feat(httpsig): body-bind IPFS responses with RFC 9530 Content-Digest. The SHA-256 is computed at cache-write time and emitted on cache hits (in CO_SIGNABLE_HEADERS, so HTTPSIG signs it). Misses stream without it; the signed ETag=CID still attests identity. - test(ipfs): cache digest round-trip + legacy (digest-less) entry coverage. - docs: rewrite the ipfs-integration.md Phase 2 section to the shipped targetProtocol design (was speculative), glossary Target Protocol entry, CLAUDE.md note. Documented the trusted-gateway-resolver protocol limitation. Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 5 +- docs/glossary.md | 7 +++ docs/ipfs-integration.md | 73 ++++++++++++++---------- src/constants.ts | 8 +++ src/ipfs/ipfs-cache.test.ts | 108 ++++++++++++++++++++++++++++++++++++ src/ipfs/ipfs-cache.ts | 14 ++++- src/ipfs/ipfs-service.ts | 21 ++++++- src/lib/httpsig.ts | 1 + src/middleware/arns.ts | 3 + src/routes/arns.ts | 10 ++++ src/routes/ipfs.ts | 20 ++++++- 11 files changed, 236 insertions(+), 34 deletions(-) create mode 100644 src/ipfs/ipfs-cache.test.ts diff --git a/CLAUDE.md b/CLAUDE.md index 4b6c01777..f7185f0ef 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -74,7 +74,10 @@ yarn service:start / stop / restart / status / logs bundle unbundling, verification, and webhooks. Controlled by `START_WRITERS`. - IPFS serving (`src/ipfs/`) is opt-in via `IPFS_ENABLED`. Uses a Kubo sidecar for content retrieval with its own cache, rate limiter, and blocklist. Routes - mount before ArNS in `app.ts`. See `docs/ipfs-integration.md`. + mount before ArNS in `app.ts`. ArNS names whose ANT record has + `targetProtocol: ipfs` resolve to a CID and are routed to the same IPFS + handler by the ArNS middleware (`src/middleware/arns.ts`); the on-demand + resolver reads `targetProtocol`. See `docs/ipfs-integration.md`. - Responses include trust headers indicating verification status. - HTTPSIG signs response headers (RFC 9421); `Content-Digest` is in `CO_SIGNABLE_HEADERS` so when present it binds the body to the signature. diff --git a/docs/glossary.md b/docs/glossary.md index 2aa786547..7c05b4e52 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -275,6 +275,13 @@ containing bundled data. human-readable names (like "my-app") to their corresponding Arweave [item IDs](#item-id). +**Target Protocol (ANT record)** - A field on an ANT record (`0` = Arweave, +`1` = IPFS, default `0`) declaring where the record's target lives. For an +Arweave target the resolved id is a 43-char transaction/data-item ID served from +the Arweave data path; for an IPFS target it is an [IPFS CID](#ipfs-cid) served +via the Kubo IPFS path. The gateway surfaces it as `X-ArNS-Protocol` and uses it +to route an ArNS name to either backend. See `docs/ipfs-integration.md`. + **Path Resolution** - The process of interpreting URL paths to determine which transaction data to serve. Includes manifest resolution (looking up paths in a manifest's routing table), index path resolution (adding index.html), and diff --git a/docs/ipfs-integration.md b/docs/ipfs-integration.md index 8a60ef8f7..f44bc19c1 100644 --- a/docs/ipfs-integration.md +++ b/docs/ipfs-integration.md @@ -44,12 +44,13 @@ placing a CID in the URL path or subdomain. The gateway validates, rate-limits, and caches the request, then proxies it to the local Kubo node. No ArNS resolution is involved. -**Phase 2 -- ArNS to CID Resolution (future):** ANT (Arweave Name Token) -records will be able to store an IPFS CID in their `transactionId` field. When -the gateway resolves an ArNS name and detects that the resolved ID is a CID -rather than an Arweave transaction ID, it routes the request to the IPFS service -instead of the Arweave data pipeline. This requires no contract changes -- CID -detection happens at the gateway level. +**Phase 2 -- ArNS to CID Resolution (implemented):** ANT (Arweave Name Token) +records carry a `targetProtocol` field (`0` = Arweave, `1` = IPFS) and the +content target may be an IPFS CID. When the gateway resolves an ArNS name whose +record targets IPFS, it routes the request to the IPFS service instead of the +Arweave data pipeline. See +[Phase 2: ArNS to IPFS Resolution](#phase-2-arns-to-ipfs-resolution) for the +full flow, headers, and caching semantics. ## Architecture @@ -451,36 +452,52 @@ serve IPFS-hosted content without the user needing to know the CID. ### How It Works -1. **ANT record stores a CID.** The Arweave Name Token contract's - `transactionId` field accepts any string. An ANT owner sets it to an IPFS CID - (e.g., `bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi`) - instead of an Arweave transaction ID. +1. **ANT record targets a CID with `targetProtocol: ipfs`.** ANT records carry + a `targetProtocol` field (`0` = Arweave, `1` = IPFS, default `0`) alongside + the content target. An ANT owner (or controller) sets the target to an IPFS + CID and `targetProtocol` to `1` -- e.g. via the AR.IO SDK: + `ant.setUndernameRecord({ undername: 'ipfs', transactionId: '', ttlSeconds: 300, targetProtocol: 1 })`. -2. **Gateway resolves the ArNS name.** The standard ArNS resolution pipeline - fetches the ANT record and extracts the `transactionId`. +2. **The on-demand resolver reads `targetProtocol`.** `OnDemandArNSResolver` + reads the record's `targetProtocol`. When it is IPFS, it validates the target + as a CID (`isValidCid`) rather than as a 43-char Arweave ID, and surfaces + `protocol: 'ipfs'` on the resolution (carried through the resolution cache). -3. **CID detection.** The gateway inspects the resolved ID. If it matches CID - format (multibase-prefixed, valid multicodec), it is classified as an IPFS - CID rather than an Arweave transaction ID. +3. **The ArNS middleware routes by protocol.** When `protocol === 'ipfs'` and + IPFS serving is enabled, the middleware sets `ipfsCid`/`ipfsPath` on the + request and hands off to the same IPFS handler used by the path/subdomain + routes -- otherwise it serves via the Arweave data path as before. -4. **Route to IPFS service.** Instead of fetching from the Arweave data pipeline - (cache, S3, peers, chunks), the gateway routes the request to the IPFS - service, which follows the same blocklist, rate limit, cache, and Kubo fetch - pipeline described above. +4. **The IPFS service serves it.** Blocklist -> rate limit -> cache -> Kubo + fetch, exactly as for a direct `/ipfs/{CID}` request. + +The response carries the full ArNS envelope (`X-ArNS-Name`, `X-ArNS-Resolved-Id` += the CID, `X-ArNS-Ant-Id`, `X-ArNS-TTL-Seconds`) plus `X-ArNS-Protocol: ipfs`, +`X-Ar-Io-Source: ipfs`, `X-Ipfs-Path`, and `ETag` = the CID. HTTPSIG signs the +ArNS binding headers and the IPFS serving headers (and `Content-Digest` on cache +hits), so the name->CID binding and the served bytes are both attested. ### Key Design Decisions -- **No contract changes.** CID detection happens entirely at the gateway level. - The ANT contract's `transactionId` field is a free-form string, so it already - accepts CIDs. +- **Explicit `targetProtocol`, not shape-sniffing.** Protocol comes from the + ANT record's `targetProtocol` field, so an Arweave TX ID and an IPFS CID are + never confused by guessing from string shape. +- **Mutable-binding cache semantics.** A direct `/ipfs/{CID}` request is cached + `immutable` (content-addressed). But an ArNS name -> CID binding is **mutable** + (the record can be repointed), so ArNS-served IPFS responses use the ArNS TTL + for `Cache-Control`, not `immutable` -- a record update is never pinned in + caches for ~a year (cf. PE-9072). - **Transparent to users.** A user visiting `my-dapp.arweave.dev` does not need - to know whether the content is on Arweave or IPFS. The URL is the same either - way. -- **Owner-controlled.** The ANT owner decides where content lives by setting - the `transactionId` to either an Arweave TX ID or an IPFS CID. Switching - between storage backends is a single contract interaction. + to know whether the content is on Arweave or IPFS; the URL is identical. +- **Owner/controller-controlled.** Switching a name between Arweave and IPFS is + a single ANT record update (target + `targetProtocol`). - **Caching and moderation apply.** All Phase 1 protections (blocklist, rate limits, cache) apply to ArNS-resolved IPFS content. +- **Resolver scope.** Protocol awareness lives in the on-demand resolver. The + trusted-gateway resolver does not yet propagate `targetProtocol` across + gateway hops, so a name whose resolution falls through to that path would be + treated as Arweave. Keep `on-demand` ahead of `gateway` in + `ARNS_RESOLVER_PRIORITY_ORDER` for IPFS-targeted names. ## Differences from Arweave Data Serving @@ -491,7 +508,7 @@ serve IPFS-hosted content without the user needing to know the CID. | **Path resolution** | Manifest JSON parsed by the gateway | UnixFS directories resolved by Kubo | | **Verification** | Merkle proofs verified by the gateway | Block hashes verified internally by Kubo | | **Caching** | Archival (operators retain data long-term, often without eviction) | LRU with bounded size (eviction when full) | -| **Cache-Control** | Varies by verification status and data source trust | `immutable` with 1-year max-age (content-addressed = never changes) | +| **Cache-Control** | Varies by verification status and data source trust | Direct CID: `immutable`, 1-year max-age (content-addressed). Via ArNS: the ArNS TTL (mutable name->CID binding) | | **Rate limiting** | Shared Arweave token bucket | Separate IPFS token bucket | | **Data source** | Multi-source fallback chain (cache, S3, peers, gateways, Arweave nodes) | Single source: local Kubo node | | **Upstream network** | Arweave protocol (block weave, mining incentives) | IPFS/libp2p (DHT, Bitswap) | diff --git a/src/constants.ts b/src/constants.ts index 4280003d8..707625a51 100644 --- a/src/constants.ts +++ b/src/constants.ts @@ -61,6 +61,14 @@ export const headerNames = { arnsBasename: 'X-ArNS-Basename', arnsRecord: 'X-ArNS-Record', arnsResolvedId: 'X-ArNS-Resolved-Id', + /** + * Storage protocol of the resolved target: `arweave` (the resolved id is an + * Arweave TX / data-item ID served from the Arweave data path) or `ipfs` + * (the resolved id is an IPFS CID served via the Kubo IPFS path). Mirrors the + * ANT record's `targetProtocol`. Lets a client know how to interpret + * `X-ArNS-Resolved-Id` (43-char TX ID vs CID) without guessing from its shape. + */ + arnsProtocol: 'X-ArNS-Protocol', dataId: 'X-AR-IO-Data-Id', /** * Identifier of the Solana program that owns the ANT mint that diff --git a/src/ipfs/ipfs-cache.test.ts b/src/ipfs/ipfs-cache.test.ts new file mode 100644 index 000000000..558a59233 --- /dev/null +++ b/src/ipfs/ipfs-cache.test.ts @@ -0,0 +1,108 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { describe, it, beforeEach, afterEach } from 'node:test'; +import { strict as assert } from 'node:assert'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import * as crypto from 'node:crypto'; + +import { createTestLogger } from '../../test/test-logger.js'; +import { IpfsFsCache } from './ipfs-cache.js'; + +const log = createTestLogger({ suite: 'IpfsFsCache' }); + +async function streamToBuffer(stream: NodeJS.ReadableStream): Promise { + const chunks: Buffer[] = []; + for await (const chunk of stream) { + chunks.push(Buffer.from(chunk)); + } + return Buffer.concat(chunks); +} + +describe('IpfsFsCache', () => { + let baseDir: string; + let cache: IpfsFsCache; + + beforeEach(() => { + baseDir = fs.mkdtempSync(path.join(os.tmpdir(), 'ipfs-cache-test-')); + cache = new IpfsFsCache({ + log, + basePath: baseDir, + maxSizeBytes: 10 * 1024 * 1024, + }); + }); + + afterEach(() => { + fs.rmSync(baseDir, { recursive: true, force: true }); + }); + + it('round-trips bytes and the content digest through putFromFile/get', async () => { + const cid = 'bafkreiefysqevlhofnppvnhaptsjt7cqi6wcjsllgpn7ml5i4256v2rbwu'; + const content = Buffer.from('hello digest test'); + const digest = crypto + .createHash('sha256') + .update(content) + .digest('base64url'); + + // The streaming writer hands putFromFile an already-written temp file under + // the cache's tmp dir; emulate that here. + const tempPath = path.join( + baseDir, + 'tmp', + crypto.randomBytes(8).toString('hex'), + ); + fs.writeFileSync(tempPath, content); + + await cache.putFromFile( + cid, + tempPath, + content.length, + 'text/plain', + undefined, + digest, + ); + + const got = await cache.get(cid); + assert.ok(got, 'expected a cache hit'); + assert.equal(got.digest, digest, 'digest should round-trip'); + assert.equal(got.size, content.length); + assert.equal(got.contentType, 'text/plain'); + assert.deepEqual(await streamToBuffer(got.stream), content); + }); + + it('returns undefined for an uncached CID', async () => { + const got = await cache.get( + 'bafkreigh2akiscaildcqabsyg3dfr6chu3fgpregiymsck7e7aqa4s52zy', + ); + assert.equal(got, undefined); + }); + + it('rebuilds a digest-less entry (older cache) without a digest', async () => { + const cid = 'bafkreiefysqevlhofnppvnhaptsjt7cqi6wcjsllgpn7ml5i4256v2rbwu'; + const content = Buffer.from('legacy entry, no digest'); + const tempPath = path.join( + baseDir, + 'tmp', + crypto.randomBytes(8).toString('hex'), + ); + fs.writeFileSync(tempPath, content); + + // No digest argument — emulates an entry written before Content-Digest support. + await cache.putFromFile( + cid, + tempPath, + content.length, + 'application/octet-stream', + ); + + const got = await cache.get(cid); + assert.ok(got); + assert.equal(got.digest, undefined, 'legacy entry should have no digest'); + assert.deepEqual(await streamToBuffer(got.stream), content); + }); +}); diff --git a/src/ipfs/ipfs-cache.ts b/src/ipfs/ipfs-cache.ts index 7aa352e85..99fa21cea 100644 --- a/src/ipfs/ipfs-cache.ts +++ b/src/ipfs/ipfs-cache.ts @@ -18,6 +18,13 @@ import { currentUnixTimestamp } from '../lib/time.js'; interface CacheEntry { size: number; contentType: string; + /** + * base64url SHA-256 of the cached bytes, computed at cache-write time. + * Emitted as an RFC 9530 Content-Digest on cache hits so the body can be + * bound into the HTTPSIG signature. Optional: pre-existing cache entries + * (written before this field existed) won't have it. + */ + digest?: string; } export class IpfsFsCache { @@ -111,7 +118,8 @@ export class IpfsFsCache { cidString: string, path?: string, ): Promise< - { stream: Readable; size: number; contentType: string } | undefined + | { stream: Readable; size: number; contentType: string; digest?: string } + | undefined > { const key = this.cacheKey(cidString, path); let entry = this.index.get(key); @@ -139,6 +147,7 @@ export class IpfsFsCache { stream, size: entry.size, contentType: entry.contentType, + digest: entry.digest, }; } catch (error: any) { this.log.error('Failed to read cached IPFS content', { @@ -203,6 +212,7 @@ export class IpfsFsCache { size: number, contentType: string, path?: string, + digest?: string, ): Promise { const key = this.cacheKey(cidString, path); @@ -211,7 +221,7 @@ export class IpfsFsCache { await fs.promises.mkdir(dataDir, { recursive: true }); await fs.promises.rename(tempPath, this.dataPath(key)); - const meta: CacheEntry = { size, contentType }; + const meta: CacheEntry = { size, contentType, digest }; await fs.promises.writeFile( this.metaPath(key), JSON.stringify(meta), diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 85d0c0c85..905a98688 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -27,6 +27,12 @@ export interface IpfsGetContentResult { size: number; contentType: string; cached: boolean; + /** + * base64url SHA-256 of the served bytes, present on cache hits (computed at + * cache-write time). Used to emit an RFC 9530 Content-Digest. Absent on cache + * misses (the body streams straight from Kubo without being hashed inline). + */ + digest?: string; } export class IpfsService { @@ -110,6 +116,7 @@ export class IpfsService { size: cached.size, contentType: cached.contentType, cached: true, + digest: cached.digest, }; } @@ -189,6 +196,9 @@ export class IpfsService { .toString('hex')}`; let bytesWritten = 0; let failed = false; + // Hash the bytes as they're written so cache hits can emit an RFC 9530 + // Content-Digest (body binding) without a re-read. + const hash = crypto.createHash('sha256'); // Create the write stream synchronously. The temp directory is created // eagerly in the IpfsFsCache constructor, so there is no async mkdir to @@ -242,6 +252,7 @@ export class IpfsService { return; } + hash.update(chunk); writeStream.write(chunk); }); @@ -250,10 +261,18 @@ export class IpfsService { cleanup(); return; } + const digest = hash.digest('base64url'); writeStream.end(() => { // Finalize: move temp file into cache this.cache - .putFromFile(cidString, tempPath, bytesWritten, contentType, path) + .putFromFile( + cidString, + tempPath, + bytesWritten, + contentType, + path, + digest, + ) .catch((error) => { this.log.error('Failed to finalize IPFS cache entry', { cid: cidString, diff --git a/src/lib/httpsig.ts b/src/lib/httpsig.ts index 816441153..e7b5723ab 100644 --- a/src/lib/httpsig.ts +++ b/src/lib/httpsig.ts @@ -37,6 +37,7 @@ export const TRIGGER_HEADERS = new Set([ 'x-arweave-tags-truncated', 'x-arns-name', 'x-arns-resolved-id', + 'x-arns-protocol', 'x-arns-ttl-seconds', 'x-arns-ant-program-id', 'x-arns-ant-id', diff --git a/src/middleware/arns.ts b/src/middleware/arns.ts index f8d2282c2..8fdadf960 100644 --- a/src/middleware/arns.ts +++ b/src/middleware/arns.ts @@ -231,6 +231,9 @@ export const createArnsMiddleware = ({ // Populate the ArNS response headers for client visibility res.header(headerNames.arnsName, arnsSubdomain); res.header(headerNames.arnsResolvedId, resolvedId); + // Tell the client how to interpret the resolved id (Arweave TX vs + // IPFS CID). Signed (x-arns-protocol is a trigger header). + res.header(headerNames.arnsProtocol, arnsProtocol); if (basename !== '') { res.header(headerNames.arnsBasename, basename); } diff --git a/src/routes/arns.ts b/src/routes/arns.ts index 811b8c767..80f4ec947 100644 --- a/src/routes/arns.ts +++ b/src/routes/arns.ts @@ -67,7 +67,13 @@ arnsRouter.get('/ar-io/resolver/:name', async (req, res) => { return; } + // Storage protocol of the target (arweave TX id vs ipfs CID). Optional on the + // resolution; absent => arweave for backward compatibility. + const protocol = + (resolved as { protocol?: 'arweave' | 'ipfs' }).protocol ?? 'arweave'; + res.header(headerNames.arnsResolvedId, resolvedId); + res.header(headerNames.arnsProtocol, protocol); res.header( headerNames.arnsTtlSeconds, (ttl ?? DEFAULT_ARNS_TTL_SECONDS).toString(), @@ -86,7 +92,11 @@ arnsRouter.get('/ar-io/resolver/:name', async (req, res) => { res.header(headerNames.arnsLimit, limit.toString()); } res.json({ + // `txId` kept for backward compatibility; for IPFS records it actually + // holds a CID. Prefer `resolvedId` + `protocol`. txId: resolvedId, + resolvedId, + protocol, ttlSeconds: ttl, antId, resolvedAt, diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index 056ef8ae9..031504873 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -31,6 +31,7 @@ import { } from '../handlers/data-handler-utils.js'; import { PaymentProcessor } from '../payments/types.js'; import { extractAllClientIPs } from '../lib/ip-utils.js'; +import { formatContentDigest } from '../lib/digest.js'; export function createIpfsRouter({ log, @@ -203,12 +204,27 @@ async function handleIpfsRequest({ if (result.size > 0) { res.setHeader('Content-Length', result.size); } - // CIDs are content-addressed — content never changes - res.setHeader('Cache-Control', 'public, max-age=29030400, immutable'); + // A direct /ipfs/{CID} or {CID}.host request is content-addressed and thus + // immutable. But when the request arrived via an ArNS name, the name->CID + // binding is MUTABLE and the ArNS middleware already set a TTL-bounded + // Cache-Control — don't override it with `immutable`, or a record update + // would be pinned in browsers/edge caches for ~a year (cf. PE-9072). + if ((req as Request & { arns?: unknown }).arns === undefined) { + res.setHeader('Cache-Control', 'public, max-age=29030400, immutable'); + } res.setHeader('ETag', `"${cidToV1Base32(cidString)}"`); res.setHeader('X-Ipfs-Path', `/ipfs/${ipfsPath}`); res.setHeader('X-Ar-Io-Source', 'ipfs'); + // Body binding (RFC 9530 Content-Digest). When a SHA-256 of the served + // bytes is known (computed at cache-write time, returned on cache hits), + // emit it — it's in CO_SIGNABLE_HEADERS, so HTTPSIG binds the body to the + // signature. Cache hits carry it for free; misses stream without it (the + // signed ETag=CID still attests content identity). + if (result.digest !== undefined) { + res.setHeader('Content-Digest', formatContentDigest(result.digest)); + } + if (result.cached) { res.setHeader('X-Cache', 'HIT'); } else { From fd4033706e145f0702b3e301cfefb017b65abd81 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 23 Jun 2026 01:11:22 +0000 Subject: [PATCH 16/47] test(arns): extract + unit-test resolved-target protocol classification MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ArNS->IPFS routing decision hinges on classifying an ANT record's target as arweave vs ipfs and validating the id accordingly. Extracted that logic from OnDemandArNSResolver into a pure, SDK-free helper (classifyResolvedTarget) and unit-tested it: arweave/ipfs by targetProtocol, undefined+unknown protocol -> arweave (fail-closed), CIDv0/v1 acceptance, and cross-format rejection (CID under arweave, TX id under ipfs, garbage). The ArNS middleware routing itself can't be unit-tested in isolation (it imports system.ts, booting the DI graph — no middleware has unit tests for this reason); it stays covered by live e2e. Also documented the three root/apex cases in ipfs-integration.md: a name's @ record and apex-via-APEX_ARNS_NAME route to IPFS; apex-via-APEX_TX_ID is Arweave-only (bypasses protocol routing). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/ipfs-integration.md | 15 ++++++ src/resolution/on-demand-arns-resolver.ts | 24 +++------ src/resolution/resolved-target.test.ts | 61 +++++++++++++++++++++++ src/resolution/resolved-target.ts | 44 ++++++++++++++++ 4 files changed, 128 insertions(+), 16 deletions(-) create mode 100644 src/resolution/resolved-target.test.ts create mode 100644 src/resolution/resolved-target.ts diff --git a/docs/ipfs-integration.md b/docs/ipfs-integration.md index f44bc19c1..13b0b47cb 100644 --- a/docs/ipfs-integration.md +++ b/docs/ipfs-integration.md @@ -499,6 +499,21 @@ hits), so the name->CID binding and the served bytes are both attested. treated as Arweave. Keep `on-demand` ahead of `gateway` in `ARNS_RESOLVER_PRIORITY_ORDER` for IPFS-targeted names. +### Root and apex names + +"Root" means three different things; only two route to IPFS: + +- **A name's root (`@`) record** -- e.g. `my-name.gateway.tld` with no + undername. The `@` record is just the `'@'` undername and goes through the + same resolution + routing, so an IPFS `@` record serves IPFS. ✅ +- **Gateway apex via `APEX_ARNS_NAME`** -- when the bare apex host has + `APEX_ARNS_NAME` set, the middleware resolves that name through the normal + path, so if its record targets IPFS the apex serves IPFS. ✅ +- **Gateway apex via `APEX_TX_ID`** -- a fixed id served directly to the Arweave + data handler, bypassing resolution and protocol routing. It is Arweave-only; + a CID there will not serve. To serve IPFS at the apex, use `APEX_ARNS_NAME` + pointing at an ANT whose `@` record targets IPFS. ❌ + ## Differences from Arweave Data Serving | Aspect | Arweave | IPFS | diff --git a/src/resolution/on-demand-arns-resolver.ts b/src/resolution/on-demand-arns-resolver.ts index d43f8ab83..d86e7cfd6 100644 --- a/src/resolution/on-demand-arns-resolver.ts +++ b/src/resolution/on-demand-arns-resolver.ts @@ -6,8 +6,7 @@ */ import winston from 'winston'; -import { isValidDataId } from '../lib/validation.js'; -import { isValidCid } from '../lib/ipfs-cid.js'; +import { classifyResolvedTarget } from './resolved-target.js'; import { NameResolution, NameResolver } from '../types.js'; import { ArNSNameDataWithName, SolanaANTReadable } from '@ar.io/sdk'; import { address, type Rpc, type SolanaRpcApiMainnet } from '@solana/kit'; @@ -118,20 +117,13 @@ export class OnDemandArNSResolver implements NameResolver { const ttl = antRecord.ttlSeconds; const index = antRecord.index; - // ANT records carry a `targetProtocol` (0 = Arweave, 1 = IPFS). For - // IPFS records the `transactionId` field holds an IPFS CID rather than - // an Arweave TX ID, so validate it against the CID format instead of - // the 43-char Arweave-ID format. (`targetProtocol` may be absent on - // older ANTs, in which case it defaults to Arweave.) - const protocol = antRecord.targetProtocol === 1 ? 'ipfs' : 'arweave'; - - if (protocol === 'ipfs') { - if (!isValidCid(resolvedId)) { - throw new Error('Invalid resolved IPFS CID'); - } - } else if (!isValidDataId(resolvedId)) { - throw new Error('Invalid resolved data ID'); - } + // Classify + validate the target against the ANT record's + // `targetProtocol` (0 = Arweave, 1 = IPFS). Throws if the id doesn't + // match the format for its protocol (treated as "name didn't resolve"). + const protocol = classifyResolvedTarget( + resolvedId, + antRecord.targetProtocol, + ); return { name, resolvedId, diff --git a/src/resolution/resolved-target.test.ts b/src/resolution/resolved-target.test.ts new file mode 100644 index 000000000..06bc3264c --- /dev/null +++ b/src/resolution/resolved-target.test.ts @@ -0,0 +1,61 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { describe, it } from 'node:test'; +import { strict as assert } from 'node:assert'; + +import { classifyResolvedTarget } from './resolved-target.js'; + +const ARWEAVE_ID = 'M2tMZzF3XAcXvyg9DR6U07Cj5HY-JLgT6tCPujdkKZ0'; // 43-char base64url +const CID_V1 = 'bafybeifx7yeb55armcsxwwitkymga5xf53dxiarykms3ygqic223w5sk3m'; +const CID_V0 = 'QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG'; + +describe('classifyResolvedTarget', () => { + describe('Arweave targets', () => { + it('classifies a 43-char Arweave id as arweave when targetProtocol is 0', () => { + assert.equal(classifyResolvedTarget(ARWEAVE_ID, 0), 'arweave'); + }); + + it('defaults to arweave when targetProtocol is undefined (older ANT)', () => { + assert.equal(classifyResolvedTarget(ARWEAVE_ID, undefined), 'arweave'); + }); + + it('treats unknown protocol numbers as arweave (fail-closed)', () => { + assert.equal(classifyResolvedTarget(ARWEAVE_ID, 2), 'arweave'); + }); + + it('rejects a non-Arweave id under an Arweave protocol', () => { + assert.throws( + () => classifyResolvedTarget(CID_V1, 0), + /Invalid resolved data ID/, + ); + }); + }); + + describe('IPFS targets', () => { + it('classifies a CIDv1 as ipfs when targetProtocol is 1', () => { + assert.equal(classifyResolvedTarget(CID_V1, 1), 'ipfs'); + }); + + it('accepts a CIDv0 under the IPFS protocol', () => { + assert.equal(classifyResolvedTarget(CID_V0, 1), 'ipfs'); + }); + + it('rejects an Arweave id under the IPFS protocol', () => { + assert.throws( + () => classifyResolvedTarget(ARWEAVE_ID, 1), + /Invalid resolved IPFS CID/, + ); + }); + + it('rejects garbage under the IPFS protocol', () => { + assert.throws( + () => classifyResolvedTarget('not-a-cid', 1), + /Invalid resolved IPFS CID/, + ); + }); + }); +}); diff --git a/src/resolution/resolved-target.ts b/src/resolution/resolved-target.ts new file mode 100644 index 000000000..a2a411d64 --- /dev/null +++ b/src/resolution/resolved-target.ts @@ -0,0 +1,44 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { isValidDataId } from '../lib/validation.js'; +import { isValidCid } from '../lib/ipfs-cid.js'; + +export type ResolvedProtocol = 'arweave' | 'ipfs'; + +/** + * Classify (and validate) an ANT record's resolved target. + * + * ANT records carry a `targetProtocol` field (`0` = Arweave, `1` = IPFS; + * absent on older ANTs, which default to Arweave). The target id is an Arweave + * TX / data-item ID for Arweave records and an IPFS CID for IPFS records, so the + * id is validated against the format implied by `targetProtocol`: + * + * - `targetProtocol === 1` -> `ipfs`; the id must be a valid CID. + * - anything else -> `arweave`; the id must be a valid 43-char id. + * + * Returns the resolved protocol, or throws if the id does not match the format + * for that protocol (the caller treats a throw as "name did not resolve"). + * + * Kept as a pure function (no SDK / network / config) so it is unit-testable in + * isolation and shared by any resolver that reads ANT records. + */ +export function classifyResolvedTarget( + resolvedId: string, + targetProtocol: number | undefined, +): ResolvedProtocol { + const protocol: ResolvedProtocol = targetProtocol === 1 ? 'ipfs' : 'arweave'; + + if (protocol === 'ipfs') { + if (!isValidCid(resolvedId)) { + throw new Error('Invalid resolved IPFS CID'); + } + } else if (!isValidDataId(resolvedId)) { + throw new Error('Invalid resolved data ID'); + } + + return protocol; +} From c7fe209c82d36f7c1053252b2ffde181e735e53b Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 04:42:26 +0000 Subject: [PATCH 17/47] feat(arns): propagate resolution protocol across trusted-gateway hops MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The trusted-gateway resolver read the ArNS envelope headers but never read X-ArNS-Protocol, and its isValidDataId gate rejected any non-43-char id — so an upstream IPFS resolution (a CID) was discarded as "invalid data ID" and the protocol classification was lost across a gateway hop. With the default ARNS_RESOLVER_PRIORITY_ORDER of `gateway,on-demand`, that meant a stock gateway silently misrouted IPFS-targeted names to the Arweave path. Read X-ArNS-Protocol from the upstream response and validate resolvedId with the shared classifyResolvedTarget (CID for ipfs, 43-char id for arweave), threading `protocol` into the returned NameResolution. An absent header defaults to arweave, so older peers are unaffected. Multi-protocol resolution now survives a trusted-gateway hop. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- .../trusted-gateway-arns-resolver.test.ts | 88 +++++++++++++++++++ .../trusted-gateway-arns-resolver.ts | 65 +++++++++----- 2 files changed, 132 insertions(+), 21 deletions(-) diff --git a/src/resolution/trusted-gateway-arns-resolver.test.ts b/src/resolution/trusted-gateway-arns-resolver.test.ts index 45ed42fa9..021fb8a77 100644 --- a/src/resolution/trusted-gateway-arns-resolver.test.ts +++ b/src/resolution/trusted-gateway-arns-resolver.test.ts @@ -187,5 +187,93 @@ describe('TrustedGatewayArNSResolver', () => { assert.equal(resolution.antId, undefined); assert.equal(resolution.resolvedId, resolvedId); }); + + it('should propagate protocol=ipfs and a CID from X-ArNS-Protocol', async () => { + const cid = 'bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi'; + interceptorId = axios.interceptors.request.use((config) => { + config.adapter = () => + Promise.resolve({ + status: 200, + statusText: 'OK', + headers: { + [headerNames.arnsResolvedId.toLowerCase()]: cid, + [headerNames.arnsProtocol.toLowerCase()]: 'ipfs', + [headerNames.arnsTtlSeconds.toLowerCase()]: '300', + [headerNames.arnsLimit.toLowerCase()]: '10', + [headerNames.arnsIndex.toLowerCase()]: '0', + }, + config, + data: null, + }); + return config; + }); + + const resolver = new TrustedGatewayArNSResolver({ + log, + trustedGatewayUrl: 'https://__NAME__.turbo-gateway.com', + }); + + const resolution = await resolver.resolve({ name: 'ipfs-name' }); + + assert.equal(resolution.resolvedId, cid); + assert.equal((resolution as { protocol?: string }).protocol, 'ipfs'); + }); + + it('should default protocol to arweave when X-ArNS-Protocol is absent', async () => { + interceptorId = axios.interceptors.request.use((config) => { + config.adapter = () => + Promise.resolve({ + status: 200, + statusText: 'OK', + headers: { + [headerNames.arnsResolvedId.toLowerCase()]: resolvedId, + [headerNames.arnsTtlSeconds.toLowerCase()]: '300', + [headerNames.arnsLimit.toLowerCase()]: '10', + [headerNames.arnsIndex.toLowerCase()]: '0', + }, + config, + data: null, + }); + return config; + }); + + const resolver = new TrustedGatewayArNSResolver({ + log, + trustedGatewayUrl: 'https://__NAME__.turbo-gateway.com', + }); + + const resolution = await resolver.resolve({ name: 'arweave-name' }); + + assert.equal(resolution.resolvedId, resolvedId); + assert.equal((resolution as { protocol?: string }).protocol, 'arweave'); + }); + + it('should fail resolution when protocol=ipfs but id is not a valid CID', async () => { + interceptorId = axios.interceptors.request.use((config) => { + config.adapter = () => + Promise.resolve({ + status: 200, + statusText: 'OK', + headers: { + // a 43-char Arweave id mislabeled as ipfs must be rejected + [headerNames.arnsResolvedId.toLowerCase()]: resolvedId, + [headerNames.arnsProtocol.toLowerCase()]: 'ipfs', + [headerNames.arnsTtlSeconds.toLowerCase()]: '300', + }, + config, + data: null, + }); + return config; + }); + + const resolver = new TrustedGatewayArNSResolver({ + log, + trustedGatewayUrl: 'https://__NAME__.turbo-gateway.com', + }); + + const resolution = await resolver.resolve({ name: 'mislabeled' }); + + assert.equal(resolution.resolvedId, undefined); + }); }); }); diff --git a/src/resolution/trusted-gateway-arns-resolver.ts b/src/resolution/trusted-gateway-arns-resolver.ts index c35dd83c3..b31868084 100644 --- a/src/resolution/trusted-gateway-arns-resolver.ts +++ b/src/resolution/trusted-gateway-arns-resolver.ts @@ -8,7 +8,7 @@ import { default as axios } from 'axios'; import winston from 'winston'; import { headerNames } from '../constants.js'; -import { isValidDataId } from '../lib/validation.js'; +import { classifyResolvedTarget } from './resolved-target.js'; import { NameResolution, NameResolver } from '../types.js'; export const DEFAULT_ARNS_TTL_SECONDS = 60 * 15; // 15 minutes @@ -84,27 +84,50 @@ export class TrustedGatewayArNSResolver implements NameResolver { const index = parseInt(response.headers[headerNames.arnsIndex.toLowerCase()]) || DEFAULT_ARNS_UNDERNAME_INDEX; - if (isValidDataId(resolvedId)) { - this.log.info('Resolved name', { name, nameUrl, resolvedId, ttl }); - return { - name, - statusCode: response.status, - resolvedId, - resolvedAt: Date.now(), - antId, - ttl, - limit, - index, - }; + // Protocol of the upstream resolution (arweave | ipfs). Older peers that + // predate multi-protocol resolution omit X-ArNS-Protocol; an absent header + // defaults to arweave, preserving prior behavior. We validate resolvedId + // against the protocol the peer claims (a CID for ipfs, a 43-char id for + // arweave) via the shared classifier, so a mislabeled or malformed target + // is rejected here rather than mis-served downstream. + const protocolHeader = + response.headers[headerNames.arnsProtocol.toLowerCase()]; + const targetProtocol = protocolHeader === 'ipfs' ? 1 : 0; + if (typeof resolvedId === 'string') { + try { + const protocol = classifyResolvedTarget(resolvedId, targetProtocol); + this.log.info('Resolved name', { + name, + nameUrl, + resolvedId, + protocol, + ttl, + }); + return { + name, + statusCode: response.status, + resolvedId, + resolvedAt: Date.now(), + antId, + ttl, + limit, + index, + protocol, + }; + } catch (error: any) { + this.log.warn('Invalid resolved target for protocol', { + name, + nameUrl, + resolvedId, + protocol: protocolHeader ?? 'arweave', + ttl, + limit, + index, + message: error.message, + }); + } } else { - this.log.warn('Invalid resolved data ID', { - name, - nameUrl, - resolvedId, - ttl, - limit, - index, - }); + this.log.warn('Missing resolved data ID', { name, nameUrl }); } } catch (error: any) { this.log.warn('Unable to resolve name:', { From 3af307a6c1fe196a22e2d514db2ee0af2723bd07 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 04:52:09 +0000 Subject: [PATCH 18/47] feat(ipfs): sandbox path-style CIDs onto per-CID origins MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A path-style GET /ipfs/{CID} served active content (HTML/JS) on the shared gateway origin — only CIDv0 was redirected (to convert to v1). That let one CID's content run in the gateway's origin, the same XSS/same-origin risk the Arweave data path avoids by forcing /{txid} to a sandbox subdomain. Generalize the redirect: any path-style CID (v0 or v1) is redirected to its per-CID sandbox subdomain {CIDv1base32}.{host}, reusing sandbox.ts's own getRequestSandbox() to skip the redirect when the request already arrived on that origin (no loop). ArNS-served IPFS is already isolated on the name's own origin and reaches the handler directly, so it's unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/middleware/sandbox.ts | 2 +- src/routes/ipfs.ts | 48 +++++++++++++++++++++++---------------- 2 files changed, 30 insertions(+), 20 deletions(-) diff --git a/src/middleware/sandbox.ts b/src/middleware/sandbox.ts index 7b65ecb97..5f9ca0fe5 100644 --- a/src/middleware/sandbox.ts +++ b/src/middleware/sandbox.ts @@ -11,7 +11,7 @@ import { base32 } from 'rfc4648'; import * as config from '../config.js'; import { fromB64Url } from '../lib/encoding.js'; -function getRequestSandbox(req: Request): string | undefined { +export function getRequestSandbox(req: Request): string | undefined { const matched = config.matchArnsRootHost(req.hostname); if ( matched !== undefined && diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index 031504873..a7e4d2aa0 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -8,14 +8,12 @@ import { Router, Request, Response, Handler } from 'express'; import { default as asyncHandler } from 'express-async-handler'; import winston from 'winston'; +import url from 'node:url'; + import * as config from '../config.js'; import * as metrics from '../metrics.js'; -import { - cidToV1Base32, - isValidCid, - parseCid, - isCidV0, -} from '../lib/ipfs-cid.js'; +import { cidToV1Base32, isValidCid } from '../lib/ipfs-cid.js'; +import { getRequestSandbox } from '../middleware/sandbox.js'; import { IpfsService } from '../ipfs/ipfs-service.js'; import { IpfsBlockedError, @@ -107,20 +105,32 @@ function createIpfsPathHandler({ return; } - // Redirect CIDv0 to CIDv1 subdomain if ArNS root hosts are configured. - // Uses {CID}.{host} (same level as ArNS names) — no .ipfs. label needed - // since CIDv1 base32 is always >51 chars (won't collide with ArNS names) - // and works with standard *.{host} wildcard TLS certificates. - const cid = parseCid(cidString); - if (cid !== null && isCidV0(cid) && config.ARNS_ROOT_HOSTS.length > 0) { + // Origin isolation. A path-style /ipfs/{CID} request must not serve active + // content on the shared gateway origin — that would let one CID's HTML/JS + // run in the gateway's origin (same XSS/same-origin risk the Arweave data + // path avoids). Redirect to the per-CID sandbox subdomain + // {CIDv1base32}.{host}, the same isolation sandbox.ts gives Arweave /{txid}, + // unless the request already arrived on that subdomain. This subsumes the + // CIDv0 case: cidToV1Base32 normalizes v0 to its case-insensitive, + // DNS/TLS-safe v1 base32 form (>51 chars, so it never collides with an ArNS + // name and works with *.{host} wildcard certs). ArNS-served IPFS is already + // isolated on the name's own origin and reaches the handler directly, not + // this route. + if (config.ARNS_ROOT_HOSTS.length > 0) { const v1Base32 = cidToV1Base32(cidString); - const rootHost = config.ARNS_ROOT_HOSTS[0].host; - const pathSuffix = path !== undefined ? `/${path}` : ''; - res.redirect( - 302, - `${config.SANDBOX_PROTOCOL ?? req.protocol}://${v1Base32}.${rootHost}${pathSuffix}`, - ); - return; + if (getRequestSandbox(req) !== v1Base32) { + const rootHost = + req.matchedArnsRootHost ?? config.ARNS_ROOT_HOSTS[0].host; + const pathSuffix = path !== undefined ? `/${path}` : ''; + const queryString = url.parse(req.originalUrl).query; + res.redirect( + 302, + `${config.SANDBOX_PROTOCOL ?? req.protocol}://${v1Base32}.${rootHost}${pathSuffix}${ + queryString !== null && queryString !== '' ? `?${queryString}` : '' + }`, + ); + return; + } } await handleIpfsRequest({ From 4f3207db9c6dd680c3a7c094252eb0de0cfa0950 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 05:02:12 +0000 Subject: [PATCH 19/47] feat(ipfs): HEAD support, content-hash blocking, and cached 404s MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bring three IPFS-path behaviors to Arweave-path parity: - HEAD /ipfs/:cid (and the subdomain/ArNS routes) now return the full header set with no body — for metadata probes and media players that HEAD before ranging. The shared handler skips the pipe, releases the upstream/cache stream, and bills zero egress for the HEAD. - Content-hash moderation: on a cache hit (where the base64url SHA-256 of the served bytes is known) the service now also checks isHashBlocked, matching the Arweave path — so a block-by-content-hash entry stops IPFS-served bytes, not only a block-by-CID. - 404s now carry Cache-Control (CACHE_NOT_FOUND_MAX_AGE, must-revalidate) like the Arweave sendNotFound, so absent CIDs aren't re-fetched from Kubo on every retry through upstream/edge caches. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/ipfs/ipfs-service.ts | 15 +++++++++++++++ src/routes/ipfs.ts | 24 +++++++++++++++++++++--- 2 files changed, 36 insertions(+), 3 deletions(-) diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 905a98688..33c69ee8c 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -104,6 +104,21 @@ export class IpfsService { // Check cache const cached = await this.cache.get(normalizedCid, path); if (cached) { + // Content-hash moderation. The cache stores the base64url SHA-256 of the + // served bytes (the same format as Arweave's data hash), so once content + // is cached we honor a block-by-hash entry too — matching the Arweave + // path's isHashBlocked enforcement — not just block-by-CID. This catches + // the same bytes blocked under an Arweave id or another identifier. + if ( + cached.digest !== undefined && + (await this.blockListValidator.isHashBlocked(cached.digest)) + ) { + cached.stream.destroy(); + metrics.ipfsBlockedTotal.inc(); + span.setAttribute('ipfs.blocked', true); + span.end(); + throw new IpfsBlockedError(`Content hash is blocked: ${normalizedCid}`); + } this.log.debug('IPFS cache hit', { cid: normalizedCid, path }); metrics.ipfsCacheHitTotal.inc(); span.setAttributes({ diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index a7e4d2aa0..cc424f86d 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -52,6 +52,10 @@ export function createIpfsRouter({ router.get('/ipfs/:cid', handler); router.get('/ipfs/:cid/*', handler); + // HEAD returns the same headers with no body — for metadata probes and media + // players that HEAD before ranging. Same handler; the body is skipped below. + router.head('/ipfs/:cid', handler); + router.head('/ipfs/:cid/*', handler); return router; } @@ -169,6 +173,7 @@ async function handleIpfsRequest({ routeType: 'path' | 'subdomain'; }): Promise { const startTime = Date.now(); + const isHead = req.method === 'HEAD'; const ipfsPath = path !== undefined ? `${cidString}/${path}` : cidString; parentLog.debug('Handling IPFS request', { cidString, path, routeType }); @@ -253,8 +258,14 @@ async function handleIpfsRequest({ metrics.ipfsContentSizeHistogram.observe(result.size); } - // Pipe stream to response - result.stream.pipe(res); + // Pipe stream to response. HEAD returns headers only — release the + // upstream/cache stream and end without a body. + if (isHead) { + result.stream.destroy(); + res.end(); + } else { + result.stream.pipe(res); + } // Adjust rate limiter tokens after response completes res.on('finish', () => { @@ -266,7 +277,7 @@ async function handleIpfsRequest({ adjustRateLimitTokens({ req, - responseSize: result.size > 0 ? result.size : contentSize, + responseSize: isHead ? 0 : result.size > 0 ? result.size : contentSize, initialResult: limitCheck, rateLimiter, }).catch((error) => { @@ -311,6 +322,13 @@ async function handleIpfsRequest({ route_type: routeType, status: 'not_found', }); + // Cache-dampen repeated 404s the way the Arweave path does (sendNotFound), + // so an absent CID isn't re-fetched from Kubo on every retry through + // upstream/edge caches. + res.setHeader( + 'Cache-Control', + `public, max-age=${config.CACHE_NOT_FOUND_MAX_AGE}, must-revalidate`, + ); res.status(404).json({ error: 'IPFS content not found' }); return; } From 3dbc5abf509bc92cf4f1494ac17ad8b472b8b3c7 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 05:06:32 +0000 Subject: [PATCH 20/47] feat(ipfs): negative-cache absent/unpinned CIDs An absent or unpinned CID re-hit Kubo on every request (latency and DoS amplification), unlike the Arweave path which short-circuits repeat misses via NegativeDataCache. Reuse that same cache in IpfsService: check isNegatively cached before the Kubo fetch and recordMiss on IpfsNotFoundError. Trips only after repeated misses (count + duration thresholds), so transient blips don't poison a CID that later pins. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/ipfs/ipfs-service.ts | 39 ++++++++++++++++++++++++++++++++------- src/system.ts | 3 +++ 2 files changed, 35 insertions(+), 7 deletions(-) diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 33c69ee8c..877490932 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -14,6 +14,7 @@ import { cidToV1Base32 } from '../lib/ipfs-cid.js'; import { startChildSpan } from '../tracing.js'; import { IpfsFsCache } from './ipfs-cache.js'; import { DataBlockListValidator } from '../types.js'; +import { NegativeDataCache } from '../data/negative-data-cache.js'; import { KuboDataSource, IpfsBlockedError, @@ -41,6 +42,7 @@ export class IpfsService { private cache: IpfsFsCache; private blockListValidator: DataBlockListValidator; private maxResponseSizeBytes: number; + private negativeCache?: NegativeDataCache; constructor({ log, @@ -48,18 +50,21 @@ export class IpfsService { cache, blockListValidator, maxResponseSizeBytes, + negativeCache, }: { log: winston.Logger; dataSource: KuboDataSource; cache: IpfsFsCache; blockListValidator: DataBlockListValidator; maxResponseSizeBytes: number; + negativeCache?: NegativeDataCache; }) { this.log = log.child({ class: this.constructor.name }); this.dataSource = dataSource; this.cache = cache; this.blockListValidator = blockListValidator; this.maxResponseSizeBytes = maxResponseSizeBytes; + this.negativeCache = negativeCache; } async getContent({ @@ -101,6 +106,18 @@ export class IpfsService { throw new IpfsNotFoundError('Invalid IPFS path'); } + // Negative cache: short-circuit CIDs we've repeatedly failed to fetch + // (absent or unpinned) so they don't re-hit Kubo on every request + // (latency / DoS amplification) — mirrors the Arweave path's negative data + // cache. Only trips after repeated misses (count + duration thresholds). + if (this.negativeCache?.isNegativelyCached(normalizedCid) === true) { + span.setAttribute('ipfs.negative_cache', 'hit'); + span.end(); + throw new IpfsNotFoundError( + `CID not found (negatively cached): ${normalizedCid}`, + ); + } + // Check cache const cached = await this.cache.get(normalizedCid, path); if (cached) { @@ -138,13 +155,21 @@ export class IpfsService { metrics.ipfsCacheMissTotal.inc(); span.setAttribute('ipfs.cache', 'miss'); - // Fetch from Kubo - const result = await this.dataSource.getContent({ - cidString: normalizedCid, - path, - signal, - parentSpan: span, - }); + // Fetch from Kubo. Record absent/unpinned CIDs in the negative cache so + // repeat requests short-circuit above instead of re-hitting Kubo. + const result = await this.dataSource + .getContent({ + cidString: normalizedCid, + path, + signal, + parentSpan: span, + }) + .catch((err) => { + if (err instanceof IpfsNotFoundError) { + this.negativeCache?.recordMiss(normalizedCid); + } + throw err; + }); span.setAttributes({ 'ipfs.size': result.size, diff --git a/src/system.ts b/src/system.ts index 998486152..f30ee1af2 100644 --- a/src/system.ts +++ b/src/system.ts @@ -1909,6 +1909,9 @@ if (config.IPFS_ENABLED) { cache: ipfsCache, blockListValidator: dataBlockListValidator, maxResponseSizeBytes: config.IPFS_MAX_RESPONSE_SIZE_BYTES, + // Reuse the shared negative cache so absent/unpinned CIDs short-circuit + // repeat Kubo fetches, as they do for absent Arweave ids. + negativeCache: negativeDataCache, }); ipfsRateLimiter = createIpfsRateLimiter(); From c9f90249a9290e99652e3c78fa30ad6c5ecfa310 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 10:57:45 +0000 Subject: [PATCH 21/47] feat(ipfs): HTTP Range (206) support MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The IPFS path served full bodies only — no Accept-Ranges, no 206, no 416 — so media seeking failed and the observer's ranged sampling of >1MiB content downloaded the whole object per sample. Kubo's gateway already supports Range, so forward a client Range header to Kubo and relay its partial response: - kubo-data-source: forward `Range`, accept 206 (capture Content-Range), map a Kubo 416 to IpfsRangeNotSatisfiableError; surface statusCode/contentRange. - ipfs-service: range requests bypass the positive cache (never serve a partial from a full cached object, never cache a partial body) and stream straight from Kubo. - route: always advertise Accept-Ranges: bytes; relay 206 + Content-Range; map IpfsRangeNotSatisfiableError to 416. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/ipfs/ipfs-service.ts | 37 ++++++++++++++++++++++++++---------- src/ipfs/kubo-data-source.ts | 29 +++++++++++++++++++++++++++- src/routes/ipfs.ts | 19 ++++++++++++++++++ 3 files changed, 74 insertions(+), 11 deletions(-) diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 877490932..4b6226c13 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -34,6 +34,10 @@ export interface IpfsGetContentResult { * misses (the body streams straight from Kubo without being hashed inline). */ digest?: string; + // 200 for a full response, 206 for a partial (Range) response. + statusCode: number; + // Present on 206 responses: the upstream Content-Range header value. + contentRange?: string; } export class IpfsService { @@ -72,11 +76,13 @@ export class IpfsService { path, signal, parentSpan, + range, }: { cidString: string; path?: string; signal?: AbortSignal; parentSpan?: Span; + range?: string; }): Promise { const span = startChildSpan( 'IpfsService.getContent', @@ -118,8 +124,12 @@ export class IpfsService { ); } - // Check cache - const cached = await this.cache.get(normalizedCid, path); + // Check cache. Range requests bypass the positive cache: we don't serve a + // partial from a full cached object here, and we must never cache a + // partial body — they're forwarded straight to Kubo (which supports Range) + // below. + const cached = + range === undefined ? await this.cache.get(normalizedCid, path) : null; if (cached) { // Content-hash moderation. The cache stores the base64url SHA-256 of the // served bytes (the same format as Arweave's data hash), so once content @@ -149,6 +159,7 @@ export class IpfsService { contentType: cached.contentType, cached: true, digest: cached.digest, + statusCode: 200, }; } @@ -163,6 +174,7 @@ export class IpfsService { path, signal, parentSpan: span, + range, }) .catch((err) => { if (err instanceof IpfsNotFoundError) { @@ -188,14 +200,17 @@ export class IpfsService { ); } - // Stream directly to the client while writing to a temp file on disk - // for caching. No memory buffering — handles files of any size. - this.streamToCache( - normalizedCid, - path, - result.stream, - result.contentType, - ); + // Stream directly to the client while writing to a temp file on disk for + // caching. No memory buffering — handles files of any size. Partial (206) + // responses are NOT cached — only full objects. + if (range === undefined && result.statusCode === 200) { + this.streamToCache( + normalizedCid, + path, + result.stream, + result.contentType, + ); + } // End span when stream completes result.stream.on('end', () => span.end()); @@ -209,6 +224,8 @@ export class IpfsService { size: result.size, contentType: result.contentType, cached: false, + statusCode: result.statusCode, + contentRange: result.contentRange, }; } catch (error: any) { if (error.name !== 'AbortError') { diff --git a/src/ipfs/kubo-data-source.ts b/src/ipfs/kubo-data-source.ts index 776793d08..acc792cb4 100644 --- a/src/ipfs/kubo-data-source.ts +++ b/src/ipfs/kubo-data-source.ts @@ -16,6 +16,10 @@ export interface IpfsContentResult { stream: Readable; size: number; contentType: string; + // 200 for a full response, 206 for a partial (Range) response. + statusCode: number; + // Present on 206 responses: the upstream Content-Range header value. + contentRange?: string; } export class KuboDataSource { @@ -46,11 +50,13 @@ export class KuboDataSource { path, signal, parentSpan, + range, }: { cidString: string; path?: string; signal?: AbortSignal; parentSpan?: Span; + range?: string; }): Promise { signal?.throwIfAborted(); @@ -104,6 +110,10 @@ export class KuboDataSource { signal: controller.signal, headers: { 'Accept-Encoding': 'identity', + // Forward a client Range to Kubo (its gateway supports Range and + // returns 206 + Content-Range). Enables media seeking and the + // observer's ranged sampling of large content. + ...(range !== undefined ? { Range: range } : {}), }, maxRedirects: 5, // Accept non-2xx so we can handle 404/408/504 ourselves @@ -127,7 +137,14 @@ export class KuboDataSource { ); } - if (response.status !== 200) { + if (response.status === 416) { + (response.data as Readable).destroy(); + throw new IpfsRangeNotSatisfiableError( + `Range not satisfiable for /ipfs/${ipfsPath}`, + ); + } + + if (response.status !== 200 && response.status !== 206) { const stream = response.data as Readable; stream.destroy(); throw new Error( @@ -173,6 +190,8 @@ export class KuboDataSource { stream, size: contentLength, contentType, + statusCode: response.status, + contentRange: response.headers['content-range'], }; } catch (error: any) { clearTimeout(connectionTimer); @@ -185,6 +204,7 @@ export class KuboDataSource { if (error instanceof IpfsNotFoundError) throw error; if (error instanceof IpfsTimeoutError) throw error; + if (error instanceof IpfsRangeNotSatisfiableError) throw error; if (error.name === 'AbortError' || error.code === 'ERR_CANCELED') { if (signal?.aborted) { @@ -239,6 +259,13 @@ export class IpfsBlockedError extends Error { } } +export class IpfsRangeNotSatisfiableError extends Error { + constructor(message: string) { + super(message); + this.name = 'IpfsRangeNotSatisfiableError'; + } +} + export class IpfsSizeLimitError extends Error { constructor(message: string) { super(message); diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index cc424f86d..034d6b077 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -18,6 +18,7 @@ import { IpfsService } from '../ipfs/ipfs-service.js'; import { IpfsBlockedError, IpfsNotFoundError, + IpfsRangeNotSatisfiableError, IpfsSizeLimitError, IpfsTimeoutError, IpfsUnavailableError, @@ -183,6 +184,7 @@ async function handleIpfsRequest({ cidString, path, signal: req.signal, + range: req.headers.range, }); // Check payment and rate limits (x402 + rate limiting in one call). @@ -219,6 +221,14 @@ async function handleIpfsRequest({ if (result.size > 0) { res.setHeader('Content-Length', result.size); } + // Advertise range support, and relay a partial (206) response from Kubo. + res.setHeader('Accept-Ranges', 'bytes'); + if (result.statusCode === 206) { + res.status(206); + if (result.contentRange !== undefined) { + res.setHeader('Content-Range', result.contentRange); + } + } // A direct /ipfs/{CID} or {CID}.host request is content-addressed and thus // immutable. But when the request arrived via an ArNS name, the name->CID // binding is MUTABLE and the ArNS middleware already set a TTL-bounded @@ -360,6 +370,15 @@ async function handleIpfsRequest({ return; } + if (error instanceof IpfsRangeNotSatisfiableError) { + metrics.ipfsRequestsTotal.inc({ + route_type: routeType, + status: 'range_not_satisfiable', + }); + res.status(416).json({ error: 'Range not satisfiable' }); + return; + } + // Client disconnected if (error.name === 'AbortError') { return; From 6e21bfd8765b43bb6aa245a3a0d4f7aa99b1a1f7 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 11:00:21 +0000 Subject: [PATCH 22/47] test(ipfs): cover Range/206/416 in kubo-data-source Adds unit coverage for the Range path: a client Range is forwarded to Kubo, a 206 relays statusCode + Content-Range, a Kubo 416 maps to IpfsRangeNotSatisfiableError, and a full response reports statusCode 200 with no Content-Range. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/ipfs/kubo-data-source.test.ts | 94 ++++++++++++++++++++++++++++++- 1 file changed, 93 insertions(+), 1 deletion(-) diff --git a/src/ipfs/kubo-data-source.test.ts b/src/ipfs/kubo-data-source.test.ts index 8a9ff690b..783d0700e 100644 --- a/src/ipfs/kubo-data-source.test.ts +++ b/src/ipfs/kubo-data-source.test.ts @@ -4,13 +4,16 @@ * * SPDX-License-Identifier: AGPL-3.0-or-later */ -import { describe, it, beforeEach } from 'node:test'; +import { describe, it, beforeEach, afterEach } from 'node:test'; import { strict as assert } from 'node:assert'; +import { Readable } from 'node:stream'; +import axios from 'axios'; import { createTestLogger } from '../../test/test-logger.js'; import { KuboDataSource, IpfsNotFoundError, + IpfsRangeNotSatisfiableError, IpfsTimeoutError, IpfsUnavailableError, } from './kubo-data-source.js'; @@ -85,6 +88,90 @@ describe('KuboDataSource', () => { }); }); + describe('range requests', () => { + const CID = 'bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi'; + let interceptorId: number; + afterEach(() => axios.interceptors.request.eject(interceptorId)); + + const readRange = (h: any): string | undefined => + typeof h?.get === 'function' ? h.get('Range') : (h?.Range ?? h?.range); + + it('forwards Range to Kubo and relays 206 + Content-Range', async () => { + let captured: any; + interceptorId = axios.interceptors.request.use((config) => { + captured = config; + config.adapter = () => + Promise.resolve({ + status: 206, + statusText: 'Partial Content', + headers: { + 'content-range': 'bytes 0-99/119762', + 'content-length': '100', + 'content-type': 'image/jpeg', + }, + config, + data: Readable.from([Buffer.alloc(100)]), + }); + return config; + }); + + const result = await kuboDataSource.getContent({ + cidString: CID, + range: 'bytes=0-99', + }); + + assert.equal(readRange(captured.headers), 'bytes=0-99'); + assert.equal(result.statusCode, 206); + assert.equal(result.contentRange, 'bytes 0-99/119762'); + assert.equal(result.size, 100); + result.stream.destroy(); + }); + + it('maps a Kubo 416 to IpfsRangeNotSatisfiableError', async () => { + interceptorId = axios.interceptors.request.use((config) => { + config.adapter = () => + Promise.resolve({ + status: 416, + statusText: 'Range Not Satisfiable', + headers: {}, + config, + data: Readable.from([]), + }); + return config; + }); + + await assert.rejects( + () => kuboDataSource.getContent({ cidString: CID, range: 'bytes=9e9-' }), + (error: any) => { + assert.equal(error.name, 'IpfsRangeNotSatisfiableError'); + return true; + }, + ); + }); + + it('returns statusCode 200 and no Content-Range for a full response', async () => { + interceptorId = axios.interceptors.request.use((config) => { + config.adapter = () => + Promise.resolve({ + status: 200, + statusText: 'OK', + headers: { + 'content-length': '119762', + 'content-type': 'image/jpeg', + }, + config, + data: Readable.from([Buffer.alloc(10)]), + }); + return config; + }); + + const result = await kuboDataSource.getContent({ cidString: CID }); + assert.equal(result.statusCode, 200); + assert.equal(result.contentRange, undefined); + result.stream.destroy(); + }); + }); + describe('error types', () => { it('IpfsNotFoundError has correct name', () => { const error = new IpfsNotFoundError('not found'); @@ -101,5 +188,10 @@ describe('KuboDataSource', () => { const error = new IpfsUnavailableError('unavailable'); assert.equal(error.name, 'IpfsUnavailableError'); }); + + it('IpfsRangeNotSatisfiableError has correct name', () => { + const error = new IpfsRangeNotSatisfiableError('range'); + assert.equal(error.name, 'IpfsRangeNotSatisfiableError'); + }); }); }); From 4d018d08a48ab0dfb45687730d55b38b52be6e60 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 11:08:21 +0000 Subject: [PATCH 23/47] docs(ipfs): document multi-protocol parity + Range, add incentive analysis Update docs/ipfs-integration.md for the behaviors added in this branch: - Resolver scope: protocol now propagates across a trusted-gateway hop (X-ArNS-Protocol read + per-protocol id validation); the on-demand-first ordering caveat is removed. - Path-style origin isolation: /ipfs/{CID} redirects to a per-CID sandbox subdomain (not just CIDv0), matching the Arweave sandbox. - HEAD, Range/206/416 + Accept-Ranges (partials bypass the cache). - Content-hash blocking (isHashBlocked) alongside CID blocking. - Negative cache for absent/unpinned CIDs and cache-controlled 404s. Add docs/drafts/ipfs-observation-incentive-analysis.md: how the AR.IO observation/incentive protocol (observer + Solana contracts) would need to adapt to observe, verify, and reward named IPFS data. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- .../ipfs-observation-incentive-analysis.md | 193 ++++++++++++++++++ docs/ipfs-integration.md | 53 +++-- 2 files changed, 229 insertions(+), 17 deletions(-) create mode 100644 docs/drafts/ipfs-observation-incentive-analysis.md diff --git a/docs/drafts/ipfs-observation-incentive-analysis.md b/docs/drafts/ipfs-observation-incentive-analysis.md new file mode 100644 index 000000000..6d1d4b910 --- /dev/null +++ b/docs/drafts/ipfs-observation-incentive-analysis.md @@ -0,0 +1,193 @@ +# AR.IO Observation & Incentive Protocol vs. Named IPFS Data — Gap Analysis + +> Research deliverable (Fable agent), for protocol-design review. Not code docs. +> Sources accessed and verified (none guessed): +> - Observer: `/programs/ar-io-observer` (Solana-adapted fork, branch +> `chore/remove-aws-observer-deploy`, running as `ar-io-node-observer-1`) +> - Contracts: `github.com/ar-io/ar-io-solana-contracts` @ `a8ef07e` +> - Gateway: `/programs/ar-io-node/wt/ipfs-sync-793` (this IPFS branch) +> - Docs: `/programs/ar-io-docs/content/learn/oip/*.mdx` +> - SDK: `/programs/ar-io-sdk/src`; live gateway at `localhost:4000` + +--- + +## 1. How observation works today + +### Observer-side (per epoch, per registered gateway) + +`ar-io-observer/src/observer.ts` runs three assessment dimensions; the composite +pass is `ownership AND names AND offsets` (`observer.ts:2233`): + +**a) Ownership** — `assessOwnership` (`observer.ts:262`): GET +`https://{host}/ar-io/info`, compare reported `wallet` to the registered wallet. + +**b) ArNS name resolution** — two name groups: +- **Prescribed names**: on-chain `Epoch.prescribed_names` via + `SolanaARIOReadable.getPrescribedNames` (`src/names/solana-names-source.ts:53`). + Contract prescribes **≤2 names/epoch** (hard cap `[[u8;32]; 2]`, + `ario-gar/src/instructions/epoch.rs:831`), selected by hashchain entropy + + linear probing over the ArNS `NameRegistry`. +- **Chosen names**: **8 per group** (`NUM_ARNS_NAMES_TO_OBSERVE_PER_GROUP`, + `config.ts:110`), sampled from the full registry using epoch entropy + (`src/names/random-arns-names-source.ts:40-67`). + +For each name the observer: +1. Gets a **reference resolution** from trusted reference gateways (default + `turbo-gateway.com`, `ar-io.net`; `config.ts:76`) or a **network consensus + resolver** (`src/reference/arns-consensus-resolver.ts`: query N gateways, + group by `resolvedId`, require `consensusThreshold`). Cached per epoch. +2. Fetches `https://{name}.{host}/` via `getArnsResolution` (`observer.ts:116`): + HEAD first; then **sha256 of the first 1 MiB** for content ≤1 MiB (or unknown + length), or **5 entropy-seeded 200-byte Range requests** for content >1 MiB. +3. Compares four properties vs. reference: `resolvedId` (`x-arns-resolved-id`), + `ttlSeconds`, `contentType`, `dataHashDigest` (`observer.ts:1907-1920`). + Header validation (`src/lib/arns-validation.ts`) does **not** enforce any ID + format — a CID in `x-arns-resolved-id` passes through untouched. +4. Names dimension passes if **≥80%** of names pass (`NAME_PASS_THRESHOLD=0.8`, + `observer.ts:63`). Continuous observer: 3 cycles/gateway, majority vote. + +**c) Offset/chunk sampling** (Arweave-specific, enforced by default): +`validateChunkAtOffset` GETs `/chunk/{offset}` and **verifies the Merkle +`data_path` against the tx `data_root`** resolved from chain +(`observer.ts:951-985,1194-1390`). `OFFSET_OBSERVATION_ENABLED=true`, rate 0.20, +4 offsets, enforcement on. + +**Is Arweave verification assumed?** Split: +- The **ArNS serving check is content-agnostic, trust-based** (sampled-digest vs. + reference/consensus). No `data_root`/tx proof. +- The **chunk/offset check is trustlessly Arweave-anchored** (`data_root` Merkle + proofs) and has **no IPFS analog**. + +### Reporting & contract accounting + +Full JSON report → Arweave via Turbo, then **`save_observations`** on-chain +(`ario-gar/src/instructions/observation.rs:8`): `gateway_results: [u8; 375]` +(1 bit/gateway pass/fail) + `report_tx_id: [u8; 32]`. Epoch pipeline +(permissionless crank): `create_epoch → tally_weights → prescribe_epoch → +save_observations → distribute_epoch → close_epoch`. Duration 24h; reward pool = +protocol balance ×0.1%; split **90% gateways / 10% observers**; a gateway is +**failed if >½ of submitting observers** mark it failed (`distribution.rs:184`). + +**Critical structural fact**: the reward machinery is **entirely +content-agnostic** — nothing in `ario-gar`/`ario-arns` knows what a name points +to. The pointer lives in the ANT program: `AntRecord.target` + +**`target_protocol: u8` (0=Arweave, 1=IPFS)**, with `is_valid_ipfs_cid` already +on-chain (`ario-ant/src/state.rs:150-152,447,574-592`). Name prescription is +**protocol-blind** — IPFS-target names can already be prescribed/chosen today. + +--- + +## 2. Where IPFS breaks or is unaddressed + +1. **Accidental, un-adjudicated inclusion.** IPFS-target names already flow into + sampling. A gateway *without* IPFS enabled sends the CID down the Arweave data + path (`middleware/arns.ts:206`), which can't retrieve it → failure. With the + 2-name prescription cap and 80% pass threshold, **one prescribed IPFS name + could fail every non-IPFS gateway in an epoch.** Nothing says IPFS serving is + mandatory. +2. **Verification model mismatch.** The observer's ArNS check is + reference-trust-based digest comparison; it ignores that **the CID is itself a + verifiable content hash.** A colluding/buggy reference set could bless wrong + bytes. The trustless dimension the protocol *does* have (chunk proofs) has no + IPFS counterpart. +3. **Range-request degradation.** For >1 MiB the observer issues 5 ranged GETs; + the IPFS route ignored Range and returned full bodies → 5× full downloads per + name/gateway/cycle, and version skew produces digest mismatches → false fails. + *(Fixed in this branch: Range/206 now supported.)* +4. **Availability/pinning risk.** No pinning anywhere; Kubo runs `--enable-gc`, + so unpinned content vanishes. Failure modes the protocol can't express: + content unpinned network-wide → reference 404s while gateways *holding* a + pinned copy get *failed* on mismatch (perverse); cold-DHT retrieval exceeds + observer timeouts non-deterministically; observation itself re-warms caches, + masking gateways that never pin. +5. **Mutable name→CID binding.** Per-epoch reference-resolution cache races with + mid-epoch record updates; IPFS update cadence is expected to be higher. +6. **No caching/pinning incentive anywhere.** `distribute_epoch` rewards + pass/fail resolution only. No instruction/account/weight rewards pinning, + persistence, or cache-hit serving. `X-Cache` is gateway-asserted, unusable as + signal. +7. **451/blocklist interplay.** The observer only special-cases 404; divergent + blocklists (451) between reference and target yield mismatch failures. + +--- + +## 3. Contract / observer gaps by requirement + +**(a) Observe IPFS-named resolution correctly** — observer-only: +capture `x-arns-protocol` on `ArnsNameAssessment`; branch `getArnsResolution` on +protocol (no Range sampling until byte-verify lands, longer timeouts, CID-format +validation); key consensus on `(resolvedId, protocol)`. + +**(b) Verify served IPFS bytes** — observer-only, *stronger than today*: verify +the multihash — hash the body for single-block/raw CIDs; fetch a random block (or +`?format=car`) for UnixFS DAGs. Trustless, no reference gateway; the IPFS analog +of chunk `data_root` validation. + +**(c) Reward resolving/caching/serving named IPFS data** — mostly **no contract +change**: the bitmap, >½ threshold, and `distribute_epoch` are content-agnostic; +if "IPFS names count like any name," rewards flow with zero contract changes. To +*deliberately* cover IPFS, `prescribe_epoch` would read `AntRecord.target_protocol` +and stratify; the 2-name cap likely enlarged. Rewarding *pinning/caching* is +genuinely new accounting. + +**(d) Availability/pinning risk** — observer policy (+ optional contract flag): +"network-wide unavailable" → **neutral** (excluded from the 80% denominator), not +failure. A `GatewaySettings` "supports IPFS" bit would let observers skip IPFS +names for non-supporting gateways if IPFS stays optional. + +--- + +## 4. Recommended adjustments, ranked + +**MUST (before any IPFS-target name lands on this registry):** +1. **Decide mandatory-vs-optional.** Either IPFS serving is required (document it; + observers fail non-supporters) or add an IPFS-capability bit to + `GatewaySettings` and make the observer skip IPFS-protocol names for gateways + without it. Otherwise one prescribed IPFS name can zero out gateway rewards + network-wide via the >½ tally. +2. **Protocol-aware observer assessment**: read `x-arns-protocol`; for `ipfs` + disable Range sampling (until (5)), extend timeout, add `protocol` to the + report (bump `formatVersion`). +3. **Neutral scoring for network-wide unavailability**: exclude un-retrievable + names from the denominator; fail only gateways that mis-serve vs. serving peers. + +**SHOULD:** +4. **Trustless CID verification** (raw-block or CAR multihash) replacing + reference-digest trust for IPFS names. +5. **IPFS block-sampling assessment** as the analog of chunk/offset validation, + staged behind config like `OFFSET_OBSERVATION_*`. +6. **Range support in `routes/ipfs.ts`** — *done in this branch.* +7. **Stratified prescription**: teach `prescribe_epoch` to read + `target_protocol`; enlarge the 2-name cap. +8. **451/blocklist policy** for the observer (treat matching 451s as neutral). + +**NO CHANGE NEEDED:** `save_observations` bitmap + `report_tx_id`, failure +threshold, `distribute_epoch` formula, weight computation, delegate accumulator; +`ario-ant` (`target_protocol` + `is_valid_ipfs_cid` already model it); `ario-arns` +registry; observer header validation (already tolerates CIDs). + +**Explicitly new design work (only if the network wants it):** a pinning/caching +incentive is *not a change to OIP* — it is a new reward category (attested +pinning, retrievability bonds, or protocol-funded pinning subsidies from ArNS +revenue). Current OIP can incentivize *serving* named IPFS data; it cannot +incentivize *persistence*, and no parameter tweak fixes that. + +## 5. Open questions / risks for the protocol team + +- **Persistence guarantee**: ArNS historically implies permanence (Arweave). Does + a name → unpinned IPFS dilute the value proposition? Should `targetProtocol=1` + registration/renewal require a pinning commitment? +- **Who must hold the data?** Is serving via Kubo's DHT fetch from a third party + "passing," or must the gateway pin? The observer can't distinguish (X-Cache is + self-asserted); rewarding "serving" may reward whoever fronts someone else's pin. +- **Observation as cache-warmer**: repeated observer fetches keep content hot; + pass rates overstate real availability. +- **Determinism across observers**: cold-content latency makes pass/fail + observer-order-dependent; borderline content could flap gateways across epochs. +- **Verification cost ceiling**: full DAG verification is expensive; block + sampling bounds cost but weakens the guarantee. +- **Mid-epoch record churn**: per-epoch reference cache races with mutable + name→CID updates; consider re-resolving on mismatch before failing. +- **Prescription capacity**: the on-chain 2-name cap makes per-protocol + stratification statistically thin; enlarging it changes the `Epoch` account + layout (zero-copy) — plan a migration. diff --git a/docs/ipfs-integration.md b/docs/ipfs-integration.md index 13b0b47cb..33658927c 100644 --- a/docs/ipfs-integration.md +++ b/docs/ipfs-integration.md @@ -144,15 +144,20 @@ label in the hostname distinguishes IPFS requests from ArNS name resolution, preventing collisions. For example, `my-app.arweave.dev` resolves as an ArNS name, while `bafybeig...arweave.dev` resolves as an IPFS CID. -### CIDv0 to CIDv1 Redirect +### Path-style origin isolation (and CIDv0 → CIDv1) -CIDv0 identifiers (base58, starting with `Qm`) cannot be used in subdomains -because they are case-sensitive and DNS is case-insensitive. When a CIDv0 is -detected in a subdomain request, the gateway issues a 301 redirect to the -equivalent CIDv1 (base32) subdomain URL. +A path-style `/ipfs/{CID}` request is redirected (302) to the per-CID sandbox +subdomain `{CIDv1base32}.{root_host}`, giving each CID its own browser origin — +the same isolation the Arweave data path gets for `/{txid}` via the sandbox +middleware. This prevents one CID's HTML/JS from executing in the shared gateway +origin. The redirect is skipped when the request already arrived on that +subdomain (no loop), and ArNS-served IPFS is already isolated on the name's own +origin (so it reaches the handler directly, not this route). -For path-based requests, both CIDv0 and CIDv1 are accepted directly without -redirection. +CIDv0 identifiers (base58, starting with `Qm`) are case-sensitive and cannot be +used in subdomains, so they are normalized to their DNS-safe CIDv1 base32 form as +part of the same redirect. Requests already on the correct `{CID}.{root_host}` +subdomain are served directly. ## Components @@ -207,6 +212,11 @@ A file-based blocklist for content moderation: notifications. Additions and removals take effect without restarting the gateway. - **Response**: Blocked CIDs return HTTP 451 (Unavailable For Legal Reasons). +- **Block by content hash too**: moderation uses the same admin API and store as + Arweave (`isIdBlocked` on the CID, and `isHashBlocked` on the served bytes' + base64url SHA-256). Because the SHA-256 is known once content is cached, a + block-by-content-hash entry stops an IPFS CID serving those same bytes, not + only a block-by-CID — matching the Arweave data path. ### `src/ipfs/ipfs-rate-limiter.ts` -- Rate Limiter @@ -251,11 +261,14 @@ Express middleware that intercepts requests based on the `Host` header: Express route handlers for path-based IPFS access: -- `GET /ipfs/:cid` and `GET /ipfs/:cid/*` routes. +- `GET` and `HEAD` `/ipfs/:cid` and `/ipfs/:cid/*` routes. - Validates the CID parameter, delegates to the IPFS service, and streams the - response with appropriate headers. -- Sets `Cache-Control`, `ETag`, and `X-Ipfs-Path` headers on successful - responses. + response with appropriate headers. `HEAD` returns the full header set with no + body (and bills zero egress). +- Sets `Cache-Control`, `ETag`, `X-Ipfs-Path`, and `Accept-Ranges: bytes`. A + client `Range` header is forwarded to Kubo and relayed as `206 Partial + Content` + `Content-Range`; an unsatisfiable range returns `416`. Partial + (206) responses are streamed straight from Kubo and are never cached. ## Data Flow @@ -292,10 +305,12 @@ The complete lifecycle of an IPFS request: | Header | Value | Purpose | |--------|-------|---------| - | `Cache-Control` | `public, max-age=31536000, immutable` | CID content never changes | + | `Cache-Control` | `public, max-age=31536000, immutable` (direct CID) / ArNS TTL (via a name) | CID content never changes; a name→CID binding is mutable | | `ETag` | `"{CID}"` | Content-addressed deduplication | | `X-Ipfs-Path` | `/ipfs/{CID}/{path}` | IPFS ecosystem interop | | `Content-Type` | Detected by Kubo or from `.meta` | Standard MIME typing | + | `Accept-Ranges` | `bytes` | Advertises Range support; a `Range` request yields `206` + `Content-Range` | + | `Content-Digest` | RFC 9530 SHA-256 (cache hits / HEAD) | Signed body binding when the hash is known | ## Caching Strategy @@ -314,6 +329,7 @@ no revalidation needed. | **Metadata** | Companion `.meta` JSON files store content type, size, and original CID | | **Cleanup** | Eviction scans run every `IPFS_CACHE_CLEANUP_THRESHOLD` seconds (default 3600) | | **Permanence** | No TTL-based expiration; entries are valid forever unless evicted for space | +| **Negative cache** | Absent/unpinned CIDs are recorded (shared `NegativeDataCache`, count+duration thresholds) so repeat requests short-circuit instead of re-hitting Kubo — as for absent Arweave ids. `404` responses also carry `Cache-Control` (`CACHE_NOT_FOUND_MAX_AGE`, `must-revalidate`). | ### Why a Separate Cache @@ -493,11 +509,14 @@ hits), so the name->CID binding and the served bytes are both attested. a single ANT record update (target + `targetProtocol`). - **Caching and moderation apply.** All Phase 1 protections (blocklist, rate limits, cache) apply to ArNS-resolved IPFS content. -- **Resolver scope.** Protocol awareness lives in the on-demand resolver. The - trusted-gateway resolver does not yet propagate `targetProtocol` across - gateway hops, so a name whose resolution falls through to that path would be - treated as Arweave. Keep `on-demand` ahead of `gateway` in - `ARNS_RESOLVER_PRIORITY_ORDER` for IPFS-targeted names. +- **Resolver scope.** Protocol propagates across resolvers, including a + trusted-gateway hop: the trusted-gateway resolver reads the signed + `X-ArNS-Protocol` header and validates the resolved id per protocol (a CID for + `ipfs`, a 43-char id for `arweave`), so an IPFS resolution survives a gateway + hop and `ARNS_RESOLVER_PRIORITY_ORDER` no longer has to keep `on-demand` ahead + of `gateway` for IPFS-targeted names. An upstream gateway that predates this + still omits the header, which the reader defaults to `arweave` — so upgrade the + upstream first for full IPFS coverage across a hop. ### Root and apex names From 161dbffa6ec566f6b931f35df7e8dfdae3982a95 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 11:24:40 +0000 Subject: [PATCH 24/47] fix(ipfs): address adversarial-review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From an adversarial review of the IPFS parity work: - HEAD/abort fd+temp leak (HIGH): streamToCache only handled 'data'/'end'/ 'error'. A HEAD destroys the read stream, which emits 'close' (not 'end'/ 'error'), so the write stream and temp file leaked — an unauthenticated fd/disk-exhaustion DoS. Add a 'close' handler that cleans up on premature close (also covers client aborts). - Hash-block bypass (HIGH): the isHashBlocked check lived only on the cache-hit branch, so a range request (which bypasses the cache) and the first uncached request served hash-blocked bytes. IPFS is streamed before its hash is known, so: (a) don't persist hash-blocked bytes (check at stream end), (b) remember the CID (bounded in-memory set) and block it pre-serve on every later request, (c) also check the cached digest on the range path. CID-level blocking remains the deterministic pre-serve primitive. - Negative-cache poisoning (MED): the IPFS path only recordMiss'd on the shared NegativeDataCache, never evict/recordSuccess — so a transiently-unavailable CID that later pinned stayed blacked out, and miss-only stats skewed Arweave health. Evict + recordSuccess on a successful fetch/cache hit. - Wrong-host redirect (MED): the path-style redirect used req.matchedArnsRootHost which is unset on the IPFS router (mounted before the ArNS middleware that sets it), so multi-host gateways always redirected to ARNS_ROOT_HOSTS[0]. Resolve the root host from req.hostname directly. - 206 signing (LOW/MED): add content-range to CO_SIGNABLE_HEADERS so a partial response's byte range is bound into the HTTPSIG signature. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/ipfs/ipfs-cache.ts | 26 +++++++++ src/ipfs/ipfs-service.ts | 118 +++++++++++++++++++++++++++++++++------ src/lib/httpsig.ts | 3 + src/routes/ipfs.ts | 7 ++- 4 files changed, 137 insertions(+), 17 deletions(-) diff --git a/src/ipfs/ipfs-cache.ts b/src/ipfs/ipfs-cache.ts index 99fa21cea..c11ca378f 100644 --- a/src/ipfs/ipfs-cache.ts +++ b/src/ipfs/ipfs-cache.ts @@ -160,6 +160,32 @@ export class IpfsFsCache { return undefined; } + /** + * Returns the cached content's base64url SHA-256 digest without opening the + * data stream (in-memory index / .meta only). Used to enforce block-by-hash + * on range requests, which bypass the positive read cache. + */ + async getDigest( + cidString: string, + path?: string, + ): Promise { + const key = this.cacheKey(cidString, path); + let entry = this.index.get(key); + if (!entry) { + try { + await fs.promises.access(this.dataPath(key), fs.constants.F_OK); + const meta = await this.readMeta(key); + if (meta) { + this.index.set(key, meta); + entry = meta; + } + } catch { + // Not cached + } + } + return entry?.digest; + } + async put( cidString: string, stream: Readable, diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 4b6226c13..8515936f0 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -47,6 +47,15 @@ export class IpfsService { private blockListValidator: DataBlockListValidator; private maxResponseSizeBytes: number; private negativeCache?: NegativeDataCache; + // CIDs whose served bytes were found to be blocked-by-hash. IPFS content is + // streamed before its SHA-256 is known, so the very first (uncached) fetch of + // never-seen hash-blocked content cannot be stopped pre-serve — but once its + // hash is known we remember the CID and block every later request pre-serve, + // without persisting the bytes. CID-level blocking (isIdBlocked) remains the + // deterministic pre-serve primitive for IPFS. Bounded, FIFO-evicted, in-memory + // (re-learned after restart). + private readonly knownHashBlockedCids = new Set(); + private static readonly MAX_KNOWN_HASH_BLOCKED = 10_000; constructor({ log, @@ -107,11 +116,37 @@ export class IpfsService { throw new IpfsBlockedError(`CID is blocked: ${normalizedCid}`); } + // Pre-serve block for CIDs previously found to carry hash-blocked bytes + // (see knownHashBlockedCids). Covers full and range requests, cached or + // not, without re-fetching. + if (this.knownHashBlockedCids.has(normalizedCid)) { + metrics.ipfsBlockedTotal.inc(); + span.setAttribute('ipfs.blocked', true); + throw new IpfsBlockedError(`Content hash is blocked: ${normalizedCid}`); + } + // Reject path traversal attempts if (path !== undefined && (path.includes('..') || path.startsWith('/'))) { throw new IpfsNotFoundError('Invalid IPFS path'); } + // Range requests bypass the positive read cache below, so also honor a + // block-by-hash here using the cached digest (available once the object + // has been cached). Without this, a `Range: bytes=0-` request could serve + // hash-blocked bytes that a full GET refuses. + if (range !== undefined) { + const cachedDigest = await this.cache.getDigest(normalizedCid, path); + if ( + cachedDigest !== undefined && + (await this.blockListValidator.isHashBlocked(cachedDigest)) + ) { + metrics.ipfsBlockedTotal.inc(); + span.setAttribute('ipfs.blocked', true); + span.end(); + throw new IpfsBlockedError(`Content hash is blocked: ${normalizedCid}`); + } + } + // Negative cache: short-circuit CIDs we've repeatedly failed to fetch // (absent or unpinned) so they don't re-hit Kubo on every request // (latency / DoS amplification) — mirrors the Arweave path's negative data @@ -146,6 +181,11 @@ export class IpfsService { span.end(); throw new IpfsBlockedError(`Content hash is blocked: ${normalizedCid}`); } + // Content is available — clear any negative-cache entry and record a + // success so a transiently-unavailable CID that later pins isn't kept in + // a negative-cache blackout, and IPFS health isn't skewed miss-only. + this.negativeCache?.evict(normalizedCid); + this.negativeCache?.recordSuccess(); this.log.debug('IPFS cache hit', { cid: normalizedCid, path }); metrics.ipfsCacheHitTotal.inc(); span.setAttributes({ @@ -200,6 +240,11 @@ export class IpfsService { ); } + // A successful fetch means the content is available — clear any + // negative-cache entry and record health (see the cache-hit path). + this.negativeCache?.evict(normalizedCid); + this.negativeCache?.recordSuccess(); + // Stream directly to the client while writing to a temp file on disk for // caching. No memory buffering — handles files of any size. Partial (206) // responses are NOT cached — only full objects. @@ -236,6 +281,16 @@ export class IpfsService { } } + /** Remember a CID whose bytes are hash-blocked (bounded, FIFO-evicted). */ + private rememberHashBlocked(cidString: string): void { + if (this.knownHashBlockedCids.has(cidString)) return; + if (this.knownHashBlockedCids.size >= IpfsService.MAX_KNOWN_HASH_BLOCKED) { + const oldest = this.knownHashBlockedCids.values().next().value; + if (oldest !== undefined) this.knownHashBlockedCids.delete(oldest); + } + this.knownHashBlockedCids.add(cidString); + } + /** * Writes stream data to a temp cache file as it flows to the client. * Non-blocking — errors are logged but don't affect the response. @@ -253,6 +308,7 @@ export class IpfsService { .toString('hex')}`; let bytesWritten = 0; let failed = false; + let finalized = false; // Hash the bytes as they're written so cache hits can emit an RFC 9530 // Content-Digest (body binding) without a re-read. const hash = crypto.createHash('sha256'); @@ -318,26 +374,46 @@ export class IpfsService { cleanup(); return; } + finalized = true; const digest = hash.digest('base64url'); writeStream.end(() => { - // Finalize: move temp file into cache - this.cache - .putFromFile( - cidString, - tempPath, - bytesWritten, - contentType, - path, - digest, - ) - .catch((error) => { - this.log.error('Failed to finalize IPFS cache entry', { - cid: cidString, + void (async () => { + // Don't persist content whose served bytes are blocked-by-hash — the + // digest is only known now, at end of stream. Mirrors the Arweave path + // refusing to cache blocked content, and prevents a subsequent request + // from serving the blocked bytes out of cache. + try { + if (await this.blockListValidator.isHashBlocked(digest)) { + this.rememberHashBlocked(cidString); + this.log.info('Refusing to cache hash-blocked IPFS content', { + cid: cidString, + path, + }); + cleanup(); + return; + } + } catch { + // If the block check itself fails, fall through and cache as before. + } + // Finalize: move temp file into cache + this.cache + .putFromFile( + cidString, + tempPath, + bytesWritten, + contentType, path, - message: error.message, + digest, + ) + .catch((error) => { + this.log.error('Failed to finalize IPFS cache entry', { + cid: cidString, + path, + message: error.message, + }); + cleanup(); }); - cleanup(); - }); + })(); }); }); @@ -345,5 +421,15 @@ export class IpfsService { failed = true; cleanup(); }); + + // Premature close without 'end'/'error' — e.g. a HEAD request destroys the + // stream after reading headers, or the client aborts mid-body. Node emits + // 'close' (not 'error'/'end') on destroy(), so without this the write stream + // and temp file leak (fd/disk exhaustion under repeated HEADs). + stream.on('close', () => { + if (!failed && !finalized) { + cleanup(); + } + }); } } diff --git a/src/lib/httpsig.ts b/src/lib/httpsig.ts index e7b5723ab..a08a78aaa 100644 --- a/src/lib/httpsig.ts +++ b/src/lib/httpsig.ts @@ -67,6 +67,9 @@ export const TRIGGER_HEADERS = new Set([ export const CO_SIGNABLE_HEADERS = new Set([ 'content-type', 'content-digest', + // Bind the served byte range into the signature on a 206, so a partial + // response can't be passed off as the full object under a signed ETag=CID. + 'content-range', 'x-cache', 'etag', 'x-ar-io-root-data-item-offset', diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index 034d6b077..5b056cdce 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -124,8 +124,13 @@ function createIpfsPathHandler({ if (config.ARNS_ROOT_HOSTS.length > 0) { const v1Base32 = cidToV1Base32(cidString); if (getRequestSandbox(req) !== v1Base32) { + // Derive the root host from the request directly. req.matchedArnsRootHost + // is set by the ArNS middleware, which is mounted AFTER this IPFS path + // router, so it is always undefined here — resolve it ourselves so a + // multi-root-host gateway redirects to the host the request arrived on. const rootHost = - req.matchedArnsRootHost ?? config.ARNS_ROOT_HOSTS[0].host; + config.matchArnsRootHost(req.hostname)?.host ?? + config.ARNS_ROOT_HOSTS[0].host; const pathSuffix = path !== undefined ? `/${path}` : ''; const queryString = url.parse(req.originalUrl).query; res.redirect( From 3a5a7b2c567f65f8bdbf0a4a6cfd58f4bfb71967 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 11:39:38 +0000 Subject: [PATCH 25/47] fix(ipfs): harden follow-ups from review (rate/DoS, https, multi-range) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Kubo concurrency cap (IPFS_KUBO_MAX_CONCURRENT_REQUESTS, default 100): bound in-flight Kubo fetches so cheap-to-issue HEAD/tiny-range requests can't amplify into unbounded upstream/DHT load; excess fail fast with 502. Slot released on stream close or error. - Multi-range requests (`bytes=0-1,5-6`) are served in full (200) instead of forwarded — Kubo doesn't reliably emit multipart/byteranges. Single-range only. - Warn at startup when TRUSTED_ARNS_GATEWAY_URL is plaintext http, since the resolver now accepts an upstream's protocol + CID (MITM could forge name→CID). - X-Ar-Io-Source is now declared in headerNames (constants) and referenced there instead of a raw literal. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/config.ts | 9 +++++++++ src/constants.ts | 3 +++ src/init/resolvers.ts | 14 ++++++++++++++ src/ipfs/kubo-data-source.ts | 28 ++++++++++++++++++++++++++++ src/routes/ipfs.ts | 11 +++++++++-- src/system.ts | 1 + 6 files changed, 64 insertions(+), 2 deletions(-) diff --git a/src/config.ts b/src/config.ts index 52a30e125..9c6c13e60 100644 --- a/src/config.ts +++ b/src/config.ts @@ -3224,6 +3224,15 @@ export const IPFS_STREAM_STALL_TIMEOUT_MS = env.positiveIntOrDefault( 30000, ); +// Max concurrent in-flight fetches to the Kubo gateway. Bounds Kubo/DHT +// amplification from cheap-to-issue requests (HEAD, tiny Range) that each force +// an upstream fetch before rate limiting is evaluated; excess requests fail fast +// with 502 rather than piling onto Kubo. 0 disables the cap. +export const IPFS_KUBO_MAX_CONCURRENT_REQUESTS = env.positiveIntOrDefault( + 'IPFS_KUBO_MAX_CONCURRENT_REQUESTS', + 100, +); + export const IPFS_CACHE_PATH = env.varOrDefault( 'IPFS_CACHE_PATH', 'data/ipfs-cache', diff --git a/src/constants.ts b/src/constants.ts index 707625a51..21e4dc2bf 100644 --- a/src/constants.ts +++ b/src/constants.ts @@ -26,6 +26,9 @@ */ export const headerNames = { hops: 'X-AR-IO-Hops', + // Which retrieval source served the body (e.g. 'ipfs'). Declared centrally so + // it's referenced consistently and is a candidate for HTTPSIG trigger headers. + arIoSource: 'X-Ar-Io-Source', origin: 'X-AR-IO-Origin', originNodeRelease: 'X-AR-IO-Origin-Node-Release', digest: 'X-AR-IO-Digest', diff --git a/src/init/resolvers.ts b/src/init/resolvers.ts index 86d45652d..1130c91fd 100644 --- a/src/init/resolvers.ts +++ b/src/init/resolvers.ts @@ -97,6 +97,20 @@ export const createArNSResolver = ({ : undefined, }; + // The trusted-gateway resolver now accepts an upstream's protocol + resolved + // id (including IPFS CIDs). A plaintext upstream can be MITM'd to bind a name + // to arbitrary content and drive this node's Kubo to fetch it — so prefer https. + if ( + trustedGatewayUrl !== undefined && + trustedGatewayUrl.startsWith('http://') + ) { + log.warn( + 'TRUSTED_ARNS_GATEWAY_URL uses plaintext http; an on-path attacker could ' + + 'forge name resolutions (incl. IPFS CIDs). Use https.', + { trustedGatewayUrl }, + ); + } + const resolvers: NameResolver[] = []; // add resolvers in the order specified by resolutionOrder diff --git a/src/ipfs/kubo-data-source.ts b/src/ipfs/kubo-data-source.ts index acc792cb4..cfc280741 100644 --- a/src/ipfs/kubo-data-source.ts +++ b/src/ipfs/kubo-data-source.ts @@ -27,22 +27,27 @@ export class KuboDataSource { private kuboUrl: string; private requestTimeoutMs: number; private streamStallTimeoutMs: number; + private maxConcurrent: number; + private inFlight = 0; constructor({ log, kuboUrl, requestTimeoutMs, streamStallTimeoutMs, + maxConcurrent = 0, }: { log: winston.Logger; kuboUrl: string; requestTimeoutMs: number; streamStallTimeoutMs: number; + maxConcurrent?: number; }) { this.log = log.child({ class: this.constructor.name }); this.kuboUrl = kuboUrl.replace(/\/$/, ''); this.requestTimeoutMs = requestTimeoutMs; this.streamStallTimeoutMs = streamStallTimeoutMs; + this.maxConcurrent = maxConcurrent; } async getContent({ @@ -60,6 +65,24 @@ export class KuboDataSource { }): Promise { signal?.throwIfAborted(); + // Concurrency cap: bound in-flight Kubo fetches so cheap-to-issue requests + // (HEAD, tiny Range) can't amplify into unbounded upstream/DHT load — excess + // requests fail fast instead of piling onto Kubo. The slot is released when + // the returned stream closes (below) or on any error (catch). + if (this.maxConcurrent > 0 && this.inFlight >= this.maxConcurrent) { + throw new IpfsUnavailableError( + `Too many concurrent IPFS fetches (${this.inFlight}/${this.maxConcurrent})`, + ); + } + this.inFlight++; + let released = false; + const release = () => { + if (!released) { + released = true; + this.inFlight--; + } + }; + // URL-encode path segments to prevent breaking the upstream request const encodedPath = path !== undefined && path !== '' @@ -186,6 +209,10 @@ export class KuboDataSource { span.end(); }); + // Release the concurrency slot when the response stream is fully consumed + // or destroyed (covers success, client abort, and downstream errors). + stream.once('close', release); + return { stream, size: contentLength, @@ -194,6 +221,7 @@ export class KuboDataSource { contentRange: response.headers['content-range'], }; } catch (error: any) { + release(); clearTimeout(connectionTimer); signal?.removeEventListener('abort', onClientAbort); diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index 5b056cdce..8cc6d0bfa 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -12,6 +12,7 @@ import url from 'node:url'; import * as config from '../config.js'; import * as metrics from '../metrics.js'; +import { headerNames } from '../constants.js'; import { cidToV1Base32, isValidCid } from '../lib/ipfs-cid.js'; import { getRequestSandbox } from '../middleware/sandbox.js'; import { IpfsService } from '../ipfs/ipfs-service.js'; @@ -180,6 +181,12 @@ async function handleIpfsRequest({ }): Promise { const startTime = Date.now(); const isHead = req.method === 'HEAD'; + // Single-range only. A multi-range request (`bytes=0-1,5-6`) is served in full + // (200) rather than forwarded — Kubo doesn't reliably emit multipart/byteranges + // and relaying it alongside our own Content-Length would be inconsistent. + const rawRange = req.headers.range; + const rangeForKubo = + typeof rawRange === 'string' && !rawRange.includes(',') ? rawRange : undefined; const ipfsPath = path !== undefined ? `${cidString}/${path}` : cidString; parentLog.debug('Handling IPFS request', { cidString, path, routeType }); @@ -189,7 +196,7 @@ async function handleIpfsRequest({ cidString, path, signal: req.signal, - range: req.headers.range, + range: rangeForKubo, }); // Check payment and rate limits (x402 + rate limiting in one call). @@ -244,7 +251,7 @@ async function handleIpfsRequest({ } res.setHeader('ETag', `"${cidToV1Base32(cidString)}"`); res.setHeader('X-Ipfs-Path', `/ipfs/${ipfsPath}`); - res.setHeader('X-Ar-Io-Source', 'ipfs'); + res.setHeader(headerNames.arIoSource, 'ipfs'); // Body binding (RFC 9530 Content-Digest). When a SHA-256 of the served // bytes is known (computed at cache-write time, returned on cache hits), diff --git a/src/system.ts b/src/system.ts index f30ee1af2..3b8864c15 100644 --- a/src/system.ts +++ b/src/system.ts @@ -1894,6 +1894,7 @@ if (config.IPFS_ENABLED) { kuboUrl: config.IPFS_KUBO_URL, requestTimeoutMs: config.IPFS_KUBO_REQUEST_TIMEOUT_MS, streamStallTimeoutMs: config.IPFS_STREAM_STALL_TIMEOUT_MS, + maxConcurrent: config.IPFS_KUBO_MAX_CONCURRENT_REQUESTS, }); const ipfsCache = new IpfsFsCache({ From 029c73e727c05e94a26458e48f4b05954144ca11 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 11:45:49 +0000 Subject: [PATCH 26/47] docs(ipfs): Arweave-parity summary, ops skill, new env, CLAUDE.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - ipfs-integration.md: add a "Parity with the Arweave Data Path" table (what is now aligned: moderation, caching, negative cache, HEAD/Range, origin isolation, rate limiting, HTTPSIG, protocol resolution, observability, error semantics) plus the inherent content-addressing/pinning differences; document IPFS_KUBO_MAX_CONCURRENT_REQUESTS. - envs.md + docker-compose.yaml: add IPFS_KUBO_MAX_CONCURRENT_REQUESTS (kept in sync per repo convention). - ar-io-gateway-operator skill: add an "IPFS serving (opt-in Kubo sidecar)" operations section (entry points, pinning/availability caveat, moderation, HEAD/Range/sandbox, verify-live + troubleshooting); correct the stale ArNS→IPFS phrasing. - CLAUDE.md: note protocol is first-class + propagated across a trusted-gateway hop, and the IPFS path is held to Arweave-path parity. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- .../skills/ar-io-gateway-operator/SKILL.md | 14 +++++++++- CLAUDE.md | 8 ++++-- docker-compose.yaml | 1 + docs/envs.md | 1 + docs/ipfs-integration.md | 28 +++++++++++++++++++ 5 files changed, 49 insertions(+), 3 deletions(-) diff --git a/.claude/skills/ar-io-gateway-operator/SKILL.md b/.claude/skills/ar-io-gateway-operator/SKILL.md index 03e92a001..c76055116 100644 --- a/.claude/skills/ar-io-gateway-operator/SKILL.md +++ b/.claude/skills/ar-io-gateway-operator/SKILL.md @@ -109,10 +109,22 @@ Pipeline diagnostic: a single `curl -sf .../ar-io/__gateway_metrics | grep -E 'q ### ArNS resolution -`CompositeArNSResolver` walks resolvers in order: `TrustedGatewayArNSResolver` (asks `TRUSTED_ARNS_GATEWAY_URL`, default `https://__NAME__.turbo-gateway.com`), then `OnDemandArNSResolver` (queries the on-chain `ario-arns` and `ario-ant` programs directly via `SOLANA_RPC_URL`). The base name set is paginated into an in-memory `ArNSNamesCache` at boot and refreshed on a debounce. Resolved IDs may be Arweave TXs (route to data path) or, on the streaming-head branch, IPFS CIDs (route to `/ipfs/` via the Kubo sidecar). Unknown names log `Unable to resolve name against all resolvers` — that's normal user-error traffic, not a service failure. +`CompositeArNSResolver` walks resolvers in order: `TrustedGatewayArNSResolver` (asks `TRUSTED_ARNS_GATEWAY_URL`, default `https://__NAME__.turbo-gateway.com`), then `OnDemandArNSResolver` (queries the on-chain `ario-arns` and `ario-ant` programs directly via `SOLANA_RPC_URL`). The base name set is paginated into an in-memory `ArNSNamesCache` at boot and refreshed on a debounce. Resolved IDs may be Arweave TXs (route to the data path) or IPFS CIDs when the ANT record sets `targetProtocol: ipfs` (route to `/ipfs/` via the Kubo sidecar; see "IPFS serving" below). The `protocol` is carried on the resolution and across a trusted-gateway hop via `X-ArNS-Protocol`. Unknown names log `Unable to resolve name against all resolvers` — that's normal user-error traffic, not a service failure. `/` and `//` requests carry `X-ArNS-*` trust headers in the response. Manifest path resolution still uses `StreamingManifestPathResolver`; the "from index" path is not implemented yet (logs warn `not implemented` then falls back to data-side resolution, which works). +### IPFS serving (opt-in Kubo sidecar) + +`IPFS_ENABLED=true` adds a Kubo sidecar (`docker compose --profile ipfs up -d`, or a compose override that adds the `kubo` service) and turns on IPFS serving. The gateway proxies, caches, moderates, and signs IPFS content alongside Arweave data, held to the same operational bar (`docs/ipfs-integration.md` → "Parity with the Arweave Data Path"). + +- **Two entry points**: direct `/ipfs/{CID}` (and `{CID}.{root_host}` subdomains), and ArNS names whose ANT record sets `targetProtocol: ipfs` — the resolved id is a CID, surfaced as `X-ArNS-Protocol: ipfs` and served via Kubo. `protocol` propagates across a trusted-gateway hop, so a name→CID binding survives even with `gateway` ahead of `on-demand` in `ARNS_RESOLVER_PRIORITY_ORDER`. +- **The node fetches from the local Kubo gateway** (`IPFS_KUBO_URL`, default `http://kubo:8080`) — it does not join the DHT itself. Availability depends on Kubo finding/holding the blocks; Kubo runs `--enable-gc`, so **unpinned content can disappear** (no on-chain permanence like Arweave). This is the key operational difference from Arweave data. +- **Caching** is a bounded LRU separate from the Arweave content cache (`IPFS_CACHE_*`); absent/unpinned CIDs are negative-cached; hash-blocked bytes are never persisted. +- **Moderation** uses the same admin API (see Block/unblock below): `PUT /ar-io/admin/block-data {"id":""}`. CID-blocking is the deterministic pre-serve primitive for IPFS (content is CID-addressed); hash-blocking also applies once a CID's served-byte SHA-256 is known. +- **HEAD, Range/`206`, and per-CID sandbox origin isolation** (`/ipfs/{CID}` → `{CID}.{root_host}`) work as on the Arweave path. `IPFS_KUBO_MAX_CONCURRENT_REQUESTS` caps in-flight Kubo fetches (amplification guard); `IPFS_MAX_RESPONSE_SIZE_BYTES` caps a single response. +- **Verify it's live**: `GET /ar-io/info` shows `ipfs.enabled: true`; `docker exec ipfs swarm peers` should list peers. +- **Troubleshooting**: a name that resolves on viewblock but `404`s here with `IPFS_ENABLED` off means the CID is being misrouted to the Arweave data path — enable IPFS + Kubo. `/ipfs/{CID}` returning `502`/`504` points at the Kubo sidecar (down, no peers, or unpinned/cold content); check swarm peers first. + ### Network identity, observer, and incentives A gateway is a registered participant in the AR.IO network, not just a piece of software. Two distinct Solana identities matter: diff --git a/CLAUDE.md b/CLAUDE.md index f7185f0ef..f3cff8ea8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -76,8 +76,12 @@ yarn service:start / stop / restart / status / logs for content retrieval with its own cache, rate limiter, and blocklist. Routes mount before ArNS in `app.ts`. ArNS names whose ANT record has `targetProtocol: ipfs` resolve to a CID and are routed to the same IPFS - handler by the ArNS middleware (`src/middleware/arns.ts`); the on-demand - resolver reads `targetProtocol`. See `docs/ipfs-integration.md`. + handler by the ArNS middleware (`src/middleware/arns.ts`). `protocol` is a + first-class field on `NameResolution` set by the on-demand resolver and + propagated across a trusted-gateway hop via the signed `X-ArNS-Protocol` + header (`src/resolution/`). The IPFS path is held to Arweave-path parity + (moderation, caching, HEAD/Range, sandbox origin isolation, HTTPSIG). See + `docs/ipfs-integration.md` ("Parity with the Arweave Data Path"). - Responses include trust headers indicating verification status. - HTTPSIG signs response headers (RFC 9421); `Content-Digest` is in `CO_SIGNABLE_HEADERS` so when present it binds the body to the signature. diff --git a/docker-compose.yaml b/docker-compose.yaml index 62b8464dc..120172db6 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -157,6 +157,7 @@ services: - IPFS_KUBO_URL=${IPFS_KUBO_URL:-http://kubo:8080} - IPFS_KUBO_REQUEST_TIMEOUT_MS=${IPFS_KUBO_REQUEST_TIMEOUT_MS:-} - IPFS_STREAM_STALL_TIMEOUT_MS=${IPFS_STREAM_STALL_TIMEOUT_MS:-} + - IPFS_KUBO_MAX_CONCURRENT_REQUESTS=${IPFS_KUBO_MAX_CONCURRENT_REQUESTS:-} - IPFS_CACHE_PATH=${IPFS_CACHE_PATH:-} - IPFS_CACHE_MAX_SIZE_BYTES=${IPFS_CACHE_MAX_SIZE_BYTES:-} - IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS=${IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS:-} diff --git a/docs/envs.md b/docs/envs.md index 9ff08d84b..d8983c7c3 100644 --- a/docs/envs.md +++ b/docs/envs.md @@ -634,6 +634,7 @@ as a Docker Compose sidecar via the `ipfs` profile). | IPFS_KUBO_URL | String | http://kubo:8080 | Kubo HTTP gateway URL | | IPFS_KUBO_REQUEST_TIMEOUT_MS | Number | 30000 | Connection timeout for Kubo requests (ms) | | IPFS_STREAM_STALL_TIMEOUT_MS | Number | 30000 | Stall timeout — max time with no data before aborting stream (ms) | +| IPFS_KUBO_MAX_CONCURRENT_REQUESTS | Number | 100 | Max concurrent in-flight Kubo fetches; excess fail fast with 502. 0 disables | | IPFS_CACHE_PATH | String | data/ipfs-cache | Directory for cached IPFS content | | IPFS_CACHE_MAX_SIZE_BYTES | Number | 10737418240 (10 GB) | Maximum cache size before LRU eviction | | IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS | Number | 3600 | Age in seconds before cached files become eviction candidates | diff --git a/docs/ipfs-integration.md b/docs/ipfs-integration.md index 33658927c..34d4d3a34 100644 --- a/docs/ipfs-integration.md +++ b/docs/ipfs-integration.md @@ -451,6 +451,7 @@ All environment variables are opt-in. The feature is disabled by default. | `IPFS_KUBO_URL` | String | `http://kubo:8080` | Base URL of the local Kubo HTTP Gateway. In Docker, this is the container name. For local development, use `http://localhost:8080`. | | `IPFS_KUBO_REQUEST_TIMEOUT_MS` | Number | `30000` | Connection timeout in milliseconds for Kubo requests (time to receive response headers). | | `IPFS_STREAM_STALL_TIMEOUT_MS` | Number | `30000` | Stall timeout in milliseconds for streaming responses from Kubo. Stream is aborted if no data is received for this duration. Actively-streaming transfers are not affected. | +| `IPFS_KUBO_MAX_CONCURRENT_REQUESTS` | Number | `100` | Maximum concurrent in-flight fetches to Kubo. Bounds upstream/DHT amplification from cheap-to-issue requests (HEAD, tiny Range) that force a fetch before rate limiting is evaluated; requests over the cap fail fast with `502`. `0` disables the cap. | | `IPFS_CACHE_PATH` | String | `data/ipfs-cache` | Directory for the IPFS filesystem cache. Relative paths are resolved from the gateway's working directory. | | `IPFS_CACHE_MAX_SIZE_BYTES` | Number | `10737418240` (10 GB) | Maximum total size of the IPFS cache directory. LRU eviction begins when this limit is exceeded. | | `IPFS_CACHE_CLEANUP_THRESHOLD` | Number | `3600` | Interval in seconds between cache eviction scans. | @@ -533,6 +534,33 @@ hits), so the name->CID binding and the served bytes are both attested. a CID there will not serve. To serve IPFS at the apex, use `APEX_ARNS_NAME` pointing at an ANT whose `@` record targets IPFS. ❌ +## Parity with the Arweave Data Path + +The IPFS path is held to the same operational bar as the Arweave data path; +cross-cutting behaviors are shared or matched: + +| Concern | Arweave path | IPFS path | +|---------|--------------|-----------| +| **Moderation** | `PUT /ar-io/admin/block-data` (id + hash) → `451` | Same admin API/store; `isIdBlocked(CID)` pre-serve, plus `isHashBlocked` once a CID's served-byte SHA-256 is known; CID-blocking is the deterministic pre-serve primitive | +| **Name blocking** | blocked ArNS name → `451` before serving | same middleware gate runs before IPFS dispatch | +| **Caching** | read-through content cache | bounded LRU IPFS cache; hash-blocked bytes are never persisted | +| **Negative cache** | absent ids short-circuit repeat lookups | absent/unpinned CIDs short-circuit (shared `NegativeDataCache`, with `evict`/`recordSuccess` on availability) | +| **HEAD / Range** | HEAD; `206`/`416`/`Accept-Ranges` | HEAD (no body); single-range `206` + `Content-Range` + `Accept-Ranges`; `416` on unsatisfiable | +| **Origin isolation** | `/{txid}` → per-id sandbox subdomain | `/ipfs/{CID}` → per-CID sandbox subdomain | +| **Rate limiting** | token bucket + IP/CIDR allowlist | separate IPFS token pools, same allowlist; Kubo concurrency cap bounds amplification | +| **HTTPSIG** | signed trust/envelope headers + `Content-Digest` | signed `X-ArNS-*` / `X-Ipfs-Path` / `X-Ar-Io-Source` / `ETag`; `Content-Range` bound on `206`; `Content-Digest` when the hash is known | +| **Protocol resolution** | — | `protocol` propagates across resolvers including a trusted-gateway hop (`X-ArNS-Protocol`) | +| **Observability** | Prometheus, OTEL, structured logs | dedicated IPFS metrics + OTEL spans + structured logs | +| **Error semantics** | `404`/`5xx` with cache-control | `404` (cache-controlled) / `451` / `413` / `416` / `502` / `504` | + +**Inherent differences** (by design, not gaps): IPFS content is +content-addressed — the CID is its integrity proof and the gateway delegates +block verification to Kubo — and pinning-dependent (it can disappear if +unpinned), whereas Arweave data is permanent and Merkle-verified against its +signed `data_root`. The first, uncached fetch of never-seen content also cannot +be hash-blocked pre-serve (the SHA-256 is only known after streaming); use +CID-level blocking for deterministic pre-serve moderation of IPFS content. + ## Differences from Arweave Data Serving | Aspect | Arweave | IPFS | From 485ea22eff1d44214aad94bf33e669cd7dc0e0b4 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 11:53:45 +0000 Subject: [PATCH 27/47] docs(ipfs): add David's-brain architectural alignment analysis Compare the shipped Kubo-sidecar Path-Gateway + ArNS->IPFS work against the architect's IPFS design (trustless gateway, CAR-ingest-to-Arweave, client-side verification) and big-picture positions. States where we align (protocol- independent addressing direction; UnixFS delegated to a paired node) and where David would push back (Path vs Trustless posture, public-IPFS vs permapinned-on- Arweave, parallel Kubo stack vs composite source, no client verification). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- docs/drafts/davids-brain-alignment.md | 107 ++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) create mode 100644 docs/drafts/davids-brain-alignment.md diff --git a/docs/drafts/davids-brain-alignment.md b/docs/drafts/davids-brain-alignment.md new file mode 100644 index 000000000..2b148ff48 --- /dev/null +++ b/docs/drafts/davids-brain-alignment.md @@ -0,0 +1,107 @@ +# Alignment check: our ArNS/IPFS/OIP work vs. David's brain + +Reading of `ar-io/davids-brain` (`big-picture.md` §1–§5; `docs/ar-io-node/ipfs-integration.md`) +against what shipped in this branch (#682 Kubo-sidecar IPFS + #793 ArNS→IPFS + +the parity/QA work here). Honest version: we align on **direction**, and diverge +from David on the **core trust/storage/serving axes**. Both are worth stating. + +## What David wants (the yardstick) + +- **§1/§2 — protocol-independent addressing.** A client asking for `bafy…` + shouldn't care whether bytes are on Arweave/S3/Filecoin, *as long as the bytes + hash back to the address and the signer checks out.* Storage contract is a + pluggable back end under a thin verified addressing layer. +- **§3 — browsers are the wrong target.** A gateway that proxies content back + over `https://` while implying it's verified "is strictly worse than a CDN." +- **§5 — verification on the client.** The gateway collapses from "trusted + query/verification oracle" to "cache + offset resolver + protocol translator." +- **His IPFS design specifically:** lead with the **Trustless Gateway** + (`GET /ipfs/:cid?format=raw` → one verifiable block, *client* checks CID), + content **stored on Arweave** via CAR ingest + a `cid → (id, offset, len, + codec)` index served through **the existing `src/data/` composite source**, and + **Path Gateway / UnixFS reassembly explicitly pushed to a *separate paired IPFS + node*, not built into ar-io-node.** "The gateway is not a trust root." + +## Where we align ✅ + +- **Direction (§1/§2).** Making `protocol` first-class on `NameResolution`, + propagating it across a trusted-gateway hop, and letting ArNS names point at + CIDs is exactly the "protocol-independent addressing" move David calls the right + instinct — "just push further." +- **We did not grow UnixFS/Path-Gateway logic into ar-io-node core.** We delegate + reassembly/dir-resolution to a **paired Kubo node** — which is precisely where + David says that responsibility belongs ("a separate paired IPFS node… rather + than building UnixFS logic into ar-io-node"). Our service boundary is right. +- **Gateway hygiene / parity.** HEAD, Range/206, per-CID sandbox origin + isolation, unified moderation, caching, rate limiting, HTTPSIG envelope — none + of this conflicts with his design; it's the table stakes he assumes. + +## Where David would call us out ⚠️ + +1. **We built a Path Gateway (trusted proxy), not a Trustless Gateway + (client-verifiable).** We serve Kubo-reassembled bytes over TLS and *sign them + with HTTPSIG* — i.e. "we won't lie to you." David's whole §3/§5 thesis is the + opposite: return verifiable blocks and let the client check the hash ("you + don't need to trust us"). **Signing IPFS responses as if attested is the exact + anti-pattern he flags.** We never expose `?format=raw` blocks or let a client + verify a CID. → *Call-out: offer a trustless `?format=raw` block path, and be + explicit that the current UnixFS path is trusted-proxy, not verified.* + +2. **Our content lives on the public IPFS network, not permapinned on Arweave.** + David's entire integration is *IPFS-content-stored-on-Arweave* ("permapin this + CID/CAR", CAR ingest → index → serve from Arweave storage). We proxy ephemeral + public-IPFS content via Kubo (`--enable-gc`, unpinned content vanishes). This + **misses §1's point** (contracts coexisting *under* Arweave's durability) and + creates the availability problem the incentive analysis independently found. + → *Call-out: the durable product is Turbo "permapin CID/CAR" + ar-io-node CAR + indexing, not a Kubo proxy to public IPFS.* + +3. **We bypassed the `src/data/` composite source.** David's serve path is "CID + lookup → range-read via the existing data source composite (S3, peers, chain, + …)". We added a **parallel Kubo stack** (`KuboDataSource`/`IpfsFsCache`) with + its own return type and cache, converging only at routing. That is the + opposite of "one thin verified addressing layer over pluggable back ends" + (§2). → *Call-out: a `cid → offset` index feeding the composite source is the + aligned shape; the Kubo sidecar is a shortcut that entrenches a second stack.* + +4. **No client-side verification and no chain anchor (§5).** We emit no + `?format=raw`, no per-block CID check, and none of the merkle-proof headers + (`X-Arweave-Chunk-Data-Path`/`-Tx-Path`/…) on IPFS routes. The gateway is a + trust root — the thing §5 exists to eliminate. His Stage 2 ("proof headers + + portable validator") is the whole payoff, and we're not on that path. + +5. **ArNS→IPFS resolving to ephemeral content dilutes the ArNS value prop.** A + name pointing at unpinned public-IPFS content can 404 tomorrow; ArNS + historically implies permanence. (Reinforced by the OIP analysis: names are + observed/rewarded assuming retrievability.) + +## OIP / observation (ties to §4/§5) + +The separate incentive analysis found the observer verifies ArNS serving by +**trust-based digest comparison against reference gateways** — the exact +"gateway as trusted oracle" pattern David wants gone. His §5 posture says the +**observer should verify CID→bytes itself** (content-addressing makes this free), +not trust a reference set. And rewarding "serving" ephemeral, non-permapinned +content has no durability anchor. So the OIP gaps and David's critique point the +same way: **verify, don't trust; anchor to the chain; reward durability, not +proxying.** + +## Honest bottom line + +What we shipped is a **pragmatic, correct, well-hardened Path-Gateway-over-Kubo** +that realizes the *addressing* direction (§2) and is genuinely useful as a +browser/convenience fallback. But on **storage (public IPFS vs Arweave), trust +(gateway-signed vs client-verified), and architecture (parallel Kubo stack vs +composite source)**, it is close to the inverse of David's design. His critical +path — **Stage 0 SQLite chunk-metadata index → Stage 1 CAR ingest + trustless +`/ipfs/:cid?format=raw` → Stage 2 proof headers + portable validator** — is a +different, larger build that our work neither advances nor blocks. + +Recommended framing for the team: **ship ours as the explicitly-trusted Path +fallback**, and **treat David's trustless/CAR/verification path as the real +roadmap** (his Stages 0–2). Two concrete, low-cost steps that move us toward him +without a rewrite: +- Add a trustless `GET /ipfs/:cid?format=raw` that serves a single block and does + **not** HTTPSIG-attest it as verified (client verifies). +- Stop implying verification on the UnixFS path — document `X-Ar-Io-Source: ipfs` + as *trusted-proxy*, and don't let the signed envelope read as a content proof. From 97ea1a6cb32997e86b245fb5c1afd66c2a482ba9 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 12:54:33 +0000 Subject: [PATCH 28/47] =?UTF-8?q?feat(ipfs):=20read-only=20IPFS=20mode=20?= =?UTF-8?q?=E2=80=94=20trustless=20format,=20named-content=20pinning,=20ho?= =?UTF-8?q?nest=20trust=20posture?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Aligns the read-only IPFS gateway with the architect's posture (trustless, client-verifiable) where it makes sense for this phase, without introducing Arweave storage (that's phase 2): - Trustless retrieval: GET /ipfs/:cid?format=raw|car (or an IPLD Accept type) is forwarded to Kubo and relayed as a verifiable raw block / CAR that the CLIENT checks against the CID. Content-Disposition: attachment, ETag=CID, and X-Ar-Io-Trustless: true. Bypasses the UnixFS cache; a mid-stream size guard bounds large CARs (which skip streamToCache); the rate-limit reserve for unknown-size responses is modest (IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES) instead of 1 GiB, which had auto-429'd every CAR. - Honest trust posture: the UnixFS proxy path is marked X-Ar-Io-Trustless: false (gateway-attested, not client-verified). The signed envelope means "a registered gateway served this", not a content proof. - Named-content pinning (IPFS_PIN_ARNS_CONTENT): pin the CIDs ArNS names resolve to, via the Kubo RPC API (IPFS_KUBO_API_URL), so named content this gateway serves stays retrievable despite Kubo GC. Best-effort, fire-and-forget, bounded FIFO (IPFS_PIN_MAX). This is the substrate an OIP "reward serving named IPFS data" incentive needs. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/app.ts | 2 + src/config.ts | 26 ++++++++ src/ipfs/ipfs-pinner.ts | 88 +++++++++++++++++++++++++++ src/ipfs/ipfs-service.ts | 47 +++++++++++++-- src/ipfs/kubo-data-source.ts | 13 +++- src/routes/arns.ts | 1 + src/routes/ipfs.ts | 113 +++++++++++++++++++++++++---------- src/system.ts | 10 ++++ 8 files changed, 264 insertions(+), 36 deletions(-) create mode 100644 src/ipfs/ipfs-pinner.ts diff --git a/src/app.ts b/src/app.ts index 615207797..ed135bf9b 100644 --- a/src/app.ts +++ b/src/app.ts @@ -135,6 +135,7 @@ if (config.IPFS_ENABLED && system.ipfsService !== undefined) { ipfsService: system.ipfsService, rateLimiter: system.ipfsRateLimiter, paymentProcessor: system.paymentProcessor, + pinner: system.ipfsPinner, }); app.use(createIpfsSubdomainMiddleware({ ipfsHandler })); app.use( @@ -143,6 +144,7 @@ if (config.IPFS_ENABLED && system.ipfsService !== undefined) { ipfsService: system.ipfsService, rateLimiter: system.ipfsRateLimiter, paymentProcessor: system.paymentProcessor, + pinner: system.ipfsPinner, }), ); } diff --git a/src/config.ts b/src/config.ts index 9c6c13e60..cc397161a 100644 --- a/src/config.ts +++ b/src/config.ts @@ -3233,6 +3233,32 @@ export const IPFS_KUBO_MAX_CONCURRENT_REQUESTS = env.positiveIntOrDefault( 100, ); +// Kubo RPC API base (distinct from the read-only gateway on 8080). Used only for +// pinning; the powerful admin API should not be exposed beyond the sidecar. +export const IPFS_KUBO_API_URL = env.varOrDefault( + 'IPFS_KUBO_API_URL', + 'http://kubo:5001', +); + +// Pin the IPFS CIDs that ArNS names resolve to, so named content this gateway +// serves stays retrievable (read-only availability) instead of being GC'd from +// Kubo when unpinned network-wide. Bounded set of "named" content only. +export const IPFS_PIN_ARNS_CONTENT = + env.varOrDefault('IPFS_PIN_ARNS_CONTENT', 'false') === 'true'; + +// Max distinct CIDs the pinner tracks/holds this process; oldest are unpinned +// (FIFO) beyond this to bound local storage. +export const IPFS_PIN_MAX = env.positiveIntOrDefault('IPFS_PIN_MAX', 10000); + +// Rate-limit reserve for a response whose size Kubo doesn't declare up front +// (CAR / chunked). Reserving the full IPFS_MAX_RESPONSE_SIZE_BYTES would exceed +// the token buckets and reject every such request; actual bytes are still bounded +// mid-stream by IPFS_MAX_RESPONSE_SIZE_BYTES. +export const IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES = env.positiveIntOrDefault( + 'IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES', + 262144, +); + export const IPFS_CACHE_PATH = env.varOrDefault( 'IPFS_CACHE_PATH', 'data/ipfs-cache', diff --git a/src/ipfs/ipfs-pinner.ts b/src/ipfs/ipfs-pinner.ts new file mode 100644 index 000000000..1b5d175fd --- /dev/null +++ b/src/ipfs/ipfs-pinner.ts @@ -0,0 +1,88 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { default as axios } from 'axios'; +import winston from 'winston'; + +import { cidToV1Base32 } from '../lib/ipfs-cid.js'; + +/** + * Best-effort pinner for "named" IPFS content — the CIDs that ArNS names resolve + * to. In read-only IPFS mode (no Arweave storage of the content) a name→CID + * binding is only as available as the public network keeps it; Kubo runs with + * GC, so unpinned content can vanish and the name 404s. Pinning the CIDs this + * gateway is responsible for serving keeps them retrievable locally. + * + * Deliberately simple: fire-and-forget (never blocks a response), idempotent + * (Kubo pin/add is idempotent; an in-memory set suppresses duplicate calls), and + * bounded (oldest pins are removed FIFO past `max` to cap local storage). The set + * is in-memory, so after a restart already-pinned CIDs are simply re-pinned on + * next resolution — harmless. Uses the Kubo RPC API (not the read-only gateway). + */ +export class IpfsPinner { + private log: winston.Logger; + private apiUrl: string; + private max: number; + // Insertion-ordered set of pinned CIDs (v1 base32) for FIFO eviction. + private pinned = new Set(); + private inFlight = new Set(); + + constructor({ + log, + apiUrl, + max, + }: { + log: winston.Logger; + apiUrl: string; + max: number; + }) { + this.log = log.child({ class: this.constructor.name }); + this.apiUrl = apiUrl.replace(/\/$/, ''); + this.max = max; + } + + /** Fire-and-forget pin of a named CID. Never throws; never blocks. */ + pin(cidString: string): void { + let cid: string; + try { + cid = cidToV1Base32(cidString); + } catch { + return; // not a valid CID — nothing to pin + } + if (this.pinned.has(cid) || this.inFlight.has(cid)) return; + this.inFlight.add(cid); + void this.doPin(cid).finally(() => this.inFlight.delete(cid)); + } + + private async doPin(cid: string): Promise { + try { + await this.rpc('pin/add', cid); + this.pinned.add(cid); + this.log.debug('Pinned named IPFS CID', { cid }); + // Bound local storage: unpin oldest beyond the cap. + while (this.pinned.size > this.max) { + const oldest = this.pinned.values().next().value; + if (oldest === undefined) break; + this.pinned.delete(oldest); + this.rpc('pin/rm', oldest).catch(() => {}); + } + } catch (error: any) { + this.log.warn('Failed to pin named IPFS CID', { + cid, + message: error?.message, + }); + } + } + + private async rpc(path: string, cid: string): Promise { + // Kubo RPC is POST-only; args go in the query string. + await axios.post( + `${this.apiUrl}/api/v0/${path}?arg=${encodeURIComponent(cid)}`, + undefined, + { timeout: 30_000 }, + ); + } +} diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 8515936f0..5f5b7496b 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -6,7 +6,7 @@ */ import fs from 'node:fs'; import crypto from 'node:crypto'; -import { Readable } from 'node:stream'; +import { Readable, Transform } from 'node:stream'; import winston from 'winston'; import { Span } from '@opentelemetry/api'; @@ -86,12 +86,14 @@ export class IpfsService { signal, parentSpan, range, + format, }: { cidString: string; path?: string; signal?: AbortSignal; parentSpan?: Span; range?: string; + format?: 'raw' | 'car'; }): Promise { const span = startChildSpan( 'IpfsService.getContent', @@ -164,7 +166,9 @@ export class IpfsService { // partial body — they're forwarded straight to Kubo (which supports Range) // below. const cached = - range === undefined ? await this.cache.get(normalizedCid, path) : null; + range === undefined && format === undefined + ? await this.cache.get(normalizedCid, path) + : null; if (cached) { // Content-hash moderation. The cache stores the base64url SHA-256 of the // served bytes (the same format as Arweave's data hash), so once content @@ -215,6 +219,7 @@ export class IpfsService { signal, parentSpan: span, range, + format, }) .catch((err) => { if (err instanceof IpfsNotFoundError) { @@ -248,7 +253,7 @@ export class IpfsService { // Stream directly to the client while writing to a temp file on disk for // caching. No memory buffering — handles files of any size. Partial (206) // responses are NOT cached — only full objects. - if (range === undefined && result.statusCode === 200) { + if (range === undefined && format === undefined && result.statusCode === 200) { this.streamToCache( normalizedCid, path, @@ -265,7 +270,13 @@ export class IpfsService { }); return { - stream: result.stream, + // Trustless format responses (CAR especially) bypass the cache and its + // mid-stream size check, and their size is often unknown up front — guard + // the stream so a large DAG can't stream unbounded. + stream: + format !== undefined + ? this.guardSize(result.stream) + : result.stream, size: result.size, contentType: result.contentType, cached: false, @@ -281,6 +292,34 @@ export class IpfsService { } } + /** + * Pipe a stream through a size-enforcing transform: abort once the response + * exceeds maxResponseSizeBytes. Used for format (raw/car) responses, which + * bypass streamToCache's mid-stream size check. + */ + private guardSize(stream: Readable): Readable { + const limit = this.maxResponseSizeBytes; + if (limit <= 0) return stream; + let seen = 0; + const guard = new Transform({ + transform(chunk: Buffer, _enc, cb) { + seen += chunk.length; + if (seen > limit) { + cb( + new IpfsSizeLimitError( + `IPFS response exceeds limit during streaming: ${seen} > ${limit}`, + ), + ); + return; + } + cb(null, chunk); + }, + }); + stream.on('error', (e) => guard.destroy(e)); + guard.on('error', () => stream.destroy()); + return stream.pipe(guard); + } + /** Remember a CID whose bytes are hash-blocked (bounded, FIFO-evicted). */ private rememberHashBlocked(cidString: string): void { if (this.knownHashBlockedCids.has(cidString)) return; diff --git a/src/ipfs/kubo-data-source.ts b/src/ipfs/kubo-data-source.ts index cfc280741..6a3c11b7f 100644 --- a/src/ipfs/kubo-data-source.ts +++ b/src/ipfs/kubo-data-source.ts @@ -56,12 +56,16 @@ export class KuboDataSource { signal, parentSpan, range, + format, }: { cidString: string; path?: string; signal?: AbortSignal; parentSpan?: Span; range?: string; + // Trustless response format passed through to Kubo: a single verifiable + // block (`raw`) or a verifiable DAG archive (`car`). Absent = UnixFS proxy. + format?: 'raw' | 'car'; }): Promise { signal?.throwIfAborted(); @@ -93,7 +97,9 @@ export class KuboDataSource { : undefined; const ipfsPath = encodedPath !== undefined ? `${cidString}/${encodedPath}` : cidString; - const url = `${this.kuboUrl}/ipfs/${ipfsPath}`; + const url = `${this.kuboUrl}/ipfs/${ipfsPath}${ + format !== undefined ? `?format=${format}` : '' + }`; const span = startChildSpan( 'KuboDataSource.getContent', @@ -137,6 +143,11 @@ export class KuboDataSource { // returns 206 + Content-Range). Enables media seeking and the // observer's ranged sampling of large content. ...(range !== undefined ? { Range: range } : {}), + // Trustless retrieval: ask Kubo for a raw block or CAR by IPLD media + // type (belt-and-suspenders with the ?format= query above). + ...(format !== undefined + ? { Accept: `application/vnd.ipld.${format}` } + : {}), }, maxRedirects: 5, // Accept non-2xx so we can handle 404/408/504 ourselves diff --git a/src/routes/arns.ts b/src/routes/arns.ts index 80f4ec947..b3aa467a3 100644 --- a/src/routes/arns.ts +++ b/src/routes/arns.ts @@ -29,6 +29,7 @@ const ipfsHandler = ipfsService: system.ipfsService, rateLimiter: system.ipfsRateLimiter, paymentProcessor: system.paymentProcessor, + pinner: system.ipfsPinner, }) : undefined; diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index 8cc6d0bfa..21a3b0e98 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -16,6 +16,7 @@ import { headerNames } from '../constants.js'; import { cidToV1Base32, isValidCid } from '../lib/ipfs-cid.js'; import { getRequestSandbox } from '../middleware/sandbox.js'; import { IpfsService } from '../ipfs/ipfs-service.js'; +import { IpfsPinner } from '../ipfs/ipfs-pinner.js'; import { IpfsBlockedError, IpfsNotFoundError, @@ -33,16 +34,36 @@ import { PaymentProcessor } from '../payments/types.js'; import { extractAllClientIPs } from '../lib/ip-utils.js'; import { formatContentDigest } from '../lib/digest.js'; +// Trustless response format (?format=raw|car or an IPLD Accept type). When set, +// the client receives verifiable bytes (a single block or a CAR) and checks them +// against the CID — the gateway neither reassembles nor attests content. +function parseIpfsFormat(req: Request): 'raw' | 'car' | undefined { + const q = req.query.format; + const fromQuery = typeof q === 'string' ? q.toLowerCase() : undefined; + const accept = ( + typeof req.headers.accept === 'string' ? req.headers.accept : '' + ).toLowerCase(); + if (fromQuery === 'raw' || accept.includes('application/vnd.ipld.raw')) { + return 'raw'; + } + if (fromQuery === 'car' || accept.includes('application/vnd.ipld.car')) { + return 'car'; + } + return undefined; +} + export function createIpfsRouter({ log, ipfsService, rateLimiter, paymentProcessor, + pinner, }: { log: winston.Logger; ipfsService: IpfsService; rateLimiter?: RateLimiter; paymentProcessor?: PaymentProcessor; + pinner?: IpfsPinner; }): Router { const router = Router(); const handler = createIpfsPathHandler({ @@ -50,6 +71,7 @@ export function createIpfsRouter({ ipfsService, rateLimiter, paymentProcessor, + pinner, }); router.get('/ipfs/:cid', handler); @@ -67,11 +89,13 @@ export function createIpfsHandler({ ipfsService, rateLimiter, paymentProcessor, + pinner, }: { log: winston.Logger; ipfsService: IpfsService; rateLimiter?: RateLimiter; paymentProcessor?: PaymentProcessor; + pinner?: IpfsPinner; }): Handler { return asyncHandler(async (req: Request, res: Response) => { const cidString = (req as any).ipfsCid as string; @@ -85,6 +109,7 @@ export function createIpfsHandler({ ipfsService, rateLimiter, paymentProcessor, + pinner, routeType: 'subdomain', }); }); @@ -95,11 +120,13 @@ function createIpfsPathHandler({ ipfsService, rateLimiter, paymentProcessor, + pinner, }: { log: winston.Logger; ipfsService: IpfsService; rateLimiter?: RateLimiter; paymentProcessor?: PaymentProcessor; + pinner?: IpfsPinner; }): Handler { return asyncHandler(async (req: Request, res: Response) => { const cidString = req.params.cid; @@ -153,6 +180,7 @@ function createIpfsPathHandler({ ipfsService, rateLimiter, paymentProcessor, + pinner, routeType: 'path', }); }); @@ -167,6 +195,7 @@ async function handleIpfsRequest({ ipfsService, rateLimiter, paymentProcessor, + pinner, routeType, }: { req: Request; @@ -177,6 +206,7 @@ async function handleIpfsRequest({ ipfsService: IpfsService; rateLimiter?: RateLimiter; paymentProcessor?: PaymentProcessor; + pinner?: IpfsPinner; routeType: 'path' | 'subdomain'; }): Promise { const startTime = Date.now(); @@ -187,23 +217,28 @@ async function handleIpfsRequest({ const rawRange = req.headers.range; const rangeForKubo = typeof rawRange === 'string' && !rawRange.includes(',') ? rawRange : undefined; + // Trustless format takes precedence over Range: verifiable block/CAR retrieval. + const format = parseIpfsFormat(req); const ipfsPath = path !== undefined ? `${cidString}/${path}` : cidString; - parentLog.debug('Handling IPFS request', { cidString, path, routeType }); + parentLog.debug('Handling IPFS request', { cidString, path, routeType, format }); try { const result = await ipfsService.getContent({ cidString, path, signal: req.signal, - range: rangeForKubo, + range: format !== undefined ? undefined : rangeForKubo, + format, }); // Check payment and rate limits (x402 + rate limiting in one call). // When Content-Length is unknown (chunked), use a conservative estimate // that gets corrected in the token adjustment after streaming. const contentSize = - result.size > 0 ? result.size : config.IPFS_MAX_RESPONSE_SIZE_BYTES; + result.size > 0 + ? result.size + : config.IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES; const limitCheck = await checkPaymentAndRateLimits({ req, res, @@ -228,44 +263,60 @@ async function handleIpfsRequest({ return; } - // Set response headers + // Common headers (both the trustless and the UnixFS-proxy paths). res.setHeader('Content-Type', result.contentType); if (result.size > 0) { res.setHeader('Content-Length', result.size); } - // Advertise range support, and relay a partial (206) response from Kubo. - res.setHeader('Accept-Ranges', 'bytes'); - if (result.statusCode === 206) { - res.status(206); - if (result.contentRange !== undefined) { - res.setHeader('Content-Range', result.contentRange); - } - } - // A direct /ipfs/{CID} or {CID}.host request is content-addressed and thus - // immutable. But when the request arrived via an ArNS name, the name->CID - // binding is MUTABLE and the ArNS middleware already set a TTL-bounded - // Cache-Control — don't override it with `immutable`, or a record update - // would be pinned in browsers/edge caches for ~a year (cf. PE-9072). - if ((req as Request & { arns?: unknown }).arns === undefined) { - res.setHeader('Cache-Control', 'public, max-age=29030400, immutable'); - } res.setHeader('ETag', `"${cidToV1Base32(cidString)}"`); res.setHeader('X-Ipfs-Path', `/ipfs/${ipfsPath}`); res.setHeader(headerNames.arIoSource, 'ipfs'); - // Body binding (RFC 9530 Content-Digest). When a SHA-256 of the served - // bytes is known (computed at cache-write time, returned on cache hits), - // emit it — it's in CO_SIGNABLE_HEADERS, so HTTPSIG binds the body to the - // signature. Cache hits carry it for free; misses stream without it (the - // signed ETag=CID still attests content identity). - if (result.digest !== undefined) { - res.setHeader('Content-Digest', formatContentDigest(result.digest)); + if (format !== undefined) { + // Trustless retrieval: the body IS the CID's content-addressed bytes (a raw + // block or a CAR), which the CLIENT verifies against the CID. The gateway + // is not a trust root here — mark it so nothing downstream reads the + // response as a gateway-attested content proof. Content-addressed, so the + // bytes are immutable regardless of any ArNS binding. + res.setHeader('Content-Disposition', 'attachment'); + res.setHeader('X-Ar-Io-Trustless', 'true'); + res.setHeader('Cache-Control', 'public, max-age=29030400, immutable'); + } else { + // UnixFS proxy path: Kubo reassembles and the gateway serves (and may sign) + // the bytes. This is TRUSTED-PROXY, not client-verifiable — the signed + // envelope attests "a registered gateway served this", not a content proof. + // Clients that want to verify should request ?format=raw|car. + res.setHeader('X-Ar-Io-Trustless', 'false'); + // Advertise range support, and relay a partial (206) response from Kubo. + res.setHeader('Accept-Ranges', 'bytes'); + if (result.statusCode === 206) { + res.status(206); + if (result.contentRange !== undefined) { + res.setHeader('Content-Range', result.contentRange); + } + } + // Direct CID content is content-addressed and immutable; via an ArNS name + // the binding is MUTABLE and the ArNS middleware already set a TTL-bounded + // Cache-Control — don't override it (cf. PE-9072). + if ((req as Request & { arns?: unknown }).arns === undefined) { + res.setHeader('Cache-Control', 'public, max-age=29030400, immutable'); + } + // RFC 9530 Content-Digest when the SHA-256 is known (cache hits); signed + // via CO_SIGNABLE_HEADERS. + if (result.digest !== undefined) { + res.setHeader('Content-Digest', formatContentDigest(result.digest)); + } + res.setHeader('X-Cache', result.cached ? 'HIT' : 'MISS'); } - if (result.cached) { - res.setHeader('X-Cache', 'HIT'); - } else { - res.setHeader('X-Cache', 'MISS'); + // Pin named (ArNS-resolved) content so it stays retrievable in read-only IPFS + // mode — best-effort, fire-and-forget, bounded. Only "named" content (an ArNS + // resolution), not arbitrary /ipfs/{CID} traffic. + if ( + pinner !== undefined && + (req as Request & { arns?: unknown }).arns !== undefined + ) { + pinner.pin(cidString); } // Track metrics diff --git a/src/system.ts b/src/system.ts index 3b8864c15..4791a4baa 100644 --- a/src/system.ts +++ b/src/system.ts @@ -1880,11 +1880,13 @@ if (dataVerificationWorker !== undefined) { import { KuboDataSource } from './ipfs/kubo-data-source.js'; import { IpfsFsCache } from './ipfs/ipfs-cache.js'; import { IpfsService } from './ipfs/ipfs-service.js'; +import { IpfsPinner } from './ipfs/ipfs-pinner.js'; import { createIpfsRateLimiter } from './ipfs/ipfs-rate-limiter.js'; import { RateLimiter } from './limiter/types.js'; export let ipfsService: IpfsService | undefined; export let ipfsRateLimiter: RateLimiter | undefined; +export let ipfsPinner: IpfsPinner | undefined; if (config.IPFS_ENABLED) { log.info('IPFS subsystem enabled, initializing...'); @@ -1917,6 +1919,14 @@ if (config.IPFS_ENABLED) { ipfsRateLimiter = createIpfsRateLimiter(); + if (config.IPFS_PIN_ARNS_CONTENT) { + ipfsPinner = new IpfsPinner({ + log, + apiUrl: config.IPFS_KUBO_API_URL, + max: config.IPFS_PIN_MAX, + }); + } + log.info('IPFS subsystem initialized', { kuboUrl: config.IPFS_KUBO_URL, cachePath: config.IPFS_CACHE_PATH, From 4457bbee7599c74ca0e259ea62c53abac73072ee Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 12:56:56 +0000 Subject: [PATCH 29/47] test(ipfs): cover pinner and trustless format passthrough - IpfsPinner: pin via Kubo pin/add RPC, dedup repeats, FIFO-evict past max, ignore invalid CIDs (added an explicit isValidCid guard). - KuboDataSource: ?format=raw is forwarded as a query param + IPLD Accept type, and the relayed content-type/statusCode are surfaced. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/ipfs/ipfs-pinner.test.ts | 81 +++++++++++++++++++++++++++++++ src/ipfs/ipfs-pinner.ts | 5 +- src/ipfs/kubo-data-source.test.ts | 37 ++++++++++++++ 3 files changed, 121 insertions(+), 2 deletions(-) create mode 100644 src/ipfs/ipfs-pinner.test.ts diff --git a/src/ipfs/ipfs-pinner.test.ts b/src/ipfs/ipfs-pinner.test.ts new file mode 100644 index 000000000..9464a2d1b --- /dev/null +++ b/src/ipfs/ipfs-pinner.test.ts @@ -0,0 +1,81 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { describe, it, afterEach } from 'node:test'; +import { strict as assert } from 'node:assert'; +import axios from 'axios'; + +import { IpfsPinner } from './ipfs-pinner.js'; +import { createTestLogger } from '../../test/test-logger.js'; + +const log = createTestLogger({ suite: 'IpfsPinner' }); +const CID_A = 'bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi'; +const CID_B = 'bafkreiem4twkqzsq2aj4shbycd4yvoj2cx72vezicletlhi7dijjciqpui'; + +const flush = () => new Promise((r) => setTimeout(r, 30)); + +describe('IpfsPinner', () => { + let interceptorId: number; + afterEach(() => axios.interceptors.request.eject(interceptorId)); + + // Capture RPC calls and stub a 200 so no real network happens. + const capture = (): string[] => { + const calls: string[] = []; + interceptorId = axios.interceptors.request.use((config) => { + calls.push(`${config.method?.toUpperCase()} ${config.url}`); + config.adapter = () => + Promise.resolve({ + status: 200, + statusText: 'OK', + headers: {}, + config, + data: null, + }); + return config; + }); + return calls; + }; + + it('pins a CID via the Kubo pin/add RPC', async () => { + const calls = capture(); + const pinner = new IpfsPinner({ log, apiUrl: 'http://kubo:5001', max: 10 }); + pinner.pin(CID_A); + await flush(); + assert.equal(calls.length, 1); + assert.match(calls[0], /^POST http:\/\/kubo:5001\/api\/v0\/pin\/add\?arg=/); + }); + + it('deduplicates repeated pins of the same CID', async () => { + const calls = capture(); + const pinner = new IpfsPinner({ log, apiUrl: 'http://kubo:5001', max: 10 }); + pinner.pin(CID_A); + await flush(); + pinner.pin(CID_A); + await flush(); + assert.equal(calls.filter((c) => c.includes('pin/add')).length, 1); + }); + + it('FIFO-evicts (unpins) the oldest past max', async () => { + const calls = capture(); + const pinner = new IpfsPinner({ log, apiUrl: 'http://kubo:5001', max: 1 }); + pinner.pin(CID_A); + await flush(); + pinner.pin(CID_B); + await flush(); + assert.ok( + calls.some((c) => c.includes('pin/rm')), + 'expected an unpin (pin/rm) once max exceeded', + ); + }); + + it('ignores invalid CIDs', async () => { + const calls = capture(); + const pinner = new IpfsPinner({ log, apiUrl: 'http://kubo:5001', max: 10 }); + pinner.pin('not-a-cid'); + await flush(); + assert.equal(calls.length, 0); + }); +}); diff --git a/src/ipfs/ipfs-pinner.ts b/src/ipfs/ipfs-pinner.ts index 1b5d175fd..eece15205 100644 --- a/src/ipfs/ipfs-pinner.ts +++ b/src/ipfs/ipfs-pinner.ts @@ -7,7 +7,7 @@ import { default as axios } from 'axios'; import winston from 'winston'; -import { cidToV1Base32 } from '../lib/ipfs-cid.js'; +import { cidToV1Base32, isValidCid } from '../lib/ipfs-cid.js'; /** * Best-effort pinner for "named" IPFS content — the CIDs that ArNS names resolve @@ -46,11 +46,12 @@ export class IpfsPinner { /** Fire-and-forget pin of a named CID. Never throws; never blocks. */ pin(cidString: string): void { + if (!isValidCid(cidString)) return; let cid: string; try { cid = cidToV1Base32(cidString); } catch { - return; // not a valid CID — nothing to pin + return; // defensive — should not happen after isValidCid } if (this.pinned.has(cid) || this.inFlight.has(cid)) return; this.inFlight.add(cid); diff --git a/src/ipfs/kubo-data-source.test.ts b/src/ipfs/kubo-data-source.test.ts index 783d0700e..6da40ee25 100644 --- a/src/ipfs/kubo-data-source.test.ts +++ b/src/ipfs/kubo-data-source.test.ts @@ -172,6 +172,43 @@ describe('KuboDataSource', () => { }); }); + describe('trustless format', () => { + const CID = 'bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi'; + let interceptorId: number; + afterEach(() => axios.interceptors.request.eject(interceptorId)); + + it('forwards ?format=raw and the IPLD Accept type to Kubo', async () => { + let captured: any; + interceptorId = axios.interceptors.request.use((config) => { + captured = config; + config.adapter = () => + Promise.resolve({ + status: 200, + statusText: 'OK', + headers: { 'content-type': 'application/vnd.ipld.raw' }, + config, + data: Readable.from([Buffer.alloc(10)]), + }); + return config; + }); + + const result = await kuboDataSource.getContent({ + cidString: CID, + format: 'raw', + }); + + assert.match(captured.url, /\/ipfs\/.*\?format=raw$/); + const accept = + typeof captured.headers?.get === 'function' + ? captured.headers.get('Accept') + : captured.headers?.Accept; + assert.equal(accept, 'application/vnd.ipld.raw'); + assert.equal(result.contentType, 'application/vnd.ipld.raw'); + assert.equal(result.statusCode, 200); + result.stream.destroy(); + }); + }); + describe('error types', () => { it('IpfsNotFoundError has correct name', () => { const error = new IpfsNotFoundError('not found'); From 824fbf667f335f85ef231e87cca8bbf5e0834ec5 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 12:59:56 +0000 Subject: [PATCH 30/47] docs(ipfs): read-only mode, trustless retrieval, pinning; new envs - ipfs-integration.md: "Read-only mode: trust posture, trustless retrieval, and pinning" section (two postures + X-Ar-Io-Trustless, ?format=raw|car, named- content pinning + the incentive tie-in); document the 4 new env vars. - envs.md + docker-compose.yaml: IPFS_KUBO_API_URL, IPFS_PIN_ARNS_CONTENT, IPFS_PIN_MAX, IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES (kept in sync). - ar-io-gateway-operator skill: read-only/pinning/trustless ops guidance. - davids-brain-alignment: note which recommendations were implemented for the read-only phase and what stays phase 2. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- .../skills/ar-io-gateway-operator/SKILL.md | 3 ++ docker-compose.yaml | 4 ++ docs/drafts/davids-brain-alignment.md | 25 +++++++++ docs/envs.md | 4 ++ docs/ipfs-integration.md | 51 +++++++++++++++++++ 5 files changed, 87 insertions(+) diff --git a/.claude/skills/ar-io-gateway-operator/SKILL.md b/.claude/skills/ar-io-gateway-operator/SKILL.md index c76055116..5ddfcfacb 100644 --- a/.claude/skills/ar-io-gateway-operator/SKILL.md +++ b/.claude/skills/ar-io-gateway-operator/SKILL.md @@ -124,6 +124,9 @@ Pipeline diagnostic: a single `curl -sf .../ar-io/__gateway_metrics | grep -E 'q - **HEAD, Range/`206`, and per-CID sandbox origin isolation** (`/ipfs/{CID}` → `{CID}.{root_host}`) work as on the Arweave path. `IPFS_KUBO_MAX_CONCURRENT_REQUESTS` caps in-flight Kubo fetches (amplification guard); `IPFS_MAX_RESPONSE_SIZE_BYTES` caps a single response. - **Verify it's live**: `GET /ar-io/info` shows `ipfs.enabled: true`; `docker exec ipfs swarm peers` should list peers. - **Troubleshooting**: a name that resolves on viewblock but `404`s here with `IPFS_ENABLED` off means the CID is being misrouted to the Arweave data path — enable IPFS + Kubo. `/ipfs/{CID}` returning `502`/`504` points at the Kubo sidecar (down, no peers, or unpinned/cold content); check swarm peers first. +- **Read-only, not permanent.** This is a read-only proxy to the *public* IPFS network — content is **not** stored on Arweave here. Durability of a named CID depends on it being pinned somewhere; there is no Arweave permanence in this mode. +- **Pin named content for availability**: `IPFS_PIN_ARNS_CONTENT=true` pins (via the Kubo RPC API, `IPFS_KUBO_API_URL`, default `http://kubo:5001`) the CIDs that ArNS names resolve to, so named content this gateway serves isn't GC'd out from under a name (bounded by `IPFS_PIN_MAX`, FIFO). Inspect with `docker exec ipfs pin ls --type=recursive`. The RPC API is powerful — keep 5001 on the internal docker network only, never host/internet. +- **Trustless retrieval**: `GET /ipfs/{CID}?format=raw` (single verifiable block) and `?format=car` (verifiable DAG) let a client check the bytes against the CID itself — the gateway isn't a trust root. Responses carry `X-Ar-Io-Trustless: true`; the reassembled UnixFS path carries `X-Ar-Io-Trustless: false` (gateway-attested, not client-verified). ### Network identity, observer, and incentives diff --git a/docker-compose.yaml b/docker-compose.yaml index 120172db6..044a4ccc3 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -158,6 +158,10 @@ services: - IPFS_KUBO_REQUEST_TIMEOUT_MS=${IPFS_KUBO_REQUEST_TIMEOUT_MS:-} - IPFS_STREAM_STALL_TIMEOUT_MS=${IPFS_STREAM_STALL_TIMEOUT_MS:-} - IPFS_KUBO_MAX_CONCURRENT_REQUESTS=${IPFS_KUBO_MAX_CONCURRENT_REQUESTS:-} + - IPFS_KUBO_API_URL=${IPFS_KUBO_API_URL:-} + - IPFS_PIN_ARNS_CONTENT=${IPFS_PIN_ARNS_CONTENT:-} + - IPFS_PIN_MAX=${IPFS_PIN_MAX:-} + - IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES=${IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES:-} - IPFS_CACHE_PATH=${IPFS_CACHE_PATH:-} - IPFS_CACHE_MAX_SIZE_BYTES=${IPFS_CACHE_MAX_SIZE_BYTES:-} - IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS=${IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS:-} diff --git a/docs/drafts/davids-brain-alignment.md b/docs/drafts/davids-brain-alignment.md index 2b148ff48..002ae335d 100644 --- a/docs/drafts/davids-brain-alignment.md +++ b/docs/drafts/davids-brain-alignment.md @@ -105,3 +105,28 @@ without a rewrite: **not** HTTPSIG-attest it as verified (client verifies). - Stop implying verification on the UnixFS path — document `X-Ar-Io-Source: ipfs` as *trusted-proxy*, and don't let the signed envelope read as a content proof. + +## Update — read-only phase alignment (implemented) + +The scope was deliberately set to a **read-only IPFS mode** (no uploading to +Arweave — that stays phase 2). David's storage-dependent items (CAR→Arweave, +chain anchor, composite-source, libp2p) are correctly deferred. The +storage-*independent* parts of his posture were adopted now: + +- **Trustless retrieval added:** `?format=raw|car` (and IPLD `Accept` types) are + forwarded to Kubo and relayed as verifiable block/CAR responses the client + checks against the CID (`X-Ar-Io-Trustless: true`, `Content-Disposition: + attachment`, `ETag`=CID). This is his Trustless-Gateway shape, on public-IPFS + content instead of Arweave-stored content. +- **Honest trust posture:** the UnixFS proxy path is marked + `X-Ar-Io-Trustless: false` and documented as gateway-attested, not a content + proof — addressing the §3 "don't pretend to be verified" critique. +- **Availability via pinning:** `IPFS_PIN_ARNS_CONTENT` pins named CIDs so + read-only named content doesn't vanish — and it's the substrate an OIP "reward + serving named IPFS data" incentive builds on (a fleet-pinning durability layer + without Arweave storage). + +Still David's phase 2 (not built): CAR→Arweave permapinning, chain-anchored +proofs, folding IPFS into the `src/data/` composite source, libp2p/Bitswap. And +the OIP §5 win — the **observer verifying CID→bytes** instead of trusting +reference gateways — remains a separate (ar-io-observer) track. diff --git a/docs/envs.md b/docs/envs.md index d8983c7c3..f42cf3335 100644 --- a/docs/envs.md +++ b/docs/envs.md @@ -635,6 +635,10 @@ as a Docker Compose sidecar via the `ipfs` profile). | IPFS_KUBO_REQUEST_TIMEOUT_MS | Number | 30000 | Connection timeout for Kubo requests (ms) | | IPFS_STREAM_STALL_TIMEOUT_MS | Number | 30000 | Stall timeout — max time with no data before aborting stream (ms) | | IPFS_KUBO_MAX_CONCURRENT_REQUESTS | Number | 100 | Max concurrent in-flight Kubo fetches; excess fail fast with 502. 0 disables | +| IPFS_KUBO_API_URL | String | http://kubo:5001 | Kubo RPC API base (pinning only). Keep internal to the sidecar — the RPC API is powerful | +| IPFS_PIN_ARNS_CONTENT | Boolean | false | Pin the CIDs that ArNS names resolve to, so named content stays retrievable (read-only availability) | +| IPFS_PIN_MAX | Number | 10000 | Max distinct CIDs the pinner holds this process; oldest are unpinned (FIFO) beyond this | +| IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES | Number | 262144 | Rate-limit reserve for a response of unknown size (CAR / chunked); actual bytes are still bounded by IPFS_MAX_RESPONSE_SIZE_BYTES | | IPFS_CACHE_PATH | String | data/ipfs-cache | Directory for cached IPFS content | | IPFS_CACHE_MAX_SIZE_BYTES | Number | 10737418240 (10 GB) | Maximum cache size before LRU eviction | | IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS | Number | 3600 | Age in seconds before cached files become eviction candidates | diff --git a/docs/ipfs-integration.md b/docs/ipfs-integration.md index 34d4d3a34..7e2938453 100644 --- a/docs/ipfs-integration.md +++ b/docs/ipfs-integration.md @@ -452,6 +452,10 @@ All environment variables are opt-in. The feature is disabled by default. | `IPFS_KUBO_REQUEST_TIMEOUT_MS` | Number | `30000` | Connection timeout in milliseconds for Kubo requests (time to receive response headers). | | `IPFS_STREAM_STALL_TIMEOUT_MS` | Number | `30000` | Stall timeout in milliseconds for streaming responses from Kubo. Stream is aborted if no data is received for this duration. Actively-streaming transfers are not affected. | | `IPFS_KUBO_MAX_CONCURRENT_REQUESTS` | Number | `100` | Maximum concurrent in-flight fetches to Kubo. Bounds upstream/DHT amplification from cheap-to-issue requests (HEAD, tiny Range) that force a fetch before rate limiting is evaluated; requests over the cap fail fast with `502`. `0` disables the cap. | +| `IPFS_KUBO_API_URL` | String | `http://kubo:5001` | Kubo **RPC API** base (distinct from the read-only gateway on 8080). Used only for pinning. The RPC API is powerful — keep it internal to the sidecar, never exposed to the host/internet. | +| `IPFS_PIN_ARNS_CONTENT` | Boolean | `false` | Pin the CIDs that ArNS names resolve to, so named content this gateway serves stays retrievable in read-only mode instead of being GC'd by Kubo. | +| `IPFS_PIN_MAX` | Number | `10000` | Max distinct CIDs the pinner holds this process; oldest are unpinned (FIFO) beyond this to bound local storage. | +| `IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES` | Number | `262144` | Rate-limit reserve for a response whose size Kubo doesn't declare up front (CAR / chunked). The full `IPFS_MAX_RESPONSE_SIZE_BYTES` would exceed the token buckets and reject every such request; actual bytes are still bounded mid-stream by `IPFS_MAX_RESPONSE_SIZE_BYTES`. | | `IPFS_CACHE_PATH` | String | `data/ipfs-cache` | Directory for the IPFS filesystem cache. Relative paths are resolved from the gateway's working directory. | | `IPFS_CACHE_MAX_SIZE_BYTES` | Number | `10737418240` (10 GB) | Maximum total size of the IPFS cache directory. LRU eviction begins when this limit is exceeded. | | `IPFS_CACHE_CLEANUP_THRESHOLD` | Number | `3600` | Interval in seconds between cache eviction scans. | @@ -534,6 +538,53 @@ hits), so the name->CID binding and the served bytes are both attested. a CID there will not serve. To serve IPFS at the apex, use `APEX_ARNS_NAME` pointing at an ANT whose `@` record targets IPFS. ❌ +## Read-only mode: trust posture, trustless retrieval, and pinning + +This integration is a **read-only IPFS mode**: the gateway serves IPFS content +that lives on the public IPFS network (via the Kubo sidecar). It does **not** +store IPFS content on Arweave — permapinning to Arweave (CAR ingest, content- +addressed indexing, chain-anchored proofs) is a separate, larger phase. In this +mode the gateway is a caching proxy with an optional client-verifiable path; the +durability of named content depends on it being pinned somewhere, not on Arweave +permanence. + +### Two trust postures (and an honest header) + +- **Trusted proxy (UnixFS path).** A plain `/ipfs/{CID}` or a name→CID request is + reassembled by Kubo and served (and, if signing is on, HTTPSIG-signed). The + signature attests *"a registered gateway served these bytes"* — it is **not** a + content proof. The response carries `X-Ar-Io-Trustless: false`. +- **Trustless (verifiable) retrieval.** `GET /ipfs/{CID}?format=raw` returns a + single verifiable block (`application/vnd.ipld.raw`); `?format=car` returns a + verifiable DAG archive (`application/vnd.ipld.car`). Equivalent IPLD `Accept` + types work too. These are forwarded to Kubo and relayed with + `Content-Disposition: attachment`, `ETag` = the CID, and + `X-Ar-Io-Trustless: true`. The **client** hashes the bytes and checks them + against the CID — the gateway is not a trust root. This is the + [IPFS Trustless Gateway](https://specs.ipfs.tech/http-gateways/trustless-gateway/) + shape. Format responses bypass the UnixFS cache; a mid-stream size guard bounds + large CARs, and the rate-limit reserve for unknown-size responses is + `IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES`. + +Clients that need to *verify* content should request `?format=raw|car` and check +the CID; the UnixFS path is a convenience for browsers and is explicitly +gateway-trusted. + +### Named-content pinning (availability) + +Because read-only content lives on the public network and Kubo runs with GC, an +unpinned CID can disappear and an ArNS name pointing at it will `404`. With +`IPFS_PIN_ARNS_CONTENT=true` the gateway pins (via the Kubo RPC API) the CIDs that +ArNS names resolve to, so the *named* content it is responsible for stays +retrievable. Pinning is best-effort and fire-and-forget (never blocks a +response), bounded to `IPFS_PIN_MAX` distinct CIDs (oldest unpinned FIFO), and +scoped to ArNS-resolved content — not arbitrary `/ipfs/{CID}` traffic. + +This is also the substrate an incentive layer would build on: rewarding gateways +for serving named IPFS data gives the fleet a reason to pin it, turning the AR.IO +gateway network into a durability layer for named IPFS content without Arweave +storage. + ## Parity with the Arweave Data Path The IPFS path is held to the same operational bar as the Arweave data path; From 3a99778bd1e78d0bd349971a2793330f87cb38c0 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 13:35:54 +0000 Subject: [PATCH 31/47] style(ipfs): prettier formatting (lint:check) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/ipfs/ipfs-service.ts | 18 ++++++++++++------ src/ipfs/kubo-data-source.test.ts | 3 ++- src/routes/ipfs.ts | 15 ++++++++++----- 3 files changed, 24 insertions(+), 12 deletions(-) diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 5f5b7496b..1394ae0fe 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -145,7 +145,9 @@ export class IpfsService { metrics.ipfsBlockedTotal.inc(); span.setAttribute('ipfs.blocked', true); span.end(); - throw new IpfsBlockedError(`Content hash is blocked: ${normalizedCid}`); + throw new IpfsBlockedError( + `Content hash is blocked: ${normalizedCid}`, + ); } } @@ -183,7 +185,9 @@ export class IpfsService { metrics.ipfsBlockedTotal.inc(); span.setAttribute('ipfs.blocked', true); span.end(); - throw new IpfsBlockedError(`Content hash is blocked: ${normalizedCid}`); + throw new IpfsBlockedError( + `Content hash is blocked: ${normalizedCid}`, + ); } // Content is available — clear any negative-cache entry and record a // success so a transiently-unavailable CID that later pins isn't kept in @@ -253,7 +257,11 @@ export class IpfsService { // Stream directly to the client while writing to a temp file on disk for // caching. No memory buffering — handles files of any size. Partial (206) // responses are NOT cached — only full objects. - if (range === undefined && format === undefined && result.statusCode === 200) { + if ( + range === undefined && + format === undefined && + result.statusCode === 200 + ) { this.streamToCache( normalizedCid, path, @@ -274,9 +282,7 @@ export class IpfsService { // mid-stream size check, and their size is often unknown up front — guard // the stream so a large DAG can't stream unbounded. stream: - format !== undefined - ? this.guardSize(result.stream) - : result.stream, + format !== undefined ? this.guardSize(result.stream) : result.stream, size: result.size, contentType: result.contentType, cached: false, diff --git a/src/ipfs/kubo-data-source.test.ts b/src/ipfs/kubo-data-source.test.ts index 6da40ee25..87cef1e06 100644 --- a/src/ipfs/kubo-data-source.test.ts +++ b/src/ipfs/kubo-data-source.test.ts @@ -141,7 +141,8 @@ describe('KuboDataSource', () => { }); await assert.rejects( - () => kuboDataSource.getContent({ cidString: CID, range: 'bytes=9e9-' }), + () => + kuboDataSource.getContent({ cidString: CID, range: 'bytes=9e9-' }), (error: any) => { assert.equal(error.name, 'IpfsRangeNotSatisfiableError'); return true; diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index 21a3b0e98..aadac265c 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -216,12 +216,19 @@ async function handleIpfsRequest({ // and relaying it alongside our own Content-Length would be inconsistent. const rawRange = req.headers.range; const rangeForKubo = - typeof rawRange === 'string' && !rawRange.includes(',') ? rawRange : undefined; + typeof rawRange === 'string' && !rawRange.includes(',') + ? rawRange + : undefined; // Trustless format takes precedence over Range: verifiable block/CAR retrieval. const format = parseIpfsFormat(req); const ipfsPath = path !== undefined ? `${cidString}/${path}` : cidString; - parentLog.debug('Handling IPFS request', { cidString, path, routeType, format }); + parentLog.debug('Handling IPFS request', { + cidString, + path, + routeType, + format, + }); try { const result = await ipfsService.getContent({ @@ -236,9 +243,7 @@ async function handleIpfsRequest({ // When Content-Length is unknown (chunked), use a conservative estimate // that gets corrected in the token adjustment after streaming. const contentSize = - result.size > 0 - ? result.size - : config.IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES; + result.size > 0 ? result.size : config.IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES; const limitCheck = await checkPaymentAndRateLimits({ req, res, From abc10871eff1591f49f99b31309e827e302c1f69 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 13:40:24 +0000 Subject: [PATCH 32/47] docs: note read-only IPFS mode, trust postures, and pinning in CLAUDE.md Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- CLAUDE.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f3cff8ea8..b7579bd49 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -80,8 +80,14 @@ yarn service:start / stop / restart / status / logs first-class field on `NameResolution` set by the on-demand resolver and propagated across a trusted-gateway hop via the signed `X-ArNS-Protocol` header (`src/resolution/`). The IPFS path is held to Arweave-path parity - (moderation, caching, HEAD/Range, sandbox origin isolation, HTTPSIG). See - `docs/ipfs-integration.md` ("Parity with the Arweave Data Path"). + (moderation, caching, HEAD/Range, sandbox origin isolation, HTTPSIG). This is a + **read-only** proxy to the public IPFS network — it does not store content on + Arweave. Two trust postures: the UnixFS path is a trusted proxy + (`X-Ar-Io-Trustless: false`), while `?format=raw|car` relays verifiable + block/CAR bytes the client checks against the CID (`true`). Optional + named-content pinning (`IPFS_PIN_ARNS_CONTENT`, `src/ipfs/ipfs-pinner.ts`) pins + the CIDs ArNS names resolve to. Uploading/permapinning IPFS content to Arweave + is a deliberate phase 2. See `docs/ipfs-integration.md`. - Responses include trust headers indicating verification status. - HTTPSIG signs response headers (RFC 9421); `Content-Digest` is in `CO_SIGNABLE_HEADERS` so when present it binds the body to the signature. From e03da75afb4e6837c5cc28723f09cfe50acd4a4d Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 20:49:14 +0000 Subject: [PATCH 33/47] fix(ipfs): address CodeRabbit review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Negative cache keyed by CID+path (was CID-only): one missing sub-path no longer blackholes the whole CID/site for the negative-cache TTL. - ETag varies by sub-path and format (raw/car), so a shared cache can't serve the wrong body under `immutable`. (Range still shares the full entity's ETag.) - Pinner keeps a CID tracked until Kubo confirms pin/rm, and stops on failure instead of silently drifting the real pin count above max. - KuboDataSource destroys the response stream when axios rejects on 5xx (was a socket/fd leak), and forwards a wall-clock cap (IPFS_KUBO_MAX_REQUEST_MS) to attachStallTimeout so a backpressure-pause-then-upstream-stall can't hold a concurrency slot forever. - classifyResolvedTarget rejects explicit unsupported targetProtocol values (and the trusted-gateway resolver rejects unknown X-ArNS-Protocol) instead of mis-serving them as Arweave; test updated. - streamToCache honors write backpressure (pause/resume on drain) and documents the load-bearing upstream pause() dependency. - Docs: align IPFS rate-limit defaults in ipfs-integration.md with config.ts; clarify that IPFS_ENABLED needs the kubo profile (else 502/504); note the IPFS_CACHE_PATH/volume coupling; broaden the block-data `id` OpenAPI schema; document IPFS_KUBO_MAX_REQUEST_MS (envs + compose). - test-ipfs.sh: follow redirects in body checks; treat an empty header expectation as a presence check (was always passing). Not changed: IpfsFsCache disk usage isn't reconciled with its in-memory index across restarts (pre-existing #682 design) — a disk sweeper / index rebuild is a separate change, tracked as a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- .../skills/ar-io-gateway-operator/SKILL.md | 2 +- docker-compose.yaml | 5 +++ docs/envs.md | 1 + docs/ipfs-integration.md | 8 ++--- docs/openapi.yaml | 6 +++- src/config.ts | 10 ++++++ src/ipfs/ipfs-pinner.ts | 18 +++++++++-- src/ipfs/ipfs-service.ts | 31 ++++++++++++++++--- src/ipfs/kubo-data-source.ts | 13 +++++++- src/resolution/resolved-target.test.ts | 7 +++-- src/resolution/resolved-target.ts | 12 ++++++- .../trusted-gateway-arns-resolver.ts | 10 +++++- src/routes/ipfs.ts | 11 ++++++- src/system.ts | 1 + test-ipfs.sh | 9 ++++-- yarn.lock | 31 ++----------------- 16 files changed, 125 insertions(+), 50 deletions(-) diff --git a/.claude/skills/ar-io-gateway-operator/SKILL.md b/.claude/skills/ar-io-gateway-operator/SKILL.md index 5ddfcfacb..5668a8046 100644 --- a/.claude/skills/ar-io-gateway-operator/SKILL.md +++ b/.claude/skills/ar-io-gateway-operator/SKILL.md @@ -115,7 +115,7 @@ Pipeline diagnostic: a single `curl -sf .../ar-io/__gateway_metrics | grep -E 'q ### IPFS serving (opt-in Kubo sidecar) -`IPFS_ENABLED=true` adds a Kubo sidecar (`docker compose --profile ipfs up -d`, or a compose override that adds the `kubo` service) and turns on IPFS serving. The gateway proxies, caches, moderates, and signs IPFS content alongside Arweave data, held to the same operational bar (`docs/ipfs-integration.md` → "Parity with the Arweave Data Path"). +Two things are required, not one: (1) `IPFS_ENABLED=true` turns on IPFS handling in `core`, and (2) the `kubo` sidecar must actually run — it's behind the Compose `ipfs` profile, so `docker compose --profile ipfs up -d` (or an override that adds the `kubo` service). **Enabling `IPFS_ENABLED` alone does not start Kubo** — the gateway will then return `502`/`504` for IPFS requests because it has no upstream to fetch from. With both in place, the gateway proxies, caches, moderates, and signs IPFS content alongside Arweave data, held to the same operational bar (`docs/ipfs-integration.md` → "Parity with the Arweave Data Path"). - **Two entry points**: direct `/ipfs/{CID}` (and `{CID}.{root_host}` subdomains), and ArNS names whose ANT record sets `targetProtocol: ipfs` — the resolved id is a CID, surfaced as `X-ArNS-Protocol: ipfs` and served via Kubo. `protocol` propagates across a trusted-gateway hop, so a name→CID binding survives even with `gateway` ahead of `on-demand` in `ARNS_RESOLVER_PRIORITY_ORDER`. - **The node fetches from the local Kubo gateway** (`IPFS_KUBO_URL`, default `http://kubo:8080`) — it does not join the DHT itself. Availability depends on Kubo finding/holding the blocks; Kubo runs `--enable-gc`, so **unpinned content can disappear** (no on-chain permanence like Arweave). This is the key operational difference from Arweave data. diff --git a/docker-compose.yaml b/docker-compose.yaml index 044a4ccc3..d0a61b650 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -54,6 +54,10 @@ services: - ${HEADERS_DATA_PATH:-./data/headers}:/app/data/headers - ${SQLITE_DATA_PATH:-./data/sqlite}:/app/data/sqlite - ${DUCKDB_DATA_PATH:-./data/duckdb}:/app/data/duckdb + # The container target must match core's IPFS_CACHE_PATH (default + # data/ipfs-cache -> /app/data/ipfs-cache). If you change IPFS_CACHE_PATH, + # point it inside this mount, or the cache is written to an unmounted path + # and is lost when the container is recreated. - ${IPFS_CACHE_DATA_PATH:-./data/ipfs-cache}:/app/data/ipfs-cache - ${TEMP_DATA_PATH:-./data/tmp}:/app/data/tmp - ${LMDB_DATA_PATH:-./data/lmdb}:/app/data/lmdb @@ -158,6 +162,7 @@ services: - IPFS_KUBO_REQUEST_TIMEOUT_MS=${IPFS_KUBO_REQUEST_TIMEOUT_MS:-} - IPFS_STREAM_STALL_TIMEOUT_MS=${IPFS_STREAM_STALL_TIMEOUT_MS:-} - IPFS_KUBO_MAX_CONCURRENT_REQUESTS=${IPFS_KUBO_MAX_CONCURRENT_REQUESTS:-} + - IPFS_KUBO_MAX_REQUEST_MS=${IPFS_KUBO_MAX_REQUEST_MS:-} - IPFS_KUBO_API_URL=${IPFS_KUBO_API_URL:-} - IPFS_PIN_ARNS_CONTENT=${IPFS_PIN_ARNS_CONTENT:-} - IPFS_PIN_MAX=${IPFS_PIN_MAX:-} diff --git a/docs/envs.md b/docs/envs.md index f42cf3335..eb7727ba0 100644 --- a/docs/envs.md +++ b/docs/envs.md @@ -635,6 +635,7 @@ as a Docker Compose sidecar via the `ipfs` profile). | IPFS_KUBO_REQUEST_TIMEOUT_MS | Number | 30000 | Connection timeout for Kubo requests (ms) | | IPFS_STREAM_STALL_TIMEOUT_MS | Number | 30000 | Stall timeout — max time with no data before aborting stream (ms) | | IPFS_KUBO_MAX_CONCURRENT_REQUESTS | Number | 100 | Max concurrent in-flight Kubo fetches; excess fail fast with 502. 0 disables | +| IPFS_KUBO_MAX_REQUEST_MS | Number | 1200000 | Hard wall-clock cap on a single Kubo fetch (backpressure-pause-then-stall safety net); 0 disables | | IPFS_KUBO_API_URL | String | http://kubo:5001 | Kubo RPC API base (pinning only). Keep internal to the sidecar — the RPC API is powerful | | IPFS_PIN_ARNS_CONTENT | Boolean | false | Pin the CIDs that ArNS names resolve to, so named content stays retrievable (read-only availability) | | IPFS_PIN_MAX | Number | 10000 | Max distinct CIDs the pinner holds this process; oldest are unpinned (FIFO) beyond this | diff --git a/docs/ipfs-integration.md b/docs/ipfs-integration.md index 7e2938453..bd699fc51 100644 --- a/docs/ipfs-integration.md +++ b/docs/ipfs-integration.md @@ -460,10 +460,10 @@ All environment variables are opt-in. The feature is disabled by default. | `IPFS_CACHE_MAX_SIZE_BYTES` | Number | `10737418240` (10 GB) | Maximum total size of the IPFS cache directory. LRU eviction begins when this limit is exceeded. | | `IPFS_CACHE_CLEANUP_THRESHOLD` | Number | `3600` | Interval in seconds between cache eviction scans. | | `IPFS_BLOCKLIST_PATH` | String | `data/ipfs-blocklist.txt` | Path to the CID blocklist file. The file is watched for changes and reloaded automatically. | -| `IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET` | Number | `50000` | Maximum tokens (bytes) per IP bucket. | -| `IPFS_RATE_LIMITER_IP_REFILL_PER_SEC` | Number | `5` | Tokens added to each IP bucket per second. | -| `IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET` | Number | `200000` | Maximum tokens (bytes) per resource (CID) bucket. | -| `IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC` | Number | `20` | Tokens added to each resource bucket per second. | +| `IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET` | Number | `100000` | Maximum tokens (bytes) per IP bucket. | +| `IPFS_RATE_LIMITER_IP_REFILL_PER_SEC` | Number | `20` | Tokens added to each IP bucket per second. | +| `IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET` | Number | `1000000` | Maximum tokens (bytes) per resource (CID) bucket. | +| `IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC` | Number | `100` | Tokens added to each resource bucket per second. | | `IPFS_MAX_RESPONSE_SIZE_BYTES` | Number | `1073741824` (1 GB) | Maximum response size for a single IPFS request. Requests exceeding this are rejected. | ## Phase 2: ArNS to IPFS Resolution diff --git a/docs/openapi.yaml b/docs/openapi.yaml index 61ea7f1af..2767d3481 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -3012,7 +3012,11 @@ paths: properties: id: type: string - description: TX ID for a transaction you want to block. + description: >- + Identifier of the content to block: an Arweave TX ID or + data-item ID (43-char base64url), an IPFS CIDv1 (base32), or + a sha-256 content hash. Blocked content is served as HTTP 451 + by the data/IPFS endpoints. notes: type: string description: Any notes or comments related to the block data. Documentation purposes only. diff --git a/src/config.ts b/src/config.ts index cc397161a..96c07887d 100644 --- a/src/config.ts +++ b/src/config.ts @@ -3233,6 +3233,16 @@ export const IPFS_KUBO_MAX_CONCURRENT_REQUESTS = env.positiveIntOrDefault( 100, ); +// Hard wall-clock cap on a single Kubo fetch. The per-chunk stall timer can't +// see the case where a stream is paused for downstream backpressure and then +// stalls upstream (the pause clears the stall timer); this bounds that wedge so +// it can't hold a concurrency slot + socket indefinitely. Generous so it doesn't +// cut legitimately-slow large transfers; 0 disables it. +export const IPFS_KUBO_MAX_REQUEST_MS = env.positiveIntOrDefault( + 'IPFS_KUBO_MAX_REQUEST_MS', + 1_200_000, +); + // Kubo RPC API base (distinct from the read-only gateway on 8080). Used only for // pinning; the powerful admin API should not be exposed beyond the sidecar. export const IPFS_KUBO_API_URL = env.varOrDefault( diff --git a/src/ipfs/ipfs-pinner.ts b/src/ipfs/ipfs-pinner.ts index eece15205..9d78286a8 100644 --- a/src/ipfs/ipfs-pinner.ts +++ b/src/ipfs/ipfs-pinner.ts @@ -63,12 +63,24 @@ export class IpfsPinner { await this.rpc('pin/add', cid); this.pinned.add(cid); this.log.debug('Pinned named IPFS CID', { cid }); - // Bound local storage: unpin oldest beyond the cap. + // Bound local storage: unpin oldest beyond the cap. Only drop it from the + // tracked set once Kubo confirms the unpin — otherwise a failed pin/rm + // would leave the CID pinned but untracked, and the real pin count could + // drift above `max`. Stop on the first failure to avoid spinning against + // an unhealthy Kubo; the entry is retried on the next eviction. while (this.pinned.size > this.max) { const oldest = this.pinned.values().next().value; if (oldest === undefined) break; - this.pinned.delete(oldest); - this.rpc('pin/rm', oldest).catch(() => {}); + try { + await this.rpc('pin/rm', oldest); + this.pinned.delete(oldest); + } catch (error: any) { + this.log.warn('Failed to unpin evicted CID; will retry later', { + cid: oldest, + message: error?.message, + }); + break; + } } } catch (error: any) { this.log.warn('Failed to pin named IPFS CID', { diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 1394ae0fe..67da3c71b 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -110,6 +110,12 @@ export class IpfsService { // Normalize CID to v1 base32 for consistent caching const normalizedCid = cidToV1Base32(cidString); span.setAttribute('ipfs.cid_normalized', normalizedCid); + // Negative-cache identity must match the content cache's (CID + path), not + // just the CID — otherwise one missing sub-path (e.g. a bad link) would + // negatively cache the whole CID and 404 every other path, including the + // root, for the TTL. + const negKey = + path !== undefined ? `${normalizedCid}/${path}` : normalizedCid; // Check blocklist (uses the same admin API as Arweave data moderation) if (await this.blockListValidator.isIdBlocked(normalizedCid)) { @@ -155,7 +161,7 @@ export class IpfsService { // (absent or unpinned) so they don't re-hit Kubo on every request // (latency / DoS amplification) — mirrors the Arweave path's negative data // cache. Only trips after repeated misses (count + duration thresholds). - if (this.negativeCache?.isNegativelyCached(normalizedCid) === true) { + if (this.negativeCache?.isNegativelyCached(negKey) === true) { span.setAttribute('ipfs.negative_cache', 'hit'); span.end(); throw new IpfsNotFoundError( @@ -192,7 +198,7 @@ export class IpfsService { // Content is available — clear any negative-cache entry and record a // success so a transiently-unavailable CID that later pins isn't kept in // a negative-cache blackout, and IPFS health isn't skewed miss-only. - this.negativeCache?.evict(normalizedCid); + this.negativeCache?.evict(negKey); this.negativeCache?.recordSuccess(); this.log.debug('IPFS cache hit', { cid: normalizedCid, path }); metrics.ipfsCacheHitTotal.inc(); @@ -227,7 +233,7 @@ export class IpfsService { }) .catch((err) => { if (err instanceof IpfsNotFoundError) { - this.negativeCache?.recordMiss(normalizedCid); + this.negativeCache?.recordMiss(negKey); } throw err; }); @@ -251,7 +257,7 @@ export class IpfsService { // A successful fetch means the content is available — clear any // negative-cache entry and record health (see the cache-hit path). - this.negativeCache?.evict(normalizedCid); + this.negativeCache?.evict(negKey); this.negativeCache?.recordSuccess(); // Stream directly to the client while writing to a temp file on disk for @@ -342,6 +348,14 @@ export class IpfsService { * Buffers early chunks in memory until the write stream is ready, * then flushes them to disk. */ + // NOTE (load-bearing): this tee relies on `stream` NOT being in flowing mode + // when it's attached — KuboDataSource calls `attachStallTimeout`, which + // `pause()`s the stream, so these 'data' listeners don't start draining it + // before the route reaches `result.stream.pipe(res)` (after its async payment/ + // rate-limit check). If that upstream `pause()` is ever removed, the cache + // writer would consume chunks before the client pipe attaches and clients + // would get truncated bodies. There is e2e coverage for the await-before-pipe + // gap; keep it. private streamToCache( cidString: string, path: string | undefined, @@ -411,7 +425,14 @@ export class IpfsService { } hash.update(chunk); - writeStream.write(chunk); + // Honor disk backpressure: if the write buffer is full, pause the source + // until it drains so a disk slower than the upstream can't grow the write + // stream's buffer without bound (the client pipe alone only throttles to + // the client's rate, not the disk's). + if (!writeStream.write(chunk)) { + stream.pause(); + writeStream.once('drain', () => stream.resume()); + } }); stream.on('end', () => { diff --git a/src/ipfs/kubo-data-source.ts b/src/ipfs/kubo-data-source.ts index 6a3c11b7f..4131ecf49 100644 --- a/src/ipfs/kubo-data-source.ts +++ b/src/ipfs/kubo-data-source.ts @@ -28,6 +28,7 @@ export class KuboDataSource { private requestTimeoutMs: number; private streamStallTimeoutMs: number; private maxConcurrent: number; + private maxRequestMs: number; private inFlight = 0; constructor({ @@ -36,18 +37,21 @@ export class KuboDataSource { requestTimeoutMs, streamStallTimeoutMs, maxConcurrent = 0, + maxRequestMs = 0, }: { log: winston.Logger; kuboUrl: string; requestTimeoutMs: number; streamStallTimeoutMs: number; maxConcurrent?: number; + maxRequestMs?: number; }) { this.log = log.child({ class: this.constructor.name }); this.kuboUrl = kuboUrl.replace(/\/$/, ''); this.requestTimeoutMs = requestTimeoutMs; this.streamStallTimeoutMs = streamStallTimeoutMs; this.maxConcurrent = maxConcurrent; + this.maxRequestMs = maxRequestMs; } async getContent({ @@ -198,7 +202,7 @@ export class KuboDataSource { response.headers['content-type'] ?? 'application/octet-stream'; // Switch from connection timeout to stall timeout - attachStallTimeout(stream, this.streamStallTimeoutMs); + attachStallTimeout(stream, this.streamStallTimeoutMs, this.maxRequestMs); span.setAttributes({ 'ipfs.content_length': contentLength, @@ -235,6 +239,13 @@ export class KuboDataSource { release(); clearTimeout(connectionTimer); signal?.removeEventListener('abort', onClientAbort); + // axios rejects for 5xx (validateStatus accepts <500 or 504). With + // responseType 'stream', error.response.data is an open Readable — destroy + // it so the socket/fd isn't leaked while Kubo returns 500/502/503. + const errStream = error?.response?.data; + if (errStream !== undefined && typeof errStream.destroy === 'function') { + errStream.destroy(); + } if (error.name !== 'AbortError') { span.recordException(error); diff --git a/src/resolution/resolved-target.test.ts b/src/resolution/resolved-target.test.ts index 06bc3264c..fd5923bd6 100644 --- a/src/resolution/resolved-target.test.ts +++ b/src/resolution/resolved-target.test.ts @@ -23,8 +23,11 @@ describe('classifyResolvedTarget', () => { assert.equal(classifyResolvedTarget(ARWEAVE_ID, undefined), 'arweave'); }); - it('treats unknown protocol numbers as arweave (fail-closed)', () => { - assert.equal(classifyResolvedTarget(ARWEAVE_ID, 2), 'arweave'); + it('rejects an unknown protocol number rather than mis-serving as arweave', () => { + assert.throws( + () => classifyResolvedTarget(ARWEAVE_ID, 2), + /Unsupported targetProtocol/, + ); }); it('rejects a non-Arweave id under an Arweave protocol', () => { diff --git a/src/resolution/resolved-target.ts b/src/resolution/resolved-target.ts index a2a411d64..a23cf7901 100644 --- a/src/resolution/resolved-target.ts +++ b/src/resolution/resolved-target.ts @@ -30,7 +30,17 @@ export function classifyResolvedTarget( resolvedId: string, targetProtocol: number | undefined, ): ResolvedProtocol { - const protocol: ResolvedProtocol = targetProtocol === 1 ? 'ipfs' : 'arweave'; + // Only an absent protocol (or an explicit 0) is a legacy Arweave record. + // Reject any other explicit value rather than silently serving a future/ + // invalid protocol through the Arweave data path. + let protocol: ResolvedProtocol; + if (targetProtocol === undefined || targetProtocol === 0) { + protocol = 'arweave'; + } else if (targetProtocol === 1) { + protocol = 'ipfs'; + } else { + throw new Error(`Unsupported targetProtocol: ${targetProtocol}`); + } if (protocol === 'ipfs') { if (!isValidCid(resolvedId)) { diff --git a/src/resolution/trusted-gateway-arns-resolver.ts b/src/resolution/trusted-gateway-arns-resolver.ts index b31868084..f42ac3094 100644 --- a/src/resolution/trusted-gateway-arns-resolver.ts +++ b/src/resolution/trusted-gateway-arns-resolver.ts @@ -92,7 +92,15 @@ export class TrustedGatewayArNSResolver implements NameResolver { // is rejected here rather than mis-served downstream. const protocolHeader = response.headers[headerNames.arnsProtocol.toLowerCase()]; - const targetProtocol = protocolHeader === 'ipfs' ? 1 : 0; + // Default to arweave only when the header is absent (older peers). An + // explicit value other than arweave/ipfs is mapped to an unsupported code + // so classifyResolvedTarget rejects it rather than mis-serving as arweave. + const targetProtocol = + protocolHeader === undefined || protocolHeader === 'arweave' + ? 0 + : protocolHeader === 'ipfs' + ? 1 + : -1; if (typeof resolvedId === 'string') { try { const protocol = classifyResolvedTarget(resolvedId, targetProtocol); diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index aadac265c..edd139cc6 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -273,7 +273,16 @@ async function handleIpfsRequest({ if (result.size > 0) { res.setHeader('Content-Length', result.size); } - res.setHeader('ETag', `"${cidToV1Base32(cidString)}"`); + // ETag must distinguish every representation that can differ under the + // `immutable` Cache-Control: sub-path and response format. (Range is + // deliberately excluded so a 206 shares the full entity's validator, which + // is what If-Range compares against.) Without this a shared cache could + // serve one sub-path/format's body for another. + const etag = + cidToV1Base32(cidString) + + (path !== undefined ? `/${path}` : '') + + (format !== undefined ? `+${format}` : ''); + res.setHeader('ETag', `"${etag}"`); res.setHeader('X-Ipfs-Path', `/ipfs/${ipfsPath}`); res.setHeader(headerNames.arIoSource, 'ipfs'); diff --git a/src/system.ts b/src/system.ts index 4791a4baa..384a4a62d 100644 --- a/src/system.ts +++ b/src/system.ts @@ -1897,6 +1897,7 @@ if (config.IPFS_ENABLED) { requestTimeoutMs: config.IPFS_KUBO_REQUEST_TIMEOUT_MS, streamStallTimeoutMs: config.IPFS_STREAM_STALL_TIMEOUT_MS, maxConcurrent: config.IPFS_KUBO_MAX_CONCURRENT_REQUESTS, + maxRequestMs: config.IPFS_KUBO_MAX_REQUEST_MS, }); const ipfsCache = new IpfsFsCache({ diff --git a/test-ipfs.sh b/test-ipfs.sh index 6f262e563..ff842bbce 100755 --- a/test-ipfs.sh +++ b/test-ipfs.sh @@ -39,7 +39,9 @@ test_body_contains() { local url="$3" local body - body=$(curl -s --max-time 30 "$url" 2>&1) + # -L: a path-style /ipfs/{CID} 302-redirects to its sandbox subdomain when + # ARNS_ROOT_HOSTS is set; without following, we'd match the redirect body. + body=$(curl -sL --max-time 30 "$url" 2>&1) local status=$? if echo "$body" | grep -q "$expected_text"; then @@ -61,7 +63,10 @@ test_header() { local actual actual=$(curl -s -I --max-time 30 "$url" 2>&1 | grep -i "^$header:" | head -1 | sed 's/^[^:]*: //' | tr -d '\r') - if echo "$actual" | grep -qi "$expected_value"; then + # An empty expected_value means "header must be present" — grep -qi "" would + # match anything (even a missing header), so require a non-empty actual value. + if { [ -z "$expected_value" ] && [ -n "$actual" ]; } || + { [ -n "$expected_value" ] && echo "$actual" | grep -qi "$expected_value"; }; then echo -e " ${GREEN}PASS${NC} $name ($actual)" ((pass++)) else diff --git a/yarn.lock b/yarn.lock index 91b424a8b..1ea218672 100644 --- a/yarn.lock +++ b/yarn.lock @@ -14359,16 +14359,7 @@ string-argv@~0.3.1: resolved "https://registry.yarnpkg.com/string-argv/-/string-argv-0.3.2.tgz#2b6d0ef24b656274d957d54e0a4bbf6153dc02b6" integrity sha512-aqD2Q0144Z+/RqG52NeHEkZauTAUWJO8c6yTftGJKO3Tja5tUgIfmIl6kExvhtxSDP7fXB6DvzkfMpCd/F3G+Q== -"string-width-cjs@npm:string-width@^4.2.0": - version "4.2.3" - resolved "https://registry.yarnpkg.com/string-width/-/string-width-4.2.3.tgz#269c7117d27b05ad2e536830a8ec895ef9c6d010" - integrity sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g== - dependencies: - emoji-regex "^8.0.0" - is-fullwidth-code-point "^3.0.0" - strip-ansi "^6.0.1" - -"string-width@^1.0.2 || 2 || 3 || 4", string-width@^4.1.0, string-width@^4.2.0, string-width@^4.2.3: +"string-width-cjs@npm:string-width@^4.2.0", "string-width@^1.0.2 || 2 || 3 || 4", string-width@^4.1.0, string-width@^4.2.0, string-width@^4.2.3: version "4.2.3" resolved "https://registry.yarnpkg.com/string-width/-/string-width-4.2.3.tgz#269c7117d27b05ad2e536830a8ec895ef9c6d010" integrity sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g== @@ -14414,14 +14405,7 @@ stringify-object@^3.2.1: is-obj "^1.0.1" is-regexp "^1.0.0" -"strip-ansi-cjs@npm:strip-ansi@^6.0.1": - version "6.0.1" - resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-6.0.1.tgz#9e26c63d30f53443e9489495b2105d37b67a85d9" - integrity sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A== - dependencies: - ansi-regex "^5.0.1" - -strip-ansi@^6.0.0, strip-ansi@^6.0.1: +"strip-ansi-cjs@npm:strip-ansi@^6.0.1", strip-ansi@^6.0.0, strip-ansi@^6.0.1: version "6.0.1" resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-6.0.1.tgz#9e26c63d30f53443e9489495b2105d37b67a85d9" integrity sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A== @@ -15373,7 +15357,7 @@ word-wrap@^1.2.5: resolved "https://registry.yarnpkg.com/word-wrap/-/word-wrap-1.2.5.tgz#d2c45c6dd4fbce621a66f136cbe328afd0410b34" integrity sha512-BN22B5eaMMI9UMtjrGd5g5eCYPpCPDUy0FJXbYsaT5zYxjFOckS53SQDE3pWkVoWpHXVb3BrYcEN4Twa55B5cA== -"wrap-ansi-cjs@npm:wrap-ansi@^7.0.0": +"wrap-ansi-cjs@npm:wrap-ansi@^7.0.0", wrap-ansi@^7.0.0: version "7.0.0" resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-7.0.0.tgz#67e145cff510a6a6984bdf1152911d69d2eb9e43" integrity sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q== @@ -15391,15 +15375,6 @@ wrap-ansi@^6.2.0: string-width "^4.1.0" strip-ansi "^6.0.0" -wrap-ansi@^7.0.0: - version "7.0.0" - resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-7.0.0.tgz#67e145cff510a6a6984bdf1152911d69d2eb9e43" - integrity sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q== - dependencies: - ansi-styles "^4.0.0" - string-width "^4.1.0" - strip-ansi "^6.0.0" - wrap-ansi@^8.1.0: version "8.1.0" resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-8.1.0.tgz#56dc22368ee570face1b49819975d9b9a5ead214" From c2b58dfd631eb3709014397f395b75f90b8882fe Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 21:47:36 +0000 Subject: [PATCH 34/47] fix(ipfs): resolve adversarial-review findings (resource, cache, rate-limit) A multi-agent adversarial review surfaced 10 confirmed defects clustered in the stream lifecycle and cache-control logic. All fixed here with tests. High: - Client disconnect mid-download now tears down the upstream Kubo stream (res.on('close') -> stream.destroy), releasing the concurrency slot, socket, and cache temp-fd immediately instead of at the ~20-min wall-clock cap. Repeated aborts could otherwise pin all IPFS_KUBO_MAX_CONCURRENT_REQUESTS slots and 502 legitimate traffic. - The trustless format=raw|car branch only sets immutable Cache-Control for direct-CID requests (req.arns === undefined), mirroring the UnixFS branch, so an ArNS name repoint is no longer masked by an ~11-month immutable entry. - The rate limiter is now charged the bytes actually streamed (a passthrough counter), not the fixed 256 KB unknown-size reserve; a 1 GB CAR previously cost ~256 tokens. Accounting also fires on 'close' (aborted transfers), not only 'finish'. Medium: - HEAD/rate-limit teardown of a format response destroys the underlying Kubo source, not just the guardSize wrapper (guard 'close' -> source.destroy). - IPFS uses its own NegativeDataCache instance, so its health window no longer gates Arweave negative-cache promotions (and vice-versa). - Vary: Accept is set on all IPFS responses (representation is negotiated on Accept: application/vnd.ipld.raw|car). - getContent spans end on 'close' too, so destroy-without-error paths (HEAD, rate-limited, guard abort) don't leak unended spans. Low: - Cache hit/miss counters increment once (in the service), not also in the route. - The cross-CID subdomain redirect preserves the query string (e.g. ?format=raw). - ArNS-root ipfsPath is normalized '' -> undefined to match the subdomain middleware's cache/ETag key. Tests: new ipfs-service, routes/ipfs, and routes/ipfs-rate-accounting suites plus a kubo-data-source slot-release case cover H1/H2/H3/M1/M4/L1 deterministically (client-abort teardown via a real socket; byte-accounting via an enabled limiter; guard/source teardown; negative-cache path key). Full suite 2148 pass / 0 fail. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/ipfs/ipfs-service.test.ts | 151 +++++++++++++++++ src/ipfs/ipfs-service.ts | 24 ++- src/ipfs/kubo-data-source.test.ts | 50 ++++++ src/ipfs/kubo-data-source.ts | 25 ++- src/middleware/arns.ts | 7 +- src/middleware/ipfs.ts | 7 +- src/routes/ipfs-rate-accounting.test.ts | 105 ++++++++++++ src/routes/ipfs.test.ts | 209 ++++++++++++++++++++++++ src/routes/ipfs.ts | 70 ++++++-- src/system.ts | 25 ++- 10 files changed, 645 insertions(+), 28 deletions(-) create mode 100644 src/ipfs/ipfs-service.test.ts create mode 100644 src/routes/ipfs-rate-accounting.test.ts create mode 100644 src/routes/ipfs.test.ts diff --git a/src/ipfs/ipfs-service.test.ts b/src/ipfs/ipfs-service.test.ts new file mode 100644 index 000000000..4901f4782 --- /dev/null +++ b/src/ipfs/ipfs-service.test.ts @@ -0,0 +1,151 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { describe, it, beforeEach, mock } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { Readable } from 'node:stream'; +import { once } from 'node:events'; + +import { createTestLogger } from '../../test/test-logger.js'; +import { IpfsService } from './ipfs-service.js'; +import { KuboDataSource, IpfsNotFoundError } from './kubo-data-source.js'; +import { IpfsFsCache } from './ipfs-cache.js'; +import { DataBlockListValidator } from '../types.js'; +import { NegativeDataCache } from '../data/negative-data-cache.js'; +import * as metrics from '../metrics.js'; + +const CID = 'bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi'; + +// A never-ending readable so the returned stream stays live until we destroy it. +function makeInfiniteStream(): Readable { + const s = new Readable({ read() {} }); + s.push(Buffer.alloc(8)); + return s; +} + +async function counterValue(counter: { + get: () => Promise<{ values: { value: number }[] }>; +}): Promise { + const m = await counter.get(); + return m.values.reduce((sum, v) => sum + v.value, 0); +} + +describe('IpfsService', () => { + const log = createTestLogger({ suite: 'IpfsService' }); + + let dataSource: KuboDataSource; + let cache: IpfsFsCache; + let blockListValidator: DataBlockListValidator; + let negativeCache: NegativeDataCache; + + const buildService = (maxResponseSizeBytes = 10_000_000) => + new IpfsService({ + log, + dataSource, + cache, + blockListValidator, + maxResponseSizeBytes, + negativeCache, + }); + + beforeEach(() => { + cache = { + get: mock.fn(async () => null), + getDigest: mock.fn(async () => undefined), + getCachePath: mock.fn(() => '/nonexistent-cache-dir'), + putFromFile: mock.fn(async () => {}), + } as unknown as IpfsFsCache; + + blockListValidator = { + isIdBlocked: mock.fn(async () => false), + isHashBlocked: mock.fn(async () => false), + } as unknown as DataBlockListValidator; + + negativeCache = { + isNegativelyCached: mock.fn(() => false), + recordMiss: mock.fn(() => {}), + recordSuccess: mock.fn(() => {}), + evict: mock.fn(() => {}), + } as unknown as NegativeDataCache; + }); + + describe('M1: guardSize teardown', () => { + it('destroys the underlying Kubo source when the returned format stream is destroyed', async () => { + const source = makeInfiniteStream(); + dataSource = { + getContent: mock.fn(async () => ({ + stream: source, + size: 0, + contentType: 'application/vnd.ipld.car', + statusCode: 200, + })), + } as unknown as KuboDataSource; + + const service = buildService(); + const result = await service.getContent({ + cidString: CID, + format: 'car', + }); + + // The returned stream is the size-guard wrapper, not the source. + assert.notEqual(result.stream, source); + assert.equal(source.destroyed, false); + + // Destroying the wrapper (as HEAD / a rate-limit teardown does) must tear + // down the underlying source so its socket + concurrency slot release. + result.stream.destroy(); + await once(source, 'close'); + assert.equal(source.destroyed, true); + }); + }); + + describe('negative cache is keyed by CID+path (regression)', () => { + it('records a miss under `${cid}/${path}`, not the bare CID', async () => { + dataSource = { + getContent: mock.fn(async () => { + throw new IpfsNotFoundError('nope'); + }), + } as unknown as KuboDataSource; + + const service = buildService(); + await assert.rejects( + () => service.getContent({ cidString: CID, path: 'sub/leaf.png' }), + (e: any) => e instanceof IpfsNotFoundError, + ); + + const recordMiss = (negativeCache.recordMiss as any).mock; + assert.equal(recordMiss.calls.length, 1); + assert.equal(recordMiss.calls[0].arguments[0], `${CID}/sub/leaf.png`); + // The bare-CID (root) key must be untouched, so a bad sub-path can't + // blackhole the whole site. + assert.notEqual(recordMiss.calls[0].arguments[0], CID); + }); + }); + + describe('L1: cache hit/miss counters increment once, in the service', () => { + it('increments the miss counter exactly once per uncached fetch', async () => { + dataSource = { + getContent: mock.fn(async () => ({ + stream: makeInfiniteStream(), + size: 8, + contentType: 'application/vnd.ipld.raw', + statusCode: 200, + })), + } as unknown as KuboDataSource; + + const before = await counterValue(metrics.ipfsCacheMissTotal); + const service = buildService(); + const result = await service.getContent({ + cidString: CID, + format: 'raw', + }); + result.stream.destroy(); + const after = await counterValue(metrics.ipfsCacheMissTotal); + + assert.equal(after - before, 1); + }); + }); +}); diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 67da3c71b..fed56ba3f 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -276,11 +276,22 @@ export class IpfsService { ); } - // End span when stream completes - result.stream.on('end', () => span.end()); + // End span when the stream terminates. 'close' is included because the + // destroy()-without-error paths (HEAD, rate-limited teardown, guardSize + // abort) emit only 'close' — not 'end'/'error' — so without it those spans + // would never end or export. endSpan() is idempotent so a normal + // 'end'-then-'close' sequence ends exactly once. + let spanEnded = false; + const endSpan = () => { + if (spanEnded) return; + spanEnded = true; + span.end(); + }; + result.stream.on('end', endSpan); + result.stream.on('close', endSpan); result.stream.on('error', (err) => { span.recordException(err); - span.end(); + endSpan(); }); return { @@ -329,6 +340,13 @@ export class IpfsService { }); stream.on('error', (e) => guard.destroy(e)); guard.on('error', () => stream.destroy()); + // A plain destroy() of the returned guard (a HEAD releasing the body, or a + // rate-limit/client-abort teardown) emits 'close', not 'error', and pipe() + // does not propagate a destroy upstream — so tear down the source here too, + // otherwise its Kubo socket + concurrency slot leak until the wall-clock cap. + guard.on('close', () => { + if (!stream.destroyed) stream.destroy(); + }); return stream.pipe(guard); } diff --git a/src/ipfs/kubo-data-source.test.ts b/src/ipfs/kubo-data-source.test.ts index 87cef1e06..2035aae20 100644 --- a/src/ipfs/kubo-data-source.test.ts +++ b/src/ipfs/kubo-data-source.test.ts @@ -7,6 +7,7 @@ import { describe, it, beforeEach, afterEach } from 'node:test'; import { strict as assert } from 'node:assert'; import { Readable } from 'node:stream'; +import { once } from 'node:events'; import axios from 'axios'; import { createTestLogger } from '../../test/test-logger.js'; @@ -210,6 +211,55 @@ describe('KuboDataSource', () => { }); }); + describe('concurrency slot release', () => { + const CID = 'bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi'; + let interceptorId: number; + afterEach(() => axios.interceptors.request.eject(interceptorId)); + + const ok200 = () => { + interceptorId = axios.interceptors.request.use((config) => { + config.adapter = () => + Promise.resolve({ + status: 200, + statusText: 'OK', + headers: { 'content-length': '4', 'content-type': 'text/plain' }, + config, + data: new Readable({ read() {} }), // stays open until destroyed + }); + return config; + }); + }; + + it('frees the slot when the response stream is destroyed, so later fetches proceed', async () => { + const ds = new KuboDataSource({ + log, + kuboUrl: 'http://localhost:8080', + requestTimeoutMs: 5000, + streamStallTimeoutMs: 5000, + maxConcurrent: 1, + }); + ok200(); + + // First fetch takes the only slot and holds it (stream left unconsumed). + const r1 = await ds.getContent({ cidString: CID }); + + // A second concurrent fetch must fail fast while the slot is held. + await assert.rejects( + () => ds.getContent({ cidString: CID }), + (e: any) => e.name === 'IpfsUnavailableError', + ); + + // Destroying the stream emits 'close', which releases the slot (and ends + // the span) — a subsequent fetch then succeeds. + r1.stream.destroy(); + await once(r1.stream, 'close'); + + const r3 = await ds.getContent({ cidString: CID }); + assert.equal(r3.statusCode, 200); + r3.stream.destroy(); + }); + }); + describe('error types', () => { it('IpfsNotFoundError has correct name', () => { const error = new IpfsNotFoundError('not found'); diff --git a/src/ipfs/kubo-data-source.ts b/src/ipfs/kubo-data-source.ts index 4131ecf49..f34353d9d 100644 --- a/src/ipfs/kubo-data-source.ts +++ b/src/ipfs/kubo-data-source.ts @@ -217,16 +217,29 @@ export class KuboDataSource { contentType, }); - // End span when stream finishes or errors - stream.on('end', () => span.end()); + // End span when the stream terminates. 'close' is included because a + // destroy()-without-error (HEAD, rate-limited teardown, client abort) emits + // only 'close' — not 'end'/'error' — so without it the span never + // ends/exports. endSpan() is idempotent so a normal 'end'-then-'close' + // sequence ends exactly once. + let spanEnded = false; + const endSpan = () => { + if (spanEnded) return; + spanEnded = true; + span.end(); + }; + stream.on('end', endSpan); stream.on('error', (err) => { span.recordException(err); - span.end(); + endSpan(); }); - // Release the concurrency slot when the response stream is fully consumed - // or destroyed (covers success, client abort, and downstream errors). - stream.once('close', release); + // Release the concurrency slot and end the span when the response stream is + // fully consumed or destroyed (covers success, client abort, and errors). + stream.once('close', () => { + endSpan(); + release(); + }); return { stream, diff --git a/src/middleware/arns.ts b/src/middleware/arns.ts index 8fdadf960..336e68656 100644 --- a/src/middleware/arns.ts +++ b/src/middleware/arns.ts @@ -206,7 +206,12 @@ export const createArnsMiddleware = ({ if (arnsProtocol === 'ipfs' && ipfsHandler !== undefined) { serveViaIpfs = true; (req as any).ipfsCid = resolvedId; - (req as any).ipfsPath = manifestPath; + // Normalize an empty root path to undefined to match the IPFS + // subdomain middleware's contract — otherwise a root request keys the + // content/negative caches and ETag as `${cid}/` here but `${cid}` via + // the subdomain path, double-fetching and double-storing the same CID. + (req as any).ipfsPath = + manifestPath === '' ? undefined : manifestPath; span.setAttribute('arns.protocol', 'ipfs'); } diff --git a/src/middleware/ipfs.ts b/src/middleware/ipfs.ts index 539883f67..6bc788479 100644 --- a/src/middleware/ipfs.ts +++ b/src/middleware/ipfs.ts @@ -82,9 +82,14 @@ export function createIpfsSubdomainMiddleware({ const targetCid = cidToV1Base32(pathCid); const rootHost = matchedEntry.host; const pathSuffix = remainder !== undefined ? `/${remainder}` : '/'; + // Preserve the query string (e.g. ?format=raw) across the redirect so + // the requested representation isn't silently dropped — matching the + // path-route redirect in routes/ipfs.ts. + const qIdx = req.originalUrl.indexOf('?'); + const queryString = qIdx >= 0 ? req.originalUrl.slice(qIdx) : ''; res.redirect( 302, - `${config.SANDBOX_PROTOCOL ?? req.protocol}://${targetCid}.${rootHost}${pathSuffix}`, + `${config.SANDBOX_PROTOCOL ?? req.protocol}://${targetCid}.${rootHost}${pathSuffix}${queryString}`, ); return; } catch { diff --git a/src/routes/ipfs-rate-accounting.test.ts b/src/routes/ipfs-rate-accounting.test.ts new file mode 100644 index 000000000..1f0146607 --- /dev/null +++ b/src/routes/ipfs-rate-accounting.test.ts @@ -0,0 +1,105 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +process.env.ENABLE_RATE_LIMITER = 'true'; +process.env.RATE_LIMITER_TYPE = 'memory'; + +import { strict as assert } from 'node:assert'; +import { describe, it, mock } from 'node:test'; +import { Readable } from 'node:stream'; +import express from 'express'; +import { default as request } from 'supertest'; + +const CID = 'bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi'; + +// Load app modules only after the env above is set. +const { createTestLogger } = await import('../../test/test-logger.js'); +const { createIpfsHandler } = await import('./ipfs.js'); +const config = await import('../config.js'); +const log = createTestLogger({ suite: 'IPFS rate accounting' }); + +const waitFor = async (pred: () => boolean, ms = 2000) => { + const start = Date.now(); + while (!pred() && Date.now() - start < ms) { + await new Promise((r) => setTimeout(r, 10)); + } +}; + +describe('IPFS route rate-limit accounting (H3)', () => { + it('confirms the rate limiter is enabled in this process', () => { + assert.equal(config.ENABLE_RATE_LIMITER, true); + }); + + it('charges the rate limiter for the actual streamed bytes, not the reserve', async () => { + // A 3-chunk body totalling 30 000 bytes. Crucially size:0 (unknown length), + // so the reserve is the 256 KB placeholder — the pre-fix code would charge + // that placeholder instead of the real 30 000. + const payload = Buffer.alloc(30_000, 0x61); + const service = { + getContent: mock.fn(async () => ({ + stream: Readable.from([ + payload.subarray(0, 10_000), + payload.subarray(10_000, 20_000), + payload.subarray(20_000), + ]), + size: 0, + contentType: 'application/octet-stream', + cached: false, + statusCode: 200, + })), + }; + + const adjustTokens = mock.fn(async () => {}); + const rateLimiter = { + isAllowlisted: mock.fn(() => false), + checkLimit: mock.fn(async () => ({ + allowed: true, + ipTokensConsumed: 1, + ipPaidTokensConsumed: 0, + ipRegularTokensConsumed: 1, + resourceTokensConsumed: 0, + resourcePaidTokensConsumed: 0, + resourceRegularTokensConsumed: 0, + })), + adjustTokens, + topOffPaidTokens: mock.fn(async () => {}), + getIpBucketState: mock.fn(async () => null), + getResourceBucketState: mock.fn(async () => null), + topOffPaidTokensForResource: mock.fn(async () => {}), + } as any; + + const app = express(); + const handler = createIpfsHandler({ + log, + ipfsService: service as any, + rateLimiter, + }); + app.get( + '/c/:cid', + (req, _res, next) => { + (req as any).ipfsCid = (req.params as any).cid; + (req as any).ipfsPath = undefined; + next(); + }, + handler, + ); + + const res = await request(app).get(`/c/${CID}`).expect(200); + assert.equal(res.body.length ?? res.text.length, 30_000); + + await waitFor(() => adjustTokens.mock.calls.length > 0); + assert.equal(adjustTokens.mock.calls.length, 1); + const ctx = (adjustTokens.mock.calls[0].arguments as any[])[1] as { + responseSize: number; + }; + assert.equal( + ctx.responseSize, + 30_000, + `expected the real 30000 streamed bytes, got ${ctx.responseSize} ` + + `(reserve was ${config.IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES})`, + ); + }); +}); diff --git a/src/routes/ipfs.test.ts b/src/routes/ipfs.test.ts new file mode 100644 index 000000000..ff3a9b38e --- /dev/null +++ b/src/routes/ipfs.test.ts @@ -0,0 +1,209 @@ +/** + * AR.IO Gateway + * Copyright (C) 2022-2025 Permanent Data Solutions, Inc. All Rights Reserved. + * + * SPDX-License-Identifier: AGPL-3.0-or-later + */ +import { strict as assert } from 'node:assert'; +import { describe, it, mock } from 'node:test'; +import http from 'node:http'; +import { Readable } from 'node:stream'; +import express from 'express'; +import { default as request } from 'supertest'; + +import { createTestLogger } from '../../test/test-logger.js'; +import { createIpfsHandler } from './ipfs.js'; +import { IpfsNotFoundError } from '../ipfs/kubo-data-source.js'; +import type { IpfsService } from '../ipfs/ipfs-service.js'; +import * as metrics from '../metrics.js'; + +const log = createTestLogger({ suite: 'IPFS routes' }); +const CID = 'bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi'; + +const body = (s: string) => Readable.from([Buffer.from(s)]); + +function makeInfiniteStream(): Readable { + const s = new Readable({ read() {} }); + s.push(Buffer.alloc(1024)); + const timer = setInterval(() => { + if (!s.destroyed) s.push(Buffer.alloc(1024)); + }, 5); + s.on('close', () => clearInterval(timer)); + return s; +} + +function makeApp({ + service, + setArns = false, +}: { + service: Partial; + setArns?: boolean; +}): express.Express { + const app = express(); + const handler = createIpfsHandler({ + log, + ipfsService: service as IpfsService, + }); + const setCtx: express.Handler = (req, _res, next) => { + (req as any).ipfsCid = (req.params as any).cid; + (req as any).ipfsPath = undefined; + if (setArns) (req as any).arns = { name: 'blog' }; + next(); + }; + app.get('/c/:cid', setCtx, handler); + app.head('/c/:cid', setCtx, handler); + return app; +} + +async function counterValue(counter: { + get: () => Promise<{ values: { value: number }[] }>; +}): Promise { + const m = await counter.get(); + return m.values.reduce((sum, v) => sum + v.value, 0); +} + +const waitFor = async (pred: () => boolean, ms = 2000) => { + const start = Date.now(); + while (!pred() && Date.now() - start < ms) { + await new Promise((r) => setTimeout(r, 10)); + } +}; + +describe('IPFS route handler', () => { + const okResult = (over: Record = {}) => ({ + stream: body('hello world'), + size: 11, + contentType: 'text/plain', + cached: false, + statusCode: 200, + ...over, + }); + + describe('H2: immutable Cache-Control is guarded by ArNS binding', () => { + it('sets immutable Cache-Control for a direct-CID trustless (format) response', async () => { + const service = { getContent: mock.fn(async () => okResult()) }; + const res = await request(makeApp({ service })) + .get(`/c/${CID}?format=raw`) + .expect(200); + assert.match(res.headers['cache-control'] ?? '', /immutable/); + }); + + it('does NOT force immutable Cache-Control when served over an ArNS name', async () => { + const service = { getContent: mock.fn(async () => okResult()) }; + const res = await request(makeApp({ service, setArns: true })) + .get(`/c/${CID}?format=raw`) + .expect(200); + assert.doesNotMatch(res.headers['cache-control'] ?? '', /immutable/); + }); + + it('does NOT force immutable Cache-Control for an ArNS UnixFS (proxy) response', async () => { + const service = { getContent: mock.fn(async () => okResult()) }; + const res = await request(makeApp({ service, setArns: true })) + .get(`/c/${CID}`) + .expect(200); + assert.doesNotMatch(res.headers['cache-control'] ?? '', /immutable/); + }); + }); + + describe('M3: Vary: Accept', () => { + it('is set on the UnixFS proxy response', async () => { + const service = { getContent: mock.fn(async () => okResult()) }; + const res = await request(makeApp({ service })).get(`/c/${CID}`); + assert.equal(res.headers['vary'], 'Accept'); + }); + + it('is set on the trustless (format) response', async () => { + const service = { getContent: mock.fn(async () => okResult()) }; + const res = await request(makeApp({ service })).get( + `/c/${CID}?format=car`, + ); + assert.equal(res.headers['vary'], 'Accept'); + }); + }); + + describe('trustless header contract', () => { + it('marks a format response trustless with an attachment disposition', async () => { + const service = { getContent: mock.fn(async () => okResult()) }; + const res = await request(makeApp({ service })).get( + `/c/${CID}?format=raw`, + ); + assert.equal(res.headers['x-ar-io-trustless'], 'true'); + assert.equal(res.headers['content-disposition'], 'attachment'); + }); + + it('marks a UnixFS proxy response NOT trustless', async () => { + const service = { getContent: mock.fn(async () => okResult()) }; + const res = await request(makeApp({ service })).get(`/c/${CID}`); + assert.equal(res.headers['x-ar-io-trustless'], 'false'); + }); + }); + + describe('HEAD releases the body stream', () => { + it('returns headers with no body and destroys the upstream stream', async () => { + const stream = makeInfiniteStream(); + const service = { + getContent: mock.fn(async () => okResult({ stream, size: 0 })), + }; + const res = await request(makeApp({ service })) + .head(`/c/${CID}`) + .expect(200); + assert.equal(res.text ?? '', ''); + assert.match(res.headers['etag'] ?? '', /"/); + await waitFor(() => stream.destroyed); + assert.equal(stream.destroyed, true); + }); + }); + + describe('L1: route does not double-count cache hit/miss', () => { + it('leaves the cache-hit counter untouched (the service owns it)', async () => { + const service = { + getContent: mock.fn(async () => okResult({ cached: true })), + }; + const before = await counterValue(metrics.ipfsCacheHitTotal); + await request(makeApp({ service })).get(`/c/${CID}`).expect(200); + const after = await counterValue(metrics.ipfsCacheHitTotal); + assert.equal(after - before, 0); + }); + }); + + describe('error mapping', () => { + it('maps IpfsNotFoundError to a cache-dampened 404', async () => { + const service = { + getContent: mock.fn(async () => { + throw new IpfsNotFoundError('absent'); + }), + }; + const res = await request(makeApp({ service })) + .get(`/c/${CID}`) + .expect(404); + assert.match(res.headers['cache-control'] ?? '', /max-age=\d+/); + }); + }); + + describe('H1: client disconnect tears down the upstream stream', () => { + it('destroys the source stream when the client aborts mid-download', async () => { + const source = makeInfiniteStream(); + const service = { + getContent: mock.fn(async () => okResult({ stream: source, size: 0 })), + }; + const server = makeApp({ service }).listen(0); + const port = (server.address() as any).port; + + await new Promise((resolve) => { + const req = http.request( + { host: '127.0.0.1', port, path: `/c/${CID}`, method: 'GET' }, + (res) => { + res.once('data', () => req.destroy()); + }, + ); + req.on('error', () => resolve()); + req.on('close', () => resolve()); + req.end(); + }); + + await waitFor(() => source.destroyed); + assert.equal(source.destroyed, true); + await new Promise((r) => server.close(() => r())); + }); + }); +}); diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index edd139cc6..4d7aacbfb 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -9,6 +9,7 @@ import { default as asyncHandler } from 'express-async-handler'; import winston from 'winston'; import url from 'node:url'; +import { Transform } from 'node:stream'; import * as config from '../config.js'; import * as metrics from '../metrics.js'; @@ -285,16 +286,26 @@ async function handleIpfsRequest({ res.setHeader('ETag', `"${etag}"`); res.setHeader('X-Ipfs-Path', `/ipfs/${ipfsPath}`); res.setHeader(headerNames.arIoSource, 'ipfs'); + // The representation is content-negotiated: the same URL yields a UnixFS + // proxy body, a raw block, or a CAR depending on the Accept header (see + // parseIpfsFormat). Tell shared caches to key on Accept so they don't serve + // one representation to a client that asked for another. + res.setHeader('Vary', 'Accept'); if (format !== undefined) { // Trustless retrieval: the body IS the CID's content-addressed bytes (a raw // block or a CAR), which the CLIENT verifies against the CID. The gateway // is not a trust root here — mark it so nothing downstream reads the - // response as a gateway-attested content proof. Content-addressed, so the - // bytes are immutable regardless of any ArNS binding. + // response as a gateway-attested content proof. res.setHeader('Content-Disposition', 'attachment'); res.setHeader('X-Ar-Io-Trustless', 'true'); - res.setHeader('Cache-Control', 'public, max-age=29030400, immutable'); + // The bytes are content-addressed and immutable, but a cache entry served + // over an ArNS name is keyed by the NAME (a mutable binding), not the CID — + // the ArNS middleware already set a TTL-bounded Cache-Control, so don't + // override it here (same PE-9072 guard as the proxy branch below). + if ((req as Request & { arns?: unknown }).arns === undefined) { + res.setHeader('Cache-Control', 'public, max-age=29030400, immutable'); + } } else { // UnixFS proxy path: Kubo reassembles and the gateway serves (and may sign) // the bytes. This is TRUSTED-PROXY, not client-verifiable — the signed @@ -333,29 +344,52 @@ async function handleIpfsRequest({ pinner.pin(cidString); } - // Track metrics + // Track metrics. Cache hit/miss counters are incremented inside + // IpfsService.getContent (the single source of truth for the cache + // decision), so they are NOT re-incremented here — doing both double-counted + // them and desynced them from ipfsRequestsTotal. const cacheStatus = result.cached ? 'hit' : 'miss'; metrics.ipfsRequestsTotal.inc({ route_type: routeType, status: 'success' }); - if (result.cached) { - metrics.ipfsCacheHitTotal.inc(); - } else { - metrics.ipfsCacheMissTotal.inc(); - } if (result.size > 0) { metrics.ipfsContentSizeHistogram.observe(result.size); } // Pipe stream to response. HEAD returns headers only — release the // upstream/cache stream and end without a body. + let bytesStreamed = 0; if (isHead) { result.stream.destroy(); res.end(); } else { - result.stream.pipe(res); + // Count the bytes actually delivered to the client so the rate limiter can + // reconcile against the reserved estimate — the reserve is only a guess for + // unknown-size (chunked/CAR) responses, which can stream far more than the + // 256 KB placeholder. + const counter = new Transform({ + transform(chunk: Buffer, _enc, cb) { + bytesStreamed += chunk.length; + cb(null, chunk); + }, + }); + result.stream.pipe(counter).pipe(res); + // A client disconnect mid-download unpipes but does NOT destroy the source, + // so the Kubo socket, its concurrency slot, and the cache temp-fd would be + // held until the wall-clock cap. Destroy the source on client close so they + // release immediately (writableFinished => the body already completed). + res.on('close', () => { + if (!res.writableFinished && !result.stream.destroyed) { + result.stream.destroy(); + } + }); } - // Adjust rate limiter tokens after response completes - res.on('finish', () => { + // Reconcile rate-limiter tokens and record duration once the response settles. + // Fires on 'finish' (completed) OR 'close' (client aborted) — whichever comes + // first — so aborted large transfers are still charged and measured. + let settled = false; + const onResponseSettled = () => { + if (settled) return; + settled = true; const durationSec = (Date.now() - startTime) / 1000; metrics.ipfsRequestDurationHistogram.observe( { route_type: routeType, cache_status: cacheStatus }, @@ -364,7 +398,13 @@ async function handleIpfsRequest({ adjustRateLimitTokens({ req, - responseSize: isHead ? 0 : result.size > 0 ? result.size : contentSize, + // Use the real streamed byte count; fall back to the known size only when + // nothing streamed (shouldn't happen for a non-HEAD success). + responseSize: isHead + ? 0 + : bytesStreamed > 0 + ? bytesStreamed + : result.size, initialResult: limitCheck, rateLimiter, }).catch((error) => { @@ -372,7 +412,9 @@ async function handleIpfsRequest({ message: error.message, }); }); - }); + }; + res.on('finish', onResponseSettled); + res.on('close', onResponseSettled); result.stream.on('error', (error) => { parentLog.error('IPFS stream error', { diff --git a/src/system.ts b/src/system.ts index 384a4a62d..6da68eba4 100644 --- a/src/system.ts +++ b/src/system.ts @@ -1907,15 +1907,34 @@ if (config.IPFS_ENABLED) { eventEmitter, }); + // Dedicated negative cache for IPFS. It short-circuits absent/unpinned CIDs + // like the Arweave one does, but MUST be a separate instance: the negative + // cache also tracks a rolling success/failure health window that gates + // promotions, and mixing IPFS traffic into the Arweave window would let one + // upstream's health mask or suppress the other's (a Kubo 404 flood suppressing + // legit Arweave promotions, or healthy IPFS traffic hiding an Arweave outage). + const ipfsNegativeCache = new NegativeDataCache({ + log, + enabled: config.NEGATIVE_CACHE_ENABLED, + maxSize: config.NEGATIVE_CACHE_MAX_SIZE, + ttlMs: config.NEGATIVE_CACHE_TTL_MS, + missCountThreshold: config.NEGATIVE_CACHE_MISS_COUNT_THRESHOLD, + missDurationMs: config.NEGATIVE_CACHE_MISS_THRESHOLD_MS, + missTrackerTtlMs: config.NEGATIVE_CACHE_MISS_TRACKER_TTL_MS, + maxTtlMs: config.NEGATIVE_CACHE_MAX_TTL_MS, + promotionHistoryTtlMs: config.NEGATIVE_CACHE_PROMOTION_HISTORY_TTL_MS, + healthWindowMs: config.NEGATIVE_CACHE_HEALTH_WINDOW_MS, + unhealthyThreshold: config.NEGATIVE_CACHE_UNHEALTHY_THRESHOLD, + minSampleSize: config.NEGATIVE_CACHE_HEALTH_MIN_SAMPLE_SIZE, + }); + ipfsService = new IpfsService({ log, dataSource: kuboDataSource, cache: ipfsCache, blockListValidator: dataBlockListValidator, maxResponseSizeBytes: config.IPFS_MAX_RESPONSE_SIZE_BYTES, - // Reuse the shared negative cache so absent/unpinned CIDs short-circuit - // repeat Kubo fetches, as they do for absent Arweave ids. - negativeCache: negativeDataCache, + negativeCache: ipfsNegativeCache, }); ipfsRateLimiter = createIpfsRateLimiter(); From e5bea625924b09895150b97644ca27f76805ef91 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 23:06:56 +0000 Subject: [PATCH 35/47] fix(metrics): label negative-cache gauges by source The M2 fix gave IPFS its own NegativeDataCache instance, but the three size gauges (negative_cache_size, miss_tracker_size, promotion_history_size) were unlabelled, so the Arweave and IPFS instances' updateGauges() calls clobbered each other. Add a `source` label (default 'arweave', 'ipfs' for the IPFS cache) so each instance reports a distinct series. Addresses CodeRabbit nitpick. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/data/negative-data-cache.ts | 14 +++++++++++--- src/metrics.ts | 3 +++ src/system.ts | 3 +++ 3 files changed, 17 insertions(+), 3 deletions(-) diff --git a/src/data/negative-data-cache.ts b/src/data/negative-data-cache.ts index 7ad0ca5be..406e25c66 100644 --- a/src/data/negative-data-cache.ts +++ b/src/data/negative-data-cache.ts @@ -31,6 +31,10 @@ export class NegativeDataCache { private healthWindowMs: number; private unhealthyThreshold: number; private minSampleSize: number; + // Distinguishes this instance's gauge series from any sibling instance (e.g. + // the separate Arweave vs IPFS caches) so their updateGauges() calls don't + // clobber the same unlabelled metric. + private metricsSource: string; constructor({ log, @@ -45,6 +49,7 @@ export class NegativeDataCache { healthWindowMs = 60_000, unhealthyThreshold = 0.8, minSampleSize = 10, + metricsSource = 'arweave', now = Date.now, }: { log: Logger; @@ -59,9 +64,11 @@ export class NegativeDataCache { healthWindowMs?: number; unhealthyThreshold?: number; minSampleSize?: number; + metricsSource?: string; now?: () => number; }) { this.log = log; + this.metricsSource = metricsSource; this.enabled = enabled; this.missCountThreshold = missCountThreshold; this.missDurationMs = missDurationMs; @@ -210,8 +217,9 @@ export class NegativeDataCache { } private updateGauges(): void { - metrics.negativeCacheSize.set(this.negativeCache.size); - metrics.missTrackerSize.set(this.missTracker.size); - metrics.promotionHistorySize.set(this.promotionHistory.size); + const source = this.metricsSource; + metrics.negativeCacheSize.set({ source }, this.negativeCache.size); + metrics.missTrackerSize.set({ source }, this.missTracker.size); + metrics.promotionHistorySize.set({ source }, this.promotionHistory.size); } } diff --git a/src/metrics.ts b/src/metrics.ts index e8587e054..06131a69e 100644 --- a/src/metrics.ts +++ b/src/metrics.ts @@ -1029,11 +1029,13 @@ export const negativeCacheEvictionsTotal = new promClient.Counter({ export const negativeCacheSize = new promClient.Gauge({ name: 'negative_cache_size', help: 'Current number of entries in the negative cache', + labelNames: ['source'], }); export const missTrackerSize = new promClient.Gauge({ name: 'miss_tracker_size', help: 'Current number of entries in the miss tracker', + labelNames: ['source'], }); export const negativeCacheRePromotionsTotal = new promClient.Counter({ @@ -1044,6 +1046,7 @@ export const negativeCacheRePromotionsTotal = new promClient.Counter({ export const promotionHistorySize = new promClient.Gauge({ name: 'promotion_history_size', help: 'Current number of entries in promotion history tracker', + labelNames: ['source'], }); export const negativeCachePromotionsSuppressedTotal = new promClient.Counter({ diff --git a/src/system.ts b/src/system.ts index 6da68eba4..1acf42d68 100644 --- a/src/system.ts +++ b/src/system.ts @@ -1926,6 +1926,9 @@ if (config.IPFS_ENABLED) { healthWindowMs: config.NEGATIVE_CACHE_HEALTH_WINDOW_MS, unhealthyThreshold: config.NEGATIVE_CACHE_UNHEALTHY_THRESHOLD, minSampleSize: config.NEGATIVE_CACHE_HEALTH_MIN_SAMPLE_SIZE, + // Distinct gauge series so IPFS and Arweave negative-cache sizes don't + // clobber each other's unlabelled metric. + metricsSource: 'ipfs', }); ipfsService = new IpfsService({ From 457725150c2dea42deaa0486ce0fa0b0403443eb Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 23:33:04 +0000 Subject: [PATCH 36/47] docs(drafts): observer IPFS change spec + first-class reconciliation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add observer-ipfs-adjustments-spec.md — a verified, observer-only plan to make named IPFS data a first-class citizen with zero smart-contract changes (protocol awareness + capability ramp, trustless CID verification, neutral scoring, block sampling). Every current-state claim is cited to live source. Reconcile the two companion drafts to the latest thinking: - ipfs-observation-incentive-analysis.md (Fable): add a dated reconciliation note — no on-chain capability bit is needed (/ar-io/info already advertises ipfs.enabled and the observer already reads it), and the framing is first-class with an adoption ramp rather than mandatory-vs-optional. Body preserved. - davids-brain-alignment.md: cross-reference the new spec; note the OIP §5 "verify, don't trust" win is now specced with zero contract changes. Working drafts; to be cleaned up later. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- docs/drafts/davids-brain-alignment.md | 7 + .../ipfs-observation-incentive-analysis.md | 18 ++ docs/drafts/observer-ipfs-adjustments-spec.md | 226 ++++++++++++++++++ 3 files changed, 251 insertions(+) create mode 100644 docs/drafts/observer-ipfs-adjustments-spec.md diff --git a/docs/drafts/davids-brain-alignment.md b/docs/drafts/davids-brain-alignment.md index 002ae335d..2044771c3 100644 --- a/docs/drafts/davids-brain-alignment.md +++ b/docs/drafts/davids-brain-alignment.md @@ -130,3 +130,10 @@ Still David's phase 2 (not built): CAR→Arweave permapinning, chain-anchored proofs, folding IPFS into the `src/data/` composite source, libp2p/Bitswap. And the OIP §5 win — the **observer verifying CID→bytes** instead of trusting reference gateways — remains a separate (ar-io-observer) track. + +> **Now specced (2026-08-04):** the §5 observer win is written up in +> `observer-ipfs-adjustments-spec.md` as trustless CID verification — and it lands +> with **zero smart-contract changes** (prescription and reward accounting are +> already content-agnostic). That doc makes named IPFS a first-class, trustlessly +> verified dimension, which is exactly David's "verify, don't trust" posture on the +> serving/observation axis, achieved now without his storage-dependent Stages 0–2. diff --git a/docs/drafts/ipfs-observation-incentive-analysis.md b/docs/drafts/ipfs-observation-incentive-analysis.md index 6d1d4b910..afc48bc5b 100644 --- a/docs/drafts/ipfs-observation-incentive-analysis.md +++ b/docs/drafts/ipfs-observation-incentive-analysis.md @@ -9,6 +9,24 @@ > - Docs: `/programs/ar-io-docs/content/learn/oip/*.mdx` > - SDK: `/programs/ar-io-sdk/src`; live gateway at `localhost:4000` +> **Reconciliation update (2026-08-04).** This report's findings were verified +> line-by-line against live source and carried into the implementation plan in +> `observer-ipfs-adjustments-spec.md`. Two points evolved with the decision to make +> **IPFS a first-class citizen with zero smart-contract changes**: +> 1. **No on-chain capability bit is needed.** §3(d)/§4.1 float a `GatewaySettings` +> "supports IPFS" bit. Superseded: the gateway already self-advertises +> `ipfs.enabled` in `/ar-io/info`, and the observer already reads that exact +> response in `assessOwnership` (`observer.ts:262`). Capability-gating is +> therefore observer-only (~5 lines), no contract change. +> 2. **Framing is first-class, not "mandatory-vs-optional."** The §4 MUST #1 +> decision is reframed as an *adoption ramp*: IPFS names are verified as a +> normal, expected dimension; names on a not-yet-enabled gateway score *neutral* +> during rollout, converging to "IPFS expected of every gateway." Because +> prescription and reward accounting are already content-agnostic (§1, §2c), this +> needs no contract change. +> +> The report body below is preserved as the original research deliverable. + --- ## 1. How observation works today diff --git a/docs/drafts/observer-ipfs-adjustments-spec.md b/docs/drafts/observer-ipfs-adjustments-spec.md new file mode 100644 index 000000000..2445bec70 --- /dev/null +++ b/docs/drafts/observer-ipfs-adjustments-spec.md @@ -0,0 +1,226 @@ +# Observer Adjustments for Named IPFS Data — Change Spec + +> Companion to `ipfs-observation-incentive-analysis.md` (the Fable gap analysis) +> and `davids-brain-alignment.md`. This is a **design spec for protocol-team +> review**, not merged code. Every current-state claim below was verified against +> the live source, cited inline. +> +> **Goal: make named IPFS data a first-class citizen** — served, resolved, and +> trustlessly verified — **with zero smart-contract changes.** Scope is +> **observer-only**; the Solana programs (`ario-gar`/`ario-arns`/`ario-ant`) are +> untouched, which is possible because prescription and reward accounting are +> already content-agnostic (§2). Contract-level work (stratified prescription, +> pinning incentives) is explicitly out of scope and called out in §7. + +Verified sources: +- Observer: `/programs/ar-io-observer` @ `chore/remove-aws-observer-deploy` + (running as `ar-io-node-observer-1`), `REPORT_FORMAT_VERSION = 2`. +- Gateway: `/programs/ar-io-node/wt/ipfs-sync-793` (PR #793, this IPFS branch). + +--- + +## 1. Why any change is needed (the safety problem) + +IPFS-target ArNS names (`AntRecord.target_protocol = 1`) are **already eligible** +for prescription and chosen-name sampling — the observer is entirely +protocol-blind. Verified: + +- No `x-arns-protocol` / CID / multihash handling exists anywhere in the observer + (repo-wide grep: zero hits). +- Header validation only checks **presence** of `x-arns-resolved-id` / + `x-arns-ttl-seconds` — a CID passes untouched (`src/lib/arns-validation.ts`). +- Content assessment is **reference-trust-based**: exact-match of four properties + (`resolvedId`, `ttlSeconds`, `contentType`, `dataHashDigest`) against a + reference/consensus gateway; any mismatch fails the name + (`src/observer.ts:1907-1920`). +- `dataHashDigest` = sha256 of the first 1 MiB, or 5×200-byte random ranges for + content > 1 MiB (`getArnsResolution`, `src/observer.ts:116`). +- Names dimension passes at ≥ 80% (`NAME_PASS_THRESHOLD`, `src/observer.ts:2225`); + composite gateway pass = `ownership AND names AND offsets` + (`src/observer.ts:2233`); a gateway is failed on-chain if **> ½** of submitting + observers fail it. + +**Consequence:** a gateway with IPFS disabled routes an IPFS-target name down the +Arweave data path (`ar-io-node middleware/arns.ts`), can't retrieve it, and fails +the name. With the ≤ 2-name prescription cap and the 80% threshold, **one +prescribed IPFS name can fail every non-IPFS gateway in an epoch** and — via the +> ½ tally — zero out their rewards. This must be fixed before any IPFS-target +name lands on the production registry. + +--- + +## 2. Two enabling facts (that shrink the work) + +**2.1 Capability is already self-advertised, and the observer already reads it.** +The gateway returns `"ipfs": { enabled: true }` in `/ar-io/info` when +`IPFS_ENABLED` (verified live; `ar-io-node src/routes/ar-io.ts:225`). The observer +**already** GETs `https://{host}/ar-io/info` and parses that JSON in +`assessOwnership` (`src/observer.ts:262`, `client.get(url).json()`). So +capability-gating needs **no new request and no contract bit** — just read +`resp.ipfs?.enabled` from the response the observer already has. + +**2.2 The gateway now exposes trustless retrieval (PR #793).** `?format=raw` +returns the raw block and `?format=car` returns a CAR, both with the client +expected to verify against the CID (`X-Ar-Io-Trustless: true`). The gateway also +emits `X-ArNS-Protocol: ipfs|arweave` on resolutions (verified live). These are +the primitives that let the observer verify IPFS bytes **trustlessly** (§4.3). + +--- + +## 3. Design decisions + +**D1 — IPFS is first-class; capability-gating is the rollout ramp.** IPFS-target +names are assessed as a normal, expected dimension — verified trustlessly (D2) and +counted toward rewards like any name (rewards are already content-agnostic, §2). +Capability-gating is the **adoption ramp**, not a permanent opt-out: while gateways +are still enabling IPFS, an IPFS name on a gateway that does not advertise +`ipfs.enabled` scores **neutral** (excluded from the names denominator) rather than +failing; the intended end-state is **IPFS expected of every gateway**. Flipping +"not-yet-enabled" from neutral to failing is a single policy flag — no contract +change, identical mechanism. *When the ramp ends is the one governance call (§8 Q1); +the code defaults safe during rollout so no gateway is wrongly failed mid-adoption.* + +**D2 — Trustless verification for IPFS, replacing reference-digest trust.** For an +IPFS name the `resolvedId` **is** a content hash, so served bytes are verified +against the CID's multihash directly. The reference gateway is still used to +establish the **name→CID binding** (which CID the name should resolve to), but the +**bytes are self-verified**, not compared to reference bytes. This is strictly +stronger than today's Arweave ArNS check and removes, for IPFS names: the +colluding/buggy-reference risk, the mid-epoch reference-cache race, and the +observation-as-cache-warmer distortion. + +**D3 — No range-sampling for IPFS.** The 5×200-byte range digest is meaningless +against a CID (a UnixFS root CID is the hash of the *root block*, not of the +reassembled file bytes). IPFS assessment uses block/CID verification (§4.3) +instead. (Range stays for the Arweave/proxy/media path.) + +**D4 — Report stays content-agnostic on-chain.** `save_observations` submits only +a pass/fail bitmap + report tx-id, so enriching the JSON report with a `protocol` +field and IPFS verification detail requires **no contract change** — only a +`REPORT_FORMAT_VERSION` bump and tolerant report consumers. + +--- + +## 4. The changes (sequenced) + +### Phase 1 — Protocol awareness + capability ramp (the safety fix) + +**Goal:** IPFS names can never wrongly fail a gateway; the report distinguishes +protocol. No verification-model change yet. + +1. **Detect protocol.** In `getArnsResolution` capture `x-arns-protocol` (falls + back to `arweave` when absent) onto `ArnsResolution`. Decode the CID from + `x-arns-resolved-id` when protocol is `ipfs` (format-validate; a malformed CID + is a *gateway* fault only if it diverges from the reference binding). +2. **Read gateway capability.** In `assessOwnership`, also read `resp.ipfs?.enabled` + and thread a `supportsIpfs: boolean` into the gateway assessment context. +3. **Gate + neutral scoring.** In `assessArnsNames`, introduce a third outcome + `neutral` alongside pass/fail. A name is neutral when: (a) its protocol is + `ipfs` and the gateway doesn't advertise `ipfs.enabled` (D1), or (b) it is + **unretrievable network-wide** — the reference/consensus resolution itself + failed to retrieve (distinguish "gateway mis-serves" from "nobody has it"). + Compute `namesPass` as `passCount / (passCount + failCount)` — **neutral names + leave the denominator** (`src/observer.ts:2225`). +4. **Report.** Add `protocol` to `ArnsNameAssessment` (`src/types.ts:148`) and a + per-name `outcome: 'pass'|'fail'|'neutral'`; bump `REPORT_FORMAT_VERSION` 2 → 3. +5. **Timeouts.** Give IPFS names a longer per-request timeout (cold DHT retrieval); + config `IPFS_ASSESSMENT_TIMEOUT_MS`. + +*Contract impact: none. Ships independently and is the minimum safe change.* + +### Phase 2 — Trustless CID verification (replaces reference-digest for IPFS) + +**Goal:** verify served IPFS bytes against the CID, no reference bytes. + +For an IPFS name whose binding (resolvedId) matches reference consensus: +- **Raw / single-block CID** (codec `raw` 0x55): fetch the content (or + `?format=raw`), compute the CID's multihash function over the bytes, compare the + digest to the multihash embedded in the CID. Pass iff equal. +- **UnixFS / dag-pb CID** (codec `dag-pb` 0x70): fetch the **root block** via + `?format=raw`, hash it, compare to the CID's multihash — this proves the gateway + serves the authentic root block. (Full-file assurance is Phase 3.) +- Use a `multiformats`-style CID decoder for codec + hash-fn + digest; support at + least sha2-256. Unknown codec/hash → neutral (can't verify), logged. + +The four-property comparison becomes protocol-branched: for IPFS, keep +`resolvedId` (binding, vs reference consensus) and `ttlSeconds`; **replace +`dataHashDigest`-vs-reference with CID self-verification**; `contentType` becomes +advisory (a trustless raw/CAR fetch has a fixed IPLD content type). + +*Contract impact: none. Depends on PR #793's `?format=raw|car`.* + +### Phase 3 — IPFS block sampling (the offset-check analog) — optional, staged + +Mirror the Arweave offset check (`OFFSET_OBSERVATION_*`): for a UnixFS DAG, fetch +`?format=car` (or walk N random child blocks), verify each block's bytes against +its CID and the link structure back to the root. Staged behind +`IPFS_BLOCK_SAMPLING_*` config, enforcement off by default. Bounds cost while +raising assurance from "authentic root" to "authentic sampled sub-DAG." + +*Contract impact: none.* + +### Phase 4 — 451/blocklist policy — small + +The observer special-cases 404 only. Divergent blocklists (target returns 451, +reference 200, or vice-versa) currently mismatch → fail. Treat a **matching** 451 +as neutral, and a target-only 451 as a policy divergence (log, don't hard-fail on +content) — same spirit as the network-wide-unavailable neutrality. + +--- + +## 5. Report compatibility + +- `REPORT_FORMAT_VERSION` 2 → 3; new fields are **additive** (`protocol`, + `outcome`, IPFS verification detail). Existing consumers that read the bitmap are + unaffected (`save_observations` is content-agnostic — D4). +- Report-parsing consumers (dashboards, `pipeline-report-sink`, tests asserting + `formatVersion: 2`) must be updated to tolerate v3. Enumerate and bump. +- Continuous observer (`src/continuous/continuous-observer.ts:824`) emits the same + `REPORT_FORMAT_VERSION` — single constant, one bump. + +--- + +## 6. Test plan + +- **Unit:** protocol detection from `x-arns-protocol`; CID decode + raw-block and + UnixFS-root verification (fixtures with known-good and tampered bytes); neutral + scoring math (denominator excludes neutral); capability gate (IPFS name + + non-IPFS gateway → neutral, not fail). +- **Integration:** against this branch's live gateway (`localhost:4000`) using the + `atomic-uat-bf09a4b3` IPFS name and a direct `bafkrei…`/`bafybei…` CID — assert + a correct gateway passes trustlessly and a byte-tampered proxy fails. +- **Regression:** Arweave-name assessment path unchanged (protocol `arweave` + falls through to today's four-property comparison). + +--- + +## 7. Contract boundary (out of scope — governance track) + +These require Solana-program changes and are **not** part of the read-only phase: +- **Stratified prescription** (`ario-gar prescribe_epoch` reading + `AntRecord.target_protocol`; enlarging the ≤ 2-name cap) — changes the zero-copy + `Epoch` account layout; needs a migration. +- **Pinning / persistence incentive** — a genuinely new reward category (attested + pinning, retrievability bonds, or ArNS-revenue-funded pinning subsidies). Current + OIP can reward *serving* named IPFS data with zero contract change, but **cannot + reward persistence** — no parameter tweak fixes that. + +Everything in §4 (Phases 1–4) ships without touching these. + +--- + +## 8. Open questions for the protocol team + +1. **Adoption curve (when does the ramp end?).** Code treats IPFS as first-class + with a neutral ramp for not-yet-enabled gateways (D1). The governance call is + *when* the observer flips "not-yet-enabled" from neutral to failing — i.e. when + IPFS support is expected of every gateway. One flag, no contract change. +2. **Persistence semantics.** ArNS historically implies Arweave permanence. Does a + name → unpinned IPFS dilute the guarantee? Should `targetProtocol = 1` + registration require a pinning commitment? (Drives whether Phase-3+ incentives + are ever needed.) +3. **"Who holds the data?"** Is serving via Kubo's DHT fetch from a third party a + pass, or must the gateway pin? The observer can't distinguish (`X-Cache` is + self-asserted); trustless verification proves *correctness*, not *who persists*. +4. **Prescription capacity.** The ≤ 2-name cap makes per-protocol stratification + statistically thin; enlarging it is the `Epoch`-layout migration in §7. From 64e74191ab10450b755b83a55394120dfa8fffcb Mon Sep 17 00:00:00 2001 From: vilenarios Date: Tue, 4 Aug 2026 23:52:46 +0000 Subject: [PATCH 37/47] docs(drafts): add provisioning & operating cost analysis to observer spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add §9 — cost/ops for first-class IPFS: observer changes are near-zero; the only real cost is running an IPFS node, satisfiable per gateway by a bundled Kubo sidecar or by pointing IPFS_KUBO_URL/IPFS_KUBO_API_URL at a shared/third-party node. Disk (10 GB cache, GC), bandwidth (DHT profile is a dial), pinning off by default; persistence is the deferred future lever. Grounded in verified config. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- docs/drafts/observer-ipfs-adjustments-spec.md | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/docs/drafts/observer-ipfs-adjustments-spec.md b/docs/drafts/observer-ipfs-adjustments-spec.md index 2445bec70..fb9cb86f7 100644 --- a/docs/drafts/observer-ipfs-adjustments-spec.md +++ b/docs/drafts/observer-ipfs-adjustments-spec.md @@ -224,3 +224,50 @@ Everything in §4 (Phases 1–4) ships without touching these. self-asserted); trustless verification proves *correctness*, not *who persists*. 4. **Prescription capacity.** The ≤ 2-name cap makes per-protocol stratification statistically thin; enlarging it is the `Epoch`-layout migration in §7. + +--- + +## 9. Provisioning & operating cost + +**Verdict:** the observer changes are near-zero; the only real new cost is running +an IPFS node. First-class means that is **on by default** — but each gateway can +either run a bundled Kubo sidecar **or** point at a shared/third-party node, so +per-gateway cost is a deployment choice, not a fixed floor. + +**Today vs. first-class default.** Currently `IPFS_ENABLED=false` and the `kubo` +service sits behind an opt-in compose profile. First-class = flip the shipped +default to IPFS-on. Both Kubo endpoints are plain env vars +(`IPFS_KUBO_URL` → gateway :8080, `IPFS_KUBO_API_URL` → RPC :5001), so the default +can be satisfied three ways: + +| Mode | What runs | Per-gateway cost | Notes | +|---|---|---|---| +| **A — Bundled Kubo sidecar** (compose default) | Each gateway runs its own Kubo | +1 container (~0.5–2 GB RAM), ~10 GB cache disk, swarm port 4001, DHT bandwidth | Self-sufficient; no external dependency; can pin | +| **B1 — Shared self-hosted Kubo** | Many gateways → one Kubo you run | ≈ 0 per gateway; cost centralizes on one node | A node to scale/secure; pinning available (you control the RPC) | +| **B2 — Third-party IPFS gateway (reads)** | `IPFS_KUBO_URL` → a provider / public trustless gateway | ≈ 0 infra | Provider dependency; no pinning (RPC :5001 is privileged, not exposed) | + +Trustless verification is client-side (the client checks bytes against the CID via +`?format=raw`), so it is unaffected by *where* Kubo runs — mode B doesn't weaken +the trust story. + +**Component cost (mode A, the heaviest):** +- **Container** `ipfs/kubo v0.32.1`, co-located — no new host. RAM ~0.5–2 GB, modest + CPU (spikes on GC / DHT provides). +- **Disk** is bounded: 10 GB read cache (`IPFS_CACHE_MAX_SIZE_BYTES`, LRU) + Kubo + datastore; `--enable-gc` prevents unpinned growth. Pinning is **off by default**; + if enabled it's capped at 10,000 CIDs (`IPFS_PIN_MAX`) × content size. +- **Bandwidth** is the only always-on new cost, in two parts: (a) serving content + (demand-driven, cache-absorbed) and (b) DHT participation. The compose default + `IPFS_PROFILE=server` makes Kubo a DHT server (background traffic even at idle); + `dhtclient` / `lowpower` cuts it to near-nothing. The idle floor is a dial. +- **No SaaS / per-GB fees** — self-hosted. + +**Observer side:** no new infrastructure; request *count* is unchanged (IPFS names +are already sampled today). Trustless verification fetches the raw **root block** +(often *fewer* bytes than today's 1 MiB digest sample); block sampling is off by +default. Net: a rounding error per gateway per epoch. + +**The real future cost lever — persistence.** Pinning named content fleet-wide so +it can't vanish is where recurring storage cost would live. That is deferred +(pinning off by default, capped) and is a separate governance/incentive decision — +not part of first-class *serving* and *verification*. From 43bdbe1e95f6b5a1875715ced70292debf30e511 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Wed, 5 Aug 2026 00:31:38 +0000 Subject: [PATCH 38/47] fix(ipfs): safe defaults when no node serves the CID (no-provider case) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An ArNS name pointing at unretrievable IPFS content (dead link, unpinned+GC'd, or offline) makes Kubo search the DHT and find no provider — Kubo never returns 404 for absent network content, it times out. Previously only 404s were negatively cached, so every repeat request for a dead CID re-ran the full IPFS_KUBO_REQUEST_TIMEOUT_MS (30s) search and held a Kubo concurrency slot; under load a single unresolvable CID could pin all IPFS_KUBO_MAX_CONCURRENT_REQUESTS slots and 502 healthy traffic. - Record a negative-cache miss on IpfsTimeoutError too, so a repeatedly-dead CID trips the (threshold-gated) negative cache and short-circuits to a fast, CDN-dampened 404 instead of re-searching. Transient cold-DHT slowness won't trip it (needs repeated misses over a window). IpfsUnavailableError / ECONNREFUSED stay uncached — those mean Kubo itself is down, not the content. - Add short Cache-Control (CACHE_NOT_FOUND_MAX_AGE, 60s) to the 504 so edges/CDNs dampen retry storms in front of the gateway. Tests: timeout records a miss; unavailable does not; 504 carries Cache-Control. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/ipfs/ipfs-service.test.ts | 44 ++++++++++++++++++++++++++++++++++- src/ipfs/ipfs-service.ts | 17 +++++++++++++- src/routes/ipfs.test.ts | 17 +++++++++++++- src/routes/ipfs.ts | 8 +++++++ 4 files changed, 83 insertions(+), 3 deletions(-) diff --git a/src/ipfs/ipfs-service.test.ts b/src/ipfs/ipfs-service.test.ts index 4901f4782..ef0768c6a 100644 --- a/src/ipfs/ipfs-service.test.ts +++ b/src/ipfs/ipfs-service.test.ts @@ -11,7 +11,12 @@ import { once } from 'node:events'; import { createTestLogger } from '../../test/test-logger.js'; import { IpfsService } from './ipfs-service.js'; -import { KuboDataSource, IpfsNotFoundError } from './kubo-data-source.js'; +import { + KuboDataSource, + IpfsNotFoundError, + IpfsTimeoutError, + IpfsUnavailableError, +} from './kubo-data-source.js'; import { IpfsFsCache } from './ipfs-cache.js'; import { DataBlockListValidator } from '../types.js'; import { NegativeDataCache } from '../data/negative-data-cache.js'; @@ -125,6 +130,43 @@ describe('IpfsService', () => { }); }); + describe('negative cache records unretrievable content (no-provider defaults)', () => { + it('records a miss on a retrieval TIMEOUT (the no-provider case)', async () => { + dataSource = { + getContent: mock.fn(async () => { + throw new IpfsTimeoutError('kubo timed out'); + }), + } as unknown as KuboDataSource; + + const service = buildService(); + await assert.rejects( + () => service.getContent({ cidString: CID }), + (e: any) => e instanceof IpfsTimeoutError, + ); + + const recordMiss = (negativeCache.recordMiss as any).mock; + assert.equal(recordMiss.calls.length, 1); + assert.equal(recordMiss.calls[0].arguments[0], CID); + }); + + it('does NOT negatively cache an IpfsUnavailableError (Kubo itself down)', async () => { + dataSource = { + getContent: mock.fn(async () => { + throw new IpfsUnavailableError('kubo down'); + }), + } as unknown as KuboDataSource; + + const service = buildService(); + await assert.rejects( + () => service.getContent({ cidString: CID }), + (e: any) => e instanceof IpfsUnavailableError, + ); + + const recordMiss = (negativeCache.recordMiss as any).mock; + assert.equal(recordMiss.calls.length, 0); + }); + }); + describe('L1: cache hit/miss counters increment once, in the service', () => { it('increments the miss counter exactly once per uncached fetch', async () => { dataSource = { diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index fed56ba3f..43f074822 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -20,6 +20,7 @@ import { IpfsBlockedError, IpfsNotFoundError, IpfsSizeLimitError, + IpfsTimeoutError, } from './kubo-data-source.js'; import * as metrics from '../metrics.js'; @@ -232,7 +233,21 @@ export class IpfsService { format, }) .catch((err) => { - if (err instanceof IpfsNotFoundError) { + // Record a negative-cache miss for content Kubo could not retrieve: + // a 404, or a retrieval TIMEOUT (the "no provider on the network" + // case — Kubo never returns 404 for absent network content, it times + // out). Without caching the timeout, every repeat request for a dead + // CID re-runs the full IPFS_KUBO_REQUEST_TIMEOUT_MS search and holds a + // concurrency slot, so a single unresolvable CID under load can pin all + // slots and 502 healthy traffic. The negative cache only trips after + // repeated misses over a window, so transient cold-DHT slowness won't + // blackhole legit-but-slow content. NOT cached: IpfsUnavailableError / + // ECONNREFUSED — those mean Kubo itself is down, not the content, and + // caching them would blackhole content once Kubo recovers. + if ( + err instanceof IpfsNotFoundError || + err instanceof IpfsTimeoutError + ) { this.negativeCache?.recordMiss(negKey); } throw err; diff --git a/src/routes/ipfs.test.ts b/src/routes/ipfs.test.ts index ff3a9b38e..331b0d039 100644 --- a/src/routes/ipfs.test.ts +++ b/src/routes/ipfs.test.ts @@ -13,7 +13,10 @@ import { default as request } from 'supertest'; import { createTestLogger } from '../../test/test-logger.js'; import { createIpfsHandler } from './ipfs.js'; -import { IpfsNotFoundError } from '../ipfs/kubo-data-source.js'; +import { + IpfsNotFoundError, + IpfsTimeoutError, +} from '../ipfs/kubo-data-source.js'; import type { IpfsService } from '../ipfs/ipfs-service.js'; import * as metrics from '../metrics.js'; @@ -178,6 +181,18 @@ describe('IPFS route handler', () => { .expect(404); assert.match(res.headers['cache-control'] ?? '', /max-age=\d+/); }); + + it('maps IpfsTimeoutError to a 504 with retry-dampening Cache-Control', async () => { + const service = { + getContent: mock.fn(async () => { + throw new IpfsTimeoutError('no provider'); + }), + }; + const res = await request(makeApp({ service })) + .get(`/c/${CID}`) + .expect(504); + assert.match(res.headers['cache-control'] ?? '', /max-age=\d+/); + }); }); describe('H1: client disconnect tears down the upstream stream', () => { diff --git a/src/routes/ipfs.ts b/src/routes/ipfs.ts index 4d7aacbfb..d7e6ad38d 100644 --- a/src/routes/ipfs.ts +++ b/src/routes/ipfs.ts @@ -467,6 +467,14 @@ async function handleIpfsRequest({ route_type: routeType, status: 'timeout', }); + // Briefly dampen retries at the edge/CDN so a burst of requests for an + // unretrievable CID doesn't re-run the full Kubo search each time (the + // negative cache handles the in-process short-circuit; this covers hops + // in front of the gateway). Short max-age since a timeout may be transient. + res.setHeader( + 'Cache-Control', + `public, max-age=${config.CACHE_NOT_FOUND_MAX_AGE}, must-revalidate`, + ); res.status(504).json({ error: 'IPFS request timed out' }); return; } From 1eb4b8ac3fd6214b8bd705037cb54e0a5c537a87 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Wed, 5 Aug 2026 00:33:14 +0000 Subject: [PATCH 39/47] docs(drafts): add observer-enforcement caveat to spec Make explicit that neutral scoring is enforced at the observer level, not the contract (the chain only tallies a per-gateway pass/fail bitmap by >1/2-observer majority). Neutral therefore protects a non-IPFS gateway only once a majority of observers run the updated code; the majority rule is the backstop for an uneven rollout. It's a software release adopted by the observer fleet, not a contract deploy, and requires no action from gateway operators. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- docs/drafts/observer-ipfs-adjustments-spec.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/drafts/observer-ipfs-adjustments-spec.md b/docs/drafts/observer-ipfs-adjustments-spec.md index fb9cb86f7..fae35f019 100644 --- a/docs/drafts/observer-ipfs-adjustments-spec.md +++ b/docs/drafts/observer-ipfs-adjustments-spec.md @@ -80,6 +80,19 @@ failing; the intended end-state is **IPFS expected of every gateway**. Flipping change, identical mechanism. *When the ramp ends is the one governance call (§8 Q1); the code defaults safe during rollout so no gateway is wrongly failed mid-adoption.* +> **Enforcement caveat (important).** Neutral scoring is enforced at the +> **observer level**, not in the contract — the chain only receives a per-gateway +> pass/fail bitmap and tallies a **> ½-of-observers majority**; it has no concept +> of names, protocols, or "neutral" (this is *why* it needs no contract change, +> §2/D4). Consequently, neutral only actually protects a non-IPFS gateway once a +> **majority of the observer fleet runs this updated observer**. The > ½ majority +> rule is the backstop that makes the rollout safe even when uneven: a lagging +> minority still running the old protocol-blind code cannot fail a non-IPFS +> gateway on an IPFS name, because it is outvoted. The rollout task is therefore +> "ship the observer update and get it adopted by the observer majority" — a +> software release, **not** a contract deploy, and **no action is required from +> gateway operators** (not-running-IPFS is self-evident via `/ar-io/info`). + **D2 — Trustless verification for IPFS, replacing reference-digest trust.** For an IPFS name the `resolvedId` **is** a content hash, so served bytes are verified against the CID's multihash directly. The reference gateway is still used to From 4926836e26c24f360754e316fbdb501911f488f3 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Wed, 5 Aug 2026 01:54:57 +0000 Subject: [PATCH 40/47] fix(ipfs): record IPFS timeouts as soft negative-cache misses (adversarial #2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The no-provider defaults commit fed IpfsTimeoutError into recordMiss, but the negative cache has a single-miss re-promotion fast path (priorPromotions>0 => effectiveCount 1, duration 0). So one transient cold-DHT timeout could instantly re-blackhole a previously-promoted-but-recovering CID for hours (exponential TTL) — contradicting the "needs repeated misses" safety claim. - NegativeDataCache.recordMiss gains an opts.softMiss flag: a soft miss never uses the single-miss fast path, always requiring the full miss count/duration threshold. Backward-compatible — Arweave callers pass nothing (hard miss). - ipfs-service records IpfsTimeoutError as a soft miss; a 404 (IpfsNotFoundError) remains a definitive hard miss. Tests: a single hard miss re-promotes after a prior promotion; a single soft miss does not. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/data/negative-data-cache.test.ts | 30 ++++++++++++++++++++++++++++ src/data/negative-data-cache.ts | 14 ++++++++++--- src/ipfs/ipfs-service.ts | 12 +++++++---- 3 files changed, 49 insertions(+), 7 deletions(-) diff --git a/src/data/negative-data-cache.test.ts b/src/data/negative-data-cache.test.ts index 6a3c1c991..c78e248f6 100644 --- a/src/data/negative-data-cache.test.ts +++ b/src/data/negative-data-cache.test.ts @@ -68,6 +68,36 @@ describe('NegativeDataCache', () => { assert.equal(cache.isNegativelyCached('id1'), true); }); + it('a single HARD miss re-promotes immediately after a prior promotion', () => { + const cache = createCache(); + cache.recordMiss('id1'); + currentTime = 6_000; + cache.recordMiss('id1'); + currentTime = 11_000; + cache.recordMiss('id1'); + // Let the negative-cache entry expire (past ttlMs) but keep promotion history. + currentTime = 11_000 + 60_001; + assert.equal(cache.isNegativelyCached('id1'), false); + // A single hard miss re-promotes immediately (the fast path). + cache.recordMiss('id1'); + assert.equal(cache.isNegativelyCached('id1'), true); + }); + + it('a single SOFT miss does NOT use the re-promotion fast path (timeout safety)', () => { + const cache = createCache(); + cache.recordMiss('id1'); + currentTime = 6_000; + cache.recordMiss('id1'); + currentTime = 11_000; + cache.recordMiss('id1'); + currentTime = 11_000 + 60_001; + assert.equal(cache.isNegativelyCached('id1'), false); + // A single SOFT miss (e.g. a transient IPFS timeout) must NOT instantly + // re-blackhole a recovering id — it still needs the full miss threshold. + cache.recordMiss('id1', { softMiss: true }); + assert.equal(cache.isNegativelyCached('id1'), false); + }); + it('evict removes from negative cache', () => { const cache = createCache(); cache.recordMiss('id1'); diff --git a/src/data/negative-data-cache.ts b/src/data/negative-data-cache.ts index 406e25c66..40d8bf9e8 100644 --- a/src/data/negative-data-cache.ts +++ b/src/data/negative-data-cache.ts @@ -130,7 +130,14 @@ export class NegativeDataCache { this.recentSuccesses++; } - recordMiss(id: string): void { + /** + * @param opts.softMiss a soft miss is a transient availability failure (e.g. + * an IPFS retrieval timeout — no live provider) rather than a definitive + * "absent" (a 404). Soft misses NEVER use the single-miss re-promotion fast + * path, so one transient timeout can't instantly re-blackhole a recovering id; + * they always require the full miss count/duration threshold. + */ + recordMiss(id: string, opts?: { softMiss?: boolean }): void { if (!this.enabled) { return; } @@ -152,8 +159,9 @@ export class NegativeDataCache { } const priorPromotions = this.promotionHistory.get(id) ?? 0; - const effectiveCount = priorPromotions > 0 ? 1 : this.missCountThreshold; - const effectiveDuration = priorPromotions > 0 ? 0 : this.missDurationMs; + const usePromotionFastPath = priorPromotions > 0 && opts?.softMiss !== true; + const effectiveCount = usePromotionFastPath ? 1 : this.missCountThreshold; + const effectiveDuration = usePromotionFastPath ? 0 : this.missDurationMs; if ( entry.count >= effectiveCount && diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index 43f074822..b57cbd209 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -244,11 +244,15 @@ export class IpfsService { // blackhole legit-but-slow content. NOT cached: IpfsUnavailableError / // ECONNREFUSED — those mean Kubo itself is down, not the content, and // caching them would blackhole content once Kubo recovers. - if ( - err instanceof IpfsNotFoundError || - err instanceof IpfsTimeoutError - ) { + if (err instanceof IpfsNotFoundError) { this.negativeCache?.recordMiss(negKey); + } else if (err instanceof IpfsTimeoutError) { + // A timeout is an availability failure (likely no live provider), + // not a definitive "absent". Record it as a SOFT miss so a single + // cold-DHT timeout can't re-blackhole a previously-promoted CID via + // the negative cache's single-miss re-promotion fast path — it still + // needs repeated timeouts over the window to trip. + this.negativeCache?.recordMiss(negKey, { softMiss: true }); } throw err; }); From 0d7d552d889ba68d2d1ca35d29cf7fec16fc2973 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Wed, 5 Aug 2026 02:35:46 +0000 Subject: [PATCH 41/47] fix(ipfs): short self-healing TTL for timeout negative-cache entries (adversarial #4) Negatively caching IpfsTimeoutError (soft miss) still blackholed recovering-but- slow content once tripped: it used the normal absent-content TTL (escalating toward maxTtlMs, hours), so a cold-DHT CID that recovered stayed cached-out with no self-healing until TTL expiry. - NegativeDataCache: a soft-miss promotion now uses a short, fixed softMissTtlMs and does NOT build the escalation/re-promotion history; hard (404) promotions keep the exponential-backoff TTL and history. Backward-compatible: softMissTtlMs defaults to ttlMs, so Arweave callers (which never pass softMiss) are unchanged. - IPFS negative cache wires softMissTtlMs = IPFS_TIMEOUT_NEGATIVE_CACHE_TTL_MS (default 60s), so a timing-out CID self-heals within ~a minute. - Corrected the ipfs-service comment to state the actual protection. Tests: a soft-miss promotion self-heals after the short TTL and builds no escalation history. negative-data-cache suite 31 pass. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- src/config.ts | 9 +++++++++ src/data/negative-data-cache.test.ts | 29 ++++++++++++++++++++++++++++ src/data/negative-data-cache.ts | 28 +++++++++++++++++++++------ src/ipfs/ipfs-service.ts | 12 +++++++----- src/system.ts | 3 +++ 5 files changed, 70 insertions(+), 11 deletions(-) diff --git a/src/config.ts b/src/config.ts index 96c07887d..944ecd4d0 100644 --- a/src/config.ts +++ b/src/config.ts @@ -3269,6 +3269,15 @@ export const IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES = env.positiveIntOrDefault( 262144, ); +// Short, non-escalating negative-cache TTL for IPFS retrieval TIMEOUTS (soft +// misses). Keeps a repeatedly-timing-out CID from re-hammering Kubo without +// blackholing recovering-but-slow content for hours — it self-heals after this +// window. Distinct from the (long, escalating) TTL used for definitive 404s. +export const IPFS_TIMEOUT_NEGATIVE_CACHE_TTL_MS = env.positiveIntOrDefault( + 'IPFS_TIMEOUT_NEGATIVE_CACHE_TTL_MS', + 60000, +); + export const IPFS_CACHE_PATH = env.varOrDefault( 'IPFS_CACHE_PATH', 'data/ipfs-cache', diff --git a/src/data/negative-data-cache.test.ts b/src/data/negative-data-cache.test.ts index c78e248f6..c976481c0 100644 --- a/src/data/negative-data-cache.test.ts +++ b/src/data/negative-data-cache.test.ts @@ -98,6 +98,35 @@ describe('NegativeDataCache', () => { assert.equal(cache.isNegativelyCached('id1'), false); }); + it('a soft-miss promotion uses the short softMissTtlMs and self-heals', () => { + const cache = createCache({ softMissTtlMs: 5_000, maxTtlMs: 120_000 }); + // Trip via the full threshold of soft misses over the duration window. + cache.recordMiss('id1', { softMiss: true }); + currentTime = 6_000; + cache.recordMiss('id1', { softMiss: true }); + currentTime = 11_000; + cache.recordMiss('id1', { softMiss: true }); + assert.equal(cache.isNegativelyCached('id1'), true); + // Self-heals after the short 5s soft TTL, NOT the 60s absent-content ttl. + currentTime = 11_000 + 5_001; + assert.equal(cache.isNegativelyCached('id1'), false); + }); + + it('a soft-miss promotion does not build escalation history', () => { + const cache = createCache({ softMissTtlMs: 5_000 }); + cache.recordMiss('id1', { softMiss: true }); + currentTime = 6_000; + cache.recordMiss('id1', { softMiss: true }); + currentTime = 11_000; + cache.recordMiss('id1', { softMiss: true }); + assert.equal(cache.isNegativelyCached('id1'), true); + currentTime = 11_000 + 5_001; // soft entry expired + // A later HARD miss must NOT get the single-miss fast path, because the soft + // promotion did not record promotion history. + cache.recordMiss('id1'); + assert.equal(cache.isNegativelyCached('id1'), false); + }); + it('evict removes from negative cache', () => { const cache = createCache(); cache.recordMiss('id1'); diff --git a/src/data/negative-data-cache.ts b/src/data/negative-data-cache.ts index 40d8bf9e8..bbb8d099c 100644 --- a/src/data/negative-data-cache.ts +++ b/src/data/negative-data-cache.ts @@ -24,6 +24,7 @@ export class NegativeDataCache { private missDurationMs: number; private baseTtlMs: number; private maxTtlMs: number; + private softMissTtlMs: number; private now: () => number; private recentSuccesses: number = 0; private recentFailures: number = 0; @@ -45,6 +46,7 @@ export class NegativeDataCache { missDurationMs, missTrackerTtlMs, maxTtlMs, + softMissTtlMs, promotionHistoryTtlMs, healthWindowMs = 60_000, unhealthyThreshold = 0.8, @@ -60,6 +62,7 @@ export class NegativeDataCache { missDurationMs: number; missTrackerTtlMs?: number; maxTtlMs?: number; + softMissTtlMs?: number; promotionHistoryTtlMs?: number; healthWindowMs?: number; unhealthyThreshold?: number; @@ -74,6 +77,10 @@ export class NegativeDataCache { this.missDurationMs = missDurationMs; this.baseTtlMs = ttlMs; this.maxTtlMs = maxTtlMs ?? ttlMs; + // Soft (timeout-origin) blackholes use a short, non-escalating TTL so a + // recovering-but-slow id self-heals quickly instead of being blackholed for + // hours. Defaults to ttlMs so callers that never pass softMiss are unaffected. + this.softMissTtlMs = softMissTtlMs ?? ttlMs; this.now = now; this.healthWindowMs = healthWindowMs; this.unhealthyThreshold = unhealthyThreshold; @@ -173,15 +180,24 @@ export class NegativeDataCache { return; } - const ttl = Math.min( - this.baseTtlMs * 2 ** Math.min(priorPromotions, 30), - this.maxTtlMs, - ); + // A soft (timeout) promotion gets a short, fixed TTL and does NOT build the + // escalation/re-promotion history — so a transient-but-recovering id can't + // be blackholed for hours (or escalate toward maxTtlMs). A hard (definitive + // 404) promotion keeps the exponential-backoff TTL and history. + const soft = opts?.softMiss === true; + const ttl = soft + ? this.softMissTtlMs + : Math.min( + this.baseTtlMs * 2 ** Math.min(priorPromotions, 30), + this.maxTtlMs, + ); this.negativeCache.set(id, true, { ttl }); - this.promotionHistory.set(id, priorPromotions + 1); + if (!soft) { + this.promotionHistory.set(id, priorPromotions + 1); + } this.missTracker.delete(id); metrics.negativeCachePromotionsTotal.inc(); - if (priorPromotions > 0) { + if (!soft && priorPromotions > 0) { metrics.negativeCacheRePromotionsTotal.inc(); } this.log.info('ID promoted to negative cache', { diff --git a/src/ipfs/ipfs-service.ts b/src/ipfs/ipfs-service.ts index b57cbd209..a072f077b 100644 --- a/src/ipfs/ipfs-service.ts +++ b/src/ipfs/ipfs-service.ts @@ -239,11 +239,13 @@ export class IpfsService { // out). Without caching the timeout, every repeat request for a dead // CID re-runs the full IPFS_KUBO_REQUEST_TIMEOUT_MS search and holds a // concurrency slot, so a single unresolvable CID under load can pin all - // slots and 502 healthy traffic. The negative cache only trips after - // repeated misses over a window, so transient cold-DHT slowness won't - // blackhole legit-but-slow content. NOT cached: IpfsUnavailableError / - // ECONNREFUSED — those mean Kubo itself is down, not the content, and - // caching them would blackhole content once Kubo recovers. + // slots and 502 healthy traffic. Timeouts trip only after repeated + // misses over the window AND are blackholed for just a short, + // non-escalating window (IPFS_TIMEOUT_NEGATIVE_CACHE_TTL_MS, ~60s), so + // recovering-but-slow content self-heals quickly rather than being + // cached-out for hours. NOT cached: IpfsUnavailableError / ECONNREFUSED + // — those mean Kubo itself is down, not the content, and caching them + // would blackhole content once Kubo recovers. if (err instanceof IpfsNotFoundError) { this.negativeCache?.recordMiss(negKey); } else if (err instanceof IpfsTimeoutError) { diff --git a/src/system.ts b/src/system.ts index 1acf42d68..37b114d7a 100644 --- a/src/system.ts +++ b/src/system.ts @@ -1922,6 +1922,9 @@ if (config.IPFS_ENABLED) { missDurationMs: config.NEGATIVE_CACHE_MISS_THRESHOLD_MS, missTrackerTtlMs: config.NEGATIVE_CACHE_MISS_TRACKER_TTL_MS, maxTtlMs: config.NEGATIVE_CACHE_MAX_TTL_MS, + // IPFS retrieval timeouts are soft misses: short, non-escalating blackhole so + // recovering-but-slow content self-heals instead of being cached-out for hours. + softMissTtlMs: config.IPFS_TIMEOUT_NEGATIVE_CACHE_TTL_MS, promotionHistoryTtlMs: config.NEGATIVE_CACHE_PROMOTION_HISTORY_TTL_MS, healthWindowMs: config.NEGATIVE_CACHE_HEALTH_WINDOW_MS, unhealthyThreshold: config.NEGATIVE_CACHE_UNHEALTHY_THRESHOLD, From d0fa6b7e716c48164be8102a64c2fe2bad640b51 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Wed, 5 Aug 2026 03:19:42 +0000 Subject: [PATCH 42/47] docs(ipfs): document timeout negative-cache TTL; reconcile observer spec to as-shipped MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add IPFS_TIMEOUT_NEGATIVE_CACHE_TTL_MS to docs/envs.md and docker-compose.yaml (the short, self-healing negative-cache TTL for IPFS retrieval timeouts). - observer-ipfs-adjustments-spec.md: prepend a "§0 As shipped" section that is now the source of truth — the design evolved through 5 adversarial passes from the original phased proposal. Captures the final model: shared assessIpfsNameTrustless on the live GatewayAssessor path, CID-based routing (not the protocol header), the PASS/FAIL/NEUTRAL scoring rules (fail only on proven-wrong bytes; availability is neutral; behavioral capability), neutral excluded from every aggregate, the IPFS_ASSESSMENT_TIMEOUT_MS deadline, and the recommended self-reference + on-demand topology. Phase 3 (leaf/DAG sampling) remains deferred. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- docker-compose.yaml | 1 + docs/drafts/observer-ipfs-adjustments-spec.md | 60 +++++++++++++++++++ docs/envs.md | 1 + 3 files changed, 62 insertions(+) diff --git a/docker-compose.yaml b/docker-compose.yaml index d0a61b650..0ea75a453 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -167,6 +167,7 @@ services: - IPFS_PIN_ARNS_CONTENT=${IPFS_PIN_ARNS_CONTENT:-} - IPFS_PIN_MAX=${IPFS_PIN_MAX:-} - IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES=${IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES:-} + - IPFS_TIMEOUT_NEGATIVE_CACHE_TTL_MS=${IPFS_TIMEOUT_NEGATIVE_CACHE_TTL_MS:-} - IPFS_CACHE_PATH=${IPFS_CACHE_PATH:-} - IPFS_CACHE_MAX_SIZE_BYTES=${IPFS_CACHE_MAX_SIZE_BYTES:-} - IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS=${IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS:-} diff --git a/docs/drafts/observer-ipfs-adjustments-spec.md b/docs/drafts/observer-ipfs-adjustments-spec.md index fae35f019..54aa20cab 100644 --- a/docs/drafts/observer-ipfs-adjustments-spec.md +++ b/docs/drafts/observer-ipfs-adjustments-spec.md @@ -19,6 +19,66 @@ Verified sources: --- +## 0. As shipped — final design (supersedes the phased proposal below) + +The plan below was implemented in `ar-io-observer` PR #112 and then hardened +through **five multi-agent adversarial-review passes**. The shipped design differs +from the original proposal in a few important ways; this section is the source of +truth, and the phased sections that follow are kept for the rationale. + +**Where it runs.** The logic lives in a shared `assessIpfsNameTrustless()` in +`observer.ts`, called by BOTH the one-shot `Observer` and the **live** +`ContinuousObserver → GatewayAssessor` path (the original proposal targeted only +`Observer`, which the running service does not use). Both paths route through the +same function so scoring cannot diverge. `REPORT_FORMAT_VERSION` is bumped 2 → 3 +(adds `protocol` + a tri-state `outcome`); on-chain submission is unchanged. + +**Routing (which names get the IPFS path).** A name is assessed via the trustless +IPFS path iff its (reference-resolved) `resolvedId` is a **valid CID** — +`isIpfsAssessable()`. We do **not** route on the `x-arns-protocol` header: it is +not part of reference consensus and older gateways may omit it. The CID form is the +ground truth (Arweave names resolve to a 43-char tx id, not a CID), and it also +blocks a poisoned reference from flipping an Arweave name onto the IPFS path. + +**Scoring (per IPFS name).** The observer fetches the target gateway's +`?format=raw` block and verifies it against the reference-bound CID: +- **PASS** — served 200 `application/vnd.ipld.raw` bytes that hash to the CID. +- **FAIL** — served 200 bytes that do NOT hash to the CID (a *proven-wrong* + answer). The fail/neutral decision never trusts the gateway's own + `x-arns-resolved-id` — a gateway controls both the bytes and that header, so a + self-minted CID proves nothing. (Consistent with the Arweave path, which also + fails on a reference `resolvedId` mismatch.) +- **NEUTRAL** (excluded from the pass/fail denominator) — anything that is not a + clean pass or a proven-wrong answer: not served / non-200 / timeout / non-raw + `Content-Type` / empty body / a multihash we can't verify (non-sha2-256). So a + gateway is **never failed for availability**, and participating in IPFS is never + riskier than abstaining. **Capability is judged behaviorally** — a non-IPFS + gateway 404s its Arweave path → neutral. (This replaces the original + self-reported `/ar-io/info` `ipfs.enabled` capability gate, which a malicious + operator could flip to exempt itself.) + +**Neutral is excluded everywhere** — `namesPass`, the ArNS metrics, the +report-selection failure rate, and the `GatewayAssessor` pass rate — via a shared +`arnsNameOutcome()`, so a neutral name can never move a gateway's on-chain result. + +**Timeouts.** The raw-block fetch has an explicit `IPFS_ASSESSMENT_TIMEOUT_MS` +(default 35s, ≥ the gateway's 30s IPFS budget) with the socket-idle timeout +overridden, and never hangs (all errors resolve as not-served → neutral). + +**Recommended reference topology (see the on-demand default, ar-io-node PR #836).** +Point the observer's reference at **its own co-located gateway** (`ARNS_ROOT_HOST`) +with that gateway resolving ArNS **on-demand** (from chain). Then the name→CID +binding is authoritative (chain-derived via your own gateway's resolver — no +re-implementation, no external-gateway or header trust) and more decentralized; +the network still aggregates observers by majority. + +**Still deferred (Phase 3).** A dag-pb/UnixFS `PASS` proves possession of the DAG +**root block** only; leaf/assembled-path verification is the IPFS block-sampling +analog of the Arweave chunk/offset proof and is not built. Contract-level items +(stratified prescription, pinning incentives) remain out of scope (§7). + +--- + ## 1. Why any change is needed (the safety problem) IPFS-target ArNS names (`AntRecord.target_protocol = 1`) are **already eligible** diff --git a/docs/envs.md b/docs/envs.md index eb7727ba0..7441ca4e3 100644 --- a/docs/envs.md +++ b/docs/envs.md @@ -640,6 +640,7 @@ as a Docker Compose sidecar via the `ipfs` profile). | IPFS_PIN_ARNS_CONTENT | Boolean | false | Pin the CIDs that ArNS names resolve to, so named content stays retrievable (read-only availability) | | IPFS_PIN_MAX | Number | 10000 | Max distinct CIDs the pinner holds this process; oldest are unpinned (FIFO) beyond this | | IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES | Number | 262144 | Rate-limit reserve for a response of unknown size (CAR / chunked); actual bytes are still bounded by IPFS_MAX_RESPONSE_SIZE_BYTES | +| IPFS_TIMEOUT_NEGATIVE_CACHE_TTL_MS | Number | 60000 | Short, non-escalating negative-cache TTL for IPFS retrieval TIMEOUTS (a CID with no live provider). Repeatedly-timing-out CIDs short-circuit for this window instead of re-searching Kubo, then self-heal — distinct from the long, escalating TTL used for definitive 404s | | IPFS_CACHE_PATH | String | data/ipfs-cache | Directory for cached IPFS content | | IPFS_CACHE_MAX_SIZE_BYTES | Number | 10737418240 (10 GB) | Maximum cache size before LRU eviction | | IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS | Number | 3600 | Age in seconds before cached files become eviction candidates | From dc733afaae94537eb213284b6ec67bc7ecf5f72c Mon Sep 17 00:00:00 2001 From: vilenarios Date: Wed, 5 Aug 2026 03:34:18 +0000 Subject: [PATCH 43/47] docs+test(ipfs): observer IPFS_ASSESSMENT_TIMEOUT_MS in compose; assert soft-miss (CodeRabbit) - docker-compose.yaml: pass IPFS_ASSESSMENT_TIMEOUT_MS through to the bundled observer service (the observer's trustless ?format=raw fetch deadline). - ipfs-service.test.ts: assert the timeout miss is recorded with { softMiss: true } so a regression to a hard miss (restoring the multi-hour blackhole) is caught. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- docker-compose.yaml | 3 +++ src/ipfs/ipfs-service.test.ts | 4 ++++ 2 files changed, 7 insertions(+) diff --git a/docker-compose.yaml b/docker-compose.yaml index 0ea75a453..565fa7f11 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -705,6 +705,9 @@ services: # the .env value — see ar-io-observer src/config.ts. - GATEWAY_ASSESSMENT_CONCURRENCY=${GATEWAY_ASSESSMENT_CONCURRENCY:-} - NAME_ASSESSMENT_CONCURRENCY=${NAME_ASSESSMENT_CONCURRENCY:-} + # Deadline for the observer's trustless IPFS `?format=raw` fetch when + # assessing IPFS-target ArNS names (>= the gateway's IPFS retrieval budget). + - IPFS_ASSESSMENT_TIMEOUT_MS=${IPFS_ASSESSMENT_TIMEOUT_MS:-} # Enable the LogReportSink — logs full per-gateway assessment detail at # info level (off by default). - ENABLE_LOG_REPORT_SINK=${ENABLE_LOG_REPORT_SINK:-} diff --git a/src/ipfs/ipfs-service.test.ts b/src/ipfs/ipfs-service.test.ts index ef0768c6a..9f288fdb5 100644 --- a/src/ipfs/ipfs-service.test.ts +++ b/src/ipfs/ipfs-service.test.ts @@ -147,6 +147,10 @@ describe('IpfsService', () => { const recordMiss = (negativeCache.recordMiss as any).mock; assert.equal(recordMiss.calls.length, 1); assert.equal(recordMiss.calls[0].arguments[0], CID); + // A timeout MUST be recorded as a soft miss (short, self-healing TTL, no + // escalation) — a regression to a hard miss would restore the multi-hour + // blackhole of recovering content. + assert.deepEqual(recordMiss.calls[0].arguments[1], { softMiss: true }); }); it('does NOT negatively cache an IpfsUnavailableError (Kubo itself down)', async () => { From 88cd8562611433214c8bed94fabceb588d14b664 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Wed, 5 Aug 2026 03:40:01 +0000 Subject: [PATCH 44/47] =?UTF-8?q?docs:=20David-alignment=20update=20?= =?UTF-8?q?=E2=80=94=20=C2=A75=20observer=20verify=20+=20on-demand=20resol?= =?UTF-8?q?ution=20shipped?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record that two of the three original call-outs are resolved (trustless ?format=raw gateway; observer verifies CID→bytes instead of trusting a reference) and the third (CAR→Arweave storage/composite-source/chain-anchored proofs) is the uploading axis, out of scope by design for read-only IPFS. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- docs/drafts/davids-brain-alignment.md | 34 +++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/docs/drafts/davids-brain-alignment.md b/docs/drafts/davids-brain-alignment.md index 2044771c3..a69c6ea02 100644 --- a/docs/drafts/davids-brain-alignment.md +++ b/docs/drafts/davids-brain-alignment.md @@ -137,3 +137,37 @@ reference gateways — remains a separate (ar-io-observer) track. > already content-agnostic). That doc makes named IPFS a first-class, trustlessly > verified dimension, which is exactly David's "verify, don't trust" posture on the > serving/observation axis, achieved now without his storage-dependent Stages 0–2. + +## Update 2 — the §5 win is now IMPLEMENTED (2026-08-05) + +Two of the three original call-outs are resolved; the third is out of scope by +design (it's the uploading axis): + +- **Call-out #1 (trustless gateway) — DONE.** `?format=raw|car` returns + client-verifiable bytes marked `X-Ar-Io-Trustless: true`; the UnixFS proxy path + is honestly marked `X-Ar-Io-Trustless: false`. This is David's Trustless-Gateway + shape and his §3 "don't imply verified." +- **§5 observer verify-don't-trust — DONE (ar-io-observer PR #112).** The observer + no longer trusts a reference gateway's bytes for IPFS names: it fetches + `?format=raw` and verifies the block against the CID's multihash. FAIL only on a + *proven-wrong* answer; availability is neutral; the gateway's self-reported + `resolvedId` is never trusted. On the IPFS axis this is now **more trustless than + the Arweave name check** (which still uses reference-digest comparison). +- **"The gateway is not a trust root" for RESOLUTION — DONE (ar-io-node PR #836).** + The default resolver is now `on-demand,gateway`: read the ANT binding from + **chain** first, hop to a trusted gateway only as a fallback. The bundled + observer references its own on-demand gateway, so the name→CID binding is + chain-derived, not oracle-trusted. + +- **Call-out #2/#3 (CAR→Arweave storage, composite-source, chain-anchored proof + headers) — out of scope by design.** These are all the *uploading/storage* axis + (David's Stages 0–2). Read-only IPFS deliberately excludes them; the parallel + Kubo stack is the *correct* architecture when there is no Arweave offset to index. + +**Net:** on every axis David cares about that does not involve uploading — trust +model, client/observer verification, protocol-independent addressing, service +boundary, decentralized (chain-authoritative) resolution — the shipped work is now +strongly aligned, and closes the two gaps he would most have flagged. The residual +divergences are entirely the storage/upload axis (deferred phase 2) plus the +inherent read-only durability trade-off (public-IPFS content can be GC'd), which is +documented and is the substrate for a future pinning/persistence incentive. From 69b2f8729f7661c9e01fe68e68d3e767cd74550d Mon Sep 17 00:00:00 2001 From: vilenarios Date: Wed, 5 Aug 2026 03:49:07 +0000 Subject: [PATCH 45/47] =?UTF-8?q?docs(drafts):=20sketch=20=E2=80=94=20AR.I?= =?UTF-8?q?O=20as=20a=20verifiable=20durability=20layer=20for=20named=20IP?= =?UTF-8?q?FS?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design sketch for gateway-to-gateway verified IPFS fetch: a gateway fetches a CID from a peer AR.IO gateway as a CAR and imports it into Kubo (which verifies blocks against the CID on import) instead of hitting public IPFS. Turns the fleet into a durable, trustless serving layer for named IPFS without Arweave storage. Key unlock: the "local-only" serve mode that powers peer-fetch also gives the observer a trustless, un-gameable PINNING signal (a 200 + verifying bytes in local-only mode proves the gateway holds the content) — closing the "X-Cache is self-asserted, can't measure pinning" gap and enabling the persistence incentive. Composes with (and steps toward) David's CAR-to-Arweave phase 2. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- docs/drafts/ipfs-peer-durability-layer.md | 165 ++++++++++++++++++++++ 1 file changed, 165 insertions(+) create mode 100644 docs/drafts/ipfs-peer-durability-layer.md diff --git a/docs/drafts/ipfs-peer-durability-layer.md b/docs/drafts/ipfs-peer-durability-layer.md new file mode 100644 index 000000000..4ea35ba0c --- /dev/null +++ b/docs/drafts/ipfs-peer-durability-layer.md @@ -0,0 +1,165 @@ +# AR.IO as a Verifiable Durability Layer for Named IPFS — Peer-Fetch Design Sketch + +> Design sketch (not a spec). Turns the AR.IO gateway fleet from a *read-only +> proxy to public IPFS* into a **durable, verifiable, decentralized serving layer** +> for named IPFS content — without uploading to Arweave. This is the "phase 1.5" +> between today's read-only proxy and David's CAR-to-Arweave permanence. + +## TL;DR + +An AR.IO gateway that needs an IPFS CID fetches it **from a peer AR.IO gateway that +already holds it**, over HTTP, and **verifies the bytes against the CID** — instead +of (or before) reaching out to the public IPFS network. Because the fetch is +content-addressed, the peer is never trusted: a wrong-bytes peer is caught by the +hash. So *any* gateway can safely serve *any* other. + +The result: **named IPFS content stays available as long as one AR.IO gateway holds +it**, independent of whether the origin public-IPFS providers are still online — and +it flows fast, over HTTP, cryptographically verified. + +## Why it matters (the value prop) + +Today read-only IPFS has an honest weakness (David and the OIP analysis both flag +it): public-IPFS content can be GC'd, so an ArNS name → IPFS CID can 404 tomorrow. +That dilutes the ArNS permanence promise. + +Peer-fetch + fleet pinning changes the durability story: + +- **Durable** — content persists as long as *any* participating gateway holds it; + the fleet is a resilient pinning cluster, not a passthrough. +- **Fast** — a peer with the content hot serves it over HTTP in ms, versus a cold + public-IPFS DHT walk (seconds, or a timeout). +- **Trustless** — served bytes are verified against the CID; no trust federation, + no allow-list. An untrusted or even malicious gateway can serve content safely. +- **Decentralized** — the value AR.IO provides (durable named content) stops + depending on the health of public IPFS. +- **Incentivizable** — and this is the key unlock below: it gives the OIP a + *trustless way to measure pinning*, which read-only IPFS was missing. + +This is a materially stronger value prop than "proxy to public IPFS": it's +durability approaching Arweave's promise, delivered by fleet replication rather than +Arweave storage. + +## Mechanism + +Everything needed already exists in the gateway; this composes it. + +1. Gateway **B** needs CID `X` (miss in its local cache; its own Kubo can't find it + or public IPFS timed out). +2. B asks one or more **peer AR.IO gateways** for the content as a CAR: + `GET https://{peer}/ipfs/{X}?format=car` with a **local-only** hint (below). +3. A peer **A** that *holds* `X` returns the CAR (the full DAG). We already serve + `?format=car` (`routes/ipfs.ts` → `KuboDataSource` `format: 'car'`). +4. B imports the CAR into its own Kubo via the RPC we already use + (`{IPFS_KUBO_API_URL}/api/v0/dag/import`, same path pattern as `pin/add`). + **Kubo verifies every block against its CID on import** — so a lying peer's CAR + fails to import and B moves to the next peer. No bespoke DAG-verify code needed; + Kubo is the verifier. +5. B now holds `X` (verified, in Kubo), serves the user, and — if + `IPFS_PIN_ARNS_CONTENT` — pins it. The content has replicated one more time. + +For a raw single-block CID this is trivially one block; for a UnixFS/dag-pb DAG the +CAR carries the whole graph, and Kubo's import verifies the block links. That also +sidesteps the multi-block trust gap (the reassembled UnixFS bytes don't hash to the +CID, but the CAR's blocks do). + +### The local-only serve mode (the load-bearing primitive) + +A peer request must **not** recurse — if B asks A and A doesn't have it, A must NOT +turn around and hit public IPFS or its own peers (latency + loops + amplification). +So peer requests carry a header, e.g. `X-Ar-Io-Local-Only: true` (or a dedicated +peer endpoint), and A serves **only from its local cache / Kubo pin**, returning +404 fast if it doesn't hold the content. This bounds the work and prevents fetch +loops across the fleet. + +### Where it slots architecturally + +IPFS currently has a single source (`KuboDataSource` → local Kubo → public IPFS). +Introduce a small **IPFS composite** mirroring the Arweave `SequentialDataSource` +pattern: + +``` +local cache → peer AR.IO gateways (verified CAR import) → local Kubo → public IPFS +``` + +Ordering is a tuning choice. A defensible v1: try a **bounded, short peer attempt** +(2–3 peers, short deadline) before the public-IPFS fallback, so hot content comes +from the fleet fast and cold/dropped content still falls through to public IPFS. + +## The killer tie-in: trustless pinning measurement → the persistence incentive + +The earlier OIP analysis had a hard blocker: *"`X-Cache` is gateway-asserted; the +observer can't distinguish a gateway serving from its own pin vs. fronting someone +else's."* The **local-only serve mode fixes exactly that**: + +> The observer requests `?format=raw|car` with `X-Ar-Io-Local-Only: true` and +> verifies the bytes against the CID. A 200 + verifying bytes **proves the gateway +> holds the content locally** — it served without reaching public IPFS. A gateway +> that only proxies returns 404 in local-only mode. + +So the same primitive that powers peer-fetch also gives the observer a **trustless, +un-gameable signal for pinning/holding** — which is precisely what a +persistence/pinning reward category needs. That closes the loop the earlier analysis +left open: + +- Observer already verifies a gateway **serves** a CID correctly (shipped). +- Local-only + CID verification lets it verify a gateway **holds** a CID. +- The OIP can then reward **holding named IPFS content** (a new reward category), + measured trustlessly. Combined with Phase-3 leaf sampling (verify random leaf + blocks, not just the root), a gateway can't game it by pinning only the tiny root. + +Peer-fetch + local-only + pinning incentive = a **decentralized, incentivized, +verifiable durability layer for named IPFS**, with zero Arweave storage. + +## Peer discovery + +Who does B ask, and how does it find who *holds* `X`? + +- **v1 (simple):** the on-chain **GAR** (gateway address registry — the gateway + already reads it; `config.ts` "in the GAR … a simple lookup") gives the peer set. + B tries a bounded random/weighted subset. Cheap, no new infra; wasteful only in + that it may ask peers that don't hold `X` (bounded by the local-only 404 fast + path). +- **v2 (routed):** a who-holds-what hint so B asks the *right* peers. Options: + reuse IPFS DHT provider records filtered to AR.IO gateways; or a lightweight + AR.IO content-routing index (gateways announce the named CIDs they pin). This is + the main scaling question and can come later. + +## Alignment with David's vision + +- **Durability without Arweave storage.** David's durability comes from CAR→Arweave + (phase 2). This delivers durability from *fleet replication + pinning* now — a + different substrate, same goal (named content persists). The two **compose**: a + gateway can fleet-pin today and permapin to Arweave in phase 2. +- **Verify, don't trust (§5).** Peer content is CID-verified (Kubo import). The + gateway remains "cache + protocol translator," never a trust root — now also a + *verifiable peer content router*. +- It's a genuine stepping stone toward his Stage 1 (CAR ingest): the CAR-transfer + + import machinery here is the same shape as CAR-to-Arweave, minus the Arweave sink. + +## Risks / open questions + +- **Recursion / amplification** — mitigated by the local-only hint (no fan-out). +- **DoS on peer-serving** — rate-limit local-only peer requests; they're cheap + (no public-IPFS recursion) but still need bounding. +- **CAR size** — a large file's CAR is large; cap the peer-import size and fall + back to public IPFS above the cap (or stream+verify in bounded chunks). +- **Binding vs. content trust** — peer-fetch is for *content* (CID→bytes, + trustless). The name→CID *binding* still comes from chain (on-demand resolution). + The two are independent and compose. +- **Cold-start / who-holds-what** — v1's blind-subset ask is fine at small fleet + size; routing (v2) matters as it grows. +- **Incentive gaming** — pinning only the root block; addressed by Phase-3 leaf + sampling on the measurement side. + +## Phasing + +- **1.5a — peer-fetch fallback source** (gateway): IPFS composite + local-only serve + mode + CAR-import-verify from a GAR-subset of peers, as a fallback when public + IPFS misses. Durability with minimal new infra. No contract change. +- **1.5b — content routing** (gateway/network): efficient who-holds-what so peers + are asked precisely rather than blindly. +- **1.5c — pinning incentive** (observer + contract): observer measures holding via + local-only + CID verification (+ leaf sampling); OIP adds a reward category for + holding named IPFS. This is the contract-level piece. +- **2 — CAR → Arweave** (David): true permanence, composed on top of the above. From 53c7ba63d74cf9209d99422d3e33f8143a80c140 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Wed, 5 Aug 2026 03:54:46 +0000 Subject: [PATCH 46/47] =?UTF-8?q?docs(drafts):=20deepen=20peer-durability?= =?UTF-8?q?=20design=20=E2=80=94=20holistic=20lifecycle=20+=20zero-contrac?= =?UTF-8?q?t=20incentive=20+=201.5a=20plan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add the end-to-end lifecycle (chain binding → peer replication → trustless serve → observer holding-probe → reward) as one self-reinforcing flywheel. - Incentive integration: the WHOLE layer ships with zero contract changes. Gateways are already rewarded for serving what ArNS points to; rewarding HOLDING folds into the existing name assessment via a local-only observer probe (no new on-chain field). Serving→holding is a ramped policy choice, not a contract choice; only a dedicated holding-weight-beyond-sampled-names touches the contract. - Content routing in depth (DHT-filtered, named-holdings announce, deterministic assignment) and a concrete, no-contract 1.5a implementation plan (local-only serve mode, IpfsPeerDataSource, IPFS composite, CAR+Kubo-import verification). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- docs/drafts/ipfs-peer-durability-layer.md | 150 ++++++++++++++++++++++ 1 file changed, 150 insertions(+) diff --git a/docs/drafts/ipfs-peer-durability-layer.md b/docs/drafts/ipfs-peer-durability-layer.md index 4ea35ba0c..9fa4ada94 100644 --- a/docs/drafts/ipfs-peer-durability-layer.md +++ b/docs/drafts/ipfs-peer-durability-layer.md @@ -163,3 +163,153 @@ Who does B ask, and how does it find who *holds* `X`? local-only + CID verification (+ leaf sampling); OIP adds a reward category for holding named IPFS. This is the contract-level piece. - **2 — CAR → Arweave** (David): true permanence, composed on top of the above. + +--- + +## The holistic lifecycle (gateway + observer + incentive as one system) + +The three systems compose into a single flywheel for a named IPFS name `foo → CID X`: + +``` + register replicate serve verify holding reward + ──────── ───────── ───── ────────────── ────── + ANT record gateways that opt to any gateway serves observer probes each OIP distributes to + foo → X, hold X pull it from X: local pin → peer gateway local-only, gateways proven to be + protocol=ipfs peers (CAR+verify) and gateways (verified) verifies bytes vs X, HOLDING named CIDs + (on-chain) pin it → public IPFS samples leaves (new reward category) + │ │ │ │ │ + └── chain is ───┘ └── trustless ──────────┘ │ + the binding (CID = proof) │ + ▼ + more gateways hold X ◄─── incentive pulls replication ┘ +``` + +Two properties make it self-reinforcing: + +1. **Everything is content-addressed, so nothing needs trust.** The binding (`foo→X`) + is chain-authoritative (on-demand resolution). The bytes are CID-verified + (Kubo import, observer probe). No gateway — serving, peering, or being observed — + is ever a trust root. +2. **The reward pulls replication.** Rewarding *holding* named content makes gateways + want to pin it; peer-fetch makes acquiring it cheap and verified; the observer + makes holding measurable. Durability rises with participation, without anyone + uploading to Arweave. + +## Incentive integration (OIP) in detail + +**The whole durability layer can ship with ZERO smart-contract changes.** Gateways +are *already* incentivized to serve whatever ArNS points to: the observer assesses +prescribed + chosen names and rewards passing gateways through the existing +distribution, and that machinery is content-agnostic (the chain only ever gets a +per-gateway pass/fail bitmap). So: + +- **Peer-fetch (1.5a)** is a gateway-side change to *how* content is acquired — + invisible to the contract. +- **Rewarding holding** folds into the EXISTING name assessment: make the observer + probe IPFS names **local-only**, so a gateway passes a name only if it actually + *holds* the content. Holding is then rewarded exactly like passing any name — no + new on-chain field, no `ario-gar` change. + +The one subtlety is a **policy choice, not a contract choice**: today's assessment +rewards *serving* (a proxy passes while public IPFS still has the content), whereas +local-only probing rewards *holding* (a proxy returns 404 and fails). Moving from +serving→holding is the durability lever, and it should be **ramped** like the +IPFS-capability ramp (neutral for not-yet-holding gateways during rollout) so honest +gateways aren't abruptly failed. Both modes are zero-contract. + +The **only** thing that would touch the contract is rewarding holding *beyond the +sampled names* — a dedicated holding weight (option (b) below). That's optional and +future; the primary path needs no contract change. + +**What the observer measures (trustless, un-gameable):** +- **Holds-it:** `GET {name}.{gw}/?format=raw` (or `?format=car`) with + `X-Ar-Io-Local-Only: true` → 200 + bytes that verify against `X` ⇒ the gateway + holds `X` locally (it served without touching public IPFS). A proxy returns 404. +- **Holds the whole thing (not just the root):** for a UnixFS DAG, sample K random + **leaf** CIDs from the DAG (reachable via `?format=car` / the root's links) and + local-only-verify each — the IPFS analog of the Arweave chunk/offset proof + (Phase 3). This stops "pin the tiny root, collect the reward." + +**What reaches chain (minimal):** the existing pass/fail bitmap is content-agnostic +and already covers *serving*. A **holding** reward needs one new signal — e.g. a +per-gateway "held-set" measure the contract can reward. Design options, cheapest +first: +- **(a) Fold into the name score.** If "holding a prescribed/chosen IPFS name" is + simply part of passing that name, no new on-chain field is needed — holding is + rewarded through the existing distribution (like any name). Simplest; ties holding + to the sampled names only. +- **(b) A dedicated holding weight.** A new per-gateway scalar (count/bytes of named + CIDs proven held) added to the weight computation — a real `ario-gar` change and + an `Epoch`/report-shape addition. More expressive (rewards holding beyond the + sampled set) but heavier; a governance decision. + +Recommend starting with **(a)** — it needs no contract change and still creates the +pull toward replication, then graduating to **(b)** if the network wants to reward +holding at scale beyond the sampled names. + +**Gaming resistance:** holding is proven by CID verification (can't fake bytes); +leaf sampling stops root-only pinning; local-only stops "front someone else's copy" +(a proxy fails the local-only probe); and the reward is bounded by the same +prescribed/chosen sampling and >½-observer majority the rest of the protocol uses. + +## Content routing in depth (the real scaling question) + +Blind-asking a GAR subset is fine at small fleet size (bounded by the local-only +fast-404), but doesn't scale. The who-holds-what problem has three tractable answers, +usable in combination: + +1. **Reuse the IPFS DHT, filtered.** Gateways that pin already advertise as + providers. A gateway resolves providers for `X` and prefers those whose peer-ids + map to AR.IO gateways (GAR-registered), then fetches via the trustless HTTP + endpoint (faster/verified) rather than Bitswap. Zero new infra; leans on IPFS's + own routing. +2. **Announce named holdings.** Because holdings are *named* (ArNS), the set is + small and enumerable. A gateway can publish "I hold {CIDs} for {names}" — via a + lightweight signed announce, or simply exposed at a well-known endpoint + (`/ar-io/ipfs/held`) that peers/observers scrape. The observer already visits + every gateway; it can build a fleet-wide holdings map as a byproduct of + assessment and expose it as a routing hint. +3. **Deterministic assignment (later).** Rendezvous-hash named CIDs to a subset of + gateways so replication is planned, not incidental — the network can guarantee N + replicas per name. This is the strongest durability guarantee and the most work; + it composes with the incentive (reward the assigned holders). + +Start with (1)+(2): DHT-filtered discovery plus an observer-built holdings hint. + +## Concrete 1.5a implementation plan (gateway, no contract change) + +The minimal shippable slice — peer-fetch as a verified fallback source: + +**New / changed components (ar-io-node):** +- **Local-only serve mode.** `routes/ipfs.ts`: honor `X-Ar-Io-Local-Only: true` (or + `?local=1`) — resolve/serve **only** from the local cache + Kubo pin, never from + public IPFS. Return 404 fast on a miss. This is the load-bearing primitive (peers + and the observer both use it). +- **`IpfsPeerDataSource`** (new, mirrors `KuboDataSource`'s interface): given a CID + + a peer list, `GET https://{peer}/ipfs/{CID}?format=car` local-only from a bounded, + short-deadline subset; on the first 200, `POST {IPFS_KUBO_API_URL}/api/v0/dag/import` + (same RPC path as `pin/add`) — **Kubo verifies blocks against the CID on import**; + reject+next-peer on import/verify failure. Returns the now-local content. +- **IPFS composite source.** Wrap `[localCache, IpfsPeerDataSource, KuboDataSource]` + in a sequential source (mirroring the Arweave `SequentialDataSource`); `IpfsService` + consumes the composite instead of `KuboDataSource` directly. +- **Peer set:** the GAR (already read by the node) → a bounded random/weighted subset; + DHT-filtered discovery is a follow-up (routing §1). + +**Config (new):** +- `IPFS_PEER_FETCH_ENABLED` (default false initially, then true once proven). +- `IPFS_PEER_FETCH_COUNT` (peers to try, e.g. 3), `IPFS_PEER_FETCH_TIMEOUT_MS` + (short), `IPFS_PEER_FETCH_MAX_CAR_BYTES` (cap; fall back to public IPFS above it). + +**Safety (reuses hardening we already did):** the peer fetch is bounded (deadline + +size cap), never recurses (local-only), verified (Kubo import), and rate-limited on +the serve side; a failing/lying peer is skipped, not trusted. Content-blocking/ +moderation applies to imported content exactly as to Kubo-fetched content. + +**Testable trustlessly end-to-end:** two gateways, one holding CID X; the other +peer-fetches X, imports+verifies, serves it — with X's public-IPFS providers offline, +proving fleet-durability independent of public IPFS. Byte-tamper a peer's CAR → import +fails → next peer. + +1.5a ships value on its own (fleet durability + faster serving) with **no contract +change**; 1.5b (routing) and 1.5c (incentive) build on the same local-only primitive. From a5e2ed076a68dc9352b5559ab411ab6fddc17c68 Mon Sep 17 00:00:00 2001 From: vilenarios Date: Wed, 5 Aug 2026 04:16:42 +0000 Subject: [PATCH 47/47] docs(drafts): concrete 1.5a+1.5b peer-fetch implementation plan Implementation-ready plan grounded in the current code: local-only serve mode, IpfsPeerDataSource (CAR + Kubo dag/import verify), SequentialIpfsSource composite, GAR-subset peer selection, config, and a testcontainers multi-node integration harness (2-3 gateways pulling verified content from each other, tamper rejection, public-IPFS-independent durability). 1.5c dropped: holding measurement rides on the local-only primitive with zero contract change. No smart-contract changes. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01QPmZYvBxMyHWxrTL85xFr7 --- ...pfs-peer-fetch-1.5a-implementation-plan.md | 523 ++++++++++++++++++ 1 file changed, 523 insertions(+) create mode 100644 docs/drafts/ipfs-peer-fetch-1.5a-implementation-plan.md diff --git a/docs/drafts/ipfs-peer-fetch-1.5a-implementation-plan.md b/docs/drafts/ipfs-peer-fetch-1.5a-implementation-plan.md new file mode 100644 index 000000000..dc0668521 --- /dev/null +++ b/docs/drafts/ipfs-peer-fetch-1.5a-implementation-plan.md @@ -0,0 +1,523 @@ +# Peer-Fetch Durability Layer — Implementation Plan (Phase 1.5a + 1.5b) + +> **Status:** implementation-ready plan. Companion to the design sketch +> `ipfs-peer-durability-layer.md` (read that first for the *why*). This doc is the +> *how*: concrete files, signatures, config, and a multi-node integration-test +> harness. Written to be handed to a fresh context that will implement it. +> +> **Scope decision (this plan):** implement **1.5a** (gateway peer-fetch + local-only +> serve mode) and **1.5b** (content routing). **1.5c is dropped as a phase** — see +> "Why 1.5c collapses" below. **No smart-contract change anywhere in this plan.** +> +> **Repo:** `ar-io-node`, branch `feat/arns-ipfs-protocol` (worktree +> `wt/ipfs-sync-793`). All file paths below are real and current as of this writing. + +--- + +## 0. Why 1.5c collapses (and this plan is a and b only) + +The original phasing had `1.5c — pinning incentive (observer + contract)`. It's not +needed as a distinct phase: + +- **Measuring holding needs no contract and no new phase.** The load-bearing + primitive in 1.5a — the **local-only serve mode** — *is* the trustless + holding-measurement. The observer already fetches `?format=raw` and verifies bytes + against the CID (shipped, PR #112). Adding `X-Ar-Io-Local-Only: true` to that + existing probe means a 200+verifying response *proves the gateway holds the content + locally* (a proxy 404s in local-only mode). That is a ~5-line observer change that + **rides on 1.5a**, not a separate contract phase. +- **Rewarding holding needs no contract.** It folds into the *existing* + content-agnostic name assessment: if the observer probes IPFS names local-only, a + gateway passes the name only if it actually holds the content — and passing names is + already rewarded through the existing distribution. Serving→holding is a **policy + ramp**, not a contract change. +- **The only contract-touching option** (a dedicated per-gateway "holding weight" + beyond the sampled names) is explicitly **optional/future/governance** — not in this + plan. + +So: build 1.5a (gateway) and 1.5b (routing). The observer holding-probe is a small +no-contract rider documented in §7, to be done alongside/after 1.5a on the observer +repo. Nothing here changes `ario-gar` or any Solana program. + +--- + +## 1. Objective + +Turn the gateway from a read-only proxy to public IPFS into a **verifiable fleet +serving layer**: when a gateway needs a CID it doesn't hold, it fetches it from a peer +AR.IO gateway that *does* hold it (as a CAR, over HTTP), and Kubo verifies every block +against the CID on import. Named IPFS content then stays available as long as **any** +participating gateway holds it — independent of public-IPFS provider health — and is +served fast and trustlessly. + +**Serving order** the composite realizes (IpfsService already handles the on-disk +cache tier above these): + +``` +1. local Kubo (offline) — do I already hold it? fast, no network +2. peer AR.IO gateways — does a fleet peer hold it? bounded, CAR+verify import +3. Kubo (public IPFS) — public DHT fallback existing behavior +``` + +Local-only requests (`X-Ar-Io-Local-Only: true`) run **tier 1 only** — never peer, +never public. This is what prevents peer-fetch recursion/amplification and what makes +holding trustlessly measurable. + +--- + +## 2. Architecture & the shape we mirror + +The IPFS stack does **not** use the Arweave `ContiguousDataSource`/`getData` interface. +It has its own source shape on `KuboDataSource`: + +```ts +// src/ipfs/kubo-data-source.ts +getContent(opts: { + cidString: string; path?: string; signal?: AbortSignal; parentSpan?: Span; + range?: string; format?: 'raw' | 'car'; +}): Promise // { stream, size, contentType, statusCode, contentRange? } +``` + +We mirror the **fall-through structure** of `src/data/sequential-data-source.ts` +(iterate ordered sources, return on first success, continue on recoverable error, +short-circuit on client-abort) but against the `getContent` shape, not `getData`. + +New abstraction: + +```ts +// src/ipfs/ipfs-content-source.ts (new) +export interface IpfsContentSource { + getContent(opts: { + cidString: string; path?: string; signal?: AbortSignal; parentSpan?: Span; + range?: string; format?: 'raw' | 'car'; localOnly?: boolean; // ← new field + }): Promise; +} +``` + +`KuboDataSource` and the new `IpfsPeerDataSource` both implement `IpfsContentSource`. +`SequentialIpfsSource` composes them. `IpfsService.dataSource` is widened from the +concrete `KuboDataSource` type to `IpfsContentSource` (see §3.6). + +--- + +## 3. Phase 1.5a — gateway work breakdown + +Ordered so each step is independently testable. Every step lists the exact file and +insertion point. + +### 3.1 Add `localOnly` to Kubo fetching (the tier-1 / local-only primitive) + +**File:** `src/ipfs/kubo-data-source.ts` + +- Extend `getContent`'s options with `localOnly?: boolean`. +- When `localOnly === true`, the fetch must be answered **only from Kubo's local + blockstore/pinset** — it must NOT trigger a public-IPFS/DHT walk. + - **Recommended mechanism (verify in a short spike — see §3.1a):** issue the fetch + against the **Kubo RPC API** (`IPFS_KUBO_API_URL`, `http://kubo:5001`) with + `offline=true`, rather than the read-only gateway (`IPFS_KUBO_URL`, :8080) which + has no per-request offline flag. Concretely: + - `format: 'car'` → `POST {api}/api/v0/dag/export?arg={cid}` (offline daemon + semantics) or `block`/`dag` reads with `&offline=true`. + - `format: 'raw'` → `POST {api}/api/v0/block/get?arg={cid}&offline=true`. + - no format (UnixFS bytes) → `POST {api}/api/v0/cat?arg={cid}&offline=true` + (+ path). Kubo returns an error fast if the blocks aren't held locally → map to + `IpfsNotFoundError`. +- Non-local-only path is unchanged (keeps hitting the :8080 gateway, which may reach + public IPFS). + +> **3.1a — REQUIRED SPIKE (do this first, ~half day):** confirm the exact Kubo +> v0.32.1 mechanism for a *per-request, provably-offline* read that returns fast on a +> local miss. Candidates in priority order: (1) RPC `offline=true` query param on +> `block/get`,`dag/export`,`cat`; (2) daemon-global `--offline` on a *second* Kubo, if +> per-request proves unreliable; (3) a pre-check via `POST /api/v0/block/stat?arg={cid} +> &offline=true` (or `pin/ls?arg={cid}&type=recursive`) before serving. The correctness +> of the whole feature (recursion prevention + holding measurement) rests on +> local-only being genuinely offline, so nail this before building on it. Kubo v0.32.1 +> is pinned in `docker-compose.yaml` (`ipfs/kubo:v0.32.1`, profile `ipfs`). + +### 3.2 Honor `X-Ar-Io-Local-Only` on the IPFS route + +**File:** `src/routes/ipfs.ts` — `handleIpfsRequest` (~L190), just before the +`ipfsService.getContent(...)` call (~L234). + +- Parse the request header: `const localOnly = req.headers['x-ar-io-local-only'] === + 'true';` (also accept `?local=1` for convenience/testing). +- Thread it into the call: `ipfsService.getContent({ cidString, path, signal: + req.signal, range: format ? undefined : rangeForKubo, format, localOnly })`. +- On a local miss the existing `IpfsNotFoundError → 404` branch (L449–463) applies — + fast 404, no fallback. No new response-status code needed. +- Optionally set a response marker on local-only hits (`X-Ar-Io-Local-Only: true`) so + the observer can assert the server honored the mode. Add the constant to + `src/constants.ts` `headerNames` if you want it symmetric with `arIoSource`. + +**File:** `src/ipfs/ipfs-service.ts` — `getContent` (L84). Add `localOnly?: boolean` to +its options and pass through to `this.dataSource.getContent({...})` (L226). Keep the +existing cache tier: when `localOnly` and the on-disk `IpfsFsCache` already has it +(the `range===undefined && format===undefined` case, L177), that's a legitimate local +hit — serve it. Otherwise delegate to the composite (which, for local-only, runs tier +1 only; see §3.5). + +### 3.3 `IpfsPeerDataSource` (tier 2 — the new source) + +**File:** `src/ipfs/ipfs-peer-data-source.ts` (new). Implements `IpfsContentSource`. + +**Constructor:** +```ts +new IpfsPeerDataSource({ + log, + peerManager: ArIOPeerManager, // the arIOPeerManager singleton (system.ts L795) + kuboApiUrl: string, // config.IPFS_KUBO_API_URL — for dag/import + re-read + kuboDataSource: KuboDataSource, // to re-serve offline after import + peerCount: number, // IPFS_PEER_FETCH_COUNT + requestTimeoutMs: number, // IPFS_PEER_FETCH_TIMEOUT_MS + maxCarBytes: number, // IPFS_PEER_FETCH_MAX_CAR_BYTES + pinner?: IpfsPinner, // pin after import if IPFS_PIN_ARNS_CONTENT +}) +``` + +**`getContent({ cidString, path, signal, format, range, localOnly })`:** +1. **If `localOnly === true`, immediately throw `IpfsNotFoundError`.** A peer source + must never run under local-only (that's tier-1-only). This is the recursion guard + at the source level (belt-and-suspenders with §3.5's composite gating). +2. Select peers: `peerManager.selectPeersForKey('ipfs', cidString, this.peerCount)` + (hash-ring gives cache locality — the same CID tends to hit the same peers, which + also warms them). Register `'ipfs'` as a `WeightCategory`. Fall back to + `selectPeers('ipfs', count)` if key-selection returns empty. +3. For each peer (bounded by `peerCount`, overall deadline `requestTimeoutMs`): + a. `GET {peer}/ipfs/{cidString}?format=car` with header + `X-Ar-Io-Local-Only: true`, a per-peer timeout, and a **byte cap** + (`maxCarBytes`; abort + skip if exceeded — a large file falls through to public + IPFS at tier 3). + b. On non-200 / timeout / cap-exceeded: `peerManager.reportFailure('ipfs', peerId)`, + next peer. + c. On 200: stream the CAR body directly into + `POST {kuboApiUrl}/api/v0/dag/import?pin-roots={IPFS_PIN_ARNS_CONTENT}` as + multipart/form-data (`file` field). **Kubo verifies every block against its CID + on import** — a tampered/lying CAR fails to import. On import error: + `reportFailure`, next peer. + d. On import success: `peerManager.reportSuccess('ipfs', peerId)`. The content is + now in local Kubo. Serve it by delegating to + `this.kuboDataSource.getContent({ cidString, path, format, range, localOnly:true })` + (now a local hit) and return that `IpfsContentResult`. Pin via `this.pinner?.pin` + if not covered by `pin-roots`. +4. All peers exhausted → throw `IpfsNotFoundError` (composite falls through to tier 3). + +**Notes:** +- Use `got`/`axios` streaming (mirror `KuboDataSource`'s HTTP client + the multipart + pattern is new — `dag/import` does not exist yet; `IpfsPinner.rpc` L93 is the closest + existing RPC call and shows the `{apiUrl}/api/v0/...` convention). A multipart CAR + upload is required (`form-data` or `got`'s form support). +- Always fetch `?format=car` from peers regardless of the client's requested format: + the CAR carries the whole DAG so Kubo can verify block links, and it sidesteps the + multi-block UnixFS trust gap. After import we re-derive the client's requested + format locally in step 3d. +- Respect `signal` (client disconnect) — abort in-flight peer fetches. + +### 3.4 `SequentialIpfsSource` (tier composition) + +**File:** `src/ipfs/sequential-ipfs-source.ts` (new). Implements `IpfsContentSource`. +Mirror `src/data/sequential-data-source.ts` fall-through logic: + +- Constructor: `{ log, sources: IpfsContentSource[] }`. +- `getContent(opts)`: iterate `sources` in order; return the first success. +- **Fall-through policy** (differs from Arweave's — IPFS has moderation semantics): + - `IpfsNotFoundError`, `IpfsTimeoutError`, `IpfsUnavailableError` → log warn, try + next source. + - `AbortError` with `signal.aborted` (genuine client disconnect) → **re-throw** + (short-circuit), same as `SequentialDataSource` L149. + - `IpfsBlockedError` (451, moderation) → **re-throw immediately** (do NOT fall + through — a blocked CID must stay blocked across all tiers). + - `IpfsSizeLimitError` / `IpfsRangeNotSatisfiableError` → re-throw (not a + availability miss). +- After all sources exhausted → throw `IpfsNotFoundError` (route maps to 404). + +### 3.5 Local-only gating in the composite + +`SequentialIpfsSource.getContent` must run **only tier 1** when `opts.localOnly`: + +- Simplest: gate at composition — when `localOnly`, only call the first source + (`KuboDataSource` in offline mode) and skip the rest. Two clean options: + 1. `SequentialIpfsSource` checks `opts.localOnly` and iterates only `sources[0]`; or + 2. each source self-guards (peer source already throws under local-only per §3.3.1; + the tier-3 public Kubo source would need the same guard). +- Recommended: **both** — composite iterates tier 1 only under local-only, *and* the + peer source self-guards. Defense in depth; the self-guard also protects any future + caller that bypasses the composite. + +### 3.6 Wire the composite in `system.ts` + +**File:** `src/system.ts`, IPFS block L1880–1961 (guarded by `config.IPFS_ENABLED`). + +- `kuboDataSource` already built at L1894. `arIOPeerManager` already in scope + (constructed L795). +- Build (only when `config.IPFS_PEER_FETCH_ENABLED`): + ```ts + const ipfsPeerDataSource = new IpfsPeerDataSource({ + log, peerManager: arIOPeerManager, kuboApiUrl: config.IPFS_KUBO_API_URL, + kuboDataSource, peerCount: config.IPFS_PEER_FETCH_COUNT, + requestTimeoutMs: config.IPFS_PEER_FETCH_TIMEOUT_MS, + maxCarBytes: config.IPFS_PEER_FETCH_MAX_CAR_BYTES, pinner: ipfsPinner, + }); + const ipfsCompositeSource = new SequentialIpfsSource({ + log, + sources: config.IPFS_PEER_FETCH_ENABLED + ? [kuboDataSource /*tier1 offline via localOnly*/, ipfsPeerDataSource, kuboDataSource /*tier3 public*/] + : [kuboDataSource], + }); + ``` + > Tier 1 and tier 3 are the *same* `KuboDataSource` instance; tier 1 is reached with + > `localOnly:true` and tier 3 without. Because `SequentialIpfsSource` under + > `localOnly` runs only `sources[0]`, and under normal mode runs all three, the + > cleanest encoding is a small wrapper that fixes `localOnly` per tier — e.g. + > `new LocalOnlyKubo(kuboDataSource)` for tier 1 (forces `localOnly:true`) and the + > raw `kuboDataSource` for tier 3. Add that 10-line wrapper in + > `sequential-ipfs-source.ts` or inline. This avoids the composite having to know + > which index is "the offline one." +- Change `IpfsService` construction (L1937): `dataSource: ipfsCompositeSource` and + widen `IpfsService`'s `dataSource` field type (ipfs-service.ts L46) from + `KuboDataSource` to `IpfsContentSource`. +- When `IPFS_PEER_FETCH_ENABLED` is false, the composite is a 1-element passthrough → + **zero behavior change** (safe default; this is how it ships dark). + +### 3.7 Config + +**File:** `src/config.ts`, IPFS block L3205–3321. Add (use `env.varOrDefault` / +`env.positiveIntOrDefault` like the neighbors): + +| Var | Default | Purpose | +|---|---|---| +| `IPFS_PEER_FETCH_ENABLED` | `false` | Master switch (ship dark, enable after the multi-node test passes). | +| `IPFS_PEER_FETCH_COUNT` | `3` | Peers to try per CID. | +| `IPFS_PEER_FETCH_TIMEOUT_MS` | `5000` | Overall peer-attempt deadline (short — public IPFS is the patient fallback). | +| `IPFS_PEER_FETCH_MAX_CAR_BYTES` | `104857600` (100 MB) | Cap; above this skip peers → public IPFS. | +| `IPFS_PEER_SERVE_LOCAL_ONLY_RATE_*` | reuse ipfs limiter | Bound inbound local-only peer-serve load (see §3.8). | + +**File:** `docker-compose.yaml` — plumb all new vars to the `core` service (the block +at L160–178 where the other 18 `IPFS_*` vars are passed), same `${VAR:-}` pattern. + +### 3.8 Serve-side hardening (inbound local-only peer requests) + +A gateway now receives `X-Ar-Io-Local-Only: true` requests from peers. These are cheap +(no public recursion) but must be bounded: +- Reuse the existing `ipfsRateLimiter` (system.ts L1946). Consider a distinct bucket + or a lighter limit for local-only requests since they're strictly cheaper than full + fetches but higher-frequency (fleet chatter). +- Local-only requests must **bypass payment/402** if any (they're intra-fleet); check + `paymentProcessor` wiring in `createIpfsRouter` (L56) — likely gate payment on + `!localOnly`. +- Moderation (`blockListValidator` / `IpfsBlockedError`) applies identically to + local-only and imported content. A blocked CID is blocked whether served, imported, + or probed. + +### 3.9 Metrics / observability + +Add Prometheus counters (mirror existing IPFS metrics): peer-fetch attempts, successes, +failures-by-reason (timeout / non-200 / import-verify-fail / cap-exceeded), CAR bytes +imported, local-only serve hits/misses. These are the signals that tell operators the +fleet layer is working and feed 1.5b routing decisions. + +--- + +## 4. Phase 1.5b — content routing (who-holds-what) + +Blind-asking a GAR subset is fine at small fleet size (bounded by the local-only +fast-404) but doesn't scale. Build on 1.5a's primitive: + +- **(1) DHT-filtered discovery.** Resolve IPFS providers for `X`, prefer peer-ids that + map to GAR-registered AR.IO gateways, fetch from them via the trustless HTTP CAR + endpoint (faster + verified) instead of Bitswap. Zero new infra. +- **(2) Announced holdings.** Because holdings are *named* (ArNS), the set is small and + enumerable. Expose a well-known endpoint `GET /ar-io/ipfs/held` listing the + named CIDs a gateway holds (locally verified). Peers/observers scrape it; the + peer-selection in §3.3.2 consults this hint instead of blind selection. The observer + already visits every gateway (see §7) so it can build a fleet-wide holdings map as a + byproduct and serve it as a routing hint. +- **(3) Deterministic assignment (later).** Rendezvous-hash named CIDs to a gateway + subset for planned N-replica durability. Strongest guarantee, most work; composes + with the incentive. Out of scope for the first cut. + +**First 1.5b cut = (1) + (2):** add `/ar-io/ipfs/held` + have `IpfsPeerDataSource` +prefer hint-matched peers, falling back to `selectPeersForKey`. This is additive to +1.5a and needs no interface changes. + +--- + +## 5. File-change checklist (implementer TL;DR) + +**New files:** +- `src/ipfs/ipfs-content-source.ts` — `IpfsContentSource` interface. +- `src/ipfs/ipfs-peer-data-source.ts` — tier-2 source (+ its `.test.ts`). +- `src/ipfs/sequential-ipfs-source.ts` — composite + local-only gating + `LocalOnlyKubo` wrapper (+ `.test.ts`). +- `test/end-to-end/ipfs-peer-fetch.test.ts` — the multi-node harness (§6). +- (1.5b) `src/routes/ar-io.ts` addition or new handler for `GET /ar-io/ipfs/held`. + +**Changed files:** +- `src/ipfs/kubo-data-source.ts` — `localOnly` option + offline RPC path (§3.1); implement `IpfsContentSource`. +- `src/ipfs/ipfs-service.ts` — `localOnly` passthrough; widen `dataSource` type to `IpfsContentSource` (L46). +- `src/routes/ipfs.ts` — parse `X-Ar-Io-Local-Only` (~L234); gate payment on `!localOnly`. +- `src/system.ts` — build peer source + composite; inject into `IpfsService` (L1937). +- `src/config.ts` — new `IPFS_PEER_FETCH_*` vars (L3205 block). +- `src/constants.ts` — optional `X-Ar-Io-Local-Only` in `headerNames`. +- `docker-compose.yaml` — plumb new vars to `core` (L160 block). +- Metrics module — peer-fetch counters (§3.9). + +**No changes to:** any Solana program, `ario-gar`, observer contract logic. (Observer +holding-probe rider is a *separate repo* change — §7.) + +--- + +## 6. Testing (the priority) — including the multi-node harness + +### 6.1 Unit tests (`node --test`, `npm run test:file `) + +- `kubo-data-source.test.ts` — extend: `localOnly:true` issues the offline RPC call and + maps a local miss to `IpfsNotFoundError` (mock the Kubo RPC with `nock`/a stub). +- `ipfs-peer-data-source.test.ts` (new): + - happy path: peer returns a valid CAR → `dag/import` called → re-serve returns the + content; `reportSuccess` called. + - **tamper:** peer returns a CAR whose bytes don't hash to the CID → `dag/import` + rejects → source tries next peer → `reportFailure` on the bad peer. (Use a real + small CAR + a corrupted copy; assert import fails. If mocking Kubo, simulate the + import 500.) + - `localOnly:true` → throws `IpfsNotFoundError` immediately, no peer calls. + - cap exceeded → aborts, next peer. + - all peers fail → `IpfsNotFoundError`. +- `sequential-ipfs-source.test.ts` (new): tier fall-through on NotFound; short-circuit + on `AbortError`; **no fall-through on `IpfsBlockedError`**; local-only runs tier 1 + only (assert tiers 2/3 are never called). +- `routes/ipfs.test.ts` — extend: `X-Ar-Io-Local-Only: true` threads `localOnly:true` + into `getContent`; local miss → 404; payment bypassed under local-only. + +### 6.2 Multi-node integration test (the key ask) + +**Goal:** prove two/three real gateways pull verified content from each other, that +durability holds with public IPFS out of the picture, and that a lying peer is +rejected. + +**Harness:** `test/end-to-end/ipfs-peer-fetch.test.ts`, using the **existing +`testcontainers` + `Network` pattern** from `test/end-to-end/data-sources.test.ts` +(which co-starts sidecars on `new Network()` with network aliases). Precedent, helpers, +and the `getCoreContainer()` build (`test/end-to-end/utils.ts` L21) already exist. + +**Topology (2 nodes to start, extend to 3):** +``` + Network "ipfs-fleet" + ┌──────────────────────────────────┐ + │ core-a ── kubo-a (holds CID X) │ + │ core-b ── kubo-b (cold) │ + └──────────────────────────────────┘ +``` +- Start `new Network()`. +- Start `kubo-a`, `kubo-b` from `ipfs/kubo:v0.32.1` (the pinned image; profile `ipfs` + in compose) with `.withNetwork(network).withNetworkAliases('kubo-a'|'kubo-b')`. + **Isolate from public IPFS** so the test proves fleet-durability, not a public + fetch: run each Kubo with an empty bootstrap list / `Swarm` disabled (e.g. + `ipfs bootstrap rm --all` before `daemon`, or `--offline` for the cold node), so + `core-b` *cannot* get X from public IPFS — only from `core-a`. +- Start `core-a`, `core-b` from `getCoreContainer()` (`GenericContainer.fromDockerfile`) + with: + - `IPFS_ENABLED=true`, `IPFS_PEER_FETCH_ENABLED=true`. + - `IPFS_KUBO_URL=http://kubo-a:8080` / `http://kubo-b:8080`, + `IPFS_KUBO_API_URL=http://kubo-a:5001` / `http://kubo-b:5001`. + - Peer discovery: point `core-b` at `core-a` as a peer. Two options — (a) stub the + GAR/`arIOPeerManager` peer list via env or a test seam so `core-b`'s `'ipfs'` peer + set = `['http://core-a:']`; or (b) set `TRUSTED_GATEWAYS_URLS` and have the + peer source read from it in tests. Prefer a **test seam**: allow + `IpfsPeerDataSource` peers to be injected/overridden by an env var + (`IPFS_PEER_FETCH_STATIC_PEERS`) for deterministic testing — this is also useful in + production for private fleets. Add that env override as part of 3.3. + - `.withNetwork(network).withNetworkAliases('core-a'|'core-b')`, + `Wait.forHttp('/ar-io/info', )`. +- **Seed:** `ipfs add` a known file into `kubo-a` (via `kubo-a` RPC), capture CID `X`; + pin it on `core-a` (or ensure held). Assert `core-a` serves it and `core-b` does not + (local-only probe on `core-b` → 404). + +**Assertions:** +1. **Peer-fetch works:** `GET http://core-b:/ipfs/{X}?format=car` (or via the + `{cidv1}.` subdomain / path handler) → 200, bytes verify against `X`. Under the + hood `core-b` fetched the CAR from `core-a`, imported+verified into `kubo-b`. +2. **Durability independent of public IPFS:** because `kubo-b` has no bootstrap/swarm + to public IPFS, a 200 proves it came from `core-a` (the fleet), not the public + network. (Belt: assert peer-fetch metric incremented on `core-b`.) +3. **Now holds it:** after the fetch, a **local-only** probe on `core-b` + (`GET http://core-b:/ipfs/{X}?format=raw` + `X-Ar-Io-Local-Only: true`) → 200 + + verifying bytes. Before the fetch the same probe → 404. This is the exact holding + signal the observer will use (§7). +4. **Tamper rejection:** stand up a **malicious peer** — a tiny HTTP stub container (or + a mocked `core-c`) that answers `/ipfs/{X}?format=car` with a CAR whose bytes don't + hash to `X`. Put it ahead of `core-a` in `core-b`'s peer list. Assert `dag/import` + on `kubo-b` rejects it, `core-b` reports the peer failed, falls through to `core-a`, + and still serves correct bytes. (Proves the "untrusted peer is safe" claim.) +5. **Recursion guard:** a `X-Ar-Io-Local-Only: true` request to `core-b` for a CID it + doesn't hold → fast 404 and (assert) no outbound peer/public fetch (metric = 0). +6. **(3-node extension):** add `core-c` holding a *different* CID `Y`; assert `core-b` + fetches `X` from `core-a` and `Y` from `core-c`, and that hash-ring + `selectPeersForKey` routes each CID to the holder. + +**Runtime notes:** these are heavy (real containers + Kubo). Gate behind +`test:e2e` (already separate from unit `test`), respect `USE_PREBUILT_IMAGE` +(`utils.ts` L149) for CI. Reuse `waitFor`/`waitForLogMessage` (`utils.ts`) for +readiness. Keep the seed file small (KB) so CAR import is instant. + +### 6.3 Manual smoke (local, before CI) + +`docker compose --profile ipfs up` two stacks on one host (or the two-node compose in +§6.2), enable peer-fetch, `ipfs add` on one, curl the other. Confirm the local-only +probe flips 404→200 after the first fetch. + +--- + +## 7. Observer holding-probe rider (separate repo, no contract, do after 1.5a) + +Not part of the gateway PR; documented here so it's not lost. In **`ar-io-observer`**: +- The live path already fetches `?format=raw` and verifies via + `assessIpfsNameTrustless` / `getIpfsRawBlock` (observer.ts). Add + `X-Ar-Io-Local-Only: true` to that request so a PASS additionally proves the gateway + *holds* (not just proxies) the content. +- Make it a **ramped policy** (like the IPFS-capability ramp): during rollout, a + not-yet-holding gateway is NEUTRAL, not FAIL, so honest operators aren't abruptly + penalized while the fleet warms. Flip to holding-required once adoption is high. +- (Phase 3, later) sample K random **leaf** CIDs from the DAG and local-only-verify + each — stops "pin only the tiny root." IPFS analog of the Arweave chunk/offset proof. +- **Zero contract change:** holding is rewarded through the existing content-agnostic + name assessment. The optional dedicated holding-weight (contract) is explicitly out + of scope. + +--- + +## 8. Rollout & sequencing + +1. **Spike §3.1a** (Kubo per-request offline) — gates everything. +2. `localOnly` in `KuboDataSource` + route header + `IpfsService` passthrough + unit + tests. Ship-able alone (enables the observer holding-probe even before peer-fetch). +3. `IpfsPeerDataSource` + `SequentialIpfsSource` + composite wiring, behind + `IPFS_PEER_FETCH_ENABLED=false`. Unit tests. +4. **Multi-node integration test (§6.2)** — the acceptance gate. Do not enable in prod + until assertions 1–5 pass. +5. Metrics + serve-side rate limiting. +6. Enable `IPFS_PEER_FETCH_ENABLED=true` on a canary node, watch metrics, then default. +7. 1.5b routing (`/ar-io/ipfs/held` + DHT-filtered discovery) as a follow-up PR. +8. Observer holding-probe rider (separate repo) once §3.2 shipped. + +Each of 2–7 is an independently reviewable PR. 1.5a's value (fleet durability + faster +serving) lands at step 6 with no contract change. + +--- + +## 9. Risks & mitigations (carried from the design sketch) + +| Risk | Mitigation | +|---|---| +| Recursion / amplification across the fleet | `localOnly` runs tier 1 only; peer source self-guards; composite gates. | +| DoS on inbound local-only serve | Rate-limit (§3.8); local-only is cheap (no public recursion) but bounded. | +| Large CAR transfer | `IPFS_PEER_FETCH_MAX_CAR_BYTES` cap → fall through to public IPFS. | +| Lying / malicious peer | Kubo verifies blocks on `dag/import`; bad CAR fails → next peer. Proven by test §6.2#4. | +| Binding vs content trust | Peer-fetch is content only (CID→bytes). Name→CID binding stays chain-authoritative (on-demand resolution). Independent, composable. | +| Cold-start / who-holds-what | v1 blind GAR-subset (bounded by local-only 404); 1.5b routing as it grows. | +| Kubo offline semantics unreliable | Spike §3.1a; fallback options listed there. | + +--- + +*Read alongside `ipfs-peer-durability-layer.md` (design rationale, incentive analysis, +lifecycle diagram). This plan deliberately excludes the announcement/social strategy, +which is kept private outside the repo.*