feat(ipfs): multi-protocol ArNS + read-only IPFS gateway (Phases 1–2)

.claude/skills/ar-io-gateway-operator/SKILL.md

@@ -109,10 +109,25 @@ Pipeline diagnostic: a single `curl -sf .../ar-io/__gateway_metrics | grep -E 'q
 
 ### ArNS resolution
 
-`CompositeArNSResolver` walks resolvers in order: `TrustedGatewayArNSResolver` (asks `TRUSTED_ARNS_GATEWAY_URL`, default `https://__NAME__.turbo-gateway.com`), then `OnDemandArNSResolver` (queries the on-chain `ario-arns` and `ario-ant` programs directly via `SOLANA_RPC_URL`). The base name set is paginated into an in-memory `ArNSNamesCache` at boot and refreshed on a debounce. Resolved IDs may be Arweave TXs (route to data path) or, on the streaming-head branch, IPFS CIDs (route to `/ipfs/<cid>` via the Kubo sidecar). Unknown names log `Unable to resolve name against all resolvers` — that's normal user-error traffic, not a service failure.
+`CompositeArNSResolver` walks resolvers in order: `TrustedGatewayArNSResolver` (asks `TRUSTED_ARNS_GATEWAY_URL`, default `https://__NAME__.turbo-gateway.com`), then `OnDemandArNSResolver` (queries the on-chain `ario-arns` and `ario-ant` programs directly via `SOLANA_RPC_URL`). The base name set is paginated into an in-memory `ArNSNamesCache` at boot and refreshed on a debounce. Resolved IDs may be Arweave TXs (route to the data path) or IPFS CIDs when the ANT record sets `targetProtocol: ipfs` (route to `/ipfs/<cid>` via the Kubo sidecar; see "IPFS serving" below). The `protocol` is carried on the resolution and across a trusted-gateway hop via `X-ArNS-Protocol`. Unknown names log `Unable to resolve name against all resolvers` — that's normal user-error traffic, not a service failure.
 
 `/<name>` and `/<name>/<path>` requests carry `X-ArNS-*` trust headers in the response. Manifest path resolution still uses `StreamingManifestPathResolver`; the "from index" path is not implemented yet (logs warn `not implemented` then falls back to data-side resolution, which works).
 
+### IPFS serving (opt-in Kubo sidecar)
+
+Two things are required, not one: (1) `IPFS_ENABLED=true` turns on IPFS handling in `core`, and (2) the `kubo` sidecar must actually run — it's behind the Compose `ipfs` profile, so `docker compose --profile ipfs up -d` (or an override that adds the `kubo` service). **Enabling `IPFS_ENABLED` alone does not start Kubo** — the gateway will then return `502`/`504` for IPFS requests because it has no upstream to fetch from. With both in place, the gateway proxies, caches, moderates, and signs IPFS content alongside Arweave data, held to the same operational bar (`docs/ipfs-integration.md` → "Parity with the Arweave Data Path").
+
+- **Two entry points**: direct `/ipfs/{CID}` (and `{CID}.{root_host}` subdomains), and ArNS names whose ANT record sets `targetProtocol: ipfs` — the resolved id is a CID, surfaced as `X-ArNS-Protocol: ipfs` and served via Kubo. `protocol` propagates across a trusted-gateway hop, so a name→CID binding survives even with `gateway` ahead of `on-demand` in `ARNS_RESOLVER_PRIORITY_ORDER`.
+- **The node fetches from the local Kubo gateway** (`IPFS_KUBO_URL`, default `http://kubo:8080`) — it does not join the DHT itself. Availability depends on Kubo finding/holding the blocks; Kubo runs `--enable-gc`, so **unpinned content can disappear** (no on-chain permanence like Arweave). This is the key operational difference from Arweave data.
+- **Caching** is a bounded LRU separate from the Arweave content cache (`IPFS_CACHE_*`); absent/unpinned CIDs are negative-cached; hash-blocked bytes are never persisted.
+- **Moderation** uses the same admin API (see Block/unblock below): `PUT /ar-io/admin/block-data {"id":"<CIDv1>"}`. CID-blocking is the deterministic pre-serve primitive for IPFS (content is CID-addressed); hash-blocking also applies once a CID's served-byte SHA-256 is known.
+- **HEAD, Range/`206`, and per-CID sandbox origin isolation** (`/ipfs/{CID}` → `{CID}.{root_host}`) work as on the Arweave path. `IPFS_KUBO_MAX_CONCURRENT_REQUESTS` caps in-flight Kubo fetches (amplification guard); `IPFS_MAX_RESPONSE_SIZE_BYTES` caps a single response.
+- **Verify it's live**: `GET /ar-io/info` shows `ipfs.enabled: true`; `docker exec <kubo container> ipfs swarm peers` should list peers.
+- **Troubleshooting**: a name that resolves on viewblock but `404`s here with `IPFS_ENABLED` off means the CID is being misrouted to the Arweave data path — enable IPFS + Kubo. `/ipfs/{CID}` returning `502`/`504` points at the Kubo sidecar (down, no peers, or unpinned/cold content); check swarm peers first.
+- **Read-only, not permanent.** This is a read-only proxy to the *public* IPFS network — content is **not** stored on Arweave here. Durability of a named CID depends on it being pinned somewhere; there is no Arweave permanence in this mode.
+- **Pin named content for availability**: `IPFS_PIN_ARNS_CONTENT=true` pins (via the Kubo RPC API, `IPFS_KUBO_API_URL`, default `http://kubo:5001`) the CIDs that ArNS names resolve to, so named content this gateway serves isn't GC'd out from under a name (bounded by `IPFS_PIN_MAX`, FIFO). Inspect with `docker exec <kubo> ipfs pin ls --type=recursive`. The RPC API is powerful — keep 5001 on the internal docker network only, never host/internet.
+- **Trustless retrieval**: `GET /ipfs/{CID}?format=raw` (single verifiable block) and `?format=car` (verifiable DAG) let a client check the bytes against the CID itself — the gateway isn't a trust root. Responses carry `X-Ar-Io-Trustless: true`; the reassembled UnixFS path carries `X-Ar-Io-Trustless: false` (gateway-attested, not client-verified).
+
 ### Network identity, observer, and incentives
 
 A gateway is a registered participant in the AR.IO network, not just a piece of software. Two distinct Solana identities matter:

.dockerignore

@@ -18,3 +18,5 @@ node_modules/
 # Test
 test/
 coverage/
+.claude/
+logs/

CLAUDE.md

@@ -1,12 +1,50 @@
 # CLAUDE.md
 
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
 AR.IO Node — Arweave gateway for accessing and indexing blockchain data, with
 caching, ANS-104 bundle unbundling, and multi-source data retrieval.
 
+## Tech stack
+
+- Node.js v20 (see `.nvmrc`), TypeScript strict mode, ESM (`"type": "module"`)
+- Test framework: **Node.js native `node:test`** (not Jest/Mocha/Vitest)
+- Transpiler: SWC (via ts-node)
+- Databases: SQLite (primary) + ClickHouse (analytics/GQL)
+- Caching: Redis, LMDB, LRU in-memory
+- HTTP: Express
+- Observability: OpenTelemetry + Prometheus + Winston
+
+## Commands
+
+```bash
+# Development
+yarn start                    # Start service (requires .env file)
+yarn watch                    # Start with nodemon (auto-restart on changes)
+yarn build                    # Clean + compile TypeScript (prod)
+
+# Testing
+yarn test                     # Run all unit tests
+yarn test:file src/path/to/file.test.ts  # Run a single test file
+yarn test:e2e                 # Run end-to-end tests (in test/ directory)
+yarn test:coverage            # Run tests with coverage report
+
+# Linting & quality
+yarn lint:check               # ESLint check
+yarn lint:fix                 # ESLint auto-fix
+yarn duplicate:check          # Detect code duplication (jscpd)
+yarn deps:check               # Detect circular dependencies (madge)
+
+# Database
+yarn db:migrate               # Run SQLite migrations
+yarn db:dump-test-schemas     # Regenerate test SQL schemas after migrations
+
+# Service management (systemd-based)
+yarn service:start / stop / restart / status / logs
+```
+
 ## Discovery points
 
-- Commands — `package.json` scripts (dev, build, service, test, lint,
-  migrations, duplicate/deps checks)
 - Documentation index — `docs/INDEX.md`
 - Env vars — `docs/envs.md` (keep this and `docker-compose.yaml` in sync when
   adding or removing env vars)
@@ -20,6 +58,8 @@ caching, ANS-104 bundle unbundling, and multi-source data retrieval.
 
 - `src/system.ts` is the central DI wiring — all services, workers, data
   sources, resolvers, and lifecycle cleanup handlers are constructed here.
+- `src/config.ts` parses all environment variables and exports typed
+  constants — this is where new env vars are added.
 - `src/data/` uses composite sources with fallback chains
   (cache → S3 → AR.IO peers → trusted gateways → Arweave nodes). Retrieval
   order is configurable via `ON_DEMAND_RETRIEVAL_ORDER` and
@@ -30,6 +70,24 @@ caching, ANS-104 bundle unbundling, and multi-source data retrieval.
 - Filters (`ANS104_UNBUNDLE_FILTER`, `ANS104_INDEX_FILTER`,
   `WEBHOOK_INDEX_FILTER`) share a composable JSON filter system — see
   `docs/filters.md`.
+- Background workers (`src/workers/`) handle block importing, data importing,
+  bundle unbundling, verification, and webhooks. Controlled by `START_WRITERS`.
+- IPFS serving (`src/ipfs/`) is opt-in via `IPFS_ENABLED`. Uses a Kubo sidecar
+  for content retrieval with its own cache, rate limiter, and blocklist. Routes
+  mount before ArNS in `app.ts`. ArNS names whose ANT record has
+  `targetProtocol: ipfs` resolve to a CID and are routed to the same IPFS
+  handler by the ArNS middleware (`src/middleware/arns.ts`). `protocol` is a
+  first-class field on `NameResolution` set by the on-demand resolver and
+  propagated across a trusted-gateway hop via the signed `X-ArNS-Protocol`
+  header (`src/resolution/`). The IPFS path is held to Arweave-path parity
+  (moderation, caching, HEAD/Range, sandbox origin isolation, HTTPSIG). This is a
+  **read-only** proxy to the public IPFS network — it does not store content on
+  Arweave. Two trust postures: the UnixFS path is a trusted proxy
+  (`X-Ar-Io-Trustless: false`), while `?format=raw|car` relays verifiable
+  block/CAR bytes the client checks against the CID (`true`). Optional
+  named-content pinning (`IPFS_PIN_ARNS_CONTENT`, `src/ipfs/ipfs-pinner.ts`) pins
+  the CIDs ArNS names resolve to. Uploading/permapinning IPFS content to Arweave
+  is a deliberate phase 2. See `docs/ipfs-integration.md`.
 - Responses include trust headers indicating verification status.
 - HTTPSIG signs response headers (RFC 9421); `Content-Digest` is in
   `CO_SIGNABLE_HEADERS` so when present it binds the body to the signature.
@@ -58,6 +116,18 @@ Always use `createTestLogger()` from `test/test-logger.ts` in test files —
 never `winston.createLogger({ silent: true })`. Test output is written to
 `logs/test.log` (overwritten each run), not the console.
 
+### Test imports
+
+Tests use `node:test` and `node:assert`:
+
+```typescript
+import { describe, it, before, after, mock } from 'node:test';
+import { strict as assert } from 'node:assert';
+```
+
+Common test stubs are in `test/stubs.ts`, SQLite helpers in
+`test/sqlite-helpers.ts`.
+
 ### Adding a database method
 
 Five coordinated edits are required:

docker-compose.yaml

@@ -54,6 +54,11 @@ services:
       - ${HEADERS_DATA_PATH:-./data/headers}:/app/data/headers
       - ${SQLITE_DATA_PATH:-./data/sqlite}:/app/data/sqlite
       - ${DUCKDB_DATA_PATH:-./data/duckdb}:/app/data/duckdb
+      # The container target must match core's IPFS_CACHE_PATH (default
+      # data/ipfs-cache -> /app/data/ipfs-cache). If you change IPFS_CACHE_PATH,
+      # point it inside this mount, or the cache is written to an unmounted path
+      # and is lost when the container is recreated.
+      - ${IPFS_CACHE_DATA_PATH:-./data/ipfs-cache}:/app/data/ipfs-cache
       - ${TEMP_DATA_PATH:-./data/tmp}:/app/data/tmp
       - ${LMDB_DATA_PATH:-./data/lmdb}:/app/data/lmdb
       - ${PARQUET_DATA_PATH:-./data/parquet}:/app/data/parquet
@@ -152,6 +157,25 @@ services:
       - RATE_LIMITER_IP_REFILL_PER_SEC=${RATE_LIMITER_IP_REFILL_PER_SEC:-}
       - RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST=${RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST:-}
       - RATE_LIMITER_ARNS_ALLOWLIST=${RATE_LIMITER_ARNS_ALLOWLIST:-}
+      - IPFS_ENABLED=${IPFS_ENABLED:-false}
+      - IPFS_KUBO_URL=${IPFS_KUBO_URL:-http://kubo:8080}
+      - IPFS_KUBO_REQUEST_TIMEOUT_MS=${IPFS_KUBO_REQUEST_TIMEOUT_MS:-}
+      - IPFS_STREAM_STALL_TIMEOUT_MS=${IPFS_STREAM_STALL_TIMEOUT_MS:-}
+      - IPFS_KUBO_MAX_CONCURRENT_REQUESTS=${IPFS_KUBO_MAX_CONCURRENT_REQUESTS:-}
+      - IPFS_KUBO_MAX_REQUEST_MS=${IPFS_KUBO_MAX_REQUEST_MS:-}
+      - IPFS_KUBO_API_URL=${IPFS_KUBO_API_URL:-}
+      - IPFS_PIN_ARNS_CONTENT=${IPFS_PIN_ARNS_CONTENT:-}
+      - IPFS_PIN_MAX=${IPFS_PIN_MAX:-}
+      - IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES=${IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES:-}
+      - IPFS_TIMEOUT_NEGATIVE_CACHE_TTL_MS=${IPFS_TIMEOUT_NEGATIVE_CACHE_TTL_MS:-}
+      - IPFS_CACHE_PATH=${IPFS_CACHE_PATH:-}
+      - IPFS_CACHE_MAX_SIZE_BYTES=${IPFS_CACHE_MAX_SIZE_BYTES:-}
+      - IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS=${IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS:-}
+      - IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET=${IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET:-}
+      - IPFS_RATE_LIMITER_IP_REFILL_PER_SEC=${IPFS_RATE_LIMITER_IP_REFILL_PER_SEC:-}
+      - IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET=${IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET:-}
+      - IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC=${IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC:-}
+      - IPFS_MAX_RESPONSE_SIZE_BYTES=${IPFS_MAX_RESPONSE_SIZE_BYTES:-}
       - NODE_MAX_OLD_SPACE_SIZE=${NODE_MAX_OLD_SPACE_SIZE:-}
       - ENABLE_FS_HEADER_CACHE_CLEANUP=${ENABLE_FS_HEADER_CACHE_CLEANUP:-}
       - ON_DEMAND_RETRIEVAL_ORDER=${ON_DEMAND_RETRIEVAL_ORDER:-}
@@ -681,6 +705,9 @@ services:
       # the .env value — see ar-io-observer src/config.ts.
       - GATEWAY_ASSESSMENT_CONCURRENCY=${GATEWAY_ASSESSMENT_CONCURRENCY:-}
       - NAME_ASSESSMENT_CONCURRENCY=${NAME_ASSESSMENT_CONCURRENCY:-}
+      # Deadline for the observer's trustless IPFS `?format=raw` fetch when
+      # assessing IPFS-target ArNS names (>= the gateway's IPFS retrieval budget).
+      - IPFS_ASSESSMENT_TIMEOUT_MS=${IPFS_ASSESSMENT_TIMEOUT_MS:-}
       # Enable the LogReportSink — logs full per-gateway assessment detail at
       # info level (off by default).
       - ENABLE_LOG_REPORT_SINK=${ENABLE_LOG_REPORT_SINK:-}
@@ -723,6 +750,22 @@ services:
     networks:
       - ar-io-network
 
+  kubo:
+    image: ipfs/kubo:${KUBO_IMAGE_TAG:-v0.32.1}
+    profiles:
+      - ipfs
+    restart: unless-stopped
+    ports:
+      - '${IPFS_SWARM_PORT:-4001}:4001/tcp'
+      - '${IPFS_SWARM_PORT:-4001}:4001/udp'
+    environment:
+      - IPFS_PROFILE=${IPFS_PROFILE:-server}
+    volumes:
+      - ${IPFS_DATA_PATH:-./data/ipfs}:/data/ipfs
+    networks:
+      - ar-io-network
+    command: ['daemon', '--enable-gc']
+
   autoheal:
     image: willfarrell/autoheal@sha256:fd2c5500ab9210be9fa0d365162301eb0d16923f1d9a36de887f5d1751c6eb8c
     network_mode: none

docs/INDEX.md

@@ -23,6 +23,12 @@ Fast, offline lookups for data item to root transaction mappings.
 | [CDB64 Tools Reference](cdb64-tools.md) | CLI tools for creating indexes |
 | [CDB64 Format Specification](cdb64-format.md) | Technical file format details |
 
+### IPFS Integration
+
+| Document | Description |
+|----------|-------------|
+| [IPFS Integration](ipfs-integration.md) | Architecture, deployment, and configuration for IPFS CID serving |
+
 ### Rate Limiting & Payments
 
 | Document | Description |