feat(ipfs): multi-protocol ArNS + read-only IPFS gateway (Phases 1–2)

.claude/skills/ar-io-gateway-operator/SKILL.md

@@ -109,10 +109,25 @@ Pipeline diagnostic: a single `curl -sf .../ar-io/__gateway_metrics | grep -E 'q
 
 ### ArNS resolution
 
-`CompositeArNSResolver` walks resolvers in order: `TrustedGatewayArNSResolver` (asks `TRUSTED_ARNS_GATEWAY_URL`, default `https://__NAME__.turbo-gateway.com`), then `OnDemandArNSResolver` (queries the on-chain `ario-arns` and `ario-ant` programs directly via `SOLANA_RPC_URL`). The base name set is paginated into an in-memory `ArNSNamesCache` at boot and refreshed on a debounce. Resolved IDs may be Arweave TXs (route to data path) or, on the streaming-head branch, IPFS CIDs (route to `/ipfs/<cid>` via the Kubo sidecar). Unknown names log `Unable to resolve name against all resolvers` — that's normal user-error traffic, not a service failure.
+`CompositeArNSResolver` walks resolvers in order: `TrustedGatewayArNSResolver` (asks `TRUSTED_ARNS_GATEWAY_URL`, default `https://__NAME__.turbo-gateway.com`), then `OnDemandArNSResolver` (queries the on-chain `ario-arns` and `ario-ant` programs directly via `SOLANA_RPC_URL`). The base name set is paginated into an in-memory `ArNSNamesCache` at boot and refreshed on a debounce. Resolved IDs may be Arweave TXs (route to the data path) or IPFS CIDs when the ANT record sets `targetProtocol: ipfs` (route to `/ipfs/<cid>` via the Kubo sidecar; see "IPFS serving" below). The `protocol` is carried on the resolution and across a trusted-gateway hop via `X-ArNS-Protocol`. Unknown names log `Unable to resolve name against all resolvers` — that's normal user-error traffic, not a service failure.
 
 `/<name>` and `/<name>/<path>` requests carry `X-ArNS-*` trust headers in the response. Manifest path resolution still uses `StreamingManifestPathResolver`; the "from index" path is not implemented yet (logs warn `not implemented` then falls back to data-side resolution, which works).
 
+### IPFS serving (opt-in Kubo sidecar)
+
+`IPFS_ENABLED=true` adds a Kubo sidecar (`docker compose --profile ipfs up -d`, or a compose override that adds the `kubo` service) and turns on IPFS serving. The gateway proxies, caches, moderates, and signs IPFS content alongside Arweave data, held to the same operational bar (`docs/ipfs-integration.md` → "Parity with the Arweave Data Path").
+
+- **Two entry points**: direct `/ipfs/{CID}` (and `{CID}.{root_host}` subdomains), and ArNS names whose ANT record sets `targetProtocol: ipfs` — the resolved id is a CID, surfaced as `X-ArNS-Protocol: ipfs` and served via Kubo. `protocol` propagates across a trusted-gateway hop, so a name→CID binding survives even with `gateway` ahead of `on-demand` in `ARNS_RESOLVER_PRIORITY_ORDER`.
+- **The node fetches from the local Kubo gateway** (`IPFS_KUBO_URL`, default `http://kubo:8080`) — it does not join the DHT itself. Availability depends on Kubo finding/holding the blocks; Kubo runs `--enable-gc`, so **unpinned content can disappear** (no on-chain permanence like Arweave). This is the key operational difference from Arweave data.
+- **Caching** is a bounded LRU separate from the Arweave content cache (`IPFS_CACHE_*`); absent/unpinned CIDs are negative-cached; hash-blocked bytes are never persisted.
+- **Moderation** uses the same admin API (see Block/unblock below): `PUT /ar-io/admin/block-data {"id":"<CIDv1>"}`. CID-blocking is the deterministic pre-serve primitive for IPFS (content is CID-addressed); hash-blocking also applies once a CID's served-byte SHA-256 is known.
+- **HEAD, Range/`206`, and per-CID sandbox origin isolation** (`/ipfs/{CID}` → `{CID}.{root_host}`) work as on the Arweave path. `IPFS_KUBO_MAX_CONCURRENT_REQUESTS` caps in-flight Kubo fetches (amplification guard); `IPFS_MAX_RESPONSE_SIZE_BYTES` caps a single response.
+- **Verify it's live**: `GET /ar-io/info` shows `ipfs.enabled: true`; `docker exec <kubo container> ipfs swarm peers` should list peers.
+- **Troubleshooting**: a name that resolves on viewblock but `404`s here with `IPFS_ENABLED` off means the CID is being misrouted to the Arweave data path — enable IPFS + Kubo. `/ipfs/{CID}` returning `502`/`504` points at the Kubo sidecar (down, no peers, or unpinned/cold content); check swarm peers first.
+- **Read-only, not permanent.** This is a read-only proxy to the *public* IPFS network — content is **not** stored on Arweave here. Durability of a named CID depends on it being pinned somewhere; there is no Arweave permanence in this mode.
+- **Pin named content for availability**: `IPFS_PIN_ARNS_CONTENT=true` pins (via the Kubo RPC API, `IPFS_KUBO_API_URL`, default `http://kubo:5001`) the CIDs that ArNS names resolve to, so named content this gateway serves isn't GC'd out from under a name (bounded by `IPFS_PIN_MAX`, FIFO). Inspect with `docker exec <kubo> ipfs pin ls --type=recursive`. The RPC API is powerful — keep 5001 on the internal docker network only, never host/internet.
+- **Trustless retrieval**: `GET /ipfs/{CID}?format=raw` (single verifiable block) and `?format=car` (verifiable DAG) let a client check the bytes against the CID itself — the gateway isn't a trust root. Responses carry `X-Ar-Io-Trustless: true`; the reassembled UnixFS path carries `X-Ar-Io-Trustless: false` (gateway-attested, not client-verified).
+
 ### Network identity, observer, and incentives
 
 A gateway is a registered participant in the AR.IO network, not just a piece of software. Two distinct Solana identities matter:

.dockerignore

@@ -18,3 +18,5 @@ node_modules/
 # Test
 test/
 coverage/
+.claude/
+logs/

CLAUDE.md

@@ -1,12 +1,50 @@
 # CLAUDE.md
 
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
 AR.IO Node — Arweave gateway for accessing and indexing blockchain data, with
 caching, ANS-104 bundle unbundling, and multi-source data retrieval.
 
+## Tech stack
+
+- Node.js v20 (see `.nvmrc`), TypeScript strict mode, ESM (`"type": "module"`)
+- Test framework: **Node.js native `node:test`** (not Jest/Mocha/Vitest)
+- Transpiler: SWC (via ts-node)
+- Databases: SQLite (primary) + ClickHouse (analytics/GQL)
+- Caching: Redis, LMDB, LRU in-memory
+- HTTP: Express
+- Observability: OpenTelemetry + Prometheus + Winston
+
+## Commands
+
+```bash
+# Development
+yarn start                    # Start service (requires .env file)
+yarn watch                    # Start with nodemon (auto-restart on changes)
+yarn build                    # Clean + compile TypeScript (prod)
+
+# Testing
+yarn test                     # Run all unit tests
+yarn test:file src/path/to/file.test.ts  # Run a single test file
+yarn test:e2e                 # Run end-to-end tests (in test/ directory)
+yarn test:coverage            # Run tests with coverage report
+
+# Linting & quality
+yarn lint:check               # ESLint check
+yarn lint:fix                 # ESLint auto-fix
+yarn duplicate:check          # Detect code duplication (jscpd)
+yarn deps:check               # Detect circular dependencies (madge)
+
+# Database
+yarn db:migrate               # Run SQLite migrations
+yarn db:dump-test-schemas     # Regenerate test SQL schemas after migrations
+
+# Service management (systemd-based)
+yarn service:start / stop / restart / status / logs
+```
+
 ## Discovery points
 
-- Commands — `package.json` scripts (dev, build, service, test, lint,
-  migrations, duplicate/deps checks)
 - Documentation index — `docs/INDEX.md`
 - Env vars — `docs/envs.md` (keep this and `docker-compose.yaml` in sync when
   adding or removing env vars)
@@ -20,6 +58,8 @@ caching, ANS-104 bundle unbundling, and multi-source data retrieval.
 
 - `src/system.ts` is the central DI wiring — all services, workers, data
   sources, resolvers, and lifecycle cleanup handlers are constructed here.
+- `src/config.ts` parses all environment variables and exports typed
+  constants — this is where new env vars are added.
 - `src/data/` uses composite sources with fallback chains
   (cache → S3 → AR.IO peers → trusted gateways → Arweave nodes). Retrieval
   order is configurable via `ON_DEMAND_RETRIEVAL_ORDER` and
@@ -30,6 +70,24 @@ caching, ANS-104 bundle unbundling, and multi-source data retrieval.
 - Filters (`ANS104_UNBUNDLE_FILTER`, `ANS104_INDEX_FILTER`,
   `WEBHOOK_INDEX_FILTER`) share a composable JSON filter system — see
   `docs/filters.md`.
+- Background workers (`src/workers/`) handle block importing, data importing,
+  bundle unbundling, verification, and webhooks. Controlled by `START_WRITERS`.
+- IPFS serving (`src/ipfs/`) is opt-in via `IPFS_ENABLED`. Uses a Kubo sidecar
+  for content retrieval with its own cache, rate limiter, and blocklist. Routes
+  mount before ArNS in `app.ts`. ArNS names whose ANT record has
+  `targetProtocol: ipfs` resolve to a CID and are routed to the same IPFS
+  handler by the ArNS middleware (`src/middleware/arns.ts`). `protocol` is a
+  first-class field on `NameResolution` set by the on-demand resolver and
+  propagated across a trusted-gateway hop via the signed `X-ArNS-Protocol`
+  header (`src/resolution/`). The IPFS path is held to Arweave-path parity
+  (moderation, caching, HEAD/Range, sandbox origin isolation, HTTPSIG). This is a
+  **read-only** proxy to the public IPFS network — it does not store content on
+  Arweave. Two trust postures: the UnixFS path is a trusted proxy
+  (`X-Ar-Io-Trustless: false`), while `?format=raw|car` relays verifiable
+  block/CAR bytes the client checks against the CID (`true`). Optional
+  named-content pinning (`IPFS_PIN_ARNS_CONTENT`, `src/ipfs/ipfs-pinner.ts`) pins
+  the CIDs ArNS names resolve to. Uploading/permapinning IPFS content to Arweave
+  is a deliberate phase 2. See `docs/ipfs-integration.md`.
 - Responses include trust headers indicating verification status.
 - HTTPSIG signs response headers (RFC 9421); `Content-Digest` is in
   `CO_SIGNABLE_HEADERS` so when present it binds the body to the signature.
@@ -58,6 +116,18 @@ Always use `createTestLogger()` from `test/test-logger.ts` in test files —
 never `winston.createLogger({ silent: true })`. Test output is written to
 `logs/test.log` (overwritten each run), not the console.
 
+### Test imports
+
+Tests use `node:test` and `node:assert`:
+
+```typescript
+import { describe, it, before, after, mock } from 'node:test';
+import { strict as assert } from 'node:assert';
+```
+
+Common test stubs are in `test/stubs.ts`, SQLite helpers in
+`test/sqlite-helpers.ts`.
+
 ### Adding a database method
 
 Five coordinated edits are required:

docker-compose.yaml

@@ -54,6 +54,7 @@ services:
       - ${HEADERS_DATA_PATH:-./data/headers}:/app/data/headers
       - ${SQLITE_DATA_PATH:-./data/sqlite}:/app/data/sqlite
       - ${DUCKDB_DATA_PATH:-./data/duckdb}:/app/data/duckdb
+      - ${IPFS_CACHE_DATA_PATH:-./data/ipfs-cache}:/app/data/ipfs-cache
       - ${TEMP_DATA_PATH:-./data/tmp}:/app/data/tmp
       - ${LMDB_DATA_PATH:-./data/lmdb}:/app/data/lmdb
       - ${PARQUET_DATA_PATH:-./data/parquet}:/app/data/parquet
@@ -152,6 +153,23 @@ services:
       - RATE_LIMITER_IP_REFILL_PER_SEC=${RATE_LIMITER_IP_REFILL_PER_SEC:-}
       - RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST=${RATE_LIMITER_IPS_AND_CIDRS_ALLOWLIST:-}
       - RATE_LIMITER_ARNS_ALLOWLIST=${RATE_LIMITER_ARNS_ALLOWLIST:-}
+      - IPFS_ENABLED=${IPFS_ENABLED:-false}
+      - IPFS_KUBO_URL=${IPFS_KUBO_URL:-http://kubo:8080}
+      - IPFS_KUBO_REQUEST_TIMEOUT_MS=${IPFS_KUBO_REQUEST_TIMEOUT_MS:-}
+      - IPFS_STREAM_STALL_TIMEOUT_MS=${IPFS_STREAM_STALL_TIMEOUT_MS:-}
+      - IPFS_KUBO_MAX_CONCURRENT_REQUESTS=${IPFS_KUBO_MAX_CONCURRENT_REQUESTS:-}
+      - IPFS_KUBO_API_URL=${IPFS_KUBO_API_URL:-}
+      - IPFS_PIN_ARNS_CONTENT=${IPFS_PIN_ARNS_CONTENT:-}
+      - IPFS_PIN_MAX=${IPFS_PIN_MAX:-}
+      - IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES=${IPFS_RATE_LIMIT_UNKNOWN_SIZE_BYTES:-}
+      - IPFS_CACHE_PATH=${IPFS_CACHE_PATH:-}
+      - IPFS_CACHE_MAX_SIZE_BYTES=${IPFS_CACHE_MAX_SIZE_BYTES:-}
+      - IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS=${IPFS_CACHE_CLEANUP_THRESHOLD_SECONDS:-}
+      - IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET=${IPFS_RATE_LIMITER_IP_TOKENS_PER_BUCKET:-}
+      - IPFS_RATE_LIMITER_IP_REFILL_PER_SEC=${IPFS_RATE_LIMITER_IP_REFILL_PER_SEC:-}
+      - IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET=${IPFS_RATE_LIMITER_RESOURCE_TOKENS_PER_BUCKET:-}
+      - IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC=${IPFS_RATE_LIMITER_RESOURCE_REFILL_PER_SEC:-}
+      - IPFS_MAX_RESPONSE_SIZE_BYTES=${IPFS_MAX_RESPONSE_SIZE_BYTES:-}
       - NODE_MAX_OLD_SPACE_SIZE=${NODE_MAX_OLD_SPACE_SIZE:-}
       - ENABLE_FS_HEADER_CACHE_CLEANUP=${ENABLE_FS_HEADER_CACHE_CLEANUP:-}
       - ON_DEMAND_RETRIEVAL_ORDER=${ON_DEMAND_RETRIEVAL_ORDER:-}
@@ -723,6 +741,22 @@ services:
     networks:
       - ar-io-network
 
+  kubo:
+    image: ipfs/kubo:${KUBO_IMAGE_TAG:-v0.32.1}
+    profiles:
+      - ipfs
+    restart: unless-stopped
+    ports:
+      - '${IPFS_SWARM_PORT:-4001}:4001/tcp'
+      - '${IPFS_SWARM_PORT:-4001}:4001/udp'
+    environment:
+      - IPFS_PROFILE=${IPFS_PROFILE:-server}
+    volumes:
+      - ${IPFS_DATA_PATH:-./data/ipfs}:/data/ipfs
+    networks:
+      - ar-io-network
+    command: ['daemon', '--enable-gc']
+
   autoheal:
     image: willfarrell/autoheal@sha256:fd2c5500ab9210be9fa0d365162301eb0d16923f1d9a36de887f5d1751c6eb8c
     network_mode: none

docs/INDEX.md

@@ -23,6 +23,12 @@ Fast, offline lookups for data item to root transaction mappings.
 | [CDB64 Tools Reference](cdb64-tools.md) | CLI tools for creating indexes |
 | [CDB64 Format Specification](cdb64-format.md) | Technical file format details |
 
+### IPFS Integration
+
+| Document | Description |
+|----------|-------------|
+| [IPFS Integration](ipfs-integration.md) | Architecture, deployment, and configuration for IPFS CID serving |
+
 ### Rate Limiting & Payments
 
 | Document | Description |