diff --git a/.gitignore b/.gitignore index beb3488..7884bbe 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,6 @@ __pycache__/ Cargo.lock .pytest_cache/ .ruff_cache/ +.op.env + +.op.apikey.env diff --git a/README.md b/README.md index e0597bf..ef85e91 100644 --- a/README.md +++ b/README.md @@ -16,8 +16,9 @@ interop test, and a working proof of CacheKit's differentiators: stores ciphertext only. - **≈ $0/month** — all third-party hosting stays inside free tiers (cost table below). -> Status: **Stage 1 spike complete** (LAB-735). Architecture locked in -> [`docs/architecture.md`](docs/architecture.md); build stages are groomed from it. +> Status: **Stage 2 components merged, Stage 3 live integration in progress** (LAB-737). +> Architecture locked in [`docs/architecture.md`](docs/architecture.md); build stages are groomed +> from it. ## Architecture @@ -29,11 +30,11 @@ flowchart LR ING[Python ingester + aggregator\ncachekit-py 0.15\n5m / 1h / 24h windows] end - ING -->|"@cache.io writes\ninterop/v1 keys"| CK[(CachekitIO\napi.cachekit.io\nnamespace: bluesky-thinking)] + ING -->|"@cache.io writes\ninterop/v1 keys"| CK[(CachekitIO\napi.dev.cachekit.io\nnamespace: bluesky-thinking)] subgraph Cloudflare edge - free plan API[TS edge API\n@cachekit-io/cachekit 0.1.3] - WASM[Rust-WASM hot path\ncachekit-rs 0.4\nkey derivation and edge compute] + WASM[Rust-WASM hot path\ncachekit-rs 0.5\nkey derivation and edge compute] DASH[Static dashboard\nWorkers Assets] end @@ -55,22 +56,23 @@ All three SDKs address the cache with **interop/v1** keys (`bluesky-thinking:{op | `cachekit-rs` compiles for `wasm32-unknown-unknown` | ✅ SDK CI recipe + downstream consumer crate | [`spike/edge-worker/`](spike/edge-worker/) | | `cachekit-rs` Worker **deploys and runs** on Cloudflare | ✅ live at `lab-735-skyline-spike.raywalker.workers.dev`, 180 KiB gzipped, 2 ms startup | [`spike/edge-worker/`](spike/edge-worker/) | | Cross-SDK key byte-compatibility | ✅ Python (PyPI), TS (npm), Rust (live CF edge) all derive `bluesky-thinking:posts_per_minute:230037de…` | [`docs/architecture.md`](docs/architecture.md#locked-key-convention) | -| CachekitIO namespace + credentials | ⏳ blocked on interactive `ck login` (human step) — runbook ready | [`docs/architecture.md`](docs/architecture.md#provisioning-runbook) | -| Free-tier hosts chosen | ✅ Oracle Always Free (ingester) · Cloudflare Workers free (edge) · Render free (fallback) | [`docs/architecture.md`](docs/architecture.md#hosting) | +| CachekitIO namespace + credentials | ✅ creds exist at `op://cachekit/ck-dev-bluesky-default`, round-trip verified against `api.dev.cachekit.io` (Stage 3) | [`docs/architecture.md`](docs/architecture.md#credentials) | +| Free-tier hosts chosen | ✅ Render free web service (ingester) · Cloudflare Workers free (edge) | [`docs/architecture.md`](docs/architecture.md#hosting) | ## Cost table (AC-8) | Component | Host | Free-tier limit | Skyline's use | Cost | | :--- | :--- | :--- | :--- | ---: | | Jetstream feed | Bluesky public infra | none (public, no auth) | 1 WebSocket consumer | $0 | -| Python ingester | Oracle Cloud Always Free (Ampere A1) | 2 OCPU / 12 GB RAM always-on¹ | ~0.25 OCPU / 512 MB | $0 | +| Python ingester | Render free web service | 750 instance-hrs/mo — enough for one continuously-running service¹ | one free web service, normally kept warm by a CF Worker cron ping | $0 | | Edge API + WASM | Cloudflare Workers free plan | 100k req/day, 10 ms CPU/invocation | cached reads, ≪ limits | $0 | | Dashboard | Cloudflare Workers Assets | static asset requests free | tiny static site | $0 | | Cache backend | CachekitIO (ours) | n/a — dogfood | one demo tenant | $0² | | **Total** | | | | **$0/mo** | -¹ Halved from 4 OCPU / 24 GB on 2026-06-15; still far more than needed. Fallback: Render free web -service (750 instance-hrs/mo) kept warm by a Cloudflare Worker cron ping. +¹ Free services spin down after 15 min without inbound traffic; a Cloudflare Worker cron trigger +pings every 10 min to keep the ingester warm ($0). Restarts lose in-memory window state, mitigated +by checkpointing aggregation state into CacheKit. ² CachekitIO is the platform being showcased — we build, run, and own it. No third-party line item. Fly.io was evaluated and **rejected**: its free tier was discontinued in 2024 (new orgs get a @@ -85,6 +87,8 @@ hotpath/ — Stage-2 Rust-WASM hot-path Worker (cachekit-rs 0.5 on interop key derivation, xxHash3 payload verification, window-slice merging — live at skyline-hotpath.raywalker.workers.dev ingester/ — Stage-2 Python ingester + window aggregator (LAB-744): Jetstream → 5m/1h/24h windows → interop/v1 aggregates +stage3/ — Stage-3 live-integration evidence harness (LAB-737): clean-namespace + audit, SDK-free raw/ciphertext reader, stampede (distributed-lock) proof spike/decorators/ — AC-3 proof: the three decorators running on cachekit 0.15.0 spike/edge-worker/ — AC-2 proof: deployable cachekit-rs Worker (the live spike) spike/roundtrip/ — AC-1 harness: CachekitIO round-trip, runs as soon as credentials exist diff --git a/docs/architecture.md b/docs/architecture.md index 264b52a..dfc42b2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -7,18 +7,13 @@ Anything not locked here is a Stage-2 implementation choice. | Component | Language / SDK | Pinned version | Host | | :--- | :--- | :--- | :--- | -| Ingester + window aggregator | Python / `cachekit` | `0.15.0` (PyPI) | Oracle Always Free VM (fallback: Render free) | +| Ingester + window aggregator | Python / `cachekit` | `0.15.0` (PyPI) | Render free web service | | Edge API | TypeScript / `@cachekit-io/cachekit` | `0.1.3` (npm) | Cloudflare Workers (free plan) | -| Edge hot path | Rust / `cachekit-rs` | git tag `cachekit-rs-v0.4.0`¹ | Cloudflare Workers, `wasm32-unknown-unknown` | +| Edge hot path | Rust / `cachekit-rs` | `0.5.0` (crates.io) | Cloudflare Workers, `wasm32-unknown-unknown` | | Dashboard | static HTML/JS | — | Cloudflare Workers Assets | -| Cache backend | CachekitIO | `api.cachekit.io` | ours (dogfood) | +| Cache backend | CachekitIO | `api.dev.cachekit.io` | ours (dogfood) | | Data source | Bluesky Jetstream | public WebSocket | e.g. `wss://jetstream2.us-east.bsky.network/subscribe` | -¹ crates.io only has `0.3.0`, which predates interop mode. `v0.4.0` is tagged and GitHub-released -but its crates.io publish failed in CI (a since-fixed debug-step quoting bug aborted the workflow -before `cargo publish` ran). Stage 2 uses the git-tag dependency; switch to the crates.io version -once the publish is re-run. Tracked separately — do not block on it. - ## Locked key convention ### The decision: interop/v1, not the 7-segment auto key @@ -36,7 +31,7 @@ answer for cross-SDK sharing is **interop/v1** (`protocol/spec/interop-mode.md`) encoding of the flat bound argument array. - Values are one plain MessagePack document (no ByteStorage envelope, no LZ4). - All three SDKs ship it: `cachekit-py` 0.15.0 (`@cache(interop=…)`), `@cachekit-io/cachekit` - 0.1.3 (`generateInteropKey` / `interop` wrap option), `cachekit-rs` v0.4.0 + 0.1.3 (`generateInteropKey` / `interop` wrap option), `cachekit-rs` 0.5.0 (`interop_key()` / `#[cachekit(interop = …)]`). - Note: the spec header in `protocol/spec/interop-mode.md` still says "NOT yet implemented in any SDK" — stale; all three implementations exist and are vector-verified. Flagged upstream. @@ -66,7 +61,7 @@ bluesky-thinking:posts_per_minute:230037def14c9a89b18603f313d982d6a3f7acd4af5147 The `("5m")` args-hash is identical across operations — same canonical argument array, same hash; the operation segment provides the identity. Spike verification: `cachekit-py` 0.15.0 (local), -`@cachekit-io/cachekit` 0.1.3 (local Node), and `cachekit-rs` v0.4.0 running **live on the +`@cachekit-io/cachekit` 0.1.3 (local Node), and `cachekit-rs` 0.5.0 running **live on the Cloudflare edge** all derived `bluesky-thinking:posts_per_minute:230037de…` byte-identically. ### The secure cache (AC-6 path) @@ -86,37 +81,57 @@ The SaaS derives its server-side namespace from the **key prefix**, not from the - Python auto-mode keys (`ns:bluesky-thinking:func:…`, used by the secure cache) → server-side namespace **`bluesky-thinking`**. -Therefore the demo API key must allow **both** namespaces. Provision the key **unrestricted** -inside a dedicated demo tenant (tenant isolation is the real boundary; a namespace allowlist adds -nothing when the whole tenant is the demo) — or, if restricting anyway, allow -`bluesky-thinking,default`. A key restricted to `bluesky-thinking` alone rejects every interop -key the demo depends on. +Therefore the demo API key must allow **both** namespaces. The existing demo key +(`op://cachekit/ck-dev-bluesky-default`) is unrestricted inside its dedicated demo tenant (tenant +isolation is the real boundary; a namespace allowlist adds nothing when the whole tenant is the +demo) — verified: it round-trips both key classes. A key restricted to `bluesky-thinking` alone +would reject every interop key the demo depends on. Distributed locking needs no enablement: `POST /v1/cache/{key}/lock` (and DELETE) are available to every authenticated caller — locking is inherent to the backend, per `protocol/spec/saas-api.md`. -## Provisioning runbook +## Credentials + +Nothing to provision — the backend is the `dev.cachekit` instance and credentials already exist +in 1Password at `op://cachekit/ck-dev-bluesky-default`: + +- `credential` field → the API key → `CACHEKIT_API_KEY` +- `encryption_key` field → 64-hex master key → `CACHEKIT_MASTER_KEY` -Requires an interactive browser login (Clerk) — **human step, ~5 minutes** (from `saas/cli`): +Load both via `op run --env-file` (or `op read` for one-off shells) — never commit them, never +echo them. The gitignored env-file templates the runbooks reference contain only `op://` +references (no secret material); recreate them at the repo root as: ```bash -ck --env production login # browser opens; or mint a machine token from the dashboard -ck --env production tenants create --name bluesky-thinking --plan free -ck --env production keys create --tenant --type sdk --name skyline-demo -# → plaintext key shown ONCE; store as CACHEKIT_API_KEY (GitHub Actions secret + host env) +# .op.env — full ingester credentials +CACHEKIT_API_KEY=op://cachekit/ck-dev-bluesky-default/credential +CACHEKIT_MASTER_KEY=op://cachekit/ck-dev-bluesky-default/encryption_key + +# .op.apikey.env — API key only (interop/evidence tooling; the master key in +# env auto-enables encryption, which interop-mode scripts must not inherit) +CACHEKIT_API_KEY=op://cachekit/ck-dev-bluesky-default/credential ``` -Then verify the round-trip end-to-end: `spike/roundtrip/roundtrip.py` (exists, runs against -`api.cachekit.io` the moment `CACHEKIT_API_KEY` is set; also exercises `@cache.io`). +`api.dev.cachekit.io` is not in the SDKs' SSRF host allowlists, so every SDK needs its +config-level custom-host override alongside the credentials: + +- Python: env `CACHEKIT_API_URL=https://api.dev.cachekit.io` + `CACHEKIT_ALLOW_CUSTOM_HOST=true` +- TS: `cachekitio({ apiUrl, allowCustomHost: true })` +- Rust: `WorkersCachekitIO::builder().api_url(...).allow_custom_host(true)` — **currently + bypassed**: `WorkersCachekitIO` panics on every wasm32 request (LAB-1079), so the hot path does a + direct `worker::Fetch` GET until the SDK fix ships (`hotpath/README.md`) + +Round-trip verified end-to-end: `spike/roundtrip/roundtrip.py` (exists, runs against +`api.dev.cachekit.io`; passed against the dev instance on 2026-07-29; also exercises `@cache.io`). ## Hosting | Decision | Rationale (verified 2026-07-24) | | :--- | :--- | -| **Ingester → Oracle Cloud Always Free** (Ampere A1) | Only true $0 *always-on* compute left. Allowance halved to 2 OCPU / 12 GB on 2026-06-15 — still 8× what the ingester needs. **Human step**: signup requires a credit card (never charged on Always Free); A1 capacity is regional — pick a low-contention region (e.g. not us-east). | -| **Ingester fallback → Render free web service** | No billing info needed. Constraint: free services spin down after 15 min without *inbound* traffic (an outbound Jetstream WebSocket doesn't count) — keep warm with a Cloudflare Worker cron trigger pinging every 10 min ($0). 750 instance-hrs/mo covers one always-on service. Restarts lose in-memory window state: mitigate by checkpointing aggregation state into CacheKit (more dogfood). | +| **Ingester → Render free web service** | No billing info needed — ray already has a Render account. Constraint: free services spin down after 15 min without *inbound* traffic (an outbound Jetstream WebSocket doesn't count) — keep warm with a Cloudflare Worker cron trigger pinging every 10 min ($0). 750 instance-hrs/mo covers one always-on service. Restarts lose in-memory window state: mitigate by checkpointing aggregation state into CacheKit (more dogfood). | +| **Oracle Cloud Always Free — earlier pick, rejected** | Would have been 8× the ingester's needs, but signup requires a credit card and ray has no Oracle account; Render needs no new signup. | | **Fly.io — rejected** | Free tier discontinued 2024; ~$2/mo minimum for an always-on machine breaks AC-8. | -| **Edge → Cloudflare Workers free plan** | 100k requests/day, 10 ms CPU/invocation — cached analytics reads are single-digit ms. Static dashboard via Workers Assets (free). Cron triggers included (used for the Render keep-alive if the fallback is active). **Proven live by this spike**: `lab-735-skyline-spike.raywalker.workers.dev` (180 KiB gzipped upload, 2 ms startup, well under the 3 MB compressed script limit). | +| **Edge → Cloudflare Workers free plan** | 100k requests/day, 10 ms CPU/invocation — cached analytics reads are single-digit ms. Static dashboard via Workers Assets (free). Cron triggers included (used for the Render keep-alive). **Proven live by this spike**: `lab-735-skyline-spike.raywalker.workers.dev` (180 KiB gzipped upload, 2 ms startup, well under the 3 MB compressed script limit). | ## Build-chain pins (from spike friction, so Stage 2 doesn't rediscover them) @@ -124,17 +139,19 @@ Then verify the round-trip end-to-end: `spike/roundtrip/roundtrip.py` (exists, r - `wasm-bindgen-cli` 0.2.126 to match the crate graph — worker-build 0.1.x auto-downloads 0.2.105 and fails; pre-install the matching CLI into its cache or PATH. - On wasm32 the CachekitIO backend is `cachekit::backend::workers::WorkersCachekitIO` (CF Fetch - API); the reqwest-based `CachekitIO` does not implement `Backend` on that target. + API); the reqwest-based `CachekitIO` does not implement `Backend` on that target. **LAB-1079**: + `WorkersCachekitIO` panics on every live wasm32 request (`SystemTime::now()` in its session + headers) — the hot path substitutes a direct `worker::Fetch` GET until the SDK fix is published. - Workers builds: `--no-default-features --features workers,cachekitio,encryption,macros` (`l1`/moka and `redis`/fred are native-only). ## Open items (flagged, not blocking the spec) -1. **CachekitIO credentials** — human runs the provisioning runbook above; then - `spike/roundtrip/roundtrip.py` closes AC-1. -2. **Oracle account** — human signup (credit card gate). Render fallback needs no billing info if - preferred. -3. **cachekit-rs 0.4.0 crates.io publish** — failed CI run; re-publish so Stage 2 can drop the - git-tag dependency. Tracked as its own issue. +1. ~~**CachekitIO credentials**~~ — resolved: creds exist at + `op://cachekit/ck-dev-bluesky-default`; `spike/roundtrip/roundtrip.py` round-trip verified + against `api.dev.cachekit.io` (2026-07-29), closing AC-1. +2. ~~**Oracle account**~~ — resolved: Render account exists (ray, 2026-07-24); Oracle dropped. +3. ~~**cachekit-rs crates.io publish**~~ — resolved by LAB-742: crates.io now carries up to + `0.8.0`; hotpath builds against `0.5.0`. 4. **protocol/spec/interop-mode.md status header** — says "NOT yet implemented in any SDK"; all three SDKs ship it. One-line doc fix for the protocol repo owners. diff --git a/edge/README.md b/edge/README.md index bd4a446..cdc72b7 100644 --- a/edge/README.md +++ b/edge/README.md @@ -35,9 +35,36 @@ npm run lint && npm run format:check && npm run type-check if it fails, key derivation drifted from the cross-SDK contract — fix the drift, not the vectors. -## Deploy (Stage 4) +## Hot-path integration (Stage 3) -`wrangler deploy` after `wrangler secret put CACHEKIT_API_KEY` (key provisioned -per the [runbook](../docs/architecture.md#provisioning-runbook)). Live -integration and production routing are Stage 3/4 concerns — nothing here -requires credentials until then. +The Worker holds a **service binding** to the Rust-WASM hot path +(`wrangler.toml [[services]]`, `env.HOTPATH` → `skyline-hotpath`). Every +payload served through `/api/{operation}` is first integrity-checked there +(`POST /v1/verify`: xxHash3-64 + strict interop/v1 decode): + +- verified → served with `x-hotpath: verified` + `x-hotpath-xxh3: <16-hex>` +- invalid → **500** `integrity_check_failed`; a corrupt entry is never served +- hot path unreachable → served with `x-hotpath: unavailable` (the aggregate + is real — it came from the backend — it just goes out unverified and says so) + +Misses never call the hot path: 404 + `X-Cache: MISS`, unchanged. + +## Deploy + +`wrangler deploy`, then set the secret (creds per +[docs/architecture.md#credentials](../docs/architecture.md#credentials)): + +```bash +op read "op://cachekit/ck-dev-bluesky-default/credential" | wrangler secret put CACHEKIT_API_KEY +``` + +Dev deployment: **https://skyline-edge.raywalker.workers.dev** (the dev +instance URL is a `[vars]` entry, `CACHEKIT_API_URL`). Production routing and +a custom domain are Stage 4. + +Two build-time accommodations for `@cachekit-io/cachekit` 0.1.3 (both retire +with the 0.1.4 WASM core, blocked on LAB-780): the `nodejs_compat` flag +(transitive node builtins), and a wrangler `[alias]` stubbing the NAPI-native +`@cachekit-io/cachekit-core-ts` — the edge never runs that path (interop +reads only, no ByteStorage envelope), and the stub throws if that ever stops +being true. diff --git a/edge/src/core-ts-stub.ts b/edge/src/core-ts-stub.ts new file mode 100644 index 0000000..c811a3b --- /dev/null +++ b/edge/src/core-ts-stub.ts @@ -0,0 +1,23 @@ +/** + * Build-time stub for `@cachekit-io/cachekit-core-ts` (wrangler [alias]). + * + * The real package is a NAPI native module (.node binaries) that can never + * run on Cloudflare Workers — bundling it is what breaks `wrangler deploy` + * on @cachekit-io/cachekit 0.1.3 (0.1.4 replaces it with a WASM core, but + * that bump is blocked upstream: LAB-780). The SDK only imports `ByteStorage` + * from it statically, and only the compression path of the Cache class ever + * constructs one. The edge never uses that path: it reads raw bytes via + * `backend.get()` and decodes interop/v1 (plain MessagePack, no ByteStorage + * envelope by contract). So the honest stub is one that throws on use — + * dead weight is aliased away, and any future code path that would silently + * depend on the native module fails loudly instead. + */ +export class ByteStorage { + constructor() { + throw new Error( + 'ByteStorage is not available on Cloudflare Workers: ' + + '@cachekit-io/cachekit-core-ts is a native NAPI module (stubbed at build time). ' + + 'The edge reads interop/v1 payloads, which never use the ByteStorage envelope.', + ); + } +} diff --git a/edge/src/handler.ts b/edge/src/handler.ts index ffd0c3b..9956133 100644 --- a/edge/src/handler.ts +++ b/edge/src/handler.ts @@ -47,6 +47,63 @@ export interface Stats { } const stats: Stats = { hits: 0, misses: 0, errors: 0 }; +/** + * The Rust-WASM hot-path Worker, reached via a Cloudflare service binding + * (wrangler [[services]]) — never a public URL. Structural type: the repo + * doesn't depend on @cloudflare/workers-types, and a binding is just an + * object with fetch(). + */ +export interface HotpathBinding { + fetch(input: string, init?: RequestInit): Promise; +} + +type HotpathVerdict = + | { state: 'verified'; xxh3: string } + | { state: 'invalid'; xxh3: string; detail: string } + | { state: 'unavailable'; detail: string }; + +/** + * Integrity-check a fetched payload on the Rust-WASM hot path + * (POST /v1/verify: xxHash3-64 + strict interop/v1 decode). Any transport or + * hot-path failure degrades to 'unavailable' — the caller decides what that + * means; this function never throws. + */ +async function verifyViaHotpath(hotpath: HotpathBinding, raw: Uint8Array): Promise { + try { + // The URL host is ignored by service bindings; only the path routes. + // Copy pins the generic to Uint8Array, which BodyInit + // accepts (backend.get returns Uint8Array). + const res = await hotpath.fetch('https://skyline-hotpath/v1/verify', { + method: 'POST', + body: new Uint8Array(raw), + }); + if (!res.ok) { + return { state: 'unavailable', detail: `hot path returned HTTP ${res.status}` }; + } + const report = (await res.json()) as { + xxh3_64?: string; + valid_interop_value?: boolean; + interop_error?: string | null; + }; + if (typeof report.xxh3_64 !== 'string') { + return { state: 'unavailable', detail: 'hot path returned no checksum' }; + } + if (!report.valid_interop_value) { + return { + state: 'invalid', + xxh3: report.xxh3_64, + detail: report.interop_error ?? 'payload is not a valid interop/v1 document', + }; + } + return { state: 'verified', xxh3: report.xxh3_64 }; + } catch (error) { + return { + state: 'unavailable', + detail: error instanceof Error ? error.message : 'hot path fetch failed', + }; + } +} + /** Test hook: reset per-isolate counters. */ export function resetStats(): void { stats.hits = 0; @@ -82,9 +139,15 @@ function isWindow(value: string | null): value is Window { * * Unknown operation → 404, missing/invalid window → 400 — both before any * cache read. Backend failures → 502, undecodable entries → 500; errors are - * surfaced, never masked with fake data. + * surfaced, never masked with fake data. Stage 3: hot-path integrity-check + * failure → 500 (integrity_check_failed); hot path unreachable or binding + * missing → served with x-hotpath: unavailable. */ -export async function handleApi(url: URL, backend: Backend): Promise { +export async function handleApi( + url: URL, + backend: Backend, + hotpath?: HotpathBinding, +): Promise { const segment = url.pathname.slice('/api/'.length); if (segment === 'stats') { @@ -140,6 +203,31 @@ export async function handleApi(url: URL, backend: Backend): Promise { ); } + // Stage 3 (AC-3): every payload served through the edge is integrity-checked + // on the Rust-WASM hot path first. 'invalid' is a hard stop — a corrupt entry + // must never be served; 'unavailable' degrades honestly: the aggregate is + // real (it came from the backend), it just goes out unverified and says so. + // A missing binding is a deploy-config fault, not a reason to go silent: + // it degrades exactly like an unreachable hot path, header and all. + const verdict: HotpathVerdict = hotpath + ? await verifyViaHotpath(hotpath, raw) + : { state: 'unavailable', detail: 'HOTPATH service binding is not configured' }; + // Structured fields: wrangler tail / Workers Logs can filter on these. + console.log('hotpath_verify', { operation: segment, window, ...verdict }); + if (verdict.state === 'invalid') { + stats.errors += 1; + return json(500, { + error: 'integrity_check_failed', + detail: verdict.detail, + key, + xxh3_64: verdict.xxh3, + }); + } + const hotpathHeaders: Record = { 'x-hotpath': verdict.state }; + if (verdict.state === 'verified') { + hotpathHeaders['x-hotpath-xxh3'] = verdict.xxh3; + } + let data: unknown; try { data = decodeInteropValue(raw); @@ -153,5 +241,5 @@ export async function handleApi(url: URL, backend: Backend): Promise { } stats.hits += 1; - return json(200, { operation: segment, window, data }, { 'x-cache': 'HIT' }); + return json(200, { operation: segment, window, data }, { 'x-cache': 'HIT', ...hotpathHeaders }); } diff --git a/edge/src/worker.ts b/edge/src/worker.ts index f238371..9727f61 100644 --- a/edge/src/worker.ts +++ b/edge/src/worker.ts @@ -7,12 +7,14 @@ * guidance: create once, reuse across requests). */ import { cachekitio, type Backend } from '@cachekit-io/cachekit'; -import { handleApi } from './handler.js'; +import { handleApi, type HotpathBinding } from './handler.js'; interface Env { CACHEKIT_API_KEY?: string; - /** Override for tests/staging; defaults to https://api.cachekit.io. */ + /** Override for the dev instance / tests; defaults to https://api.cachekit.io. */ CACHEKIT_API_URL?: string; + /** Service binding to the Rust-WASM hot-path Worker (wrangler [[services]]). */ + HOTPATH?: HotpathBinding; } let backend: Backend | null = null; @@ -42,9 +44,12 @@ export default { backend ??= cachekitio({ apiKey: env.CACHEKIT_API_KEY, - ...(env.CACHEKIT_API_URL ? { apiUrl: env.CACHEKIT_API_URL } : {}), + // A non-default apiUrl (the dev instance) is outside the SDK's SSRF + // allowlist; the value comes from wrangler config, so opting out is + // an operator decision, not a request-time one. + ...(env.CACHEKIT_API_URL ? { apiUrl: env.CACHEKIT_API_URL, allowCustomHost: true } : {}), }); - return handleApi(url, backend); + return handleApi(url, backend, env.HOTPATH); }, }; diff --git a/edge/test/handler.test.ts b/edge/test/handler.test.ts index 6ce5309..61ee917 100644 --- a/edge/test/handler.test.ts +++ b/edge/test/handler.test.ts @@ -11,7 +11,13 @@ import { BackendError, type Backend, } from '@cachekit-io/cachekit'; -import { handleApi, resetStats, NAMESPACE, OPERATIONS } from '../src/handler.js'; +import { + handleApi, + resetStats, + NAMESPACE, + OPERATIONS, + type HotpathBinding, +} from '../src/handler.js'; function mockBackend(get: Backend['get']): Backend { return { @@ -166,3 +172,72 @@ describe('GET /api/stats', () => { expect(await res.json()).toEqual({ hits: 0, misses: 0, errors: 0, hit_rate: null }); }); }); + +describe('hot-path integrity verification (Stage 3, AC-3)', () => { + const aggregate = { window: '5m', ppm: 42.0 }; + const key = generateInteropKey(NAMESPACE, 'posts_per_minute', ['5m']); + const backend = () => storeOf({ [key]: aggregate }); + + function mockHotpath(respond: () => Promise): HotpathBinding { + return { fetch: respond }; + } + + it('serves verified payloads with x-hotpath headers', async () => { + const hotpath = mockHotpath(async () => + Response.json({ + xxh3_64: 'deadbeefcafef00d', + valid_interop_value: true, + interop_error: null, + }), + ); + const res = await handleApi(api('/api/posts_per_minute?window=5m'), backend(), hotpath); + expect(res.status).toBe(200); + expect(res.headers.get('x-cache')).toBe('HIT'); + expect(res.headers.get('x-hotpath')).toBe('verified'); + expect(res.headers.get('x-hotpath-xxh3')).toBe('deadbeefcafef00d'); + }); + + it('refuses to serve a payload the hot path calls invalid', async () => { + const hotpath = mockHotpath(async () => + Response.json({ + xxh3_64: 'deadbeefcafef00d', + valid_interop_value: false, + interop_error: 'trailing bytes', + }), + ); + const res = await handleApi(api('/api/posts_per_minute?window=5m'), backend(), hotpath); + expect(res.status).toBe(500); + const body = (await res.json()) as { error: string }; + expect(body.error).toBe('integrity_check_failed'); + }); + + it('degrades honestly when the hot path is unavailable: real data, unverified header', async () => { + const hotpath = mockHotpath(async () => { + throw new Error('binding unavailable'); + }); + const res = await handleApi(api('/api/posts_per_minute?window=5m'), backend(), hotpath); + expect(res.status).toBe(200); + expect(res.headers.get('x-hotpath')).toBe('unavailable'); + expect(res.headers.get('x-hotpath-xxh3')).toBeNull(); + const body = (await res.json()) as { data: { ppm: number } }; + expect(body.data.ppm).toBe(42.0); + }); + + it('keeps the miss contract untouched: 404 + X-Cache MISS, no hot-path call', async () => { + let called = false; + const hotpath = mockHotpath(async () => { + called = true; + return Response.json({}); + }); + const res = await handleApi(api('/api/posts_per_minute?window=1h'), backend(), hotpath); + expect(res.status).toBe(404); + expect(res.headers.get('x-cache')).toBe('MISS'); + expect(called).toBe(false); + }); + + it('reports unavailable when no HOTPATH binding is configured at all', async () => { + const res = await handleApi(api('/api/posts_per_minute?window=5m'), backend()); + expect(res.status).toBe(200); + expect(res.headers.get('x-hotpath')).toBe('unavailable'); + }); +}); diff --git a/edge/wrangler.toml b/edge/wrangler.toml index 13e796b..7e7a61a 100644 --- a/edge/wrangler.toml +++ b/edge/wrangler.toml @@ -1,11 +1,34 @@ -# Skyline edge API + dashboard. Deploy/routing is Stage 4; this config is -# complete for `wrangler dev` and a plain `wrangler deploy` on the free plan. -# The CACHEKIT_API_KEY secret is provisioned in Stage 3 -# (docs/architecture.md provisioning runbook): -# wrangler secret put CACHEKIT_API_KEY +# Skyline edge API + dashboard. Production routing is Stage 4; this config +# deploys the dev worker on the free plan. +# The CACHEKIT_API_KEY secret comes from op://cachekit/ck-dev-bluesky-default +# (docs/architecture.md#credentials): +# op read "op://cachekit/ck-dev-bluesky-default/credential" | wrangler secret put CACHEKIT_API_KEY name = "skyline-edge" main = "src/worker.ts" compatibility_date = "2026-07-01" +# nodejs_compat: @cachekit-io/cachekit 0.1.3 transitively requires node +# builtins (string_decoder via redis-parser) that esbuild can't resolve +# for Workers without it. +compatibility_flags = ["nodejs_compat"] + +# @cachekit-io/cachekit 0.1.3 statically imports its NAPI core (.node +# binaries — unbundleable and unrunnable on Workers). The edge never executes +# that path (interop reads only); alias it to a throwing stub. 0.1.4 fixes +# this properly with a WASM core but is uninstallable today (LAB-780). +[alias] +"@cachekit-io/cachekit-core-ts" = "./src/core-ts-stub.ts" + +[vars] +# The dev.cachekit instance. Not in the SDK's SSRF host allowlist, hence the +# explicit config-level opt-out (trusted operator config, same trust model as +# the Python CACHEKIT_ALLOW_CUSTOM_HOST env var). +CACHEKIT_API_URL = "https://api.dev.cachekit.io" + +# Stage 3 (AC-3): the Rust-WASM hot path, reached via service binding — not +# a public URL. Every served aggregate is integrity-checked there first. +[[services]] +binding = "HOTPATH" +service = "skyline-hotpath" # Static dashboard. Asset requests are served before the worker script runs; # unmatched paths (/api/*) fall through to the fetch handler. diff --git a/hotpath/Cargo.lock b/hotpath/Cargo.lock index 94f7d1d..7af8851 100644 --- a/hotpath/Cargo.lock +++ b/hotpath/Cargo.lock @@ -2,43 +2,6 @@ # It is not intended for manual editing. version = 4 -[[package]] -name = "aead" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" -dependencies = [ - "crypto-common", - "generic-array", -] - -[[package]] -name = "aes" -version = "0.8.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" -dependencies = [ - "cfg-if", - "cipher", - "cpufeatures 0.2.17", - "zeroize", -] - -[[package]] -name = "aes-gcm" -version = "0.10.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "831010a0f742e1209b3bcea8fab6a8e149051ba6099432c8cb2cc117dec3ead1" -dependencies = [ - "aead", - "aes", - "cipher", - "ctr", - "ghash", - "subtle", - "zeroize", -] - [[package]] name = "anstream" version = "1.0.0" @@ -166,35 +129,15 @@ version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ee6235f73aefb0dc66b9cd0d333b9da928c8192e4234357a739fe756c2f8f23" dependencies = [ - "aes", - "aes-gcm", "byteorder", "bytes", "cbindgen", - "generic-array", - "getrandom 0.2.17", - "hkdf", - "hmac", "lz4_flex", - "ring", "rmp-serde", "serde", "serde_bytes", - "sha2", "thiserror 2.0.19", "xxhash-rust", - "zeroize", -] - -[[package]] -name = "cachekit-macros" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712c2fb36da45d979a5ef178f455aa6395fac894382acbb5761f936d3da5cce1" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.119", ] [[package]] @@ -207,7 +150,6 @@ dependencies = [ "blake2", "bytes", "cachekit-core", - "cachekit-macros", "getrandom 0.2.17", "hex", "js-sys", @@ -271,8 +213,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" dependencies = [ "cfg-if", - "cpufeatures 0.3.0", - "rand_core 0.10.1", + "cpufeatures", + "rand_core", ] [[package]] @@ -286,16 +228,6 @@ dependencies = [ "wasm-bindgen", ] -[[package]] -name = "cipher" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" -dependencies = [ - "crypto-common", - "inout", -] - [[package]] name = "clap" version = "4.6.4" @@ -329,15 +261,6 @@ version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d07550c9036bf2ae0c684c4297d503f838287c83c53686d05370d0e139ae570" -[[package]] -name = "cpufeatures" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280" -dependencies = [ - "libc", -] - [[package]] name = "cpufeatures" version = "0.3.0" @@ -354,19 +277,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" dependencies = [ "generic-array", - "rand_core 0.6.4", "typenum", ] -[[package]] -name = "ctr" -version = "0.9.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835" -dependencies = [ - "cipher", -] - [[package]] name = "digest" version = "0.10.7" @@ -519,20 +432,10 @@ dependencies = [ "js-sys", "libc", "r-efi", - "rand_core 0.10.1", + "rand_core", "wasm-bindgen", ] -[[package]] -name = "ghash" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1" -dependencies = [ - "opaque-debug", - "polyval", -] - [[package]] name = "hashbrown" version = "0.17.1" @@ -551,24 +454,6 @@ version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" -[[package]] -name = "hkdf" -version = "0.12.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7" -dependencies = [ - "hmac", -] - -[[package]] -name = "hmac" -version = "0.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" -dependencies = [ - "digest", -] - [[package]] name = "http" version = "1.4.2" @@ -780,15 +665,6 @@ dependencies = [ "hashbrown", ] -[[package]] -name = "inout" -version = "0.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" -dependencies = [ - "generic-array", -] - [[package]] name = "ipnet" version = "2.12.0" @@ -901,12 +777,6 @@ version = "1.70.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" -[[package]] -name = "opaque-debug" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" - [[package]] name = "percent-encoding" version = "2.3.2" @@ -939,18 +809,6 @@ version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" -[[package]] -name = "polyval" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25" -dependencies = [ - "cfg-if", - "cpufeatures 0.2.17", - "opaque-debug", - "universal-hash", -] - [[package]] name = "potential_utf" version = "0.1.5" @@ -1048,16 +906,7 @@ checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80" dependencies = [ "chacha20", "getrandom 0.4.3", - "rand_core 0.10.1", -] - -[[package]] -name = "rand_core" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" -dependencies = [ - "getrandom 0.2.17", + "rand_core", ] [[package]] @@ -1072,7 +921,7 @@ version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "caa0f4137e1c0a72f4c651489402276c8e8e1cf081f3b0ba156d2cbeef09e86a" dependencies = [ - "rand_core 0.10.1", + "rand_core", ] [[package]] @@ -1308,17 +1157,6 @@ dependencies = [ "serde", ] -[[package]] -name = "sha2" -version = "0.10.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" -dependencies = [ - "cfg-if", - "cpufeatures 0.2.17", - "digest", -] - [[package]] name = "shlex" version = "2.0.1" @@ -1335,6 +1173,7 @@ dependencies = [ "hex", "serde", "serde_json", + "urlencoding", "wasm-bindgen", "worker", ] @@ -1650,16 +1489,6 @@ version = "1.0.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" -[[package]] -name = "universal-hash" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea" -dependencies = [ - "crypto-common", - "subtle", -] - [[package]] name = "untrusted" version = "0.9.0" diff --git a/hotpath/Cargo.toml b/hotpath/Cargo.toml index 6573557..c2bc16f 100644 --- a/hotpath/Cargo.toml +++ b/hotpath/Cargo.toml @@ -28,8 +28,13 @@ hex = "0.4" worker = "0.4" serde_json = "1" base64 = "0.22" +# Same key encoding WorkersCachekitIO uses; already in the tree via cachekit-rs. +# Direct dep for the LAB-1079 workaround fetch in lib.rs. +urlencoding = "2" # Pin the bindgen ABI to the pre-installed CLI (docs/architecture.md build pins). wasm-bindgen = "=0.2.126" -# Workers feature set exactly as spike-proven; WorkersCachekitIO is the only -# CachekitIO backend that implements Backend on wasm32. -cachekit-rs = { version = "0.5.0", default-features = false, features = ["workers", "cachekitio", "encryption", "macros"] } +# `workers, cachekitio` are staged for the LAB-1079 swap-back to +# WorkersCachekitIO — nothing uses them today (the workaround fetches via the +# `worker` crate directly; only cachekit::interop is consumed, and that module +# is unconditional). `encryption`/`macros` were never used: dropped. +cachekit-rs = { version = "0.5.0", default-features = false, features = ["workers", "cachekitio"] } diff --git a/hotpath/README.md b/hotpath/README.md index 3eeeccd..5d8464e 100644 --- a/hotpath/README.md +++ b/hotpath/README.md @@ -13,7 +13,7 @@ Dev deployment: **https://skyline-hotpath.raywalker.workers.dev** | interop/v1 key derivation | `GET /v1/key/:operation/:window` | The five locked operations × `5m`/`1h`/`24h` (contract: [`docs/architecture.md`](../docs/architecture.md)). Returns the key + locked TTL. Off-contract input → 400. | | Payload integrity | `POST /v1/verify[?expected=<16-hex>]` | Body = raw cached payload. Returns xxHash3-64 (big-endian hex, the `StorageEnvelope` convention) + strict interop/v1 validity (single MessagePack document, no trailing bytes, CK frames flagged with a diagnostic). | | Window-slice aggregation | `POST /v1/merge` | JSON `{"slices": ["", …], "top": 50}` → merged top-N counts (count desc, key asc) + the canonical interop/v1 MessagePack of the result, ready to write back byte-identically. | -| Cache read + verify | `GET /v1/cache/:operation/:window` | Derives the key, fetches via `WorkersCachekitIO`, checksums + strict-decodes the payload. **503 until the Stage-3 `CACHEKIT_API_KEY` secret is provisioned** — by design, not a bug. | +| Cache read + verify | `GET /v1/cache/:operation/:window` | Derives the key, fetches the live backend, checksums + strict-decodes the payload. 503 only if the `CACHEKIT_API_KEY` secret is missing (set since Stage 3). The fetch is a direct `worker::Fetch` GET — **LAB-1079 workaround**: `WorkersCachekitIO` (cachekit-rs ≤ 0.8.0) panics on wasm32 (`SystemTime::now()` in its session headers); swap back once the SDK fix ships. Key derivation, interop decode and checksum stay on cachekit-rs / cachekit-core. | | Service info | `GET /` | Contract summary + endpoint list; doubles as a health check. | Example — the byte-locked spike vector, derived live on the edge: @@ -47,5 +47,12 @@ $ worker-build --release # reproducible wasm32 build (th $ npx wrangler deploy # runs worker-build itself, then uploads ``` -Secrets: `CACHEKIT_API_KEY` via `wrangler secret put CACHEKIT_API_KEY` once the Stage-3 -provisioning runbook has been run. Nothing else is configurable. +Secrets: `CACHEKIT_API_KEY` via `wrangler secret put CACHEKIT_API_KEY`, from +`op://cachekit/ck-dev-bluesky-default/credential` +([docs/architecture.md#credentials](../docs/architecture.md#credentials)). +Config: `CACHEKIT_API_URL` (`wrangler.toml [vars]`) points at the dev +instance. Nothing else is configurable. + +Stage 3 also bound this Worker into the TS edge's serving path: the edge +holds a service binding (`env.HOTPATH`) and `POST /v1/verify`s every payload +it serves — see [`edge/README.md`](../edge/README.md). diff --git a/hotpath/src/lib.rs b/hotpath/src/lib.rs index fec457d..5550c13 100644 --- a/hotpath/src/lib.rs +++ b/hotpath/src/lib.rs @@ -17,8 +17,6 @@ pub mod compute; #[cfg(target_arch = "wasm32")] mod edge { use base64::Engine as _; - use cachekit::backend::workers::WorkersCachekitIO; - use cachekit::backend::Backend as _; use serde::Deserialize; use serde_json::json; use worker::*; @@ -140,6 +138,44 @@ mod edge { } } + /// Raw CachekitIO GET over `worker::Fetch`. + /// + /// LAB-1079 workaround: `WorkersCachekitIO` (cachekit-rs ≤ 0.8.0) panics on + /// every request — its `session_headers()` calls `SystemTime::now()`, which + /// is unimplemented on wasm32-unknown-unknown. Until the SDK fix ships, + /// the hot path does the one HTTP verb it needs directly; key derivation, + /// strict interop/v1 decode and the xxHash3 checksum stay on cachekit-rs / + /// cachekit-core. Swap back to `WorkersCachekitIO::builder()` (with + /// `.api_url(...).allow_custom_host(true)` for the dev instance) once + /// LAB-1079 is fixed and published. + async fn backend_get(api_url: &str, api_key: &str, key: &str) -> std::result::Result>, String> { + let url = format!("{}/v1/cache/{}", api_url.trim_end_matches('/'), urlencoding::encode(key)); + let mut headers = Headers::new(); + headers + .set("Authorization", &format!("Bearer {api_key}")) + .map_err(|e| format!("failed to set auth header: {e}"))?; + // SDK-class keys require an L1 status on every /v1/cache request. + headers + .set("X-CacheKit-L1-Status", "disabled") + .map_err(|e| format!("failed to set header: {e}"))?; + + let mut init = RequestInit::new(); + init.with_method(Method::Get).with_headers(headers); + let request = Request::new_with_init(&url, &init).map_err(|e| format!("failed to build request: {e}"))?; + let mut resp = Fetch::Request(request) + .send() + .await + .map_err(|e| format!("fetch failed: {e}"))?; + + match resp.status_code() { + 200 => Ok(Some(resp.bytes().await.map_err(|e| format!("failed to read body: {e}"))?)), + 404 => Ok(None), + // Never echo the response body: it is not ours and error bodies + // can carry request context. Status code only. + status => Err(format!("backend returned HTTP {status}")), + } + } + async fn cache_handler(ctx: &RouteContext<()>) -> Result { let (operation, window) = path_params(ctx)?; let key = match compute::derive_key(operation, window) { @@ -149,14 +185,14 @@ mod edge { let Ok(api_key) = ctx.secret("CACHEKIT_API_KEY") else { return json_error( 503, - "CACHEKIT_API_KEY secret not configured — live cache reads activate in Stage 3", + "CACHEKIT_API_KEY secret not configured — see docs/architecture.md#credentials", ); }; - let backend = WorkersCachekitIO::builder() - .api_key(api_key.to_string()) - .build() - .map_err(|e| Error::RustError(e.to_string()))?; - let bytes = match backend.get(&key).await { + let api_url = ctx + .var("CACHEKIT_API_URL") + .map(|v| v.to_string()) + .unwrap_or_else(|_| "https://api.cachekit.io".to_string()); + let bytes = match backend_get(&api_url, &api_key.to_string(), &key).await { Ok(Some(bytes)) => bytes, Ok(None) => { return Response::from_json(&json!({ "key": key, "found": false })) diff --git a/hotpath/wrangler.toml b/hotpath/wrangler.toml index 3aa1419..4bea2b6 100644 --- a/hotpath/wrangler.toml +++ b/hotpath/wrangler.toml @@ -7,6 +7,14 @@ compatibility_date = "2026-07-01" [build] command = "worker-build --release" +[vars] +# The dev.cachekit instance (docs/architecture.md#credentials). The LAB-1079 +# raw-fetch workaround in lib.rs bypasses the SDK's SSRF allowlist entirely — +# this value is trusted operator config, a deploy-time decision. Restore +# `.api_url(...).allow_custom_host(true)` when swapping back to +# WorkersCachekitIO. +CACHEKIT_API_URL = "https://api.dev.cachekit.io" + # Secrets (set via `wrangler secret put`, never in this file): -# - CACHEKIT_API_KEY — activates GET /v1/cache/…; absent until the Stage-3 -# provisioning runbook runs (docs/architecture.md). +# - CACHEKIT_API_KEY — activates GET /v1/cache/…; from +# op://cachekit/ck-dev-bluesky-default/credential (docs/architecture.md#credentials). diff --git a/ingester/README.md b/ingester/README.md index aa509cc..a539f53 100644 --- a/ingester/README.md +++ b/ingester/README.md @@ -11,19 +11,25 @@ byte-identical to what the TS edge API and Rust-WASM hot path read. cd ingester uv sync -# Live: writes real CachekitIO entries (key from the provisioning runbook) -CACHEKIT_API_KEY=ck_live_... uv run skyline-ingester +# Live: writes real CachekitIO entries. Creds per docs/architecture.md#credentials: +CACHEKIT_API_URL=https://api.dev.cachekit.io CACHEKIT_ALLOW_CUSTOM_HOST=true \ + op run --env-file=../.op.env -- uv run skyline-ingester # Dry-run: no key -> same pipeline, in-process backend, every write logged uv run skyline-ingester ``` -Configuration (env or `.env`, via pydantic-settings; secrets are `SecretStr`): +Configuration (env or `.env`, via pydantic-settings; secrets are `SecretStr`). +In live mode the backend itself is built by the SDK's env-config path, so the +`CACHEKIT_*` backend variables must be real process env vars (`op run` +provides that) — the SDK does not read this service's `.env` file: | Variable | Default | Meaning | | :--- | :--- | :--- | | `CACHEKIT_API_KEY` | unset | CachekitIO key. Unset → dry-run mode. | | `CACHEKIT_MASTER_KEY` | unset | 64-hex master key for the `@cache.secure` sentiment cache. **Required in live mode** (fail closed — a live deploy without it refuses to start); unset in dry-run → secure cache disabled with a warning. | +| `CACHEKIT_API_URL` | `https://api.cachekit.io` | Backend endpoint (the demo uses the dev instance, `https://api.dev.cachekit.io`). | +| `CACHEKIT_ALLOW_CUSTOM_HOST` | unset | Required `true` for the dev instance — its hostname is outside the SDK's SSRF allowlist. | | `JETSTREAM_URL` | `wss://jetstream2.us-east.bsky.network/subscribe` | Jetstream endpoint. | | `PUBLISH_TICK_SECONDS` | `15` | Publish-loop poll interval. | | `CHECKPOINT_INTERVAL_SECONDS` | `120` | Window-state checkpoint cadence. | diff --git a/ingester/src/skyline_ingester/__main__.py b/ingester/src/skyline_ingester/__main__.py index 07708ac..d291f69 100644 --- a/ingester/src/skyline_ingester/__main__.py +++ b/ingester/src/skyline_ingester/__main__.py @@ -9,6 +9,7 @@ import asyncio import logging +import os import time from skyline_ingester.backends import MemoryBytesBackend @@ -28,8 +29,18 @@ def build_publisher(settings: Settings, store: WindowStore) -> Publisher: raise RuntimeError("CACHEKIT_MASTER_KEY is required in live mode: the secure sentiment cache must fail closed") from cachekit.backends.cachekitio import CachekitIOBackend - backend = CachekitIOBackend(api_key=settings.cachekit_api_key.get_secret_value()) - logger.info("live mode: publishing to CachekitIO") + # No-args = the SDK's env-config path (CACHEKIT_API_KEY, plus optional + # CACHEKIT_API_URL / CACHEKIT_ALLOW_CUSTOM_HOST for the dev instance). + # Passing api_key alone is rejected by the SDK ("Both api_url and + # api_key required if using manual config"), so live mode never came up + # before this fix. + if "CACHEKIT_API_KEY" not in os.environ: + # Settings reads .env; the SDK's env config reads process env only. + # Without this guard a .env-only key selects live mode and then dies + # in the SDK with a misleading "api_key Field required". + raise RuntimeError("CACHEKIT_API_KEY must be a real environment variable in live mode (the SDK does not read .env)") + backend = CachekitIOBackend() + logger.info("live mode: publishing to CachekitIO at %s", os.environ.get("CACHEKIT_API_URL", "https://api.cachekit.io")) else: backend = MemoryBytesBackend(log_writes=True) logger.warning("CACHEKIT_API_KEY not set — dry-run mode, writes are logged only") diff --git a/ingester/tests/test_secure.py b/ingester/tests/test_secure.py index 118df5c..f2c0d51 100644 --- a/ingester/tests/test_secure.py +++ b/ingester/tests/test_secure.py @@ -1,7 +1,13 @@ """AC-6 groundwork: the secure sentiment cache stores ciphertext only.""" +import pytest +from pydantic import SecretStr + from skyline_ingester import NAMESPACE +from skyline_ingester.__main__ import build_publisher +from skyline_ingester.config import Settings from skyline_ingester.publisher import Publisher +from skyline_ingester.windows import WindowStore from .conftest import MASTER_KEY, NOW @@ -35,13 +41,40 @@ def test_no_master_key_disables_secure_cache(store, backend): def test_live_mode_without_master_key_fails_closed(): """Epic decision (ray, 2026-07-24): a live deploy missing the secure-cache master key must refuse to start, not come up with AC-6 silently absent.""" - import pytest - from pydantic import SecretStr - - from skyline_ingester.__main__ import build_publisher - from skyline_ingester.config import Settings - from skyline_ingester.windows import WindowStore - settings = Settings(cachekit_api_key=SecretStr("ck_test_not_a_real_key"), cachekit_master_key=None) with pytest.raises(RuntimeError, match="fail closed"): build_publisher(settings, WindowStore()) # raises before any backend is constructed + + +def test_live_mode_builds_backend_from_env(monkeypatch): + """Live mode must construct CachekitIOBackend via the SDK's env-config path. + + Regression (LAB-737): passing api_key alone to the constructor raises + "Both api_url and api_key required if using manual config", so the + pre-Stage-3 live path could never start. Env config also carries the + CACHEKIT_API_URL / CACHEKIT_ALLOW_CUSTOM_HOST overrides the dev instance + (api.dev.cachekit.io — not in the SDK's SSRF host allowlist) needs. + """ + monkeypatch.setenv("CACHEKIT_API_KEY", "ck_test_not_a_real_key") + monkeypatch.setenv("CACHEKIT_API_URL", "https://api.dev.cachekit.io") + monkeypatch.setenv("CACHEKIT_ALLOW_CUSTOM_HOST", "true") + settings = Settings( + cachekit_api_key=SecretStr("ck_test_not_a_real_key"), + cachekit_master_key=SecretStr("a" * 64), + ) + # Old code raised ValueError here; construction makes no network calls. + publisher = build_publisher(settings, WindowStore()) + assert publisher.secure_enabled + + +def test_live_mode_requires_key_in_process_env(monkeypatch): + """A .env-only key selects live mode but the SDK's env config reads process + env only — the guard must fail with a clear message, not the SDK's + misleading "api_key Field required" (panel finding, LAB-737).""" + monkeypatch.delenv("CACHEKIT_API_KEY", raising=False) + settings = Settings( + cachekit_api_key=SecretStr("ck_test_from_dotenv_only"), + cachekit_master_key=SecretStr("a" * 64), + ) + with pytest.raises(RuntimeError, match="real environment variable"): + build_publisher(settings, WindowStore()) diff --git a/stage3/README.md b/stage3/README.md new file mode 100644 index 0000000..bbd5c9b --- /dev/null +++ b/stage3/README.md @@ -0,0 +1,21 @@ +# Stage-3 live-integration evidence harness (LAB-737) + +Repeatable proofs against the live `dev.cachekit` backend. Credentials per +[`docs/architecture.md#credentials`](../docs/architecture.md#credentials); +every command below runs under `op run` so nothing secret touches disk or +logs. All three scripts run from `ingester/` — `derive_keys.py` and +`stampede.py` reuse its venv; `raw_read.py` is deliberately SDK-free and +brings its own deps via `--with`: + +```bash +cd ingester +export CACHEKIT_API_URL=https://api.dev.cachekit.io CACHEKIT_ALLOW_CUSTOM_HOST=true +``` + +| Script | Proof | Run | +| :--- | :--- | :--- | +| `derive_keys.py` | Prints all 17 keys the ingester writes (15 interop/v1 aggregates + auto-mode checkpoint + `@cache.secure` sentiment), from the real decorator machinery. No network. | `uv run python ../stage3/derive_keys.py` | +| `raw_read.py` | SDK-free reader over the raw HTTP API. `--expect absent` = AC-2 clean-namespace audit; default = AC-4 byte/checksum evidence (xxHash3-64, strict single-document MessagePack check); `--expect ciphertext --forbid …` = AC-6 zero-knowledge check; `--delete` = cleanup. | `op run --env-file=../.op.apikey.env -- uv run --with httpx --with xxhash --with msgpack python ../stage3/raw_read.py [flags] KEY…` | +| `stampede.py` | AC-5: N=12 concurrent async callers on one cold key → exactly one recompute, real `POST/DELETE …/lock` SaaS traffic in the httpx log. The cached function must be async — cachekit-py's sync wrapper does no distributed locking. | `op run --env-file=../.op.apikey.env -- uv run python ../stage3/stampede.py` | + +Recorded Stage-3 results (2026-07-29) live on LAB-737. diff --git a/stage3/derive_keys.py b/stage3/derive_keys.py new file mode 100644 index 0000000..e52d94c --- /dev/null +++ b/stage3/derive_keys.py @@ -0,0 +1,38 @@ +"""Print every cache key the ingester writes (LAB-737 evidence tooling). + +Drives the real Publisher against the in-process bytes backend, so the keys +come from the same decorator machinery the live service uses — the 15 +interop/v1 aggregate keys, the auto-mode checkpoint key, and the auto-mode +@cache.secure sentiment key. No network, no credentials. + + cd ingester && uv run python ../stage3/derive_keys.py +""" + +from __future__ import annotations + +import hashlib +import time + +from skyline_ingester.backends import MemoryBytesBackend +from skyline_ingester.publisher import Publisher +from skyline_ingester.windows import WINDOW_TTLS, WindowStore + +# Inert 64-hex value that unlocks the Publisher's secure-cache code path so +# its key NAME gets derived — key derivation never mixes the master key in, +# so any value yields the same keys. Runtime-derived from a public string so +# no literal in source is, or could be swapped for, real key material. +PLACEHOLDER_MASTER_KEY = hashlib.sha256(b"skyline-derive-keys-placeholder-not-a-secret").hexdigest() + + +def main() -> None: + backend = MemoryBytesBackend() + publisher = Publisher(WindowStore(), backend, master_key=PLACEHOLDER_MASTER_KEY, now_fn=time.time) + for window in WINDOW_TTLS: + publisher.publish_window(window) + publisher.checkpoint() + for key in sorted(backend.stored_keys()): + print(key) + + +if __name__ == "__main__": + main() diff --git a/stage3/raw_read.py b/stage3/raw_read.py new file mode 100644 index 0000000..31bc56f --- /dev/null +++ b/stage3/raw_read.py @@ -0,0 +1,123 @@ +"""Independent raw reader for Stage-3 evidence (LAB-737). + +Fetches cache entries straight off the SaaS HTTP API — no SDK, no decorator, +no decryption — so what it prints is exactly what the backend stores. Used +for three proofs: + +- AC-2: the namespace is clean before the first live run (`--expect absent`) +- AC-4: the payload bytes a reader fetches are what the ingester wrote + (prints xxHash3-64 big-endian hex, the StorageEnvelope convention) +- AC-6: the @cache.secure value is ciphertext (`--expect ciphertext` asserts + the bytes are NOT a valid MessagePack document and contain none of the + `--forbid` plaintext markers) + +Env: CACHEKIT_API_KEY (required), CACHEKIT_API_URL (default api.cachekit.io). + + # from ingester/ (matches stage3/README.md; SDK-free — deps come from --with) + op run --env-file=../.op.apikey.env -- \ + uv run --with httpx --with xxhash --with msgpack \ + python ../stage3/raw_read.py [--expect absent|present|ciphertext] \ + [--forbid TEXT ...] [--hexdump N] KEY [KEY ...] + +Exit code 0 only if every key satisfies --expect. +""" + +from __future__ import annotations + +import argparse +import os +import sys +from urllib.parse import quote + +import httpx +import msgpack +import xxhash + + +def fetch(client: httpx.Client, api_url: str, key: str) -> tuple[int, bytes]: + resp = client.get(f"{api_url}/v1/cache/{quote(key, safe='')}") + return resp.status_code, resp.content + + +def is_msgpack_document(data: bytes) -> bool: + """Strict single-document check (interop/v1 rule: no trailing bytes).""" + try: + unpacker = msgpack.Unpacker(strict_map_key=False) + unpacker.feed(data) + unpacker.unpack() + return unpacker.tell() == len(data) + except Exception: + return False + + +def check_key(client: httpx.Client, api_url: str, key: str, args: argparse.Namespace) -> bool: + status, body = fetch(client, api_url, key) + if status == 404: + print(f"ABSENT {key}") + return args.expect == "absent" + if status != 200: + # Status only — error bodies are not ours and can carry request + # context; this output gets recorded as evidence on tickets. + print(f"ERROR {key}: HTTP {status}") + return False + + digest = xxhash.xxh3_64(body).intdigest() + plain_msgpack = is_msgpack_document(body) + print(f"PRESENT {key}") + print(f" size={len(body)} xxh3_64={digest:016x} msgpack_document={plain_msgpack}") + if args.hexdump: + prefix = body[: args.hexdump] + print(f" hex[:{len(prefix)}]={prefix.hex()}") + + if args.expect == "absent": + return False + ok = True + for text in args.forbid: + if text.encode() in body: + print(f" FORBIDDEN plaintext marker {text!r} found in stored bytes") + ok = False + if args.expect == "ciphertext" and plain_msgpack: + print(" FAIL: stored bytes decode as a plain MessagePack document") + ok = False + return ok + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("keys", nargs="+") + parser.add_argument("--expect", choices=["absent", "present", "ciphertext"], default="present") + parser.add_argument("--forbid", action="append", default=[], help="plaintext that must NOT appear in stored bytes") + parser.add_argument("--hexdump", type=int, default=0, metavar="N", help="print first N stored bytes as hex") + parser.add_argument("--delete", action="store_true", help="DELETE the keys instead of reading them (AC-2 cleanup)") + args = parser.parse_args() + + api_key = os.environ.get("CACHEKIT_API_KEY") + if not api_key: + sys.exit("CACHEKIT_API_KEY not set") + api_url = os.environ.get("CACHEKIT_API_URL", "https://api.cachekit.io").rstrip("/") + + headers = { + "Authorization": f"Bearer {api_key}", + # SDK-class keys require an L1 status on every cache request. + "X-CacheKit-L1-Status": "disabled", + } + failures = 0 + with httpx.Client(headers=headers, timeout=10.0) as client: + if args.delete: + for key in args.keys: + resp = client.delete(f"{api_url}/v1/cache/{quote(key, safe='')}") + print(f"DELETE {resp.status_code} {key}") + if resp.status_code not in (200, 204, 404): + failures += 1 + print(f"{'PASS' if failures == 0 else 'FAIL'}: deleted {len(args.keys) - failures}/{len(args.keys)} keys") + return 1 if failures else 0 + for key in args.keys: + if not check_key(client, api_url, key, args): + failures += 1 + verdict = "PASS" if failures == 0 else "FAIL" + print(f"{verdict}: {len(args.keys) - failures}/{len(args.keys)} keys satisfied --expect {args.expect}") + return 1 if failures else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/stage3/stampede.py b/stage3/stampede.py new file mode 100644 index 0000000..70fd0ae --- /dev/null +++ b/stage3/stampede.py @@ -0,0 +1,101 @@ +"""AC-5 (epic AC-5): the CachekitIO SaaS distributed lock prevents a stampede. + +N concurrent invocations hit one cold key; the SaaS lock +(`POST /v1/cache/{key}/lock`) must serialize the recompute so exactly one +executes and the rest are served the winner's value. + +The cached function is **async**, and that is load-bearing: cachekit-py's +sync wrapper does not do distributed locking at all ("Sync wrappers don't +support distributed locking (backend protocol is async-only)" — +decorators/wrapper.py). Only the async wrapper takes the +`hasattr(backend, "acquire_lock")` path this AC exercises. Empirically +confirmed on 0.15.0: the sync variant of this probe recorded 12/12 +recomputes and zero lock traffic. + +Instrumentation, per the AC: +- a recompute counter inside the cached function (must read exactly 1); +- httpx request logging (INFO) — the actual `POST …/lock` / `DELETE …/lock` + SaaS traffic, straight from the SDK's HTTP client, no mocking. + +Design notes (LAB-737 "two design gotchas"): +- the wrapper's lock waiters block up to 5 s, and the lock self-expires at + 30 s — so the recompute sleeps well under 5 s. A slower recompute would + make a *correct* system report > 1 execution. +- L1 is left enabled deliberately: it cannot mask a stampede on a cold key + (nothing is in L1 before the first recompute), and the wrapper's + double-checked locking is the property under test. + +Env: CACHEKIT_API_KEY (required), CACHEKIT_API_URL / CACHEKIT_ALLOW_CUSTOM_HOST +for the dev instance. + + op run --env-file=../.op.apikey.env -- \ + uv run python ../stage3/stampede.py +""" + +from __future__ import annotations + +import asyncio +import logging +import sys +import time + +from cachekit import cache +from cachekit.backends.cachekitio import CachekitIOBackend +from cachekit.interop import generate_interop_key + +NAMESPACE = "bluesky-thinking" +OPERATION = "stampede_probe" # dedicated key: never collides with the five locked aggregates +N = 12 +RECOMPUTE_SECONDS = 2.0 # < 5 s blocking_timeout, < 30 s lock_timeout + +recomputes = 0 + + +async def main() -> int: + logging.basicConfig(level=logging.WARNING, format="%(asctime)s %(name)s %(message)s") + logging.getLogger("httpx").setLevel(logging.INFO) # the SaaS lock traffic evidence + + backend = CachekitIOBackend() # env config, explicit instance + + @cache(interop=OPERATION, namespace=NAMESPACE, ttl=60, backend=backend) + async def stampede_probe(window: str) -> dict: + global recomputes + recomputes += 1 + generation = recomputes + await asyncio.sleep(RECOMPUTE_SECONDS) + return {"window": window, "generation": generation} + + key = generate_interop_key(NAMESPACE, OPERATION, ["5m"]) + backend.delete(key) # cold start — the whole point + print(f"cold key: {key}") + + started = time.perf_counter() + try: + results = await asyncio.gather(*(stampede_probe("5m") for _ in range(N))) + # Stop the clock before cleanup: elapsed measures the stampede window + # only, not the cleanup round-trip. + elapsed = time.perf_counter() - started + finally: + # Leave the namespace as we found it, even on failure — but never let + # a failing cleanup mask the probe's own exception. + try: + backend.delete(key) + except Exception: + logging.exception("probe-key cleanup failed: %s", key) + + distinct = {tuple(sorted(r.items())) for r in results} + print(f"\n{N} concurrent invocations finished in {elapsed:.2f}s") + print(f"recomputes: {recomputes}") + print(f"distinct results: {len(distinct)} -> {results[0]}") + if recomputes != 1: + print("FAIL: expected exactly one recompute") + return 1 + if len(distinct) != 1: + print("FAIL: callers saw different values") + return 1 + print("PASS: exactly one recompute; all callers served the winner's value") + return 0 + + +if __name__ == "__main__": + sys.exit(asyncio.run(main()))