diff --git a/CHANGELOG.md b/CHANGELOG.md index 8dc4e06..d4c140b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,13 @@ All notable changes to the CacheKit Protocol Specification. ### Specs +- Key rotation specified (not yet implemented, LAB-516): client-side keyring — + one forward-only current key plus ≤3 decrypt-only master keys; fingerprint-based + key selection where per-entry identity exists (cachekit-py frames), sequential + same-AAD attempts elsewhere; no wire change. Retires the never-written 32-byte + `RotationAwareHeader` from the spec. Rationale, rejected options, and operator + runbooks in [decisions/key-rotation.md](decisions/key-rotation.md). + - Interop mode promoted from draft to specified (interop/v1): flat canonical argument array (named→positional binding), number canonicalization (integral float64 → int, the only rule implementable in JS), code-point map-key ordering, encoded-byte set diff --git a/README.md b/README.md index 5ae4ae3..ef68fc7 100644 --- a/README.md +++ b/README.md @@ -70,6 +70,7 @@ layer's own store/retrieve flows are specified in | [spec/saas-api.md](spec/saas-api.md) | REST API endpoints, binary wire protocol, error codes, metrics headers | | [spec/interop-mode.md](spec/interop-mode.md) | Cross-SDK cache sharing — language-neutral key format, canonical argument normalization *(specified, not yet implemented in any SDK)* | | [sdk-feature-matrix.md](sdk-feature-matrix.md) | Feature parity tracking across Python, Rust, TypeScript, and PHP SDKs | +| [decisions/key-rotation.md](decisions/key-rotation.md) | Decision records — master-key rotation via client-side keyring (rationale, rejected options, operator runbooks) | --- diff --git a/decisions/key-rotation.md b/decisions/key-rotation.md new file mode 100644 index 0000000..df556b0 --- /dev/null +++ b/decisions/key-rotation.md @@ -0,0 +1,191 @@ +**[Protocol](../README.md)** > **Decisions** > **Key Rotation** + +# Decision Record: Master-Key Rotation — Client-Side Keyring with Grace Window + +| | | +| :--- | :--- | +| **Status** | Proposed (accepted on merge) | +| **Date** | 2026-07-23 | +| **Ticket** | LAB-516 (filed by the LAB-275 cross-SDK feature-gap audit) | +| **Normative spec** | [`spec/encryption.md` → Key Rotation (Keyring)](../spec/encryption.md#key-rotation-keyring) — the spec section owns the rules; this record owns the rationale and runbooks. | +| **Implementation** | Not yet shipped in any SDK — tracked as LAB-516 sub-issues. The [feature matrix](../sdk-feature-matrix.md) rotation row is corrected to ❌ fleet-wide by the LAB-275 audit PR ([protocol#29](https://github.com/cachekit-io/protocol/pull/29)) and flips per SDK only as each implementation ships. | + +--- + +## Context + +CacheKit's differentiator is zero-knowledge client-side encryption, which makes +"rotate the master key" a when-not-if operational event: compliance cadence, +employee offboarding, suspected compromise. As of 2026-07-23 no SDK ships a +usable rotation path (code-verified by the LAB-275 audit): + +- **cachekit-core 0.3.0**: `rotate_key()` returns + `EncryptionError::NotImplemented` (`src/encryption/core.rs:492`). + `RotationAwareHeader` and `KeyRotationState` exist as types but are + constructed only in tests — **no SDK writes the 32-byte header to the wire**, + despite `spec/encryption.md` describing it as the rotation mechanism. +- **cachekit-py**: PyO3 exposes `KeyRotationState` bindings + (`rust/src/python_bindings.rs:256-305`) that zero Python code references. The + only live behaviour is per-entry key-fingerprint **mismatch detection** + (`serializers/encryption_wrapper.py:341-354`): fail-open logs and lets + AES-GCM authentication fail (→ miss), fail-closed raises. There is no + decrypt-with-previous-key path. +- **cachekit-rs**: no rotation surface at all. +- **cachekit-ts**: nonce-exhaustion monitoring whose error text says "key + rotation required" (`src/errors.ts:99-103`) — with no rotation mechanism to + point at. + +Consequently, rotating `CACHEKIT_MASTER_KEY` today invalidates every encrypted +entry at once: fail-open readers take a fleet-wide miss storm (billable on the +SaaS metered-misses model, and a thundering herd at origin); fail-closed +readers take hard errors. + +Two structural facts drive the design: + +1. **The server can never help.** Zero-knowledge means the backend stores + ciphertext it cannot decrypt, so it cannot re-encrypt. Any rotation + mechanism is necessarily client-side. +2. **Cache entries are ephemeral and reproducible.** Every entry expires (TTL) + or can be recomputed from origin. Rotation never risks data loss — only + availability and cost. + +## Options considered + +### A. Client-side keyring with a grace window — **chosen** + +Configuration gains a small ordered list of decrypt-only master keys alongside +the current key. Writes always use the current key. Reads decrypt with +whichever keyring key encrypted the entry. Old-key entries age out via TTL or +get re-encrypted on the next write; after the longest TTL in use has elapsed, +the operator removes the retired key. + +- No wire-format change: AAD v0x03 does not include key identity, and the + ciphertext layout (`nonce ‖ ciphertext ‖ tag`) is untouched. Every deployed + ciphertext remains format-compatible and decryptable for as long as the key + that encrypted it is retained in the keyring (dropping a key — retirement, + compromise cut-over — is the deliberate exception). +- No server change, no SaaS involvement. +- No state machine and no rotation API: rotation state **is** configuration. +- Contains option B as its degenerate case: an empty decrypt-only list is a + hard cut-over. + +### B. Documented invalidate-on-rotate — rejected as the *only* answer + +"Caches are ephemeral; rotation = planned cold start." Zero new decrypt logic — +but it makes a routine compliance event a fleet-wide miss storm that is +billable under metered-misses pricing, hammers origin, and punishes exactly +the customers who bought the product for its encryption: the perverse +incentive is to *avoid* rotating. It remains the correct trade for compromise +response, where it appears as the empty-keyring configuration. + +### C. `RotationAwareHeader` on the wire (`key_version` byte) — rejected + +The direction the dead code in cachekit-core pointed at. Rejected on three +grounds: + +1. **It is a wire migration.** No SDK writes the header today, so shipping it + means every reader must handle header-present and header-absent entries + forever, for no benefit over what per-entry metadata and a bounded keyring + already provide. +2. **The versioning scheme is broken.** `KeyRotationState.decryption_key()` + interprets `key_version` *relatively* (`0` = original, `1` = rotated). Data + at rest cannot be renumbered, so the scheme cannot survive a second + rotation. Absolute key identity already exists — the 16-byte key + fingerprint (`SHA-256("key_fingerprint_v1" ‖ key)[0..16]`). +3. **It is redundant.** cachekit-py already stores the fingerprint per entry + as frame metadata; surfaces without per-entry metadata (cachekit-ts, + cachekit-rs, interop mode) are covered by bounded sequential key attempts. + +## Decision + +Adopt **option A: a client-side keyring with an operator-controlled grace +window**. The normative rules — keyring shape and cap, forward-only current +key, fingerprint-based selection vs sequential attempts, failure semantics, +key hygiene — live in +[`spec/encryption.md` → Key Rotation (Keyring)](../spec/encryption.md#key-rotation-keyring). +Points that are decision, not mechanism: + +- **The current key is forward-only.** A master key that has ever occupied the + encrypting slot is never re-promoted; backing out a bad rotation means + rotating forward to a fresh key. (Re-promotion resumes a used, unknowable + nonce budget — catastrophic for AES-GCM. The spec carries the MUST NOT.) + A stateless SDK cannot detect re-promotion once the retired key has left the + supplied configuration, so the invariant is **operator-enforced**: treat + retired key material as destroyed. SDKs enforce the detectable subset only — + a configuration where the current key also appears in the decrypt-only list + is rejected at load. +- **The cap is 3 decrypt-only keys.** Bounds worst-case sequential decrypt + attempts and memory while allowing a forced mid-window second rotation + (e.g. an offboarding landing during a long-TTL compliance window). + Exceeding the cap is a configuration error, rejected at load. +- **Dead code is deleted, not preserved**: `rotate_key()`, + `KeyRotationState`, `RotationAwareHeader` (and its `EncryptionHeader` + alias) leave cachekit-core; the unreferenced `KeyRotationState` PyO3 + bindings leave cachekit-py. No `NotImplemented` stub remains. + +### Runbooks (normative for docs) + +**Scheduled rotation** (compliance cadence, offboarding without suspected +exfiltration) — three phases; the mechanism supports zero rotation-caused +misses only with this choreography: + +0. *Precondition*: audit for non-expiring encrypted entries (e.g. writes with + no TTL where the backend permits them). Flush them or assign TTLs first — + otherwise the migration window never closes and, under fail-closed, those + entries become permanent errors at step 3. +1. Add the incoming key k₂ to every instance's decrypt-only list (writes + still use k₁). Deploy fleet-wide and wait for completion — every reader + can now decrypt both keys before any writer switches. +2. Promote k₂ to current, demote k₁ to the decrypt-only list. Deploy. + Mixed writers during rollout are safe — all readers already hold both + keys. **The migration-window clock starts when this deploy completes + fleet-wide** (the moment the last writer stopped encrypting under k₁). +3. After at least the longest TTL in use has elapsed since step 2 completed, + remove k₁. Deploy. + +A single-deploy swap (new current + old into decrypt-only in one step) is +**not** zero-miss: during the rollout, a not-yet-deployed reader holding only +k₁ cannot decrypt entries already written under k₂. + +**Compromise**: deploy a fresh current key with an **empty** decrypt-only list +immediately, and flush encrypted namespaces. Old entries that survive the +flush become authentication failures (misses under fail-open, errors under +fail-closed) — a deliberate cold start, because availability ranks below +confidentiality here. + +**Honest limit** (state it in every doc surface): rotation cannot +retroactively protect ciphertext already captured by an attacker who holds +the old key. That is cryptographic reality, not a CacheKit limitation. +Rotation bounds exposure going forward — hence the unconditional flush above. + +**Keyring exposure is all-keys exposure**: master-key material handled by +managed-language runtimes (environment variables, interpreter strings) cannot +be reliably scrubbed from memory. During a grace window the environment holds +the current *and* previous keys — operators MUST treat exposure of the keyring +configuration as exposure of every key in it. + +## Consequences (→ LAB-516 sub-issues) + +- **protocol**: normative spec section (ships with this record); keyring test + vectors — one value encrypted under key₁ and key₂, verified decryptable + with keyring `[k₂, k₁]` and rejected with keyring `[k₂]` — wired into the + existing `tools/encryption-verify.py` CI check. +- **cachekit-core**: delete `rotate_key()`, `KeyRotationState`, + `RotationAwareHeader`/`EncryptionHeader`; add the minimal multi-key decrypt + helper the SDK bindings share, so decrypt-only keys stay in native memory. + Removing public API is a breaking change: next 0.x minor. +- **cachekit-py**: `encryption.previous_master_keys` (list of `SecretStr`, + env `CACHEKIT_PREVIOUS_MASTER_KEYS`, comma-separated hex); fingerprint-based + keyring selection in `EncryptionWrapper`; delete the dead PyO3 bindings. +- **cachekit-ts**: `previousMasterKeys: string[]`; keyring loop behind the + NAPI boundary; link `NonceExhaustedError` guidance to the rotation runbook. +- **cachekit-rs**: builder `.previous_master_keys(...)`; sequential-attempt + keyring in the decrypt path. +- **docs**: rotation runbook on docs.cachekit.io + SDK READMEs; feature-matrix + row flips to ✅ per SDK only as each implementation ships. + +## Out of scope + +Nonce-exhaustion handling itself (separate, already monitored in ts); +per-tenant key-derivation changes; SaaS-side anything (zero-knowledge keeps +the server out of this by construction). diff --git a/spec/encryption.md b/spec/encryption.md index 2436a35..dfb7714 100644 --- a/spec/encryption.md +++ b/spec/encryption.md @@ -19,7 +19,7 @@ - [Nonce Generation](#nonce-generation) - [Ciphertext Format](#ciphertext-format) - [Additional Authenticated Data (AAD)](#additional-authenticated-data-aad) -- [Encryption Header (Key Rotation)](#encryption-header-rotationawareheader) +- [Key Rotation (Keyring)](#key-rotation-keyring) - [Encryption Flow](#encryption-flow) - [Decryption Flow](#decryption-flow) - [Compliance](#compliance) @@ -31,7 +31,7 @@ CacheKit provides **optional** client-side encryption using AES-256-GCM. When enabled, the backend (Redis or SaaS) stores opaque ciphertext and never has access to keys or plaintext. > [!NOTE] -> Encryption is not configurable by design. AES-256-GCM is the only supported algorithm. See the [Encryption Algorithm Decision Record](../decisions/encryption-algorithm.md) for rationale. +> Encryption is not configurable by design. AES-256-GCM is the only supported algorithm. | Property | Value | | :--- | :--- | @@ -131,6 +131,10 @@ For key identification without revealing key material: fingerprint = SHA-256("key_fingerprint_v1" || key)[0..16] // First 16 bytes ``` +For per-entry fingerprints — the value cachekit-py stores in CK frame metadata and +the input to [keyring selection](#key-rotation-keyring) — `key` is the HKDF-derived +per-tenant **encryption key** (`tenant_keys.encryption_key`), never the master key. + --- ## Nonce Generation @@ -336,37 +340,51 @@ by `tools/encryption-verify.py`. --- -## Encryption Header (RotationAwareHeader) - -When key rotation is in use, a 32-byte header is prepended to identify which key encrypted the data. - -``` -┌──────┬──────┬──────────────────┬───────────────┬────────┬─────────┬──────────┐ -│ ver │ algo │ key_fingerprint │ tenant_hash │ domain │ key_ver │ reserved │ -└──────┴──────┴──────────────────┴───────────────┴────────┴─────────┴──────────┘ - 1 B 1 B 16 bytes 8 bytes 4 B 1 B 1 B -│◄──────────────────────── 32 bytes total ──────────────────────────────────────►│ -``` - -| Field | Offset | Size | Description | -| :--- | :---: | :---: | :--- | -| `version` | 0 | 1 | Header version (must be `1`) | -| `algorithm` | 1 | 1 | Encryption algorithm (`0` = AES-256-GCM, only valid value) | -| `key_fingerprint` | 2 | 16 | First 16 bytes of SHA-256("key_fingerprint_v1" \|\| key) | -| `tenant_id_hash` | 18 | 8 | Tenant identifier hash | -| `domain` | 26 | 4 | Domain context (e.g., `"ench"` for encryption) | -| `key_version` | 30 | 1 | `0` = original key, `1` = rotated key | -| reserved | 31 | 1 | Reserved (must be `0x00`) | +## Key Rotation (Keyring) -### Key Rotation Strategy - -``` -1. Start rotation → set new key, keep old key for decryption -2. New encryptions → use new key (key_version=1) -3. Decryption → try both keys based on key_version byte -4. Migration window → run until all old-key entries expire -5. Complete → remove old key (complete_rotation()) -``` +> [!WARNING] +> **Status: specified, not yet implemented** (decision record: +> [decisions/key-rotation.md](../decisions/key-rotation.md), 2026-07-23; +> implementation tracked under LAB-516). Until an SDK ships this section, rotating +> its master key invalidates every encrypted entry — fail-open readers take misses, +> fail-closed readers take errors. The [feature matrix](../sdk-feature-matrix.md) +> reflects per-SDK implementation status. +> +> Earlier revisions of this spec described a 32-byte `RotationAwareHeader` prepended +> to ciphertext. **That header was never written by any SDK and is not part of the +> protocol**; it is retired — rationale in the decision record. + +Rotation is a client-side concern by construction: the backend stores ciphertext it +cannot decrypt (zero-knowledge), so it can never re-encrypt. The mechanism is a +**keyring** — configuration, not a state machine: + +| Rule | Requirement | +| :--- | :--- | +| Keyring | One **current** master key (encrypts and decrypts) plus an ordered list of **at most 3 decrypt-only** master keys — typically previous keys; during a [two-phase rollout](../decisions/key-rotation.md#runbooks-normative-for-docs), the incoming key. SDKs MUST reject configuration exceeding the cap at load — never silently truncate. | +| Configuration | Cross-SDK env vars: `CACHEKIT_MASTER_KEY` (current, [as above](#master-key)) and `CACHEKIT_PREVIOUS_MASTER_KEYS` (decrypt-only list, comma-separated hex, same per-key validation). Programmatic naming is SDK-local. | +| Forward-only current key | A master key that has ever occupied the current (encrypting) slot MUST NOT be re-promoted to current; it may re-appear only in the decrypt-only list. Backing out a rotation means rotating **forward** to a fresh key. Re-promotion resumes a used, unknowable nonce budget and risks catastrophic AES-GCM nonce reuse (plaintext recovery and forgery). Enforcement: a stateless SDK cannot know whether a newly supplied current key was used before, so this invariant is **operator-enforced** (treat retired key material as destroyed); SDKs MUST reject the detectable subset — a configuration where the current key also appears in the decrypt-only list. | +| Derivation | Each keyring entry independently derives per-tenant keys via the [HKDF construction above](#key-derivation). Salts, domains, and fingerprints are unchanged. | +| Encrypt | Always the current key. A new master key yields freshly derived keys with a fresh per-key nonce budget — which is why rotation (always forward, to a *new* key) is the remedy when nonce-exhaustion monitoring fires. | +| Decrypt — with per-entry key identity | Where the SDK stores a per-entry [key fingerprint](#key-fingerprint) (cachekit-py's CK frame metadata), select the keyring entry by exact fingerprint match; never trial-decrypt across the keyring. **The fingerprint is computed over the HKDF-derived per-tenant encryption key, not the master key**: selection derives the tenant's keys for each keyring entry, fingerprints each derived encryption key, and compares. A match is binding — if the matched key fails AES-GCM authentication, the failure is terminal (straight to the fail-open / fail-closed policy; no further keyring entries). No match → the SDK's existing fingerprint-mismatch semantics (fail-closed raises; fail-open attempts the current key only and treats the authentication failure as a miss). | +| Decrypt — without per-entry key identity | (cachekit-ts, cachekit-rs, [interop mode](interop-mode.md)) Attempt each keyring key in order, current first, **rebuilding the identical AAD for every attempt**. Retrying across keys is permitted; retrying across AAD variants remains forbidden — the [no-retry rule](#additional-authenticated-data-aad) binds AAD inputs, not key count. | +| Exhaustion | All permitted attempts fail → AES-GCM authentication failure → the SDK's existing fail-open / fail-closed policy. No new failure mode. | +| Key hygiene | Derived keys held behind the native boundary zeroize on drop, decrypt-only keys included. Master-key material handled by managed-language runtimes (env vars, interpreter strings) cannot be reliably scrubbed — treat exposure of the keyring configuration as exposure of every key in it. | + +Nothing on the wire changes: the [ciphertext format](#ciphertext-format) and +[AAD v0x03](#additional-authenticated-data-aad) contain no key identity, so every +existing entry remains format-compatible — and decryptable for as long as the key +that encrypted it is retained in the keyring — and cross-SDK interop is unaffected. Keyring +conformance vectors will accompany the implementations (test plan in the +[decision record](../decisions/key-rotation.md#consequences--lab-516-sub-issues)). + +**Rotation window**: writes during the window re-encrypt under the current key; +old-key entries age out via TTL. Run the window at least as long as the longest TTL +in use, **measured from the moment the last writer stopped encrypting under the old +key** (fleet deploy of the promotion phase complete), then drop the retired key. An +empty decrypt-only list is legal — the hard cut-over used for compromise response. +Rollout choreography, preconditions (non-expiring entries), and the compromise +runbook are normative in the +[decision record](../decisions/key-rotation.md#runbooks-normative-for-docs). --- @@ -419,24 +437,34 @@ process but fails authentication for any correct second reader (see ## Decryption Flow ``` -Input: ciphertext, stored metadata (format, compressed, optional original_type), - master_key, tenant_id, cache_key - -1. Derive: tenant_keys = derive_tenant_keys(master_key, tenant_id) -2. Build AAD: aad = create_aad(tenant_id, cache_key, +Input: ciphertext, stored metadata (format, compressed, optional original_type, + optional per-entry key_fingerprint — SDK-internal storage, e.g. + cachekit-py's CK frame header), keyring (current master_key + + decrypt-only master keys), tenant_id, cache_key + +1. Select key: per [Key Rotation (Keyring)](#key-rotation-keyring) — fingerprint + match where the stored key_fingerprint is present (binding: no + further keys after a match), else sequential keyring attempts, + current first — an authentication failure on an intermediate key + continues to the next permitted key; the hard-error semantics + below apply once permitted attempts are exhausted. Single-key + configurations reduce to the current key. +2. Derive: tenant_keys = derive_tenant_keys(selected_master_key, tenant_id) +3. Build AAD: aad = create_aad(tenant_id, cache_key, stored.format, stored.compressed, stored.original_type) -3. Decrypt: plaintext_bytes = aes_256_gcm_decrypt( +4. Decrypt: plaintext_bytes = aes_256_gcm_decrypt( ciphertext = ciphertext, // nonce(12) || encrypted || tag(16) key = tenant_keys.encryption_key, aad = aad ) -4. Unenvelope: serialized_bytes = unenvelope(plaintext_bytes) // per the reader's configured +5. Unenvelope: serialized_bytes = unenvelope(plaintext_bytes) // per the reader's configured // serializer/mode — see wire-format.md -5. Deserialize: user_data = deserialize(serialized_bytes) +6. Deserialize: user_data = deserialize(serialized_bytes) ``` -The AAD in step 2 is rebuilt from the **stored cleartext metadata**; tampering makes -step 3 fail authentication, which is a hard error — the no-retry and no-sniffing +The AAD in step 3 is rebuilt from the **stored cleartext metadata**; tampering makes +step 4 fail authentication, which is a hard error once the key attempts permitted by +step 1 are exhausted — the no-retry and no-sniffing rules in [AAD v0x03 Format](#additional-authenticated-data-aad) apply. How each SDK stores this metadata (e.g. cachekit-py's CK frame header) is SDK-internal; the only cross-SDK-normative AAD inputs are interop mode's pinned four components. diff --git a/spec/interop-mode.md b/spec/interop-mode.md index 9cce4fe..ddf50ab 100644 --- a/spec/interop-mode.md +++ b/spec/interop-mode.md @@ -337,8 +337,10 @@ Three interop-specific pins: AAD — an SDK carrying auto-mode's type hint into the AAD would fail cross-SDK authentication. Interop AAD is always exactly four components. -Key derivation (HKDF-SHA256), nonces, the ciphertext layout -(`nonce ‖ ciphertext ‖ tag`), and the RotationAwareHeader are all unchanged. +Key derivation (HKDF-SHA256), nonces, and the ciphertext layout +(`nonce ‖ ciphertext ‖ tag`) are all unchanged. Interop entries carry no +per-entry key identity, so keyring readers use sequential key attempts — see +[encryption.md → Key Rotation (Keyring)](encryption.md#key-rotation-keyring). Two vectors substantiate this end-to-end, not just by construction: