-
Notifications
You must be signed in to change notification settings - Fork 0
docs(protocol): specify File backend format (LAB-430) #42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
ef577e4
docs(protocol): specify File backend format (LAB-430)
27Bslash6 ff73906
Merge remote-tracking branch 'origin/main' into agent/sonnet/19e31eda…
27Bslash6 b1ac9e9
fix(tools): fail closed under python -O and handle vector-load errors
27Bslash6 bee3c27
fix(spec): atomic TTL refresh, reserved/expiry negotiation vectors, r…
27Bslash6 348f15c
style(tools): emit verifier success line via logging per team rule
27Bslash6 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| # File Backend Storage Format | ||
|
|
||
| ## Scope | ||
|
|
||
| This optional transport format lets File backends in different CacheKit SDKs share one cache directory. It specifies only the file name and container header; the payload is the opaque byte sequence supplied by the SDK backend. It does not make an SDK auto-mode value representation cross-language compatible. | ||
|
|
||
| ## File name | ||
|
|
||
| For a logical backend key encoded as UTF-8, calculate `BLAKE2b-128(key_utf8)` and write its lowercase hexadecimal encoding. The result is a 32-character file name in a flat cache directory. | ||
|
|
||
| The File backend does not add a key prefix. Interop mode must therefore reject a configured hidden prefix before this mapping is used. | ||
|
|
||
| ## Entry layout | ||
|
|
||
| Every entry is a 14-byte header followed by zero or more payload bytes. | ||
|
|
||
| | Offset | Size | Field | Required value | | ||
| | ---: | ---: | --- | --- | | ||
| | 0 | 2 | Magic | ASCII `CK` (`0x43 0x4b`) | | ||
| | 2 | 1 | Version | `1` | | ||
| | 3 | 1 | Reserved | `0` | | ||
| | 4 | 2 | Flags | unsigned 16-bit big-endian; `0` | | ||
| | 6 | 8 | Expiry | unsigned 64-bit big-endian Unix seconds; `0` means no expiry | | ||
| | 14 | variable | Payload | opaque backend bytes | | ||
|
|
||
| Writers for version 1 MUST set the reserved byte and flags to zero. The expiry is the integer UTC Unix-second deadline. An entry with a nonzero expiry is expired when the reader wall clock reaches that timestamp; readers MAY lazily remove expired entries. | ||
|
|
||
| ## Version and flag negotiation | ||
|
|
||
| A truncated header or wrong magic/version is corrupt and may be removed as a cache miss. A nonzero reserved byte or flag value is different: it can indicate a future payload transform. A reader that does not implement every indicated transform MUST return a miss and MUST NOT delete, rewrite, or return the payload. This fails closed rather than exposing transformed bytes as plaintext. | ||
|
|
||
| A future nonzero flag assignment requires a protocol update and canonical test vector. A writer MUST NOT set an unknown flag. A reader that implements a future flag must preserve the established fields and verify the transform before returning payload bytes. | ||
|
|
||
| ## Write and TTL behavior | ||
|
|
||
| Entries are written through a temporary file in the same directory and atomically renamed into place. Refreshing TTL rewrites only bytes 6 through 13 and MUST NOT expose a torn expiry to a concurrent reader: a refresh either rewrites the entry through the same temporary-file and atomic-rename path, or updates the expiry with a single positioned 8-byte write on platforms that guarantee read/write atomicity for regular files (POSIX.1-2017 §2.9.7) — in which case readers MUST load the 14-byte header in a single read. Writers calculate the absolute deadline and store its whole-second Unix timestamp; a positive sub-second TTL can therefore expire within the current second. | ||
|
|
||
| The canonical examples are in [`test-vectors/file-backend.json`](../test-vectors/file-backend.json). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| { | ||
| "version": "1.1.0", | ||
| "generator": "cachekit-py FileBackend format v1", | ||
| "format": "File name = BLAKE2b-128(UTF-8 key) lowercase hex; header = CK | version | reserved | flags u16 BE | expiry u64 BE | payload. reader_action is evaluated at reader_now_unix_seconds (default 0): miss_preserve for nonzero reserved/flags, miss_expired when now reaches a nonzero expiry, else return_payload. reserved defaults to 0.", | ||
| "vectors": [ | ||
| { | ||
| "name": "permanent_ascii", | ||
| "key_utf8": "test-key", | ||
| "filename": "0e2a03b49262c15a063c04d5a29c0158", | ||
| "header_hex": "434b010000000000000000000000", | ||
| "payload_hex": "68656c6c6f", | ||
| "file_hex": "434b01000000000000000000000068656c6c6f", | ||
| "expiry_unix_seconds": 0, | ||
| "flags": 0, | ||
| "reader_action": "return_payload" | ||
| }, | ||
| { | ||
| "name": "future_expiry", | ||
| "key_utf8": "unicode-ключ-🔑", | ||
| "filename": "f13b0df717f97cf1d2fe3da650525c79", | ||
| "header_hex": "434b0100000000000000f4865700", | ||
| "payload_hex": "68656c6c6f", | ||
| "file_hex": "434b0100000000000000f486570068656c6c6f", | ||
| "expiry_unix_seconds": 4102444800, | ||
| "flags": 0, | ||
| "reader_action": "return_payload" | ||
| }, | ||
| { | ||
| "name": "unknown_flag_preserved", | ||
| "key_utf8": "test-key", | ||
| "filename": "0e2a03b49262c15a063c04d5a29c0158", | ||
| "header_hex": "434b010000010000000000000000", | ||
| "payload_hex": "667574757265", | ||
| "file_hex": "434b010000010000000000000000667574757265", | ||
| "expiry_unix_seconds": 0, | ||
| "flags": 1, | ||
| "reader_action": "miss_preserve" | ||
| }, | ||
| { | ||
| "name": "reserved_nonzero_preserved", | ||
| "key_utf8": "test-key", | ||
| "filename": "0e2a03b49262c15a063c04d5a29c0158", | ||
| "header_hex": "434b010200000000000000000000", | ||
| "payload_hex": "7265736572766564", | ||
| "file_hex": "434b0102000000000000000000007265736572766564", | ||
| "expiry_unix_seconds": 0, | ||
| "flags": 0, | ||
| "reserved": 2, | ||
| "reader_action": "miss_preserve" | ||
| }, | ||
| { | ||
| "name": "expired_entry", | ||
| "key_utf8": "test-key", | ||
| "filename": "0e2a03b49262c15a063c04d5a29c0158", | ||
| "header_hex": "434b01000000000000003b9aca00", | ||
| "payload_hex": "65787069726564", | ||
| "file_hex": "434b01000000000000003b9aca0065787069726564", | ||
| "expiry_unix_seconds": 1000000000, | ||
| "flags": 0, | ||
| "reader_now_unix_seconds": 1000000000, | ||
| "reader_action": "miss_expired" | ||
| } | ||
| ] | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| #!/usr/bin/env python3 | ||
| """Validate the protocol-owned File backend vectors with Python stdlib.""" | ||
|
|
||
| from __future__ import annotations | ||
|
27Bslash6 marked this conversation as resolved.
|
||
|
|
||
| import hashlib | ||
| import json | ||
| import logging | ||
| from pathlib import Path | ||
| import struct | ||
| import sys | ||
|
|
||
| ROOT = Path(__file__).resolve().parents[1] | ||
| VECTORS = ROOT / "test-vectors" / "file-backend.json" | ||
|
|
||
|
|
||
| def check(condition: bool, name: str, detail: str) -> None: | ||
| """Fail closed even under ``python -O`` (asserts would be stripped).""" | ||
| if not condition: | ||
| raise ValueError(f"{name}: {detail}") | ||
|
|
||
|
|
||
| def expected_reader_action(reserved: int, flags: int, expiry: int, now: int) -> str: | ||
| """spec/file-backend-format.md: unknown reserved/flags fail closed before expiry.""" | ||
| if reserved != 0 or flags != 0: | ||
| return "miss_preserve" | ||
| if expiry != 0 and now >= expiry: | ||
| return "miss_expired" | ||
| return "return_payload" | ||
|
|
||
|
|
||
| def verify(document: dict) -> int: | ||
| for vector in document["vectors"]: | ||
| name = vector["name"] | ||
| key = vector["key_utf8"].encode("utf-8") | ||
| filename = hashlib.blake2b(key, digest_size=16).hexdigest() | ||
| check(filename == vector["filename"], name, "filename mismatch") | ||
|
|
||
| image = bytes.fromhex(vector["file_hex"]) | ||
| check(len(image) >= 14, name, "file shorter than 14-byte header") | ||
| check(image[:14].hex() == vector["header_hex"], name, "header_hex mismatch") | ||
| check(image[14:].hex() == vector["payload_hex"], name, "payload_hex mismatch") | ||
| check(image[:2] == b"CK" and image[2] == 1, name, "bad magic or version") | ||
|
|
||
| reserved = image[3] | ||
| flags = struct.unpack(">H", image[4:6])[0] | ||
| expiry = struct.unpack(">Q", image[6:14])[0] | ||
| check(reserved == vector.get("reserved", 0), name, "reserved mismatch") | ||
| check(flags == vector["flags"], name, "flags mismatch") | ||
| check(expiry == vector["expiry_unix_seconds"], name, "expiry mismatch") | ||
|
|
||
| now = vector.get("reader_now_unix_seconds", 0) | ||
| expected = expected_reader_action(reserved, flags, expiry, now) | ||
| check(vector["reader_action"] == expected, name, "reader_action mismatch") | ||
|
|
||
| return len(document["vectors"]) | ||
|
|
||
|
|
||
| def main() -> None: | ||
| try: | ||
| document = json.loads(VECTORS.read_text(encoding="utf-8")) | ||
| except OSError as exc: | ||
| sys.exit(f"cannot read {VECTORS}: {exc}") | ||
| except json.JSONDecodeError as exc: | ||
| sys.exit(f"invalid JSON in {VECTORS}: {exc}") | ||
|
|
||
| try: | ||
| count = verify(document) | ||
| except ValueError as exc: | ||
| sys.exit(f"invalid vector file: {exc}") | ||
| except (KeyError, TypeError, AttributeError) as exc: | ||
| sys.exit(f"invalid vector file: malformed structure ({exc!r})") | ||
|
|
||
| logging.info("validated %d File backend vectors", count) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| logging.basicConfig(level=logging.INFO, format="%(message)s") | ||
| main() | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.