Summary
StorageEnvelope.compressed_data is a plain Vec<u8> with no #[serde(with = "serde_bytes")], so rmp_serde::to_vec encodes it as a msgpack array of integers (one element per byte, 2 bytes on wire for values ≥ 0x80) instead of msgpack bin.
Measured impact (cachekit-py, cachekit-core 0.3.0)
- 256MB incompressible payload → 404.8MB envelope (1.58× inflation: ~1.5× from array-of-ints on random bytes, ~1.004× LZ4, headers).
- Store throughput ~150MB/s end-to-end — per-element serialization dominates, not LZ4 (~GB/s) or xxh3 (~36GB/s).
Found while working #45 (cachekit-py#223): after releasing the GIL, msgpack encode is the remaining hot-path cost.
Why this is not a quick fix
The array-of-ints encoding is the current on-wire format — every SDK byte-verifies against it. Switching to serde_bytes (msgpack bin) is a wire-format change: it needs a protocol RFC, cross-SDK coordination (py/ts/rs readers), versioning/migration for stored envelopes, and the crypto/protocol review gate.
Payoff if done: ~35% smaller envelopes and several-fold faster store/retrieve for incompressible payloads.
Summary
StorageEnvelope.compressed_datais a plainVec<u8>with no#[serde(with = "serde_bytes")], sormp_serde::to_vecencodes it as a msgpack array of integers (one element per byte, 2 bytes on wire for values ≥ 0x80) instead of msgpackbin.Measured impact (cachekit-py, cachekit-core 0.3.0)
Found while working #45 (cachekit-py#223): after releasing the GIL, msgpack encode is the remaining hot-path cost.
Why this is not a quick fix
The array-of-ints encoding is the current on-wire format — every SDK byte-verifies against it. Switching to
serde_bytes(msgpackbin) is a wire-format change: it needs a protocol RFC, cross-SDK coordination (py/ts/rs readers), versioning/migration for stored envelopes, and the crypto/protocol review gate.Payoff if done: ~35% smaller envelopes and several-fold faster store/retrieve for incompressible payloads.