perf(l1): cut the cost of cold contract-code access - #7095
Conversation
A sorted `Arc<[u32]>` of JUMPDEST offsets, persisted next to the bytecode, costs ~4x the code size for jumpdest-dense contracts (97,906 B of value for 24,576 B of code) and ~10 ns per entry to rebuild on every code-cache miss: 246 us for a max-size runtime that is almost all JUMPDESTs. A bitmap is len/8 bytes at any density, decodes as a memcpy (0.5 us), and turns jump validation into a bit test rather than a binary search (12.9 -> 0.37 ns). Values written in the older form need no migration: the RLP item header tells a list from a byte string, and for a list the bitmap is rebuilt from the bytecode (26.6 us, still ~9x cheaper than decoding the list). `Code::size()` now counts the bytecode allocation it excluded, so the cache honors its byte budget instead of holding orders of magnitude more than it accounts for; the budget goes from 64 MiB that bounded only the jump tables to 256 MiB of actual bytecode. Re-inserting a cached hash no longer inflates the accounted size.
`account_codes` was the only exact-key point-lookup CF on the execution read path without a bloom filter, so a get had to read a data block per candidate level to discover the key was absent; with blob files enabled the SST value is only a blob reference, so its 32KB blocks bought nothing. `account_code_metadata` fell into the default arm for the same reason. Both now match the trie-node and flat-KV CFs: 4KB blocks, 10 bits per key. Write-time option: applies to newly flushed/compacted SSTs, existing SSTs are read as-is.
`get_code_metadata` loaded the whole bytecode to return `.len()`, so a size query materialized up to 24KB out of the blob store; the size-only `ACCOUNT_CODE_METADATA` table was plumbed through every layer but reached only from a mempool check. Read it instead, falling back to code already loaded for another reason. Witness generation stays complete: a metadata read observes the bytecode, so the logger records the code hash and the witness still carries the code, which is what stateless validation recomputes the length from.
|
github-actions
Bot
added
L1
🤖 Kimi Code ReviewThis is a high-quality optimization PR. The switch from a sorted Correctness & Security
Performance & Memory
Code Quality
Minor Notes
Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt |
Lines of code reportTotal lines added: Detailed view |
Greptile SummaryThe PR reduces cold contract-code access costs by replacing jump-target lists with compact bitmaps, correcting bytecode-cache accounting, tuning RocksDB code tables, and serving EXTCODESIZE from code metadata.
Confidence Score: 5/5The PR appears safe to merge, with the changed storage formats, cache accounting, VM jump validation, and witness behavior preserving their required contracts. Normal code construction and persistence produce matching jump-destination bitmaps, legacy database records are explicitly decoded, metadata lookup retains a correct fallback, and metadata-only execution reads still include bytecode in witnesses.
|
| Filename | Overview |
|---|---|
| crates/common/types/account.rs | Replaces jump-target offsets with a compact bitmap, updates Code serialization and memory accounting, and preserves logical-bytecode padding. |
| crates/storage/store.rs | Adds legacy-compatible bitmap decoding, corrects code-cache accounting and eviction, and uses the code-metadata table with a correct full-code fallback. |
| crates/storage/backend/rocksdb.rs | Configures filtered 4 KiB blocks for account code and code-metadata point lookups. |
| crates/vm/backends/levm/db.rs | Records metadata-only code observations so execution witnesses still include the corresponding bytecode. |
| crates/vm/levm/src/db/gen_db.rs | Answers code-length queries from cached code or metadata instead of always loading full bytecode. |
| crates/vm/levm/src/opcode_handlers/stack_memory_storage_flow.rs | Validates JUMP and JUMPI destinations through the new bitmap while retaining existing gas and tracing behavior. |
| test/tests/common/code_serde_tests.rs | Updates Code serde coverage for bitmap preservation and bytecode padding. |
| test/tests/common/jumpdest_bitmap_tests.rs | Covers valid destinations, PUSH immediates, bounds, empty maps, and bitmap sizing. |
| test/tests/common/mod.rs | Registers the new jump-destination bitmap test module. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
Opcode[EXTCODESIZE] --> GDB[GeneralizedDatabase]
GDB --> Meta[Code metadata cache/table]
Meta --> Length[Return code length]
Meta -. legacy metadata miss .-> CodeStore[ACCOUNT_CODES]
CodeStore --> Decode[Decode bytecode and bitmap]
Decode --> Cache[Bounded code cache]
Jump[JUMP or JUMPI] --> Bitmap[JUMPDEST bitmap check]
Reviews (1): Last reviewed commit: "perf(l1): answer EXTCODESIZE from the co..." | Re-trigger Greptile
🤖 Codex Code Review
I could not run the targeted tests here because the toolchain tries to write under read-only Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
🤖 Claude Code ReviewNo cleanup needed, that was just a tool-output cache file. Here's the review. Review:
|
Benchmark Results ComparisonNo significant difference was registered for any benchmark run. Detailed ResultsBenchmark Results: BubbleSort
Benchmark Results: ERC20Approval
Benchmark Results: ERC20Mint
Benchmark Results: ERC20Transfer
Benchmark Results: Factorial
Benchmark Results: FactorialRecursive
Benchmark Results: Fibonacci
Benchmark Results: FibonacciRecursive
Benchmark Results: ManyHashes
Benchmark Results: MstoreBench
Benchmark Results: Push
Benchmark Results: SstoreBench_no_opt
|
Indexing the bitmap put a bounds-check panic path in the scan loop, which cost more than the scan itself: 24KB of jumpdest-free initcode took 55.4 us to analyse against 11.2 us for the offset-list version it replaced, and benchmarkoor's test_jumpdest_analysis[00] lost 69-78% throughput. Accumulate a byte's bits in a register and store it once the scan leaves that byte; the monotonically increasing index makes the single store safe. Against the offset-list version, per 24KB of code: dense 49.8 -> 17.0 us, jumpdest-free 11.1 -> 11.8 us, 1-in-32 12.5 -> 12.0 us, PUSH-heavy unchanged.
Benchmark Block Execution Results Comparison Against Main
|
| length: code_length, | ||
| let length = match self.codes.get(&code_hash) { | ||
| Some(code) => code.len() as u64, | ||
| None => self.store.get_code_metadata(code_hash)?.length, |
There was a problem hiding this comment.
This is the change that makes ACCOUNT_CODE_METADATA an execution-path read, and there's a pre-existing wrinkle behind it worth knowing about before merge.
Store::get_code_metadata (crates/storage/store.rs, the else branch on a table miss) auto-migrates: it loads the full code, derives the length, and fires a tokio::task::spawn to write the metadata back. That code is on main already, so it isn't yours — but until now it was, in your words, "reached only from a mempool check". This PR puts it under EXTCODESIZE.
Two things change character as a result:
Runtime context. tokio::task::spawn panics if no reactor is running on the calling thread. A mempool check is comfortably inside the async runtime; block execution is not always — anything reached via spawn_blocking or a sync execution path would panic on a metadata miss rather than degrade. Worth confirming every caller of EXTCODESIZE is on a runtime thread.
Miss volume. A miss is the steady state on any node whose codes predate the metadata table, and the burst is worst on exactly the shape this PR targets: the DIFF_MAX fixture is unique max-size contracts, so a block of them is one spawned write task per contract, concurrent with execution's own writes. Fine once migrated, potentially a large unbounded burst on the first blocks after upgrade.
Neither is a reason to hold the perf work, and the fallback is correct. But since it's newly hot, a bounded or synchronous backfill — or just confirming the panic case is unreachable — seems worth doing in this PR rather than discovering it on a mainnet upgrade. A one-off backfill at startup would also make the fallback cold again.
There was a problem hiding this comment.
Good catch, the panic case was reachable: execution runs on rayon workers, which have no runtime, so a metadata miss aborted the node. Fixed in 6dd6ca8, the spawn is gated on Handle::try_current() and skipping it just costs one code read next time.
Added a regression test that is deliberately a plain #[test] rather than #[tokio::test], since the absence of a runtime is the condition. Verified it fails without the guard.
That also bounds the burst you describe: on the DIFF_MAX shape the reads come from rayon workers, so no task is spawned at all. A startup backfill would still be worth having to make the fallback cold, but it is out of scope here.
EXTCODESIZE now resolves its length through Store::get_code_metadata, whose miss path spawned the metadata backfill with tokio::task::spawn. Execution runs on rayon workers, which are not in a runtime, and that spawn panics there, so a hash with no metadata row would abort the node on the first EXTCODESIZE against it. No migration backfills that column family, so any database written before it existed is in exactly that state. Spawn only when a runtime handle is reachable. Skipping the backfill costs one code read the next time the hash is asked for.
Witness: code_accessed records one entry per read, and a contract read for both its bytecode and its length is recorded twice, so the builder embedded the same bytecode more than once. Dedup before embedding. Code metadata: EXTCODESIZE made this the execution read path, where the cache was an unbounded map behind a single mutex that serialized the parallel executor. Bound it to a derived entry count, and answer from resident code in the caching layer so a length read usually never reaches it. Schema: bump the store version with a no-op migration. The value change needs no rewrite forwards, but an older binary cannot decode a bitmap, and the bump makes it warn instead of failing on the first code read. RocksDB: the 4KB block size was justified by blob indirection, which the metadata CF does not have; keep its 16KB and let the bytecode CF skip its last-level filter, whose lookups are almost always positive. Also make the legacy-decode test build its expectation from the offsets the old format names rather than from the function under test.
| pub jump_targets: Arc<[u32]>, | ||
| jumpdests: Arc<[u8]>, |
There was a problem hiding this comment.
This changes the serialization of Code and therefore of AccountUpdate, breaking store_account_updates_by_block_number.
There was a problem hiding this comment.
Right, thanks. Fixed in 9a4c1fd by dropping the jump destinations from the wire format entirely and recomputing them on deserialize: they are a pure function of the bytecode, so carrying them coupled AccountUpdates stored format to how they happen to be represented. The format is now hash plus bytecode, so a future representation change cannot break it again, and the payload loses a byte per eight bytes of code. Added a test asserting the field set.
Note this still does not make rows written by an older binary readable, and the rollup store has no schema version to gate on. Since those rows are consumed at batch commit the window is small, but tell me if you want a version guard instead.
There was a problem hiding this comment.
I would rather have a migration and not break things.
We could simply keep the field on CodeSerde and do the migration later if you want. For the encode path, transforming from a bitmap to a list should be easy.
Code's serde form carried the jump destinations, so changing how they are represented changed the wire format of everything embedding a Code. That includes AccountUpdate, which the L2 rollup store persists with bincode and the committer reads back to re-apply state, so rows written by an older binary would not decode. They are a pure function of the bytecode, so drop them from the format and recompute on deserialize. The format is now hash plus bytecode, which cannot be broken again by a representation change, and the payload loses a byte per eight bytes of code.
| /// is cheaper than decoding the list. | ||
| fn decode_jumpdests(code: &[u8], encoded: &[u8]) -> Result<Arc<[u8]>, StoreError> { | ||
| let (is_list, payload, _) = decode_rlp_item(encoded)?; | ||
| if is_list { |
| /// letting it fail on the first code read. | ||
| fn migrate_3_to_4(_backend: &dyn StorageBackend) -> Result<(), StoreError> { | ||
| Ok(()) | ||
| } |
There was a problem hiding this comment.
Since there is no migration, the "recompute everything" fallback will be used on every cold read. This is likely going to cause a big performance hit.
| pub jump_targets: Arc<[u32]>, | ||
| jumpdests: Arc<[u8]>, |
There was a problem hiding this comment.
I would rather have a migration and not break things.
We could simply keep the field on CodeSerde and do the migration later if you want. For the encode path, transforming from a bitmap to a list should be easy.
4 participants
Motivation
ethrex is red on
COLD_ACCOUNT_CODE_ACCESSin the EIP-8038 repricing fit: 7680 gas against that page's 3000 goal, second worst of six clients. Every other cold-access parameter is green.The binding fixture is EEST
test_account_accesswithAccountMode.EXISTING_CONTRACT_DIFF_MAX, where every target address holds a unique max-size runtime that is 24544 of 24576 bytesJUMPDEST. Splitting the measurement by account mode isolates the cost: a shared 24KB code runs as fast as a 1-byte code, so the code cache hits by code hash, andEXTCODEHASH/BALANCEstay fast, so the account path is fine (ethrex is the fastest of the six onNON_EXISTING). All of it is the cost of ingesting a newly seen bytecode, ~84 us per unique 24KB contract against ~17.6 for reth, ~18.5 besu, ~21.3 geth, ~25.3 nethermind.Description
Jump destinations are a bitmap, not an RLP list of
u32offsets. The list was ~4x the size of the code it described and was rebuilt on every code-cache miss. Per 24KB jumpdest-dense contract, the stored value goes from 97,906 B (3.98x code) to 27,654 B (1.13x), a load from 224 us to 0.48 us (17.4 us for a legacy value, rebuilt from the bytecode), and aJUMPvalidity check from 12.3 ns to 0.23 ns. No data rewrite is needed: the RLP item header tells a list from a byte string, so existing values still read. The store schema version is bumped with a no-op migration for the other direction, since an older binary cannot decode a bitmap and should warn rather than fail on the first code read. Building the bitmap is also cheaper than building the offset list, which matters for initcode analysis on every CREATE, 17.0 us against 49.8 us per 24KB of dense JUMPDESTs. The bits of a byte are accumulated in a register and stored once, so the scan loop carries neither a read-modify-write nor a bounds-check panic path.The code cache counts the bytecode it holds.
Code::size()countedsize_of::<Bytes>()instead of the heap allocation, so the byte budget excluded the only thing that makes an entry large. The budget goes from 64 MiB of jump tables to 256 MiB of bytecode, and re-inserting a cached hash no longer inflates the counter.Bloom filter for
account_codesandaccount_code_metadata, 4KB blocks for the former.account_codeswas the only exact-key point-lookup CF on the execution read path without a filter, and its 32KB blocks buy nothing when the SST value is just a blob reference. The metadata CF has no blob indirection, so it keeps 16KB; the bytecode CF also skips its last-level filter, whose lookups are almost always positive because the hash comes from an account that references it.EXTCODESIZEreads the code-length table. It was loading the full bytecode to return.len(), whileACCOUNT_CODE_METADATAwas plumbed through every layer and reached only from a mempool check. Witnesses stay complete: a metadata read records the code hash, so the bytecode still lands in the witness and stateless validation recomputes the length from it.Measured
Stateful suite
3f6a0898955dff4f, run1785777240_7cd53c4f(2026-08-03T17:14Z) against the run before it, refit through the same pipeline the repricing site uses (benchmarkoor-fetch0.3.1 andevm-gasfit0.3.0 on the site's ownfit.yaml, scoped to ethrex):COLD_ACCOUNT_CODE_ACCESSCOLD_ACCOUNT_CODE_WRITECOLD_ACCOUNT_NOCODE_ACCESSBinding model n=11, R² 0.997, confidence intervals disjoint (98.8-105.7 us against 59.0-63.8 us). Refitting the pre-fix run gives 7736 where the site publishes 7680, so these are comparable to it. In raw wall time on the 300M
DIFF_MAXfixture every code-loading opcode drops ~38% (EXTCODESIZE68%, since (4) removes its bytecode read) whileBALANCEandEXTCODEHASH, which never materialize the bytecode, sit flat at ~1940 and ~1950 ms across three runs.The
COLD_ACCOUNT_ACCESSgoal is 3000 and needs both the CODE and NOCODE variants under it, so ethrex goes from 2.58x the goal to 1.55x and the goal stays unmet. Clearing it means 40.0 us per access at this fit's anchor, against the 62.0 measured here. What is left is the ~105 us blob read per unique contract, plus the 17.4 us legacy rebuild on this snapshot only. The blob read wants the BAL code prefetch issued as batchedmulti_gets rather than 22.9k point gets, which is a separate change.Notes
glamsterdam-devnet-7with the same commits: 11616 EEST blockchain tests pass, plus clippy and the workspace and storage suites.