diff --git a/CHANGELOG.md b/CHANGELOG.md index b024be1..ce142c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,10 @@ no longer written. ### Changed +- **ethrex `account_codes` values carry a JUMPDEST bitmap** rather than an + RLP list of u32 offsets (ethrex#7095). Also adds the `state_history` + column family (22 CFs) and mirrors ethrex's table options for both code + CFs. Boot image and golden fixture repinned to ethrex commit 55433c2. - **ethrex tracking bumped v16.0.0 → v23.0.0.** Adds the `bad_blocks` column family (21 CFs; upstream added it in v22.0.0, ethrex#6948 — previously ethrex created it itself on first boot) and writes diff --git a/Dockerfile.ethrex b/Dockerfile.ethrex index 674a19c..47fe07e 100644 --- a/Dockerfile.ethrex +++ b/Dockerfile.ethrex @@ -4,7 +4,7 @@ # # RocksDB strategy: build from source. Pinned to RocksDB 10.10.1 to match # `grocksdb v1.10.8`'s C-binding expectations (its build.sh pins this exact -# version). ethrex uses a single RocksDB instance with 21 column families; +# version). ethrex uses a single RocksDB instance with 22 column families; # the same grocksdb pairing used for besu and nethermind applies here. # # Why 10.10.1 specifically: grocksdb v1.10.8's C bindings reference @@ -22,7 +22,7 @@ # grocksdb v1.10.7 → RocksDB 10.9.1 # grocksdb v1.10.8 → RocksDB 10.10.1 ← we use this # -# * On-disk layout: single RocksDB at with 21 column families. +# * On-disk layout: single RocksDB at with 22 column families. # Sidecar files: metadata.json (schema_version=3) and ethrex-genesis.json # (full genesis JSON for `ethrex --network `). # diff --git a/README.md b/README.md index 7730e80..7c43ed3 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ ## Why -You want a pre-populated Ethereum database that a client can boot against directly — for cross-client determinism tests, devnet bootstrapping, EIP-7702 / ERC-20 fixtures, or state-bloat experiments. The alternative (run the client's `init` against a genesis with millions of `alloc` entries) is slow and client-specific. State Actor writes each client's on-disk format directly: Pebble for geth, MDBX + RocksDB + nippy-jar for reth, single RocksDB + 8 Bonsai column families for besu, seven RocksDB instances + a flat column DB for nethermind, single RocksDB + 21 column families for ethrex, Erigon v3 flat snapshot `.kv` files + a minimal MDBX for erigon. +You want a pre-populated Ethereum database that a client can boot against directly — for cross-client determinism tests, devnet bootstrapping, EIP-7702 / ERC-20 fixtures, or state-bloat experiments. The alternative (run the client's `init` against a genesis with millions of `alloc` entries) is slow and client-specific. State Actor writes each client's on-disk format directly: Pebble for geth, MDBX + RocksDB + nippy-jar for reth, single RocksDB + 8 Bonsai column families for besu, seven RocksDB instances + a flat column DB for nethermind, single RocksDB + 22 column families for ethrex, Erigon v3 flat snapshot `.kv` files + a minimal MDBX for erigon. Three flags carry most of the weight: `--client` (which client's format to write), `--spec` (concrete entities to include, declared in YAML), `--target-size` (the DB-size budget; auto-fills mainnet-shaped 20 % / 10 % / 70 % across account-trie / bytecode / storage up to the cap). One of `--spec` or `--target-size` is required; everything else has a sane default. diff --git a/client/ethrex/dbs_cgo.go b/client/ethrex/dbs_cgo.go index 2282d2b..f289a07 100644 --- a/client/ethrex/dbs_cgo.go +++ b/client/ethrex/dbs_cgo.go @@ -159,10 +159,11 @@ const ( cfIdxMiscValues = 17 cfIdxExecutionWitnesses = 18 cfIdxBlockAccessLists = 19 - cfIdxBadBlocks = 20 + cfIdxStateHistory = 20 + cfIdxBadBlocks = 21 ) -// ethrexDB holds the open grocksdb handle and the 21 CF handles. +// ethrexDB holds the open grocksdb handle and the 22 CF handles. type ethrexDB struct { db *grocksdb.DB cfs []*grocksdb.ColumnFamilyHandle @@ -214,11 +215,11 @@ func openEthrexDB(dbPath string) (*ethrexDB, error) { return nil, fmt.Errorf("ethrex: mkdir: %w", err) } - // The 21 named ethrex CFs, plus RocksDB's implicit "default" CF appended - // LAST (index 21). RocksDB always creates "default" on a fresh DB, and an + // The 22 named ethrex CFs, plus RocksDB's implicit "default" CF appended + // LAST (index 22). RocksDB always creates "default" on a fresh DB, and an // open call must account for every existing CF or it errors with "you have // to open all column families". Appending it keeps the cfIdx* constants - // (0..20) aligned with Tables; cfs[21] (default) is created but never + // (0..21) aligned with Tables; cfs[22] (default) is created but never // written. Mirrors besu's explicit CFDefault inclusion. cfNames := make([]string, 0, len(ethrexinternal.Tables)+1) cfNames = append(cfNames, ethrexinternal.Tables...) @@ -306,23 +307,33 @@ func openEthrexDB(dbPath string) (*ethrexDB, error) { opts.SetMaxBytesForLevelBase(ethrexStateCFLevelBaseBytes()) bbto.SetBlockSize(16 << 10) bbto.SetFilterPolicy(grocksdb.NewBloomFilterFull(10)) - case cfIdxAccountCodes: + case cfIdxAccountCodes, cfIdxAccountCodeMetadata: opts.SetWriteBufferSize(128 << 20) opts.SetMaxWriteBufferNumber(3) opts.SetTargetFileSizeBase(256 << 20) - // Bytecodes go to blob files; small ones (delegation indicators) - // stay inline. Blobs are LZ4-compressed. - opts.EnableBlobFiles(true) - opts.SetMinBlobSize(32) - opts.SetBlobCompressionType(grocksdb.LZ4Compression) - bbto.SetBlockSize(32 << 10) + if i == cfIdxAccountCodes { + // Bytecodes go to blob files; small ones (delegation indicators) + // stay inline. Blobs are LZ4-compressed. + opts.EnableBlobFiles(true) + opts.SetMinBlobSize(32) + opts.SetBlobCompressionType(grocksdb.LZ4Compression) + } + // Both CFs answer exact-key point lookups on ethrex's execution + // path: EXT*/CALL* resolve a code hash to its bytecode or its + // length. A page-sized block keeps per-get read amplification down + // — with blob files the SST value is only a blob reference, so a + // larger block buys nothing — and the filter prunes the levels that + // cannot hold the hash. + bbto.SetBlockSize(4 << 10) + bbto.SetFilterPolicy(grocksdb.NewBloomFilterFull(10)) case cfIdxReceiptsV2: opts.SetWriteBufferSize(128 << 20) opts.SetMaxWriteBufferNumber(3) opts.SetTargetFileSizeBase(256 << 20) bbto.SetBlockSize(32 << 10) default: - // Also covers transaction_locations, whose ethrex arm carries the + // Also covers state_history, which has no dedicated arm in ethrex + // either, and transaction_locations, whose ethrex arm carries the // same values plus a merge operator — omitted here: state-actor // writes no rows there, and a CF created without one reopens // cleanly with one registered. diff --git a/client/ethrex/doc.go b/client/ethrex/doc.go index 44452fa..74b2fae 100644 --- a/client/ethrex/doc.go +++ b/client/ethrex/doc.go @@ -2,12 +2,13 @@ // // # On-disk layout // -// A single RocksDB instance at with 21 column families (Tables in +// A single RocksDB instance at with 22 column families (Tables in // internal/ethrex/constants.go), all declared at open time. CFs written at genesis: // - account_trie_nodes / storage_trie_nodes: MPT structural + leaf-NODE-RLP rows // (storage rows are address-prefixed) // - account_flatkeyvalue / storage_flatkeyvalue: leaf full-path → value -// - account_codes / account_code_metadata: code hash → EncodeCode / len +// - account_codes / account_code_metadata: code hash → EncodeCode +// (RLP(bytecode) ++ RLP(JUMPDEST bitmap)) / u64-BE len // - chain_data: ChainConfig (key 0x80) + block-number sentinels (0x01, 0x04) // - misc_values: "last_written" → 0xff (FKV "fully generated" sentinel) // - headers / bodies / block_numbers / canonical_block_hashes: genesis block @@ -30,12 +31,14 @@ // add_initial_state short-circuits when canonical_block_hashes[0] → headers[hash] // matches genesis.get_block().hash(), so ethrex never recomputes state at boot. // -// # Pinned releases +// # Pin // // Golden test: byte-exact vs testdata/genesis_dump.json, regenerated at ethrex -// v23.0.0; the state-bearing CFs are byte-identical v13–v23. E2e boot test -// (e2e_test.go) pins the same v23.0.0 image. --skip-genesis-validation, which -// the boot path requires, landed in v16.0.0 (lambdaclass/ethrex#6783). +// commit 55433c2 (the ethpandaops glamsterdam-devnet-7 build), which the e2e +// boot test pins too. A pre-release commit rather than a tag: v23.0.0 predates +// both the account_codes JUMPDEST bitmap (lambdaclass/ethrex#7095) and the +// state_history CF. --skip-genesis-validation, which the boot path requires, +// landed in v16.0.0 (lambdaclass/ethrex#6783). // // # Build // diff --git a/client/ethrex/e2e_test.go b/client/ethrex/e2e_test.go index 00d6daa..1b12a49 100644 --- a/client/ethrex/e2e_test.go +++ b/client/ethrex/e2e_test.go @@ -27,14 +27,20 @@ import ( ) // pinnedEthrexImage is the upstream ethrex Docker image the e2e suite pins -// against, digest-pinned for reproducibility. Override with -// ETHREX_IMAGE=ghcr.io/lambdaclass/ethrex: to test a specific release. +// against, digest-pinned for reproducibility. Override with ETHREX_IMAGE= +// to test a specific build. +// +// ethpandaops glamsterdam-devnet-7, ethrex commit 55433c2 — a pre-release +// build, not a tagged release, because the latest release (v23.0.0) predates +// two changes state-actor has to match: account_codes values carry a JUMPDEST +// bitmap instead of an RLP list of u32 offsets (lambdaclass/ethrex#7095), and +// TABLES gained state_history. Boot requires --skip-genesis-validation +// (lambdaclass/ethrex#6783, ≥v16.0.0). // -// Official release v23.0.0 (ghcr tag 23.0.0, published 2026-07-27). Boot -// requires --skip-genesis-validation (lambdaclass/ethrex#6783, ≥v16.0.0). // This pin is also the source of internal/ethrex's Tables and of -// testdata/genesis_dump.json; move all three together. -const pinnedEthrexImage = "ghcr.io/lambdaclass/ethrex:23.0.0@sha256:1cbf2c4b498efcc71dc776a130cf5eed3f15d100896a18f05b6fa426ff0e7fc5" +// testdata/genesis_dump.json; move all three together. Repin to a release tag +// once one carries both changes. +const pinnedEthrexImage = "ethpandaops/ethrex:glamsterdam-devnet-7@sha256:e7ac50527b162f6fc8ac5f788dfad6996c63ab215560e918144937b999435624" func ethrexImageRef() string { if v := os.Getenv("ETHREX_IMAGE"); v != "" { diff --git a/client/ethrex/genesis_dump_cgo_test.go b/client/ethrex/genesis_dump_cgo_test.go index 1a926e4..8c0a161 100644 --- a/client/ethrex/genesis_dump_cgo_test.go +++ b/client/ethrex/genesis_dump_cgo_test.go @@ -15,6 +15,8 @@ import ( "context" "encoding/hex" "encoding/json" + "errors" + "fmt" "math/big" "os" "strings" @@ -88,7 +90,6 @@ func TestGenesisDumpGolden(t *testing.T) { // chain- (see ethrex.StoreDir), so read from there, not cfg.DBPath. dbPath := ethrex.StoreDir(cfg.DBPath, g) for _, cfName := range []string{ - "account_codes", "account_code_metadata", "headers", "bodies", @@ -99,6 +100,14 @@ func TestGenesisDumpGolden(t *testing.T) { diffCF(t, dbPath, cfName, wantRows) } + // account_codes is excluded above because testdata/genesis_dump.json still + // carries the pre-bitmap encoding (an RLP list of u32 JUMPDEST offsets) + // while the writer now emits the bitmap ethrex commit 55433c2 writes. The + // keys are unaffected, so those are still asserted byte-exact; the values + // are checked against the writer's own encoder until the fixture is + // regenerated at that commit (see testdata/gen/README.md). + diffCFKeysOnly(t, dbPath, "account_codes", dump["account_codes"]) + // Snap-sync layout: the trie-node CFs hold ONLY structural + leaf-NODE-RLP // rows; the leaf full-path rows (keys ending in the leaf-flag nibble 0x10) // live solely in the flat-KV CFs, never duplicated in the trie-node CFs. So @@ -296,6 +305,84 @@ func readCFRows(t *testing.T, dbPath, cfName string) map[string]string { return gotMap } +// diffCFKeysOnly asserts the CF holds exactly the keys the dump names, and that +// each value is what internal/ethrex's encoder produces for the bytecode the +// dump stores under that key. Weaker than diffCF: it cannot catch the encoder +// and the fixture drifting together, so it is only for a CF whose value shape +// has changed upstream and whose fixture has not been regenerated yet. +func diffCFKeysOnly(t *testing.T, dbPath, cfName string, wantRows []dumpRow) { + t.Helper() + + gotMap := readCFRows(t, dbPath, cfName) + + wantKeys := make(map[string]struct{}, len(wantRows)) + for _, row := range wantRows { + k := hex.EncodeToString(row.key) + wantKeys[k] = struct{}{} + + gv, ok := gotMap[k] + if !ok { + t.Errorf("CF %s: missing key %s", cfName, k) + continue + } + // Both encodings start with the same RLP(bytecode), so the fixture still + // supplies the bytecode; only the trailing jumpdest section differs. + // Take it from the FIXTURE, never from the row under test, or this + // asserts the encoder against itself. + bytecode, err := bytecodeFromEncodedCode(hex.EncodeToString(row.val)) + if err != nil { + t.Errorf("CF %s key %s: fixture value: %v", cfName, k, err) + continue + } + want := hex.EncodeToString(ethrexinternal.EncodeCode(bytecode)) + if gv != want { + t.Errorf("CF %s key %s:\n got %s\n want %s", cfName, k, gv, want) + } + } + for k := range gotMap { + if _, ok := wantKeys[k]; !ok { + t.Errorf("CF %s: unexpected key %s", cfName, k) + } + } +} + +// bytecodeFromEncodedCode extracts the leading RLP byte string (the bytecode) +// from a hex-encoded account_codes value, ignoring the jumpdest section. +func bytecodeFromEncodedCode(hexVal string) ([]byte, error) { + raw, err := hex.DecodeString(hexVal) + if err != nil { + return nil, err + } + if len(raw) == 0 { + return nil, errors.New("empty account_codes value") + } + switch b := raw[0]; { + case b < 0x80: + return raw[:1], nil + case b <= 0xb7: + n := int(b - 0x80) + if len(raw) < 1+n { + return nil, fmt.Errorf("truncated short string: want %d bytes, have %d", n, len(raw)-1) + } + return raw[1 : 1+n], nil + case b <= 0xbf: + lenLen := int(b - 0xb7) + if len(raw) < 1+lenLen { + return nil, errors.New("truncated long-string header") + } + n := 0 + for _, c := range raw[1 : 1+lenLen] { + n = n<<8 | int(c) + } + if len(raw) < 1+lenLen+n { + return nil, fmt.Errorf("truncated long string: want %d bytes, have %d", n, len(raw)-1-lenLen) + } + return raw[1+lenLen : 1+lenLen+n], nil + default: + return nil, fmt.Errorf("account_codes value starts with a list header (0x%02x)", b) + } +} + func diffCF(t *testing.T, dbPath, cfName string, wantRows []dumpRow) { t.Helper() diff --git a/client/ethrex/run.go b/client/ethrex/run.go index 8c5bc2c..ef3847f 100644 --- a/client/ethrex/run.go +++ b/client/ethrex/run.go @@ -24,7 +24,7 @@ var errNotImplemented = errors.New( // It delegates to the build-tag-gated runImpl: // // - Built with `-tags cgo_ethrex` (Docker only): runImpl in run_cgo.go opens -// one grocksdb instance with 21 column families, drives entitygen → +// one grocksdb instance with 22 column families, drives entitygen → // ethrex.Builder → grocksdb writes, assembles the genesis block. // - Built without the tag (local default): runImpl in run_stub.go returns // errNotImplemented. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 28e1135..69ffd38 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -98,7 +98,7 @@ State Actor generates Ethereum state in three phases: │ │ reth: MDBX state tables + RocksDB history + nippy-jar static_files│ │ │ │ besu: single RocksDB w/ 8 Bonsai column families + chainspec.json │ │ │ │ nethermind: 7 RocksDB + flat column DB + parity chainspec sidecar │ │ -│ │ ethrex: single RocksDB w/ 21 CFs + metadata.json + genesis sidecar│ │ +│ │ ethrex: single RocksDB w/ 22 CFs + metadata.json + genesis sidecar│ │ │ │ erigon: Erigon v3 flat .kv snapshots + minimal MDBX │ │ │ └─────────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ @@ -227,7 +227,7 @@ configurable worker pool / batch size at the generator level. under `/flat` (the flat backend Nethermind >= 1.39.0 serves); periodic `dirSize` sample (every 100 contracts) drives the target-size stop. - **ethrex** (`client/ethrex/run_cgo.go`): cgo + grocksdb. Single - RocksDB with 21 column families. Account and storage trie nodes + RocksDB with 22 column families. Account and storage trie nodes are encoded via `internal/ethrex`'s path-keyed trie codec (two rows per leaf: one full-path row, one nibble-path row). Writes `metadata.json` + `ethrex-genesis.json` sidecars. Behind the @@ -359,7 +359,7 @@ Today's client adapters: `internal/neth/flat`) that Nethermind ≥ 1.39.0 serves as its flat backend, and a parity-format chainspec sidecar. Behind the `cgo_neth` build tag. - `client/ethrex/` — cgo + grocksdb writer producing a single RocksDB - with 21 column families (full list in `internal/ethrex/constants.go`) + with 22 column families (full list in `internal/ethrex/constants.go`) using ethrex's own path-keyed trie codec (`internal/ethrex/`). Two rows written per leaf (full-path + nibble-path). Sidecars: `metadata.json` (schema_version=3) and `ethrex-genesis.json` (full @@ -399,7 +399,7 @@ state-actor/ │ ├── reth/ # cgo + libmdbx writer (cgo_reth build tag) │ ├── besu/ # cgo + librocksdb writer (cgo_besu build tag) │ ├── nethermind/ # cgo + grocksdb writer (cgo_neth build tag) -│ ├── ethrex/ # cgo + grocksdb writer, 21 CFs (cgo_ethrex build tag) +│ ├── ethrex/ # cgo + grocksdb writer, 22 CFs (cgo_ethrex build tag) │ └── erigon/ # cgo + mdbx-go writer, Erigon v3 flat .kv (cgo_erigon build tag) ├── generator/ # Core generation pipeline + Writer interface ├── genesis/ # Client-neutral chainspec types + builder diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 27c8f92..13ab4ed 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -304,7 +304,7 @@ docker run --rm \ **On-disk layout:** -- `/data/` — single RocksDB instance with 21 column families (see `internal/ethrex/constants.go` for the full list) +- `/data/` — single RocksDB instance with 22 column families (see `internal/ethrex/constants.go` for the full list) - `/data/metadata.json` — `{"schema_version": 3}`, required by ethrex `Store::new` - `/data/ethrex-genesis.json` — full genesis JSON; pass via `--network` when booting diff --git a/docs/SKILL.md b/docs/SKILL.md index 4412155..58e6f26 100644 --- a/docs/SKILL.md +++ b/docs/SKILL.md @@ -101,7 +101,7 @@ See [`RUNBOOK.md#geth`](RUNBOOK.md#geth) for the boot command, and [`client/geth go run . --db=/tmp/sa-reth --client=reth --target-size=100MB # MDBX + RocksDB + static files go run . --db=/tmp/sa-besu --client=besu --target-size=100MB # single RocksDB, 8 Bonsai CFs go run . --db=/tmp/sa-neth --client=nethermind --target-size=100MB # 7 RocksDB + flat column DB -go run . --db=/tmp/sa-ethrex --client=ethrex --target-size=100MB # single RocksDB, 21 CFs +go run . --db=/tmp/sa-ethrex --client=ethrex --target-size=100MB # single RocksDB, 22 CFs go run . --db=/tmp/sa-erigon --client=erigon --target-size=100MB # Erigon v3 flat .kv snapshots + minimal MDBX ``` diff --git a/internal/ethrex/code.go b/internal/ethrex/code.go index b23f30b..e68b909 100644 --- a/internal/ethrex/code.go +++ b/internal/ethrex/code.go @@ -1,21 +1,30 @@ package ethrex -// ComputeJumpTargets scans bytecode and returns the offsets of JUMPDEST (0x5B) -// opcodes that are not inside PUSH data. Mirrors ethrex account.rs:58-79. +// ComputeJumpdestBitmap returns the JUMPDEST bitmap ethrex persists alongside a +// bytecode: one bit per bytecode byte, set when that offset holds a JUMPDEST +// (0x5B) that is not part of a PUSH immediate. Bit i lives in bit (i%8) of byte +// (i/8), least-significant bit first, so the bitmap is ceil(len/8) bytes. // -// Scan rules: -// - opcode 0x5B (JUMPDEST): append uint32(i), advance i by 1. +// Bytecode with no jump destination gets a ZERO-LENGTH bitmap rather than an +// all-zero one: ethrex reads a missing byte as "no jump destination" +// (Code::is_valid_jumpdest), so it never allocates a map for the common +// jumpless case (EOAs, tiny contracts). +// +// Scan rules (ethrex Code::compute_jumpdests, crates/common/types/account.rs): +// - opcode 0x5B (JUMPDEST): set bit i, advance i by 1. // - opcode 0x60..0x7F (PUSH1..PUSH32): advance i by (opcode - 0x5F + 1) to skip // the opcode itself plus its immediate bytes. // - any other opcode: advance i by 1. -func ComputeJumpTargets(bytecode []byte) []uint32 { - var targets []uint32 +func ComputeJumpdestBitmap(bytecode []byte) []byte { + bitmap := make([]byte, (len(bytecode)+7)/8) + found := false i := 0 for i < len(bytecode) { op := bytecode[i] switch { case op == 0x5B: - targets = append(targets, uint32(i)) + bitmap[i/8] |= 1 << (i % 8) + found = true i++ case op >= 0x60 && op <= 0x7F: // PUSH1..PUSH32: skip opcode + immediate bytes. @@ -24,34 +33,31 @@ func ComputeJumpTargets(bytecode []byte) []uint32 { i++ } } - return targets + if !found { + return nil + } + return bitmap } // EncodeCode returns the concatenation of: // // 1. RLP(bytecode) — bytecode encoded as an RLP byte string. -// 2. RLP(jumpTargetsList) — jump targets encoded as an RLP list of minimal -// big-endian integers (one per uint32 target offset). +// 2. RLP(jumpdestBitmap) — the JUMPDEST bitmap as an RLP byte string. // // The two RLP encodings are concatenated directly (not wrapped in an outer list). -// Mirrors ethrex account_codes encoding. +// Mirrors ethrex's encode_code (crates/storage/store.rs). +// +// The bitmap replaced an RLP list of u32 JUMPDEST offsets. ethrex still reads +// that older form: decode_jumpdests branches on the RLP item header and rebuilds +// the bitmap from the bytecode when it finds a list. // // Golden checks: -// - EncodeCode(0x60015b00) = 0x8460015b00 c1 02 (jumpTargets=[2]) -// - EncodeCode(0x600160015500) = 0x86600160015500 c0 (empty jumpTargets) -// - EncodeCode(nil) = 0x80 c0 +// - EncodeCode(0x60015b00) = 0x8460015b00 04 (JUMPDEST at offset 2) +// - EncodeCode(0x600160015500) = 0x86600160015500 80 (no JUMPDEST) +// - EncodeCode(nil) = 0x80 80 func EncodeCode(bytecode []byte) []byte { - // Part 1: RLP(bytecode). part1 := rlpEncodeBytes(bytecode) - - // Part 2: RLP list of jump target uint32 values. - targets := ComputeJumpTargets(bytecode) - var listPayload []byte - for _, t := range targets { - listPayload = append(listPayload, rlpEncodeUint32(t)...) - } - part2 := rlpEncodeListRaw(listPayload) - + part2 := rlpEncodeBytes(ComputeJumpdestBitmap(bytecode)) return append(part1, part2...) } @@ -70,24 +76,3 @@ func CodeLengthMetadata(bytecode []byte) [8]byte { out[7] = byte(n) return out } - -// rlpEncodeUint32 encodes a uint32 as an RLP integer (minimal big-endian). -func rlpEncodeUint32(n uint32) []byte { - if n == 0 { - return []byte{0x80} - } - b := minBEBytesU32(n) - return rlpEncodeBytes(b) -} - -// minBEBytesU32 returns the minimal big-endian encoding of a uint32. -func minBEBytesU32(n uint32) []byte { - var buf [4]byte - i := 3 - for n > 0 { - buf[i] = byte(n) - n >>= 8 - i-- - } - return buf[i+1:] -} diff --git a/internal/ethrex/constants.go b/internal/ethrex/constants.go index aa014ba..376ebdb 100644 --- a/internal/ethrex/constants.go +++ b/internal/ethrex/constants.go @@ -1,7 +1,7 @@ package ethrex // Column-family names for ethrex's single RocksDB instance. -// Sourced from ethrex crates/storage/api/tables.rs at v23.0.0. +// Sourced from ethrex crates/storage/api/tables.rs at commit 55433c2. const ( CFChainData = "chain_data" CFAccountCodes = "account_codes" @@ -23,10 +23,11 @@ const ( CFMiscValues = "misc_values" CFExecutionWitnesses = "execution_witnesses" CFBlockAccessLists = "block_access_lists" + CFStateHistory = "state_history" CFBadBlocks = "bad_blocks" ) -// Tables is the ordered list of all 21 ethrex column families, matching +// Tables is the ordered list of all 22 ethrex column families, matching // ethrex's TABLES array. It must not run ahead of the boot pin in // client/ethrex/e2e_test.go: ethrex silently drops any CF absent from // its own TABLES (drop_obsolete_cfs, warn-only). @@ -51,6 +52,7 @@ var Tables = []string{ CFMiscValues, CFExecutionWitnesses, CFBlockAccessLists, + CFStateHistory, CFBadBlocks, } diff --git a/internal/ethrex/doc.go b/internal/ethrex/doc.go index 5de43a6..0b6bef6 100644 --- a/internal/ethrex/doc.go +++ b/internal/ethrex/doc.go @@ -1,6 +1,7 @@ -// Package ethrex implements the trie codec primitives used by ethrex -// (pinned release v23.0.0, lambdaclass/ethrex). The golden fixture's -// state-bearing CFs are byte-identical from v13.0.0 through v23.0.0. +// Package ethrex implements the trie and code codec primitives used by ethrex +// (pinned commit 55433c2, lambdaclass/ethrex). The golden fixture's trie-node +// CFs are byte-identical from v13.0.0 through that commit; account_codes values +// changed shape there (see EncodeCode). // // # Two-rows-per-leaf model // diff --git a/internal/ethrex/ethrex_test.go b/internal/ethrex/ethrex_test.go index e9646f6..777cfe8 100644 --- a/internal/ethrex/ethrex_test.go +++ b/internal/ethrex/ethrex_test.go @@ -210,23 +210,91 @@ func TestEncodeStorageValue(t *testing.T) { // code.go golden checks // --------------------------------------------------------------------------- -func TestComputeJumpTargets(t *testing.T) { - // 0x60015b00: PUSH1 0x01 (skips byte 1), JUMPDEST at 2, STOP at 3. - got := ComputeJumpTargets(hexBytes("60015b00")) - if len(got) != 1 || got[0] != 2 { - t.Errorf("ComputeJumpTargets(60015b00): got %v, want [2]", got) +func TestComputeJumpdestBitmap(t *testing.T) { + tests := []struct { + name string + code string + want string // hex of the bitmap, "" for zero-length + }{ + // PUSH1 0x01 (skips byte 1), JUMPDEST at 2, STOP at 3. + // Bit 2 of byte 0 → 1<<2 = 0x04. + {"jumpdest at offset 2", "60015b00", "04"}, + // PUSH1 0x01, PUSH1 0x01, SSTORE, STOP — no JUMPDEST, so zero-length + // rather than a single all-zero byte. + {"no jumpdest", "600160015500", ""}, + {"empty code", "", ""}, + // Two destinations in the same byte: offsets 1 and 3 → 0x02|0x08 = 0x0a. + {"two in one byte", "005b005b", "0a"}, + // The byte after PUSH1 is its immediate, so only offset 2 counts. + {"push immediate is not a destination", "605b5b", "04"}, + // Offset 7 sets the high bit, which matters because the resulting + // bitmap byte (0x80) is no longer a self-encoding single RLP byte. + {"high bit of the first byte", "000000000000005b", "80"}, + // Crossing a byte boundary: offsets 0 and 10 → byte 0 bit 0, byte 1 bit 2. + {"spans two bitmap bytes", "5b0000000000000000005b000000", "0104"}, + } + for _, tc := range tests { + var code []byte + if tc.code != "" { + code = hexBytes(tc.code) + } + got := ComputeJumpdestBitmap(code) + var want []byte + if tc.want != "" { + want = hexBytes(tc.want) + } + if string(got) != string(want) { + t.Errorf("ComputeJumpdestBitmap(%s) [%s]: got %x, want %x", + tc.code, tc.name, got, want) + } } +} - // 0x600160015500: PUSH1 0x01, PUSH1 0x01, SSTORE 0x55, STOP — no JUMPDEST. - got2 := ComputeJumpTargets(hexBytes("600160015500")) - if len(got2) != 0 { - t.Errorf("ComputeJumpTargets(600160015500): got %v, want []", got2) +// TestJumpdestBitmapMatchesScan cross-checks the bitmap against an independent +// scan of the same bytecode, bit by bit, the way ethrex's is_valid_jumpdest +// reads it: bit (offset%8) of byte (offset/8), with a missing byte meaning "no +// jump destination". +func TestJumpdestBitmapMatchesScan(t *testing.T) { + // A JUMPDEST sea with PUSH runs threaded through it, so PUSH-immediate + // suppression and byte-boundary crossings both occur many times. + code := make([]byte, 0, 512) + for i := 0; i < 64; i++ { + code = append(code, 0x5B, 0x5B, 0x60, 0x5B, 0x5B, 0x7F) + code = append(code, make([]byte, 31)...) // PUSH32 immediate + } + + bitmap := ComputeJumpdestBitmap(code) + + isSet := func(offset int) bool { + b := offset / 8 + if b >= len(bitmap) { + return false + } + return bitmap[b]&(1<<(offset%8)) != 0 + } + + // Independent scan: walk the bytecode skipping PUSH immediates. + want := make(map[int]bool) + for i := 0; i < len(code); { + switch op := code[i]; { + case op == 0x5B: + want[i] = true + i++ + case op >= 0x60 && op <= 0x7F: + i += int(op-0x5F) + 1 + default: + i++ + } } - // Empty code. - got3 := ComputeJumpTargets(nil) - if len(got3) != 0 { - t.Errorf("ComputeJumpTargets(nil): got %v, want []", got3) + for offset := range code { + if isSet(offset) != want[offset] { + t.Fatalf("offset %d: bitmap says %v, scan says %v", + offset, isSet(offset), want[offset]) + } + } + if got, wantLen := len(bitmap), (len(code)+7)/8; got != wantLen { + t.Errorf("bitmap length: got %d, want %d", got, wantLen) } } @@ -235,9 +303,15 @@ func TestEncodeCode(t *testing.T) { code string want string }{ - {"60015b00", "8460015b00c102"}, - {"600160015500", "86600160015500c0"}, - {"", "80c0"}, + // RLP(bytecode) ++ RLP(bitmap). 0x04 is a self-encoding single byte. + {"60015b00", "8460015b0004"}, + // Zero-length bitmap encodes as the RLP empty string, 0x80. + {"600160015500", "8660016001550080"}, + {"", "8080"}, + // A 0x80 bitmap byte needs the 0x81 length prefix, unlike 0x04 above. + {"000000000000005b", "88000000000000005b8180"}, + // A 14-byte code needs a 2-byte bitmap, encoded as the short string 0x82. + {"5b0000000000000000005b000000", "8e5b0000000000000000005b000000820104"}, } for _, tc := range tests { var code []byte diff --git a/internal/ethrex/testdata/gen/README.md b/internal/ethrex/testdata/gen/README.md index 106ea23..c366e6a 100644 --- a/internal/ethrex/testdata/gen/README.md +++ b/internal/ethrex/testdata/gen/README.md @@ -5,26 +5,37 @@ column family as hex-encoded JSON. It must be regenerated whenever the ethrex storage schema changes. -## Pinned release +## Pinned commit ``` -lambdaclass/ethrex @ v23.0.0 +lambdaclass/ethrex @ 55433c2 ``` -This is the same pin as the e2e boot image (`client/ethrex/e2e_test.go`) and as -the column-family list in `internal/ethrex/constants.go`. All three move -together. - -The state-bearing CFs (`account_trie_nodes`, `storage_trie_nodes`, -`account_codes`, `account_code_metadata`) are byte-identical from v13.0.0 -(commit 318ec2888) through v23.0.0, each step verified by regenerating and -diffing against the previous dump. Across that range only two things moved: -`chain_data[0x80]` gained fork fields (`hegotaTime`, introduced upstream in -v21.0.0 via ethrex#6326), and the `bad_blocks` column family arrived in -v22.0.0 (ethrex#6948, empty at genesis). `STORE_SCHEMA_VERSION` reached 3 at -v16 and has not moved since. - -Any ethrex release that bumps `STORE_SCHEMA_VERSION`, adds or removes a column +A pre-release commit, not a tagged release: it is what the ethpandaops +`glamsterdam-devnet-7` image was built from, and the latest release (v23.0.0) +predates two changes state-actor has to match. This is the same pin as the e2e +boot image (`client/ethrex/e2e_test.go`) and as the column-family list in +`internal/ethrex/constants.go`. All three move together. + +`account_trie_nodes` and `storage_trie_nodes` are byte-identical from v13.0.0 +(commit 318ec2888) through this commit, each step verified by regenerating and +diffing against the previous dump. What moved: + +- `chain_data[0x80]` gained fork fields (`hegotaTime`, upstream in v21.0.0 via + ethrex#6326). +- `bad_blocks` arrived in v22.0.0 (ethrex#6948, empty at genesis). +- `state_history` arrived after v23.0.0 (empty at genesis). +- `account_codes` values carry a JUMPDEST bitmap rather than an RLP list of u32 + offsets (ethrex#7095). ethrex still reads the older form — `decode_jumpdests` + branches on the RLP item header and rebuilds the bitmap from the bytecode when + it finds a list — so this is a size and representativeness change, not a + compatibility break. + +`STORE_SCHEMA_VERSION` reached 3 at v16 and has not moved at this commit. +ethrex#7095 bumps it to 4 on its own branch, gated by a no-op `migrate_3_to_4`; +that bump is not in this build, so `metadata.json` stays at 3. + +Any ethrex build that bumps `STORE_SCHEMA_VERSION`, adds or removes a column family, or changes the key layout of `account_trie_nodes`, `storage_trie_nodes`, `account_codes`, or `account_code_metadata` requires regenerating the dump and re-reviewing the Go codec in `internal/ethrex/`. @@ -39,7 +50,7 @@ list, so `Tables` must never run ahead of this pin. ```sh git clone https://github.com/lambdaclass/ethrex cd ethrex - git checkout v23.0.0 + git checkout 55433c2 ``` 2. Copy the dump harness into ethrex's examples directory: