Skip to content

fix(l1): add reachable sepolia bootnodes - #7079

Merged
ilitteri merged 1 commit into
mainfrom
fix/sepolia-bootnodes
Aug 1, 2026
Merged

fix(l1): add reachable sepolia bootnodes#7079
ilitteri merged 1 commit into
mainfrom
fix/sepolia-bootnodes

Conversation

@ilitteri

Copy link
Copy Markdown
Collaborator

Motivation

All five of sepolia's EF execution-layer bootnodes stopped answering UDP discovery on 2026-07-30, in a window between 12:39 and 15:44 UTC, and have not come back. They are the only entries in cmd/ethrex/networks/sepolia/bootnodes.json, so a node starting with an empty peer table has no way into the DHT at all: it re-pings those five addresses forever and sits at zero peers, unable to begin snap sync even though the network itself is healthy and the consensus client peers normally.

The impact is broad, because everything that starts from a fresh datadir hits it:

  • Snapsync CI: both sepolia jobs fail every run, each burning its full 3h30m budget. Run 30594206688 is representative — sepolia - Lighthouse and sepolia - Prysm both failed at 3h31m while both hoodi jobs passed. That is over 7h of a 6h cron window, so runs now queue behind each other.
  • The multisync host: the sepolia leg has failed every cycle since 2026-07-30 23:15 UTC, having never failed before in that log (the previous failure of any network was 2026-06-10). The currently stuck container had downloaded not one header or account after eight hours, having sent ~143k discovery packets to those five addresses and received nothing back, while hoodi and mainnet synced normally beside it.
  • Users, on any cold start.

This is not a regression and not a fork mismatch: commit 2f1593f2 passed sepolia twice before failing on the same SHA, and sepolia's eth_config reports its current fork with next: null, matching our genesis exactly. The bootnode hosts are alive — they answer ICMP, and one still serves consensus-layer discv5 — so only the execution-layer discovery service is gone. Upstream eth-clients/sepolia still lists all five unchanged, so nothing has been retired on paper.

Description

Keeps the five EF entries and appends twelve reachable nodes taken from sepolia's public DNS node list.

  • The EF entries stay because they are the canonical bootnodes, upstream still publishes them unchanged, and they may well come back.
  • Each added node was verified individually, by starting a node with that bootnode as its only entry and DNS-based discovery unavailable, then confirming inbound discovery replies could only have come from it. 23 of the 24 candidates tested were responsive; the one that was not was dropped.
  • They are spread across distinct providers and regions deliberately. The five EF bootnodes were all on a single provider, which is precisely why losing them was all-or-nothing.

Verified end to end on this branch with no command-line overrides: a cold start reaches peers within 30 seconds, and the failed to find target block header loop that characterises the outage does not appear at all (0 occurrences, against 906 on the stuck multisync container).

This buys back the ability to sync sepolia today, and it covers CI, the multisync host and users in one change. It is explicitly not a substitute for reading that node list at runtime — hardcoded addresses are exactly what failed here, and these are community nodes rather than dedicated infrastructure, so they will drift over time. #7077 is the durable fix.

Supersedes #7076, which worked around the same outage in the snapsync action only; that PR can be closed in favour of this one.

Checklist

  • Updated STORE_SCHEMA_VERSION (crates/storage/lib.rs) if the PR includes breaking changes to the Store requiring a re-sync.

Not applicable — no storage change, and no resync is required.

… the network.

All five of sepolia's EF execution-layer bootnodes stopped answering UDP
discovery on 2026-07-30, in a window between 12:39 and 15:44 UTC, and have not
come back. They are the only entries in this file, so a node starting with an
empty peer table has no way into the DHT at all: it re-pings those five
addresses forever and sits at zero peers, unable to begin snap sync even though
the network itself is healthy.

The effect is not subtle. Both sepolia snapsync jobs now fail every run, each
burning its full 3h30m budget, which puts that workflow over its 6h cron and
queues runs behind themselves. On the multisync host the sepolia leg has failed
every cycle since 23:15 UTC that same day -- having never failed before -- and
the stuck container had downloaded not one header or account after eight hours,
having sent ~143k discovery packets to those five addresses and received nothing
back, while hoodi and mainnet synced normally beside it.

Keep the EF entries: they are the canonical bootnodes, upstream still publishes
them unchanged, and they may well come back. Append twelve nodes taken from the
network's public DNS node list, each verified individually by starting a node
with that bootnode as its only entry and confirming discovery replies come back
-- 23 of the 24 candidates tested were responsive. They are spread across
distinct providers and regions on purpose: the five EF bootnodes were all on one
provider, which is why losing them was all-or-nothing.

Verified end to end on this branch, with no command-line overrides: a cold start
now reaches peers within 30 seconds, and the "failed to find target block
header" loop that characterises the outage does not appear at all.

This buys back the ability to sync sepolia today. It is not a substitute for
being able to read that node list at runtime, since hardcoded addresses are what
just failed here; that is a separate change.
@ilitteri
ilitteri requested a review from a team as a code owner July 31, 2026 15:35
Copilot AI review requested due to automatic review settings July 31, 2026 15:35
@github-actions github-actions Bot added the L1 Ethereum client label Jul 31, 2026
@github-actions

Copy link
Copy Markdown

⚠️ Known Issues — intentionally skipped tests

Source: docs/known_issues.md

Stateless (zkEVM) Amsterdam+ EF tests skipped

Where: tooling/ef_tests/blockchain/test_runner.rsparse_and_execute skips
fixtures with network >= Fork::Amsterdam when running with a stateless backend.
Affects make test-stateless (the vectors_zkevm/ run); make test-levm is
unaffected.

Why: The stateless run uses the tests-zkevm@v0.5.0 bundle, filled against
glamsterdam-devnet v6.1.0, which predeploys the EIP-8282 builder deposit/exit
contracts at the OLD addresses (0x0000884d…d9008282 / 0x000014574a…0f008282).
This client uses the devnet-7 addresses (0x0000bff4…300d8282 /
0x000064d6…800e8282, matching the live tests-glamsterdam-devnet@v7.2.0 bundle
used by make test-levm). Every Amsterdam+ block runs the end-of-block EIP-8282
builder system call; with the new addresses absent from the v0.5.0 bundle, each
stateless Amsterdam+ block fails with
SystemContractCallFailed("System contract: 0x0000…8282 has no code after deployment").
The skip is by fork rather than by test name, since cross-fork directories such as
for_amsterdam/prague/... still execute at the Amsterdam fork.

Removal: Delete the skip_stateless_amsterdam branch in parse_and_execute
once a tests-zkevm bundle filled with the devnet-7 builder predeploy addresses is
released and .fixtures_url_zkevm is bumped to it.

@github-actions

Copy link
Copy Markdown

🤖 Kimi Code Review

File: cmd/ethrex/networks/sepolia/bootnodes.json

Security Concern: Unverified Bootnode Sources
The PR adds 13 new bootnodes without documenting their source. Bootnodes are critical infrastructure for peer discovery; adding untrusted nodes exposes users to eclipse attacks or network partitioning.

  • Line 6-18: Verify these enodes match an official source (e.g., go-ethereum's params/bootnodes.go, Ethereum Foundation infrastructure, or Sepolia community consensus).
  • Action required: Include a comment or commit message referencing the source repository/version (e.g., "Sync with go-ethereum v1.14.0 sepolia bootnodes").

Code Quality: OK

  • JSON syntax is valid (proper comma placement after the previously last element).
  • Enode format is correct (128-hex-character Node ID + valid public IP + standard 30303 port).
  • No duplicate entries detected between new and existing bootnodes.

Recommendation: Require source attribution before merging. If these are from an official upstream, the change is straightforward. If manually curated, they need verification against multiple reputable sources.


Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

That's fine — file ending format is a trivial nit, not worth pursuing further. I have enough to write the review.

Review: fix(l1): add reachable sepolia bootnodes (PR #7079)

This is a pure data change to cmd/ethrex/networks/sepolia/bootnodes.json — no Rust code is touched. I validated the file mechanically:

  • Valid JSON (jq empty passes).
  • All 17 enode entries (5 original + 12 new) have correctly-formed 128-hex-char public keys.
  • No duplicate public keys and no duplicate ip:port pairs across the full list.
  • ip:port syntax is well-formed for every new entry.

Given the change is purely additive (existing 5 EF entries untouched, 12 appended) and scoped to a single static config file, the risk surface is minimal — bootnodes are just a discovery-bootstrap hint list, so a bad entry here fails soft (unreachable peer, tried and skipped) rather than affecting consensus or state correctness.

Observations

  1. No consensus/correctness risk. Bootnodes are not part of any protocol validation path — worst case an unreachable entry just wastes a discovery ping, which is exactly the problem already being solved here (relative to a network that's completely unreachable).

  2. Trust in the new entries is procedural, not cryptographic. The PR body says each of the 24 candidates was verified individually by node identity (pubkey match on discovery reply), which is the right approach — this avoids blindly trusting a DNS/list source. Worth confirming in review (can't be validated from the diff alone) that the verification was done against the genuinely expected sepolia network (correct chain/fork), not just "any UDP discovery response," since an attacker-controlled node could still answer discovery correctly but later misbehave at the eth/68 protocol level. That said, this is inherent to bootnode-based discovery in general and not something this PR makes worse — bootnodes only seed the DHT, later peer/protocol handshakes still apply normal validation.

  3. Provider/geographic diversity is a good call. Explicitly noted in the PR: the outage happened because all 5 EF nodes were single-provider. Spreading the 12 new entries across distinct providers directly addresses the root cause (correlated failure) rather than just patching the symptom.

  4. Acknowledged as a stopgap, not a fix. The PR is explicit that this doesn't replace runtime DNS-based discovery (tracked separately as feat(l1): add EIP-1459 DNS discovery for bootstrap resilience #7077 per the description) — appropriate scoping; hardcoding more addresses is a mitigation, not a durable solution, and the PR doesn't overclaim otherwise.

  5. Checklist item on STORE_SCHEMA_VERSION is correctly marked not applicable — this is a static config list, not a store schema/migration change.

Minor/non-blocking

  • No automated test/lint enforces enode format for this file (e.g., a unit test asserting all entries parse via the existing enode parser). Not a blocker for a config-only fix, but if bootnode churn becomes routine (which the PR itself predicts, since these are community nodes "not dedicated infrastructure" and "will drift over time"), a small #[test] that loads and parses each network's bootnodes.json through the real enode parser would catch a malformed entry at CI time rather than at cold-start. Worth considering, not required for this PR.

Overall: correct, well-verified, minimal-blast-radius fix for a real and urgent operational issue. No code-level concerns.


Automated review by Claude (Anthropic) · sonnet · custom prompt

@greptile-apps

greptile-apps Bot commented Jul 31, 2026

Copy link
Copy Markdown

Greptile Summary

Adds twelve geographically and provider-diverse Sepolia execution-layer bootnodes while retaining the five canonical EF entries, restoring discovery options for fresh nodes when the canonical endpoints are unavailable.

Confidence Score: 5/5

The PR appears safe to merge.

The added enode URLs satisfy the repository parser’s required key and socket-address format, preserve valid JSON, and introduce no identified startup or discovery regression.

Important Files Changed

Filename Overview
cmd/ethrex/networks/sepolia/bootnodes.json Adds twelve syntactically valid enode entries to improve Sepolia bootstrap availability; no actionable defect was identified.

Reviews (1): Last reviewed commit: "Add reachable sepolia bootnodes so a col..." | Re-trigger Greptile

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates ethrex’s bundled Sepolia execution-layer bootnode list to mitigate the current discovery outage where the five canonical EF bootnodes are not responding to UDP discovery, preventing cold-start nodes from joining the DHT.

Changes:

  • Keeps the existing 5 EF Sepolia bootnodes.
  • Appends 12 additional reachable Sepolia bootnodes to restore bootstrap connectivity for fresh datadirs/CI.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No findings.

The PR is a static bootnode list update in cmd/ethrex/networks/sepolia/bootnodes.json. I checked how the file is consumed (serde_json into Vec<Node>) and validated that the resulting Sepolia list has 17 unique enode:// entries with no duplicate pubkeys or socket addresses, so I don’t see a correctness, security, gas/EVM, or consensus-risk regression from this change.

Residual risk is operational rather than code-level: peer discovery quality now depends on these new endpoints actually being reachable and maintained on Sepolia. I couldn’t verify liveness from this environment.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-project-automation github-project-automation Bot moved this to In Review in ethrex_l1 Jul 31, 2026
@ilitteri
ilitteri added this pull request to the merge queue Aug 1, 2026
Merged via the queue into main with commit 79954db Aug 1, 2026
66 checks passed
@ilitteri
ilitteri deleted the fix/sepolia-bootnodes branch August 1, 2026 04:23
@github-project-automation github-project-automation Bot moved this from In Review to Done in ethrex_l1 Aug 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

L1 Ethereum client

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants