feat(node): probe foreign-chain RPC providers on startup by haiyuechen-nearone · Pull Request #3848 · near/mpc

haiyuechen-nearone · 2026-07-17T07:11:59Z

Part of #3764. Wires the identity-based check into the node. Merge the identity-probe stack (#3932 → #3962 → #3963 → #3964) first, then this trio bottom-up:

feat(node): probe foreign-chain RPC providers on startup #3848 (this PR) — probe every configured provider on startup + per-provider logs
feat(node): expose foreign-chain RPC provider health metric #3849 — per-chain configured / healthy Prometheus gauges
feat(node): surface healthy provider counts on /debug/node_config #3850 — healthy counts on /debug/node_config

Spawns a detached startup task that concurrently probes every configured foreign-chain RPC provider once — verifying its chain identity and inspecting a recent transaction, the same inspector and auth the real verification path uses — and logs a per-provider result plus an x/y providers healthy summary. The probe never blocks startup, and failure reasons aren't logged (they can carry secrets).

Expected identities come from foreign_chain_health_check.identities (no built-ins, so any network — including local — is checkable; a configured chain with no identity fails). ExpectedIdentities moves to node-config so the node and config-tester share one type.

claude · 2026-07-22T11:53:07Z

Pull request overview

Adds a startup health check that probes every configured foreign-chain RPC provider once against a golden reference transaction, then logs per-provider results and a x/y providers healthy summary. Runs in a detached task so it never blocks node startup. Mainnet/testnet always use a built-in golden set; local/sandbox chains only run the check when the operator supplies their own golden via a new foreign_chain_health_check_golden config field.

Changes:

New HealthCheckGoldenConfig (and BlockHashGolden/AptosGolden/SuiGolden) in mpc-node-config, plus a per-node foreign_chain_health_check_golden field.
Refactor golden.rs to represent the built-in mainnet/testnet sets as HealthCheckGoldenConfig values (unified type with the operator-supplied path).
New NetworkKind / HealthCheckRoute / SkipReason in network.rs, and resolve_network_from_config that trusts near_init.chain_id first and falls back to the contract-id suffix.
check_all_providers_with_golden runs the 10 per-chain probes concurrently via join_all, replacing the prior sequential mark_skipped / mark_not_configured mutation style.
New crates/node/src/foreign_chain_health.rs wires the route decision, logs results with secret-safe fields, and warns when a golden is set on mainnet/testnet (ignored).
New e2e test startup_foreign_chain_health.rs and mock-server plumbing (setup_evm_mock_with_block_hash, wait_for_log_substring) exercise the config-supplied golden path.
foreign-chain-config-tester now shares network_from_contract_id.

Reviewed changes

Per-file summary

File	Description
`crates/node/src/foreign_chain_health.rs`	New: startup entrypoint; picks a `HealthCheckRoute`, runs it, logs summary + per-provider status without leaking failure reasons.
`crates/node/src/run.rs`	Resolves network kind and spawns `run_startup_health_check` detached before indexer starts.
`crates/node/src/lib.rs` / `tests.rs` / `web.rs`	Module registration + test wiring for the new config field.
`crates/foreign-chain-health-check/src/lib.rs`	Adds `check_all_providers_with_golden`; converts per-chain runners to `Option<&Cfg>` + `Option<&Golden>` returning `Vec<ProviderResult>`; fans out via `join_all`.
`crates/foreign-chain-health-check/src/network.rs`	Adds `NetworkKind`, `HealthCheckRoute::decide`, `SkipReason`, `resolve_network_from_config`, `network_from_contract_id` and tests.
`crates/foreign-chain-health-check/src/golden.rs`	Reworks built-in reference tables to `HealthCheckGoldenConfig`.
`crates/foreign-chain-health-check/src/checks.rs`	Signature changes: `&str` params where owned refs used to be passed.
`crates/foreign-chain-config-tester/src/config.rs`	Shares `network_from_contract_id` with the node.
`crates/node-config/src/foreign_chains.rs` / `lib.rs`	New `HealthCheckGoldenConfig` + per-chain golden types; adds `foreign_chain_health_check_golden` to `ConfigFile`.
`crates/e2e-tests/src/cluster.rs` / `mpc_node.rs` / `foreign_chain_mock.rs` / `README.md`	Per-node golden plumbing, log-tail helper, EVM mock override for block hash.
`crates/e2e-tests/tests/startup_foreign_chain_health.rs`	New e2e verifying pass/fail per provider and skip-on-no-golden for a second node.
`docs/foreign-chain-transactions.md`	Documents the new startup probe behavior.

Findings

Non-blocking (nits, follow-ups, suggestions):

crates/node/src/run.rs:216 — The spawned health-check task's JoinHandle is dropped, so any panic inside a probe (e.g., a future bug in an inspector) silently disappears with no operator signal. Since this is diagnostic-only code the risk is small, but wrapping the future with FutureExt::catch_unwind (or spawning a small supervisor that logs on panic) would make failure modes debuggable. Consider aligning with the pattern used for _web_server_join_handle on line 210 if there's an established convention.
crates/node-config/src/foreign_chains.rs:63-85 — HealthCheckGoldenConfig field order (bitcoin, abstract, starknet, bnb, base, arbitrum, hyper_evm, polygon, aptos, sui) is neither alphabetical nor grouped by family. This will render in the same order in serialized TOML/JSON output and in the Debug impl. Alphabetizing would make config diffs easier to read and match the order in ForeignChainsConfig (if that one is alphabetized).
crates/foreign-chain-health-check/src/network.rs:90 — The mainnet contract-id fallback is a single hardcoded contract_id == \"v1.signer\" string. If a future mainnet deployment introduces another bare (non-.near) name, it will silently become Undetermined and skip the probe. Not a bug today, just a fragile piece worth a doc note or a small allowlist.
crates/foreign-chain-health-check/src/lib.rs:82-94 — Ethereum/Solana are hardcoded as "not yet supported by the node". When either becomes supported, the branch here needs to be dropped in lockstep with verify_foreign_tx/sign.rs; the existing referring comment mitigates but a TODO(#issue) linking the tracking issue (if one exists) would be safer against silent drift.
crates/e2e-tests/tests/startup_foreign_chain_health.rs:99 — The test asserts on the exact wording \"foreign-chain RPC provider health check complete: 1/2 providers healthy\", which pins the log string across two crates. Not wrong, but any future wording tweak in foreign_chain_health.rs will break this e2e; a contains on both \"providers healthy\" and \"1/2\" would decouple the check.
crates/node/src/foreign_chain_health.rs:83 — The TODO(#2350) comment is good; consider verifying that Define a systematic approach to secret redaction #2350 explicitly tracks "log health-check failure reasons after redaction lands" so the follow-up is not lost when Define a systematic approach to secret redaction #2350 gets closed on the general redaction work.

✅ Approved

haiyuechen-nearone · 2026-07-22T12:26:22Z

@claude

haiyuechen-nearone · 2026-07-22T12:42:59Z

+pub fn resolve_network_from_config(
+    near_init: Option<&NearInitConfig>,
+    contract_id: &str,
+) -> NetworkKind {
+    match near_init {
+        Some(near_init) => match near_init.chain_id {
+            ChainId::Mainnet => NetworkKind::Public(Network::Mainnet),
+            ChainId::Testnet => NetworkKind::Public(Network::Testnet),
+            ChainId::Localnet | ChainId::Sandbox | ChainId::Custom(_) => NetworkKind::Local,
+        },
+        None => match network_from_contract_id(contract_id) {
+            Some(network) => NetworkKind::Public(network),
+            None => NetworkKind::Undetermined,
+        },
+    }
+}
+
+/// Classify mainnet/testnet from a contract-id suffix.
+pub fn network_from_contract_id(contract_id: &str) -> Option<Network> {
+    if contract_id.ends_with(".testnet") {
+        Some(Network::Testnet)
+    } else if contract_id.ends_with(".near") || contract_id == "v1.signer" {
+        Some(Network::Mainnet)
+    } else {
+        None
+    }
+}


To reviewers: please double check that I have the correct understanding of our network and contract naming conventions

netrome

Started looking at this now, some initial thoughts:

We should not expect RPCs to retain individual TX information forever (blocking)
I'd prefer to minimize network-specific code paths. Ideally we should not have any network-specific code paths, but I can see the point of providing defaults for mainnet and testnet for convenience.

netrome · 2026-07-22T19:26:04Z

+    // A config-supplied golden is only meaningful on a local chain; on
+    // mainnet/testnet the built-in set is always used.


Why don't we provide golden configurations on mainnet/testnet? Hard-coding values feels like a time bomb, as many RPCs might not serve requests for old transactions.

My understanding was that we want to always check for the same transaction, which is baked in to the binary, maybe I misunderstood.

@anodar Judging from the foreign chain tester CLI code I assumed that we would like to continue using hard coded golden transactions for health check, maybe you had something else in mind?

netrome · 2026-07-22T19:29:39Z

+            "foreign_chain_health_check_golden is set in config but ignored on \
+             mainnet/testnet; the built-in golden set is always used — remove it \
+             from config.yaml"


Wouldn't it make sense to override the golden set if provided manually? The hard-coded values could exist as a convenience so we don't have to ask all operators to manually add these but retaining the ability to always override sounds like it would simplify our code. I'd prefer if we have as few network-specific code paths as possible. Ideally none.

netrome · 2026-07-22T19:31:53Z

+/// Hashes are hex, with or without a `0x` prefix.
+#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq)]
+pub struct BlockHashGolden {
+    pub tx: String,
+    pub block_hash: String,
+}


I think this is a dangerous construction. Many RPCs won't serve transactions forever. It's better to ask for genesis hash or chain ID, like we do for Sui.

Replace Starknet's pinned golden transaction with the Sui-style hybrid probe: verify the chain identity (starknet_chainId, a genesis-derived constant that is never pruned), then run the real inspector over a transaction from the latest L1-accepted block at AcceptedOnL1 finality. This removes the archive-node dependency and the per-network golden-vector maintenance while keeping the production-path smoke test (receipt -> finality -> canonicality). AcceptedOnL1 matches what real verify-foreign-tx requests use and actually exercises the finality check (the AcceptedOnL2 branch is a no-op). Discovery uses the l1_accepted block tag, so the probe requires provider JSON-RPC v0.9+. To stay robust against load-balanced providers whose backends disagree on the L1-accepted frontier, the probe walks back from the head — doubling the step each time, capped at MAX_WALKBACK_BLOCKS — past empty blocks and past receipts a lagging backend still reports below L1 finality. Expected identities carry no built-in values: they come from configuration (foreign_chain_health_check.identities), so any network — including local or custom ones — is checkable, and a configured identity-probed chain fails until its identity is set. Sui's built-in reference moves to configuration on the same terms. Operator templates ship with the identities prefilled.

…ed transaction Extend the identity-based probe to the 6 EVM chains (base, bnb, arbitrum, polygon, hyper_evm, abstract): verify eth_chainId against the configured expected identity, then run the inspector over a transaction from a block 10 below the finalized head, walking back past empty blocks. The EVM golden vectors are deleted: expected chain ids come from configuration like the other identity chains (well-known values in the config-tester README).

…ction Extend the identity-based probe to Bitcoin: verify the genesis block hash (getblockhash 0 — never pruned, and a proof-of-data identity since the provider must actually hold block 0) against the configured expected identity, then run the inspector over the coinbase of a block 10 below the tip. Tip height comes from getbestblockhash + getblock rather than getblockcount: some provider edges serve getblockcount as a JSON-RPC 1.0-style response the 2.0 transport rejects. The pinned Bitcoin golden vectors are deleted; the expected genesis comes from configuration like the other identity chains.

…action Extend the identity-based probe to Aptos, the last golden-transaction chain: verify the ledger chain_id (GET /v1; 1 = mainnet, 2 = testnet) against the configured expected identity, then run the inspector over a committed transaction 100 ledger versions behind the tip. With every chain now identity-based and configuration the sole source of expected identities, the golden module's vectors and the entire network machinery are deleted: the Network enum, the CLI's --network flag, and the chain-id/contract-id auto-detection all existed only to pick between built-in reference sets. The golden module becomes a small parse module of identity decoders.

Spawn a detached, diagnostic-only startup task that concurrently probes every configured foreign-chain RPC provider against its expected chain identity and logs a per-provider result plus an `x/y providers healthy` summary — so config typos and un-enabled API keys surface at startup instead of on the first real verification request. The probe never blocks startup, and failure reasons are not logged (they can carry secrets). Expected identities are read from the node's typed config (`foreign_chain_health_check.identities`); `ExpectedIdentities` moves to node-config so the node and the config-tester share one type.

haiyuechen-nearone · 2026-07-30T23:39:39Z

Superseded by #4013

This was referenced Jul 17, 2026

feat(node): expose foreign-chain RPC provider health metric #3849

Merged

feat(node): surface healthy provider counts on /debug/node_config #3850

Merged

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 131a75f to e9e4b81 Compare July 17, 2026 08:03

haiyuechen-nearone force-pushed the 3764-extract-foreign-chain-health-check-crate branch from 05a55bc to ad30f88 Compare July 17, 2026 08:03

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch 2 times, most recently from 4eeed22 to 8cdfbfa Compare July 17, 2026 08:35

haiyuechen-nearone force-pushed the 3764-extract-foreign-chain-health-check-crate branch from ad30f88 to a1145b2 Compare July 17, 2026 08:35

This was referenced Jul 17, 2026

Foreign chain RPC provider health checks: validate config without curated reference data #3764

Open

refactor: extract foreign-chain-health-check library from config-tester #3797

Merged

haiyuechen-nearone force-pushed the 3764-extract-foreign-chain-health-check-crate branch from a1145b2 to 9031612 Compare July 20, 2026 09:27

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 8cdfbfa to 36e439e Compare July 20, 2026 09:57

haiyuechen-nearone self-assigned this Jul 20, 2026

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch 7 times, most recently from a2805eb to 3817516 Compare July 20, 2026 18:21

haiyuechen-nearone mentioned this pull request Jul 20, 2026

Run foreign-chain RPC health check on localnet (developer-supplied endpoints + golden) + seed provider whitelist #3860

Closed

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 30c77ea to 208c837 Compare July 21, 2026 09:24

Base automatically changed from 3764-extract-foreign-chain-health-check-crate to main July 21, 2026 09:52

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 208c837 to 7958f4b Compare July 21, 2026 11:30

haiyuechen-nearone marked this pull request as ready for review July 22, 2026 11:48

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 8e97b7e to 47f58d5 Compare July 22, 2026 12:32

haiyuechen-nearone commented Jul 22, 2026

View reviewed changes

haiyuechen-nearone requested review from anodar and netrome July 22, 2026 13:18

netrome requested changes Jul 22, 2026

View reviewed changes

haiyuechen-nearone marked this pull request as draft July 23, 2026 08:19

haiyuechen-nearone mentioned this pull request Jul 23, 2026

feat: probe Starknet by chain id and a recent transaction #3932

Open

netrome mentioned this pull request Jul 27, 2026

config tester: reuse the node's identity probe and drop golden transactions #3969

Open

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 485eb48 to 315514c Compare July 27, 2026 16:35

haiyuechen-nearone changed the base branch from main to 3764-foreign-chain-health-hybrid-aptos July 27, 2026 16:35

haiyuechen-nearone mentioned this pull request Jul 27, 2026

feat(health-check): probe Aptos by ledger chain id and a recent transaction #3964

Closed

haiyuechen-nearone added 5 commits July 30, 2026 21:21

haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 315514c to 748639d Compare July 30, 2026 19:23

haiyuechen-nearone changed the base branch from 3764-foreign-chain-health-hybrid-aptos to main July 30, 2026 19:23

haiyuechen-nearone closed this Jul 30, 2026

haiyuechen-nearone mentioned this pull request Aug 5, 2026

feat: report foreign chain RPC provider health at Node startup #4071

Open

Navigation Menu

Uh oh!

feat(node): probe foreign-chain RPC providers on startup - #3848

feat(node): probe foreign-chain RPC providers on startup#3848
haiyuechen-nearone wants to merge 5 commits into
mainfrom
3764-node-startup-health-check

haiyuechen-nearone commented Jul 17, 2026 •

edited

Loading

Uh oh!

claude Bot commented Jul 22, 2026

Uh oh!

haiyuechen-nearone commented Jul 22, 2026

Uh oh!

haiyuechen-nearone Jul 22, 2026

Uh oh!

netrome left a comment

Uh oh!

netrome Jul 22, 2026

Uh oh!

haiyuechen-nearone Jul 23, 2026

Uh oh!

netrome Jul 22, 2026

Uh oh!

netrome Jul 22, 2026

Uh oh!

haiyuechen-nearone commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		// A config-supplied golden is only meaningful on a local chain; on
		// mainnet/testnet the built-in set is always used.

Uh oh!

Conversation

haiyuechen-nearone commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

claude Bot commented Jul 22, 2026

Pull request overview

Reviewed changes

Findings

Uh oh!

haiyuechen-nearone commented Jul 22, 2026

Uh oh!

haiyuechen-nearone Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

netrome left a comment

Choose a reason for hiding this comment

Uh oh!

netrome Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

haiyuechen-nearone Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

netrome Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

netrome Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

haiyuechen-nearone commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

haiyuechen-nearone commented Jul 17, 2026 •

edited

Loading