cluster: make cluster/fork work under WASIX (fork IPC + reuseport scheduling) - #115
cluster: make cluster/fork work under WASIX (fork IPC + reuseport scheduling)#115Arshia001 wants to merge 7 commits into
Conversation
3dcb534 to
3b232c0
Compare
|
Rebased onto #144, so the stack is now #143 → #144 → #115. Merge bottom-up; if #115 doesn't land, #144 still greens main on its own. Two commits dropped as already-upstream — both were
main's pin One conflict, in the Makefile. #144 adds Re-validated against current main (its previous green run was 2026-07-15, ~54 commits back). Rebuilt the quickjs-wasix guest and ran the 67 passed / 0 failed, up from 60 on #144 — the extra 7 are the cluster/fork tests this PR un-skips. Both dgram shared-port tests complete in ~1s each, versus the 10-minute timeout on #144's base. Worth recording why this only became urgent recently: the known_issues test was green until 2026-08-05 because cluster IPC was completely broken. libuv's IPC read was stubbed Note the trade-offs this PR already documents in |
child_process.exec() and execSync() do not spawn the command directly -- they run it through `/bin/sh -c`. The root package ships no shell, so every exec() in the V8 WASIX package failed to spawn one. The QuickJS package has declared this dependency since it was created; this one never did. Surfaced by test-http-chunk-problem and test-http-full-response on the v8-wasix lane, which #115 stops skipping. Both exec an external tool and both are written to cope with it being absent, but they never got the chance: with no shell to resolve, the spawn failed before their callback ran. They pass on quickjs-wasix, which has the shell. The uncaught throw those tests hit is a second, separate bug -- proc_spawn reported ENOEXEC rather than ENOENT for an unresolvable command, and Node only converts a known errno set into an 'error' event (wasmerio/wasmer#6877). Either fix alone makes these two tests pass; both are worth having, since this one also makes exec() work at all.
WASIX cannot pass listen handles between processes (no SCM_RIGHTS), which made both of Node cluster's scheduling strategies unusable there: round robin passes every accepted connection to a worker, and the shared-handle mode passes the listen handle itself. SO_REUSEPORT, however, works end to end under WASIX (wasix-libc -> sock_set_opt_flag -> wasmer applies it to the host socket), with genuine kernel-level connection balancing across forked guest processes. Add a third scheduling strategy for WASIX. Project policy keeps the Node lib/ tree byte-identical to upstream, so it lives in native code (src/edge_cluster_wasix.cc) and is installed from EdgeRuntime before the main builtin executes: in WASIX cluster workers (NODE_UNIQUE_ID still present at that point), an embedded script replaces the worker-side cluster._getServer — an exported, replaceable property — so TCP port listens bind their own UV_TCP_REUSEPORT handle instead of asking the primary for one, and report the 'listening' act for the primary's bookkeeping. No primary-side changes are needed at all: the primary never learns a handle key, so its registry and cleanup paths are untouched. UDP, fd, and pipe listens keep the upstream path, as does everything on native targets (compile-time gate). Known limitation inherited from the child-only shape: _getServerData/ _setServerData is not round-tripped through the primary (e.g. TLS session ticket keys stay per-worker). Verified under WASIX: cluster workers balance raw TCP connections 13/11, cluster 'listening' events fire in the primary, and js-firekylin (ThinkJS, cluster-served) passes the framework test. Regressions green: js-firekylin native (round robin unchanged), js-hedgedoc/js-rssmonster WASIX, js-svelte native. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With fork IPC (libuv-wasix plain read) and the cluster reuseport strategy in place, 11 of the 17 tests in WASIX_SKIP_CLUSTER_FORK_TESTS pass and are removed from the skip list, including TCP cluster serving (test-http-server-drop-connections-in-cluster, test-tls-ticket-cluster) and the child_process fork/messaging tests (test-diagnostics-channel-process, the domain and http fork harnesses). Two entries were misfiled and move to their real groups: test-http-client-with-create-connection fails on a unix-socket listen (unix-socket group) and test-crypto-secure-heap fails on OpenSSL secure heap (crypto group). What remains cluster-specific is UDP cluster listens, which still go through shared-handle passing, plus the known_issues negative test whose error-swallowing path (exit 0 on non-success worker messages) engages now that fork IPC delivers messages — the upstream known issue is not observable under WASIX. Full wasix quickjs suite locally: 1681 passed, 0 failed (baseline before: 1671 with the old skip list). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
UDP cluster listens went through shared-handle passing and were the last cluster capability broken under WASIX. The worker-side override now also covers udp4/udp6 port listens via dgram._createSocketHandle with UV_UDP_REUSEPORT; the kernel distributes datagrams between the workers by source hash (flows pin to a worker) instead of shared-socket delivery. Two contract details surfaced by the upstream tests: - _getServer callbacks must stay asynchronous (an IPC round trip upstream); the override now defers cb via process.nextTick, which test-dgram-cluster-close-during-bind's close-during-bind window depends on. - dgram passes the raw bind() arguments through: options.port can be null, undefined, or the bind callback function (socket.bind(cb)); all of those mean an ephemeral-port listen per the bind([port][, address][, callback]) signature. WASIX_SKIP_CLUSTER_FORK_TESTS is now empty: every cluster/fork test in the wasix lanes passes, including the known_issues negative test (back to failing-as-expected: with reuseport the port-0 rebind scenario behaves deterministically again). Verified: cluster UDP echo distributes 11/5 across two workers; full wasix quickjs suite 1686 passed / 0 failed; js-firekylin green on WASIX and native. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The test spawns an external cksum binary in the guest; the first exec cold-downloads and LLVM-compiles wasmer/coreutils, which exceeds the per-test timeout on CI runners with an empty wasmer cache. It passes locally with a warm cache, so this is an environment cost, not a cluster/fork or subprocess capability gap. Filed under the subprocess-shell group. (CI wasix suite was otherwise green: 1685 passed / 1 failed.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same class as test-http-chunk-problem: the test execs ab through a shell,
and on CI runners with an empty wasmer cache the first external exec
cold-downloads and compiles wasmer/bash + wasmer/coreutils, exceeding the
per-test timeout. Locally the test self-skips gracefully ('problem
spawning ab') because the warm-cached shell starts fast enough. These two
are the only external-binary tests among the recent cluster/fork unskips;
the remaining nine are node-child-only and passed CI twice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ipping test-http-chunk-problem (spawns cat) and test-http-full-response (execs ab through a shell) rely on guest binaries that ARE available (wasmer/bash, wasmer/coreutils); their CI timeouts came from the first exec cold- downloading and compiling those packages, not from a capability gap. Give them the scaled timeout (WASIX_SLOW_TESTS, 12x) and keep the coverage rather than skipping. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reuseport strategy let each worker bind its own socket, which is
correct for a fixed port but wrong for two things a worker cannot decide
alone.
A port-0 listen means "any port", and a cluster only stays one server if
every worker lands on the same one. Upstream gets that for free by
binding once in the primary and sharing the descriptor; binding
independently gave each worker a *different* ephemeral port. Only one is
ever reported to the primary, so callers that discover and advertise
their port point at a single worker while the rest listen where nobody
is looking. Measured before this change: 4 workers, 4 different ports.
Server data (obj._getServerData/_setServerData, i.e. TLS session ticket
keys) was likewise never round-tripped, so each worker generated its own
key and a session resumed only on the worker that issued it.
Both are now settled by a broker in the primary. Workers ask it for a
listen key; the first caller for a port-0 key is elected to bind
ephemerally and report back, later callers are parked until it does, and
everyone receives the same port and the same server data. Only numbers
and JSON cross the channel, never a descriptor, so this works where the
upstream shared-handle path cannot.
Fixed-port listens take the round trip too. Upstream always makes one, so
this is not extra cost, and it is what carries the server data.
The broker is installed lazily: edge_process_wrap sees NODE_UNIQUE_ID in
a spawn's env, which only cluster.fork() sets, and that is the first
moment the process is known to be a cluster primary. `cluster` is already
loaded by then so the require is a cache hit -- requiring it eagerly at
startup measured 21 ms on every process. createWorkerProcess() runs
before cluster.emit('fork'), so the broker still sees the worker whose
spawn installed it.
Verified on quickjs-wasix:
- 4 workers, listen(0): one shared port, connections spread across all
four (was: 4 distinct ports, all traffic to one worker).
- test-tls-ticket-cluster passes with 15/16 resumed across 4 distinct
workers. It passed before only because listen(0) funnelled every
connection to a single worker; forced onto a shared port with the
old code it resumed 4/16, one per worker.
- Categories green with CI's skip set: node:dgram 67, node:tls 182,
node:http + node:https + node:diagnostics_channel 537.
Remaining deviation: UDP datagrams are distributed by the kernel's
reuseport source hash, so flows pin to a worker instead of the
shared-socket delivery upstream gets.
3b232c0 to
9f56862
Compare
|
Rebased onto #145; the stack is now #143 → #144 → #145 → #115. Clean rebase, no conflicts. Adds one commit — a port broker in the primary — and inherits two fixes for the failures the previous run exposed. New: agree on the ephemeral port and the server dataThe reuseport strategy let each worker bind its own socket. Right for a fixed port, wrong for two things a worker cannot decide alone:
A broker in the primary now settles both. Workers ask for a listen key; the first caller for a port-0 key is elected to bind ephemerally and report back, later callers park until it does, and everyone gets the same port and the same data. Only numbers and JSON cross the channel, never a descriptor — which is the whole reason the upstream shared-handle path cannot work here. Fixed-port listens take the round trip too. Upstream always makes one, so it is not extra cost, and it is what carries the server data. The broker installs lazily. On test-tls-ticket-clusterIt passed before this change, but vacuously: Inherited from #145The previous run failed on both wasix lanes for reasons unrelated to the broker:
VerificationRebuilt on the rebased base and run with CI's full skip set: Remaining deviation from upstream: UDP datagrams are distributed by the kernel's reuseport source hash, so flows pin to a worker rather than the shared-socket delivery upstream gets. |
1 participant
Split out of #111.
Makes Node's
cluster(andchild_process.forkIPC) functional under WASIX:read()for IPC streams — the IPC read path was stubbed to ENOSYS under__wasi__, killing every fork/cluster message channel. Fork IPC now works end-to-end (online handshake, bidirectionalprocess.send, listen negotiation).uv__sock_reuseportfailed UV_ENOTSUP before consulting the runtime, though the whole path below works (wasix-libc →sock_set_opt_flag→ wasmer applies it to the host socket before bind). (libuv-wasix changes: wasix: enable SO_REUSEPORT in uv__sock_reuseport wasix-org/libuv#9)UV_TCP_REUSEPORT/UV_UDP_REUSEPORThandles and the host kernel balances connections. Implemented insrc/edge_cluster_wasix.cc(project policy keepslib/byte-identical to upstream) as an embedded script that replaces the worker-sidecluster._getServer; no primary-side changes. Compile-time gated to WASIX.WASIX_SKIP_CLUSTER_FORK_TESTSis now empty: every cluster/fork test passes in the wasix lanes. The two tests that exec external guest binaries (test-http-chunk-problem,test-http-full-response) move to the scaled-timeout slow bucket (cold wasmer cache compiles on CI runners), not the skip list.Known limitation:
_getServerData/_setServerDatais not round-tripped through the primary (TLS session ticket keys stay per-worker).Verified under WASIX: raw TCP connections balance 13/11 across two workers, UDP echo distributes 11/5, full wasix quickjs suite 1686 passed / 0 failed.
🤖 Generated with Claude Code