test(persistence): G1 cross-plane kill-9 crash matrix — 37 cells, 4 RED root-cause groups (kernel M3 stage 1)

CHANGELOG.md

@@ -6,6 +6,72 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added — cross-plane kill-9 crash-matrix suite (kernel M3 stage 1 / G1)
+
+New `tests/crash_matrix_cross_plane.rs` + `tests/crash_matrix_cross_plane/`
+harness: 40 kill-9 crash-recovery cells spanning every persistence plane
+(KV/AOF, KV disk-offload, graph/vector/WS/MQ WAL v3, cross-store MULTI/EXEC,
+mixed-plane concurrent workloads, and a checkpoint-`Finalize`-window kill)
+across the `{shards=1, shards=4} x {appendonly, disk-offload}` matrix. This
+is a RED-first tripwire suite — 29 cells run GREEN by default; 11 known-RED
+cells are gated behind a runtime guard (`harness::red_guard`, checked via
+`MOON_CRASH_MATRIX_RED=1`) rather than `#[ignore = "RED: ..."]`, because the
+suite's own `--ignored` invocation convention (every cell needs a release
+binary) makes an ignore-reason string alone non-functional as a gate. A
+`MOON_CRASH_MATRIX_ITERS` env var (default 1) enables ad-hoc soak runs for
+probabilistic findings.
+
+A second review round caught a vacuous-precondition bug and a gating
+granularity bug before this suite could be trusted as a tripwire:
+`kv_spilled_isolated`'s filler (3000×512B against a 4 MiB cap) never
+actually forced a cold-tier spill at either shard count, silently
+degenerating to the same coverage as `kv_isolated` — fixed by mirroring
+`crash_recovery_cold_del_resurrection.rs`'s proven filler ratios
+(16,000×600B against 8 MiB) plus a hard `heap-*.mpf` precondition assert so
+a future load-parameter regression fails loud instead of vacuously passing.
+`txn_isolated` was split into `txn_isolated_committed` (still RED, task #52
+below) and `txn_isolated_atomicity` (a transaction queued but killed before
+`EXEC` must apply nothing) — the two halves have unrelated root causes and
+sharing one `red_guard` hid a working, GREEN regression tripwire on
+`prod_s1`/`prod_s4` by default; the atomicity half now runs ungated on
+those two configs (still RED on `legacy_yes_s1`, same pre-existing
+legacy-graph gap as everything else there).
+
+Two real, unresolved findings surfaced and are intentionally left RED (not
+fixed — that is out of scope for this stage) for the kernel M3 backlog:
+
+- **Cross-store TXN graph leg not durable (task #52):** a committed
+  `MULTI`/`EXEC` mixing `SET` + `GRAPH.ADDNODE` survives kill-9 on the KV
+  leg (PR #247's `persist_txn_aof`) but the graph leg is lost. Deterministic
+  repro (3x consecutive at shards=1 and shards=4): kill immediately after
+  the post-`EXEC` WAL-v3 sync wait, with `--checkpoint-timeout 3600` so no
+  periodic checkpoint can incidentally rescue the graph leg. See
+  `scenarios::txn_isolated_committed`'s doc comment for the minimal repro.
+- **Checkpoint-`Finalize` window can total-loss the graph plane (new):**
+  some kill offsets in the 0-150ms window after `BGSAVE` lose the *entire*
+  already-wal-v3-synced graph batch (not just an unsynced tail) while
+  KV/vector/WS/MQ all survive in the same run. Probabilistic
+  (~1-in-7 to 1-in-12 per iteration) — a single default run can report
+  false-green even with `MOON_CRASH_MATRIX_RED=1`; reproduce reliably with
+  `MOON_CRASH_MATRIX_RED=1 MOON_CRASH_MATRIX_ITERS=20`. Strong P0 candidate
+  for kernel M3 stage 2 (K2). See `scenarios::mixed_mid_checkpoint`'s doc
+  comment.
+
+Also confirmed GREEN (not RED, contrary to the brief's grouping at the
+design-doc level): `WS DROP` durability under kill-9 — `WorkspaceDrop` WAL
+records replay correctly in order, unlike the sibling MQ generic-`DEL`
+resurrection gap (still RED, same bug class as the already-fixed KV/vector
+cold-plane resurrection, PR #257, not yet applied to MQ).
+
+`tests/common/mod.rs` gains `find_moon_binary()`/`sigkill()`/
+`wait_for_port_down()` shared helpers this suite depends on (the latter now
+panics on loop exhaustion instead of silently returning — a tripwire
+codebase should not have silent-pass verification helpers). Shared helpers
+added; migration of the 5 existing `crash_recovery_*.rs` suites' own local
+copies to the shared versions is deferred, not done in this stage.
+
+Test-only; no production code changed in this stage.
+
 ### Added — memory + tier accounting spine (kernel M2 stage 2 / K4)
 
 `resident_bytes()` is now implemented by every storage plane, and the

tests/common/mod.rs

@@ -25,6 +25,7 @@
 
 use std::collections::HashSet;
 use std::net::TcpStream;
+use std::path::PathBuf;
 use std::process::Child;
 use std::sync::{LazyLock, Mutex};
 use std::time::{Duration, Instant};
@@ -108,3 +109,104 @@ pub fn spawn_listening(mut spawn: impl FnMut(u16) -> Child) -> (Child, u16) {
          not a port race; read the server stderr log in the test's --dir"
     );
 }
+
+// ---------------------------------------------------------------------------
+// Kernel M3 / G1 (task #18 follow-up): consolidated binary-resolution + kill
+// helpers. Every crash suite used to carry its own copy of these three
+// functions (grepped: `crash_recovery_graph_durability.rs`,
+// `crash_recovery_wal_recycle_legacy.rs`, `crash_recovery_vector_durability.rs`
+// duplicated all three; `crash_recovery_mq_effects.rs` /
+// `crash_recovery_temporal_mq.rs` already used `spawn_listening` above but
+// still kept a local `find_moon_binary`). Consolidated here as the mechanical,
+// behavior-preserving union of every variant found (MOON_BIN override with a
+// non-empty guard, then the compile-time `CARGO_BIN_EXE_moon` path Cargo sets
+// for this test binary, then `target/{release,debug}/moon` as a last resort
+// for ad-hoc `cargo test --test <name>` invocations outside the normal
+// harness). `tests/aof_multidb_kill9.rs`'s `moon_bin()` returns a bare
+// `./target/release/moon` default with NO binary-existence check and no
+// `CARGO_BIN_EXE_moon` fallback — a real behavioral difference, so it is
+// deliberately left alone here rather than force-migrated (see the kernel M3
+// brief, Stage 1, decision #5: "if any suite needs behavioral adaptation,
+// skip it there and note it for a follow-up task instead").
+// ---------------------------------------------------------------------------
+
+/// Resolve the `moon` server binary for crash/integration suites.
+///
+/// Precedence: `MOON_BIN` env var (only if non-empty and the path exists) →
+/// `CARGO_BIN_EXE_moon` (the exact binary Cargo built for THIS test run —
+/// right profile, right `CARGO_TARGET_DIR`, right `.exe` suffix on Windows)
+/// → `target/release/moon` → `target/debug/moon`. Panics with a actionable
+/// message if none resolve.
+pub fn find_moon_binary() -> PathBuf {
+    if let Ok(bin) = std::env::var("MOON_BIN")
+        && !bin.trim().is_empty()
+    {
+        let p = PathBuf::from(&bin);
+        if p.exists() {
+            return p;
+        }
+    }
+    let cargo_bin = PathBuf::from(env!("CARGO_BIN_EXE_moon"));
+    if cargo_bin.exists() {
+        return cargo_bin;
+    }
+    let manifest = env!("CARGO_MANIFEST_DIR");
+    let release = PathBuf::from(format!("{manifest}/target/release/moon"));
+    if release.exists() {
+        return release;
+    }
+    let debug = PathBuf::from(format!("{manifest}/target/debug/moon"));
+    if debug.exists() {
+        return debug;
+    }
+    panic!(
+        "No moon binary found. Build with `cargo build --release` or set \
+         MOON_BIN=/path/to/moon."
+    );
+}
+
+/// SIGKILL a spawned child and reap it (never SIGTERM — SIGTERM +
+/// SO_REUSEPORT is a documented hang, see CLAUDE.md / the harness-speed
+/// gotcha ledger). `Child::kill()` is documented to send SIGKILL on Unix,
+/// so no raw `libc::kill` (and no `unsafe`, no cfg split) is needed.
+pub fn sigkill(child: &mut Child) {
+    let _ = child.kill();
+    let _ = child.wait();
+}
+
+/// Wait until nothing is accepting on `port` — required before a same-port
+/// restart, or the new listener can race the dying process's socket
+/// teardown. moon binds `SO_REUSEPORT` per shard, so a plain bind-based
+/// check is useless here; two consecutive refused connects is the signal.
+pub fn wait_for_port_down(port: u16) {
+    let addr = format!("127.0.0.1:{port}");
+    let mut consecutive_refused = 0;
+    for _ in 0..120 {
+        match TcpStream::connect_timeout(&addr.parse().expect("addr"), Duration::from_millis(100)) {
+            Ok(_) => {
+                consecutive_refused = 0;
+                std::thread::sleep(Duration::from_millis(100));
+            }
+            Err(_) => {
+                consecutive_refused += 1;
+                if consecutive_refused >= 2 {
+                    return;
+                }
+                std::thread::sleep(Duration::from_millis(50));
+            }
+        }
+    }
+    // No silent-pass verification helpers in a tripwire codebase (review
+    // round 3, P2): a caller that proceeds to bind the SAME port after this
+    // returns without the port actually being down races the dying
+    // process's socket teardown, which can manifest as a flaky bind failure
+    // or — worse — a same-port respawn silently talking to the wrong
+    // (still-dying) process. Loop exhaustion after 120 iterations (~18s
+    // worst case) is not a benign timeout here; it means the port never
+    // went down and every caller's assumption is false.
+    panic!(
+        "wait_for_port_down: port {port} never stopped accepting connections \
+         after 120 poll iterations (~18s) — the old process may still be \
+         alive, or SO_REUSEPORT is masking a listener that never exited"
+    );
+}