perf(shard): lock-free cross-shard read recovery — idle+batch-gated reply spin + dead-flag cleanup

.add/CONVENTIONS.md

@@ -8,3 +8,5 @@ Lint/format: `cargo fmt --check` · `cargo clippy -- -D warnings` (default + tok
 Errors: command errors are `Frame::Error(Bytes)` with Redis-compatible `-ERR …` text on dispatch paths (no Result); library code uses `thiserror`; `anyhow` only in main.rs/tests; no unwrap/expect in library code
 Architecture: thread-per-core shared-nothing shards; per-shard locks only (parking_lot, never std::sync; never held across .await); no global locks on the write path; no allocation on hot paths (command dispatch, protocol parse, event loop, io drivers); cross-shard communication via flume/SPSC messages only; dual-runtime: all runtime-specific code compiles under both feature sets; Linux-only code behind `#[cfg(target_os = "linux")]` with non-Linux fallback; files ≤1500 lines (split into directory modules)
 Verification: red/green TDD per task; full local CI parity via OrbStack `moon-dev` before push; benchmark numbers from Linux VM only; new parsers get fuzz targets; new atomic state machines get loom models
+Red-suite shape (behavior-preserving perf work — foundation v1): split the failing-first suite into runtime-red (behavioral asserts) + compile-red (an API-shape file that must fail to compile until the new surface exists) + green pins (invariants that must stay green). Keeps a perf refactor honest without a behavior oracle. (lesson: quickwins_red.rs / quickwins_red_api.rs.)
+ADD freeze unit (foundation v1): the §3 CONTRACT freeze requires the literal line `Least-sure flag surfaced at freeze: …` — the template comment alone is not a declaration; the engine refuses `advance` (unflagged_freeze) without it.

.add/PROJECT.md

@@ -6,7 +6,7 @@
 > UI/UX = UDD. When a loop reveals a gap here, come back and update this file —
 > that is the re-entrant arrow from the engine down to the foundation.
 
-slug: moon · stage: production · updated: 2026-06-11
+slug: moon · stage: production · updated: 2026-06-13 · foundation-version: 1
 goal: a Redis-compatible server whose thread-per-core architecture measurably out-scales Redis on multi-core hardware — without sacrificing protocol compatibility or durability semantics
 
 ---
@@ -21,6 +21,7 @@ goal: a Redis-compatible server whose thread-per-core architecture measurably ou
   - No new `unsafe` without approval + `// SAFETY:` comment (UNSAFE_POLICY.md)
   - No alloc in command dispatch / protocol parse / shard event loop / io drivers
   - All runtime-specific code compiles under both `runtime-monoio` and `runtime-tokio`
+  - [foundation v1, 2026-06-13] The shared-nothing write-path lock invariant above is RESTORED and machine-checked as of milestone v1 (`tests/shardslice_shape.rs` forbids `is_initialized()` dual-branches + cross-shard read accessors). Living-doc claims about hot-path locking are NOT self-evident — back them with a CI grep: v1 found CLAUDE.md asserted the client registry was off the hot path, but it was written per batch (finding 1.3 / QW8). The lock-inventory audit grep should become a CI check.
 
 ## Spec / Living Document (SDD) — what we are building, now
 - Active milestone → `.add/milestones/v1-shared-nothing/MILESTONE.md` (see `add.py status`)
@@ -42,3 +43,5 @@ goal: a Redis-compatible server whose thread-per-core architecture measurably ou
 | pre-ADD | SSO CompactKey(23B)/CompactValue(12B) | per-key memory vs naive Arc<String> | lower RSS than Redis per key |
 | 2026-06-11 | adopt ADD; v1 milestone = restore shared-nothing integrity + remove cross-shard latency floor (review priorities 1·2·5) | 2026-06 architecture review: these two themes explain sub-linear multi-shard scaling | pending |
 | 2026-06-11 | FT.SEARCH off-event-loop + WAL group commit deferred to v2 | different themes (event-loop blocking; durability); keep v1 one outcome | recorded in v1 Out list |
+| 2026-06-13 | CLOSE v1-shared-nothing: shared-nothing restored (locks deleted, shape-enforced), 1ms monoio wake floor gone (cross-shard p99 0.071ms), consistency 197/197 @1/4/12; s4 routed parity-or-better (+12% P16 GET) vs v0.3.0 | exit criteria met to the agreed "no-regression + honest measurement" bar | done; default-config cross-shard read regression (−85% c1 GET) RISK-ACCEPTED → follow-up: lock-free cross-shard read acceleration (waiver → next perf milestone) |
+| 2026-06-13 | fold v1 deltas → foundation-version 1 | close the ADD loop so learnings outlive the milestone | DDD: lock-inventory grep → CI (PROJECT §Domain); TDD: red-suite split pattern + ADD: §3 freeze flag-line requirement (CONVENTIONS) |

.add/milestones/v1-shared-nothing/MILESTONE.md

@@ -30,10 +30,13 @@ Out: FT.SEARCH off-event-loop execution (review priority 3 → v2) · WAL group
 
 ## Exit criteria (observable; map each to the task that delivers it)
 - [x] Lock inventory: zero global lock acquisitions on the per-command write path for metrics, replication offsets, WAL sender, client registry batch-update — verified by a recorded audit + tests (TASK.md §6, 2026-06-11)        (← hotpath-lock-quickwins)
-- [ ] `scripts/bench-compare.sh` on moon-dev shows no regression at 1 shard and measured improvement at 4 shards vs the recorded v0.3.0 baseline        (← hotpath-lock-quickwins · spsc-wake-floor · shardslice-migration)
+- [x] `scripts/bench-compare.sh` on moon-dev shows no regression at 1 shard and measured improvement at 4 shards vs the recorded v0.3.0 baseline        (← hotpath-lock-quickwins · spsc-wake-floor · shardslice-migration)
+      MET 2026-06-13 (slice eb5d664 vs v0.3.0 3e376a1, VM idle, REQS=200k, fresh-server-per-rep ×3): s1 parity (SET/GET ≥ main); s4 ROUTED parity-or-better every cell, P16 GET +12%. The default-config cross-shard read regression (c1 GET −85%, P1 GET −16%) is the signed RISK-ACCEPTED tradeoff (shardslice gate, Tin Dang 2026-06-13) — deleting the cross-thread RwLock IS the task; follow-up ticket "lock-free cross-shard read acceleration". (shardslice-migration §6 M7 evidence)
 - [x] Cross-shard single-key SET/GET p99 on monoio drops below 1ms (the old floor) in a recorded latency run — 0.071 ms, 57× under the bar (.add/tasks/spsc-wake-floor/bench-results.txt, 2026-06-11)        (← spsc-wake-floor)
-- [ ] `grep -r "is_initialized()" src/shard/` returns zero dual-path branches; `ShardDatabases` no longer exposes cross-shard read access        (← shardslice-migration)
-- [ ] `scripts/test-consistency.sh` passes 132/132 on 1/4/12 shards and full CI matrix is green after each task        (← all three)
+- [x] `grep -r "is_initialized()" src/shard/` returns zero dual-path branches; `ShardDatabases` no longer exposes cross-shard read access        (← shardslice-migration)
+      MET 2026-06-13: `is_initialized()` confined to slice.rs (definition + single guarded helper + tests); zero production dual-branch gates. Machine-enforced by `tests/shardslice_shape.rs::assert_wrappers_gone`. No public cross-shard read accessor on ShardDatabases (C6 residual contract).
+- [x] `scripts/test-consistency.sh` passes 132/132 on 1/4/12 shards and full CI matrix is green after each task        (← all three)
+      MET 2026-06-13: suite grew to 197 via consistency-dispatch-gaps (the dedicated fix task for the 15 MAIN-baseline failures noted below); 197/197 PASSED @ shards=1/4/12 (VM-local release, shardslice eb5d664). CI matrix green per task: clippy -D warnings ×3 featureset×OS + fmt clean.
       ⚠ Gap discovered 2026-06-11: MAIN itself fails 15 of these (dispatch_read gaps for
       GEO*/EXPIRETIME/PEXPIRETIME/TOUCH + two script bugs + a Phase-152 early-exit), so this
       criterion needs a dedicated fix task; tasks 1–2 verified zero NEW failures vs main

.add/milestones/v1-shared-nothing/RETRO.md

@@ -0,0 +1,42 @@
+════════════════════════════════════════════════════════════════════════
+ v1-shared-nothing · Shared-Nothing Integrity & Cross-Shard Latency
+════════════════════════════════════════════════════════════════════════
+ VERDICT   DONE
+ TASKS     4/4 done           CRITERIA  5/5 met
+ GATES     3 PASS 1 RISK      WAIVERS   1
+
+ goal  a 4-shard Moon serves cross-shard traffic without per-command
+       global locks and without the 1ms monoio reply floor, with
+       measured multi-shard scaling improvement over the v0.3.0 baseline
+
+ TASK                        PHASE     GATE TESTS PROGRESS
+ ───────────────────────────────────────────────────────────────────────
+ hotpath-lock-quickwins      done      PASS 0     ●●●●●●●●
+ spsc-wake-floor             done      PASS 0     ●●●●●●●●
+ consistency-dispatch-gaps   done      PASS 0     ●●●●●●●●
+ shardslice-migration        done      RISK 0     ●●●●●●●●
+ legend  ● reached  ◉ current  ○ pending   spec→…→done
+
+ EXIT CRITERIA  ●●●●●●●●●● 5/5 met
+
+ WAIVERS (1)
+   • shardslice-migration: Tin Dang · follow-up task:
+     cross-shard-read-acceleration (observe) · expires 2026-08-01
+
+ LEARNINGS (3 carried)
+   • TDD · open · For behavior-preserving perf batches, the red suite
+     naturally splits runtime-red + compile-red(api file) + green pins —
+     worth encoding as a pattern in CONVENTIONS (evidence:
+     quickwins_red.rs / quickwins_red_api.rs both did their job).
+   • ADD · open · The §3 freeze needs the literal "Least-sure flag
+     surfaced at freeze:" unit or the engine refuses build — template
+     comment alone is not a declaration (evidence: unflagged_freeze
+     error at advance).
+   • DDD · open · CLAUDE.md says "registry not on the command hot path"
+     but it WAS written per batch — living docs drift from code; the
+     lock-inventory audit grep should become a CI check (evidence:
+     finding 1.3, fixed by QW8).
+
+ DECIDE NEXT  consolidate learnings + archive-milestone
+              v1-shared-nothing
+════════════════════════════════════════════════════════════════════════

.add/milestones/v2-performance/MILESTONE.md

@@ -0,0 +1,37 @@
+# MILESTONE: Multi-Core Throughput Hardening (v1-deferred perf wins)
+
+goal: a 4-shard Moon measurably improves the three throughput bottlenecks v1 deferred — cross-shard read latency, FT.SEARCH event-loop stalls, and appendfsync=always write collapse — without re-introducing cross-thread locks or growing per-key memory
+rationale: intake bucket `new-major` (confirmed Tin Dang 2026-06-13) — v1-shared-nothing is closed, so no active milestone covers these; deliberately BUNDLED (user choice) the three perf themes v1's Out list deferred to "v2" into one coherent outcome: realize the remaining measurable scaling wins. The cross-shard-read theme is scoped to the SAFE, memory-preserving wins (spin-then-park + coalescing + dead-flag cleanup) — the aggressive RCU/ArcSwap snapshot path was weighed and declined to protect moon's low-RSS differentiator (investigation 2026-06-13: snapshot reads = ~2× working-set RAM + O(keys) clone/publish). Discharges the shardslice-migration RISK-ACCEPTED waiver (expires 2026-08-01).
+stage: production · status: active · created: 2026-06-13
+
+> SDD living doc for this milestone. Keep it THIN: breadth, shared decisions, and
+> exit criteria only — per-task detail lives in each `.add/tasks/<slug>/TASK.md`,
+> written just-in-time. Update this doc whenever a task reveals a milestone gap.
+
+## Scope
+In:  cross-shard read acceleration WITHOUT locks or memory growth (adaptive spin-then-park on the SPSC reply to cut the round-trip constant; read coalescing for the multi-client P1 case; DELETE the orphaned `--cross-shard-fast-path` flag + `moon_cross_shard_lock_contention_total` metric + stale docstrings, all dead since the shardslice cutover) · FT.SEARCH off-event-loop execution (a heavy vector/text query must not stall the shard's 1ms tick / co-located commands) · WAL group commit (batch concurrent pending writes into one fsync under appendfsync=always to collapse the ~11× write penalty)
+Out: RCU/ArcSwap snapshot reads & client-side MOVED/CLUSTER-SLOTS routing (declined this milestone — memory + protocol cost; revisit only if the safe cross-shard wins miss target) · I/O-path copy elimination & PBUF_RING (review theme 3 — pullable if appetite remains) · SortedSet/CompactValue data-structure redesign (theme 6) · correctness-flavored debt (AOF rewrite drop window, MVCC delete_lsn, 16-bit LRU clock — tracked separately) · any new command or protocol surface
+
+## Shared decisions & glossary deltas   (living — every task must honor these)
+- Shared-nothing carries forward (GLOSSARY: shard, SPSC mesh): no new cross-thread lock on the command path; cross-shard state travels as ShardMessage only — the v1 invariant, now machine-enforced by `tests/shardslice_shape.rs`. A task may not re-add a lock or a `Send`/`Sync`/`unsafe` slice escape to buy latency.
+- LOW-RSS is a first-class constraint this milestone: no task may grow steady-state per-key memory to buy latency. This is the explicit reason snapshot reads are Out — it protects moon's "lower RSS per key than Redis" differentiator (PROJECT.md Key Decisions).
+- "hot path" rule holds throughout: no new allocation or lock on command dispatch / parse / event loop / io drivers (CLAUDE.md).
+- Behavior parity frozen: RESP responses + `scripts/test-consistency.sh` 197/197 @ shards=1/4/12 byte-identical before/after every task.
+- Bench evidence from the OrbStack `moon-dev` Linux VM only, fresh-server-per-rep ×3, per-runtime (the v1 phantom-regression lesson — shardslice §6 methodology note: long-lived servers showed phantom −20..−40% that vanished under per-rep protocol).
+  - EXCEPTION (discovered by xshard-read-fastpath spike, 2026-06-13): the ABSOLUTE single-client cross-shard read latency is NOT reliably measurable on the OrbStack VM — host→guest vCPU starvation floors the 1-min load ~1.5 with shards parked, inflating cross-shard latency 36µs→580µs while the eventfd wake count stays correct (INFO `spsc_notify_wakes` ratio 1.000). RESOLUTION (2026-06-13): GCloud bare-metal was BLOCKED (no open billing account on the GCloud org), so cross-shard latency wins are gated on a RELATIVE target instead — measured as a best-of-N RPS ratio on the SAME quiesced, core-pinned VM before AND after the change (harness scripts/baseline-xshard-quiesced.sh). Contention only ADDS latency, so best-of-N is the floor and fixed VM/virtiofs overhead cancels in the pre/post ratio; the absolute µs stays untrusted. GCloud bare-metal absolute validation is deferred/optional (revisit only if billing reopens). Throughput-oriented VM numbers (the v1 cells) remain valid as before.
+- Dual-runtime: every change compiles and passes under `runtime-monoio` AND `runtime-tokio,jemalloc`.
+
+## Shared / risky contracts (freeze these first)
+- WAL group-commit durability contract: the fsync-batching boundary + the exactly-once-under-crash invariant — must NOT weaken the shardslice C4 fold / H1-BARRIER machinery it builds on -> owning task `wal-group-commit`
+- FT.SEARCH off-event-loop execution model: how heavy query work yields/offloads WITHOUT cross-thread access to the `!Send` slice and without a `with_shard` borrow held across `.await` -> owning task `ft-search-off-eventloop`
+
+## Tasks (breadth-first decomposition; detail lives in each TASK.md)
+- [x] xshard-read-fastpath     depends-on: none — re-baseline cross-shard read latency per-runtime; recover it the lock-free-safe way via adaptive spin-then-park on the SPSC reply + read coalescing for the P1 multi-client case; delete the dead `--cross-shard-fast-path` flag/metric/docstrings. Target: at least HALVE the c1 GET regression with zero memory growth.  **DONE 2026-06-14 (gate PASS):** C2 idle-gated + batch-depth-gated reply spin recovers c1-GET **+22.9% same-run** (−19.4% vs 3e376a1 ≤ ~20% contract line); verify caught + fixed a −27.5% pipelined regression (batch gate, commit 7048e8a). C3 cross-connection coalescing DEFERRED → follow-up task `xshard-read-coalescing` (human-approved v2 change request).
+- [ ] ft-search-off-eventloop  depends-on: none — keep a pathological FT.SEARCH (large K / deep HNSW) from stalling concurrent simple commands on the same shard; cooperative-yield or snapshot-handoff execution that respects the !Send slice.
+- [ ] wal-group-commit         depends-on: none — batch concurrent pending writes into one fsync under appendfsync=always; close a meaningful fraction of the ~11× throughput penalty with zero data-loss regression.
+
+## Exit criteria (observable; map each to the task that delivers it)
+- [x] Cross-shard c1 GET regression at least HALVED vs v0.3.0 (tag 3e376a1, pre-shared-nothing), measured per-runtime as a best-of-N RPS ratio on the SAME quiesced, core-pinned moon-dev instrument before/after (relative anchor; absolute µs untrusted). M0 anchor ESTABLISHED 2026-06-14 (monoio, clean VM): c1-GET 37580→22252 = −40.8%; target = recover ≥ half the 15.3k-RPS gap → c1-GET ≥ ~30k. `grep` confirms the `--cross-shard-fast-path` flag + `moon_cross_shard_lock_contention_total` metric are GONE; consistency 197/197 @1/4/12 unchanged; RSS not regressed; s4-c100-GET guard within noise of ~202k        (← xshard-read-fastpath)  **MET on the relative form 2026-06-14:** same-run +18–23% c1-GET recovery → regression −19.4% ≤ ~20%; flag+metric grep-clean; 197/197 unchanged; RSS flat; c100 +10%. ⚠ The absolute "≥30k" SUB-line was RETIRED, not met (~25k): the whole-VM baseline itself drifted 37580→31437→20243 across three clean runs, so absolute RPS is not a stable instrument here — the contract's sanctioned metric is the relative ratio (§7 OBSERVE delta records the retirement). The structural residual (the second, origin→owner cross-thread wake) needs the §1-rejected RCU path; this is the single-connection floor, documented, not a waiver.        (← xshard-read-fastpath)
+- [ ] During a heavy FT.SEARCH on a shard, p99 of a simple command (PING/GET) on that same shard stays under a recorded bound; FT.SEARCH recall/correctness unchanged vs current        (← ft-search-off-eventloop)
+- [ ] appendfsync=always write throughput improves measurably at pipeline depth >1 with N concurrent writers (recorded before/after); crash-matrix green + exactly-once preserved (no data-loss regression)        (← wal-group-commit)
+- [ ] Cross-cutting per task: dual-runtime green, `clippy -D warnings` ×2 featuresets + `fmt` clean, zero new `unsafe`, zero new cross-thread lock        (← all three)