From 4938ecd29f35ea1a94e3c87e0c154f5a7b2d5e88 Mon Sep 17 00:00:00 2001 From: Tin Dang Date: Wed, 15 Jul 2026 14:31:48 +0700 Subject: [PATCH 1/5] release(v0.7.0): Replication GA for multi-shard masters MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Real Redis-compatible async replication with WAIT/ACK semantics: a multi-shard master (--shards N) streams a merged, exactly-once command feed to a single-shard streaming replica across all six data planes — KV, vector/text index, graph, WS, MQ, and temporal. Every write plane is crash-durable and survives kill -9 on either side. Folds in the full v0.6.1 hardening scope (WAL v3 storage-kernel M1–M4) and a supply-chain CI gate (cargo audit + cargo deny). Gate: 24h replication kill-9 soak (REPL-SOAK-01) PASSED, zero acked-write loss — SOAK-PASS duration=86400s cycles=114 acked=82044 inflight=7 master_kills=57 replica_kills=57 (2026-07-15, RC e2d87893, run dir moon-soak/runs/20260714-141946). Version bump 0.6.0 -> 0.7.0 (Cargo.toml + Cargo.lock), CHANGELOG [0.7.0] roll-up with the #71a replica-TTL disclosure, RELEASES.md entry, PRODUCTION-CONTRACT REPL-SOAK-01 + SUPPLY-01 ticked, shardslice-migration waiver retired (cross-shard read now tracked as open GA gap XSHARD-READ-01, not time-boxed). Known limitation: streaming replica is single-shard only (--shards 1); multi-shard replicas roadmapped for v0.8/v0.9. Closes #62, #65. Discloses #71a (replica TTL semantics; absolute-rewrite in v0.7.1 / #71b). author: Tin Dang --- CHANGELOG.md | 31 +++++++++++++++++++++++++++++++ Cargo.lock | 2 +- Cargo.toml | 2 +- RELEASES.md | 5 +++++ docs/PRODUCTION-CONTRACT.md | 4 ++-- 5 files changed, 40 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3df02db8..e07c4fc0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.7.0] — 2026-07-15 + +**Replication GA for multi-shard masters.** Moon now supports real Redis-compatible +asynchronous replication with `WAIT`/`ACK` acknowledgement semantics: a multi-shard +master (`--shards N`) streams a merged, exactly-once command feed to a single-shard +streaming replica across all data planes — KV, vector/text index, graph, workspace (WS), +message-queue (MQ), and temporal. Every write plane is crash-durable and survives +`kill -9` on either side, validated by a 24h continuous-load kill-9 soak (alternating +master/replica restarts every 12 min) that asserts zero loss of any `WAIT`-acknowledged +write. Also folds in the full v0.6.1 hardening scope (WAL v3 storage-kernel M1–M4: +cross-plane crash matrix, unified per-shard WAL-recycle floor, atomic durable writes, +FTS term-dict durability) and a supply-chain CI gate (`cargo audit` + `cargo deny`). + +Soak evidence (release gate REPL-SOAK-01): `SOAK-PASS duration=86400s cycles=114 +acked=82044 inflight=7 master_kills=57 replica_kills=57` — 82,044 WAIT-acked writes +preserved across 114 alternating kill-9 cycles, zero acked-write loss (2026-07-15, +run dir `moon-soak/runs/20260714-141946`, RC `e2d87893`). + +Known limitation: the streaming replica is single-shard only (`--shards 1`) — the +multi-shard work in this release is master-side (merged N-shard PSYNC feed). Multi-shard +replicas are roadmapped for v0.8/v0.9. + +Replica TTL semantics (disclosure): relative-expire commands (`EXPIRE`, `SETEX`, `PEXPIRE`, +`GETEX` with a relative TTL) currently replicate verbatim rather than being rewritten to +absolute `PEXPIREAT` on the master, and replicas run their own active-expiry cycle +regardless of role. In practice keys expire correctly on both sides under normal clock +sync, but a master/replica clock skew can shift a relative-TTL key's expiry moment between +the two by up to that skew. Absolute-expiry rewrite + role-gated passive expiry land in +v0.7.1 (task #71b). Applications needing exact cross-node expiry parity should set +absolute deadlines with `PEXPIREAT` until then. + ### Fixed — `segment_plane_scan` missed v0.6.0's nested-Command plane framing, risking WS/MQ/temporal data loss on upgrade (task #69) `WalWriterV3::recycle_aggressive`/`recycle_segments_before` gate deletion of diff --git a/Cargo.lock b/Cargo.lock index 76407ae0..09aa2992 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1820,7 +1820,7 @@ dependencies = [ [[package]] name = "moon" -version = "0.6.0" +version = "0.7.0" dependencies = [ "anyhow", "arc-swap", diff --git a/Cargo.toml b/Cargo.toml index 83fc2872..5ff4115d 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "moon" -version = "0.6.0" +version = "0.7.0" edition = "2024" rust-version = "1.94" description = "A high-performance Redis-compatible server written in Rust" diff --git a/RELEASES.md b/RELEASES.md index 6c29af59..c283d328 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -1,5 +1,10 @@ # Releases +## v0.7.0 — 2026-07-15 +milestones: v0.7.0 "Replication GA for multi-shard masters" (soak-gated tag; v0.6.1 hardening folded in) +waivers: none — the v0.6.0 `shardslice-migration` waiver (was: expires 2026-08-01) is **retired**; lock-free cross-shard read work is now tracked as an open GA gap (`XSHARD-READ-01`, ROADMAP R4), not a time-boxed waiver. +evidence: Real async replication with WAIT/ACK across all six data planes (KV, vector/text, graph, WS, MQ, temporal); multi-shard master → single-shard streaming replica; every plane crash-durable under kill -9. Plus full v0.6.1 hardening (WAL v3 storage-kernel M1–M4) and a supply-chain CI gate (cargo audit + cargo deny). Release gate — 24h replication kill-9 soak (REPL-SOAK-01), zero acked-write loss: `SOAK-PASS duration=86400s cycles=114 acked=82044 inflight=7 master_kills=57 replica_kills=57` (2026-07-15; run dir moon-dev:~/moon-soak/runs/20260714-141946; RC main e2d87893; 114 alternating master/replica kill -9 cycles, 82,044 WAIT-acked writes preserved). Known limitation: streaming replica is single-shard only. Replica relative-TTL semantics: verbatim replication (no PEXPIREAT rewrite) + role-agnostic active expiry — absolute-rewrite lands in v0.7.1 (task #71b). + ## v0.6.0 — 2026-07-08 milestones: v0.6.0-release (PR #249, six workstreams) waivers: shardslice-migration (expires 2026-08-01; retirement planned with v0.7 L4 cross-shard-read work) diff --git a/docs/PRODUCTION-CONTRACT.md b/docs/PRODUCTION-CONTRACT.md index fbbd45e3..c86096d9 100644 --- a/docs/PRODUCTION-CONTRACT.md +++ b/docs/PRODUCTION-CONTRACT.md @@ -89,7 +89,7 @@ per the CI gate · Blocking `—` = tracked but never blocks a tag (see Out of S | ✅ | CHANGELOG-01 | CHANGELOG.md CI gate on every PR (skip-changelog label escape hatch) | `ci.yml` step `CHANGELOG check` | GA | | ✅ | REL-LEDGER-01 | Release tag requires a matching `RELEASES.md` entry | `release.yml` step `Require RELEASES.md entry for this tag` | GA | | ✅ | CONTRACT-01 | This document is a checked ledger with a CI-wired gate | `scripts/check-production-contract.sh`, `release.yml` | GA | -| ⬜ | SUPPLY-01 | `cargo audit` + `cargo deny` CI-blocking | `deny.toml` exists but no CI job runs the tools. Dedicated `supply-chain.yml` workflow in flight (task #63, 2026-07-14) — tick when merged and green. | GA | +| ✅ | SUPPLY-01 | `cargo audit` + `cargo deny` CI-blocking | Shipped (task #63, PR #326): `.github/workflows/supply-chain.yml` runs `cargo audit` + `cargo deny` on push/PR with `deny.toml` policy. Folded into v0.7.0. | GA | ### B. Correctness Hardening @@ -126,7 +126,7 @@ per the CI gate · Blocking `—` = tracked but never blocks a tag (see Out of S | ✅ | REPL-MULTISHARD-01 | Multi-shard master replication (a `--shards N>1` master can be replicated at all) | R2 (task #20): `ShardMessage::PrepareReplicaSync` per-shard atomic snapshot legs + merged Redis-format RDB + per-record SELECT framing on the merged wire. monoio only; replicas run `--shards 1`; partial resync degrades to full at N>1. `tests/replication_multishard.rs` (2/4/8-shard resync, interleaved multi-db parity, graph, partial→full). | GA | | ✅ | WAIT-01 | `WAIT` reflects real replica ACK state | R1 (task #19, PR #282): replica 1s `REPLCONF ACK` ticker on the split PSYNC socket; master `ack_read_loop` + `drain_ack_offsets` record into `ReplicaInfo.ack_offsets`; connection-layer `try_handle_wait` blocks until ACK ≥ target or timeout. `wait_returns_acked_replica_count` e2e; exact on multi-shard masters too (summed snapshot offset). | GA | | ✅ | REPL-PLANES-01 | Every write plane replicates, not just KV: eviction/expiry DELs, Lua effects, graph, vector/text index defs+contents, WS.*, MQ.*, TEMPORAL.* | Wave A (PR #285): eviction/expiry DELs + Lua effects to both planes (EVAL was previously durable in neither). Wave B (PR #294 + task #34): WS/MQ deterministic records + replica apply + PSYNC registry blob. Graph plane (task #25): live GRAPH.* streaming + snapshot backfill. Suites: `tests/replication_planes.rs`, `replication_graph.rs`, `replication_mq.rs`, `replication_readonly_ws_mq.rs`. Unified poison-record policy for replica apply (task #48). | GA | -| ⬜ | REPL-SOAK-01 | 24h replication soak: kill -9 either side under WAIT-confirmed load, zero acked-write loss | Harness in flight (task #61: `scripts/soak-replication-24h.sh`); the run (task #62) gates the v0.7.0 tag. | GA | +| ✅ | REPL-SOAK-01 | 24h replication soak: kill -9 either side under WAIT-confirmed load, zero acked-write loss | **PASSED 2026-07-15** — `SOAK-PASS duration=86400s cycles=114 acked=82044 inflight=7 master_kills=57 replica_kills=57`; 82,044 WAIT-acked writes preserved across 114 alternating kill-9 cycles, zero acked-loss. Run dir `moon-soak/runs/20260714-141946`, RC `e2d87893`. Gates the v0.7.0 tag (task #65). | GA | | ⬜ | KEYSPACE-NOTIF-01 | `notify-keyspace-events` keyspace notifications | No implementation found in `src/`. ROADMAP v0.7.0 workstream R5 — deferred to v0.7.1 (one-headline rule). | GA | | ⬜ | MONITOR-01 | `MONITOR` command | No implementation found in `src/command/`. ROADMAP v0.7.0 workstream R5. | GA | | ⬜ | XSHARD-READ-01 | Lock-free cross-shard read path (retire the shardslice waiver) | Waiver **expires 2026-08-01** per `RELEASES.md` v0.6.0 entry and ROADMAP §5; L4 redesign (`tmp/MULTISHARD-REDESIGN.md`) unstarted. ROADMAP v0.7.0 workstream R4. | GA | From 4f2ea610d81b9d5d264a2643aac4d10cb5c1e0ea Mon Sep 17 00:00:00 2001 From: Tin Dang Date: Wed, 15 Jul 2026 14:42:47 +0700 Subject: [PATCH 2/5] docs(replication): v0.7 replication config + tuning guide; fix stale clustering walkthrough MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A "Replication GA" release shipped without replication coverage in the config/tuning reference docs, and the clustering guide's replica walkthrough had rotted: - guides/clustering.md showed the PRE-GA topology (single-shard leader, multi-shard replica) — the reverse of the shipped v0.7 shape (multi-shard master, --shards 1 replica) — directly contradicting its own info box. Corrected the walkthrough. - Its "WS.*/MQ.* not replicated yet" note was stale: Wave B (task #34) replicates all six planes (KV, vector/text, graph, WS, MQ, temporal). Corrected. - Added: WAIT-durability ladder, replica promotion (REPLICAOF NO ONE), INFO replication monitoring (offset/lag, master_link_status), replica-TTL caveat (#71a). New "Replication" section in docs/configuration.md (topology + durability flags + REPLICAOF-is-runtime-only) and "Replication durability" section in guides/tuning.md (RPO/latency ladder, read-scaling with more replicas, lag alerting). Cross-linked. All facts verified against the code (handshake.rs INFO fields, apply.rs Wave B WS/MQ arms, metadata.rs command table). Docs-only; ships in the v0.7.0 tag. Refs #65. author: Tin Dang --- CHANGELOG.md | 11 ++++++ docs/configuration.md | 19 +++++++++++ docs/guides/clustering.md | 70 ++++++++++++++++++++++++++++++++------- docs/guides/tuning.md | 28 ++++++++++++++++ 4 files changed, 116 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e07c4fc0..a518a21f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,17 @@ the two by up to that skew. Absolute-expiry rewrite + role-gated passive expiry v0.7.1 (task #71b). Applications needing exact cross-node expiry parity should set absolute deadlines with `PEXPIREAT` until then. +### Documentation — replication configuration & tuning + +Documented the v0.7 replication surface: corrected the stale `guides/clustering.md` +replica walkthrough (it still showed the pre-GA topology — single-shard leader, +multi-shard replica — the reverse of the shipped shape) and its outdated +"WS/MQ not replicated yet" note (all six planes replicate as of Wave B). Added the +`WAIT`-durability ladder, replica promotion (`REPLICAOF NO ONE`), `INFO replication` +monitoring, and the replica-TTL caveat. New **Replication** section in +`configuration.md` and a **Replication durability** section in `guides/tuning.md` +(RPO/latency trade-offs, read-scaling, lag alerting). + ### Fixed — `segment_plane_scan` missed v0.6.0's nested-Command plane framing, risking WS/MQ/temporal data loss on upgrade (task #69) `WalWriterV3::recycle_aggressive`/`recycle_segments_before` gate deletion of diff --git a/docs/configuration.md b/docs/configuration.md index 43097785..19557478 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -58,6 +58,25 @@ All options are available as command-line flags. Run `moon --help` for the full | `--cluster-enabled` | `false` | Enable cluster mode | | `--cluster-node-timeout` | `15000` | Node timeout in ms | +## Replication + +Replication (v0.7 GA) is initiated at runtime with the `REPLICAOF ` +command — there is **no startup flag**. The relevant startup flags shape the +topology and durability of the pair: + +| Flag | On | Effect for replication | +|------|-----|------------------------| +| `--shards N` | master | Multi-core writer; the master merges all shards into one exactly-once replication feed | +| `--shards 1` | replica | **Required** — replicas are single-shard; scale reads by adding replicas | +| `--appendonly yes` | both | Persist the AOF so a restarted node recovers before re-syncing | +| `--appendfsync always` | master | RPO 0 on the master; pair with `WAIT N` for zero-RPO cross-node durability | + +Replicas are read-only (`slave_read_only:1`; writes return `-READONLY`). `WAIT +numreplicas timeout` reports real replica ACKs. Full setup, `WAIT` durability, +promotion (`REPLICAOF NO ONE`), and the replica TTL caveat: see the +**[clustering & replication guide](guides/clustering.md#replication)** and the +[tuning guide](guides/tuning.md#replication-durability). + ## ACL | Flag | Default | Description | diff --git a/docs/guides/clustering.md b/docs/guides/clustering.md index 757588ed..d6088ba6 100644 --- a/docs/guides/clustering.md +++ b/docs/guides/clustering.md @@ -30,42 +30,88 @@ Moon implements PSYNC2-compatible replication with per-shard WAL streaming and p - `WAIT N timeout` reflects real replica ACKs (1s `REPLCONF ACK` cadence). - `master_link_status` in `INFO replication` reflects the handshake state — use it to detect a failed REPLICAOF. - `CLIENT LIST TYPE replica` has no predicate yet; returns all clients. - - WS.\*/MQ.\* planes are **not replicated yet** (the master logs one warning - when a replica is attached); vector/text/graph planes replicate fully. + - **All six data planes replicate** (v0.7 GA): KV, vector/text index, graph, + workspace (WS.\*), message-queue (MQ.\*), and temporal. A full resync ships + each plane's snapshot; the live stream carries every plane's effect records. ### Set up a replica -### Start the leader (must be --shards 1 in v0.1.x) +**Topology (v0.7):** the **master** runs any `--shards N` (multi-core writer); each +**replica** runs `--shards 1`. Scale reads by adding replicas, not replica shards. +Replication is initiated at runtime with the `REPLICAOF` command — there is no +startup flag; operators script it after the replica is up (e.g. via an init hook). + +#### 1. Start the master (any shard count) ```bash -./target/release/moon --port 6379 --shards 1 +./target/release/moon --port 6379 --shards 4 --appendonly yes --appendfsync always ``` -### Start the replica (any shard count) +#### 2. Start the replica (must be --shards 1) ```bash -./target/release/moon --port 6380 --shards 4 +./target/release/moon --port 6380 --shards 1 --appendonly yes ``` -### Connect the replica to the leader +#### 3. Attach the replica to the master ```bash redis-cli -p 6380 REPLICAOF 127.0.0.1 6379 ``` -### Verify link status +The replica performs a full resync (one merged Redis-format RDB across all planes), +then applies the live stream. Replicas are **read-only**: writes return +`-READONLY` (`INFO replication` reports `slave_read_only:1`). + +#### 4. Verify the link ```bash redis-cli -p 6380 INFO replication | grep master_link_status -# Expect: master_link_status:up +# Expect: master_link_status:up (anything else = handshake not complete) +``` + +### Acknowledged writes with WAIT + +`WAIT numreplicas timeout` blocks until at least `numreplicas` replicas have ACKed +every write issued on the connection (replicas send `REPLCONF ACK` on a ~1 s +cadence). Combine it with `appendfsync always` on both sides for a **zero-RPO, +cross-node durable write**: + +```bash +redis-cli -p 6379 SET k v +redis-cli -p 6379 WAIT 1 1000 # returns the number of replicas that ACKed within 1000 ms ``` +On the master, `INFO replication` lists each replica's `offset` and `lag`; on the +replica it reports `slave_repl_offset`. See the [tuning guide](tuning.md#replication-durability) +for the durability/latency trade-offs. + +### Promote a replica (failover) + +```bash +redis-cli -p 6380 REPLICAOF NO ONE # replica stops applying and becomes a writable master +``` + +Repoint surviving replicas at the new master with `REPLICAOF `. +There is no automatic replication failover outside cluster mode (see below). + ### Replication features +- **All six data planes** — KV, vector/text index, graph, WS, MQ, temporal (v0.7 GA) - **PSYNC2 protocol** — compatible with Redis replication clients -- **Per-shard WAL streaming** — each shard streams its own WAL independently (once connected) -- **Partial resync** — reconnecting replicas resume from where they left off via the replication backlog -- **Lazy backlog** — the replication backlog is only allocated when the first replica handshake begins (REPLCONF), saving ~12 MB baseline memory +- **Per-shard WAL streaming** — each master shard streams its own WAL; a multi-shard master merges them into one exactly-once feed +- **Partial resync** — single-shard masters resume reconnecting replicas from the backlog window; multi-shard masters answer every reconnect with a full resync +- **Lazy backlog** — allocated only when the first replica handshake begins (REPLCONF), saving ~12 MB baseline memory +- **Validated** — 24 h continuous-load kill-9 soak (alternating master/replica restarts), zero loss of any WAIT-acknowledged write + +!!! warning "Replica TTL semantics (v0.7)" + Relative-expire commands (`EXPIRE`, `SETEX`, `PEXPIRE`, `GETEX` with a relative + TTL) replicate verbatim rather than being rewritten to absolute `PEXPIREAT`, and + replicas run their own active-expiry cycle. Under normal clock sync keys expire + correctly on both sides, but master/replica clock skew can shift a relative-TTL + key's expiry moment by up to that skew. For exact cross-node expiry parity, set + absolute deadlines with `PEXPIREAT`. Absolute-rewrite + role-gated expiry land in + v0.7.1. ## Cluster mode diff --git a/docs/guides/tuning.md b/docs/guides/tuning.md index 4f297d06..7a7bed20 100644 --- a/docs/guides/tuning.md +++ b/docs/guides/tuning.md @@ -157,6 +157,34 @@ data loss on a host crash; then use `always` and size expectations to your disk' rate (or provision faster storage). Both preserve their guarantee under SIGKILL — validated by the crash-recovery matrix (100% of acked writes recovered). +## Replication durability + +AOF durability protects a **single node** against a crash. Replication (v0.7 GA) +adds a **second node** so a write survives losing the master's disk entirely. The +two combine on a latency/RPO ladder — pick the rung your workload needs: + +| Goal | Master | Client | RPO | Cost | +|------|--------|--------|-----|------| +| Fast, replica for read-scaling/DR | `--appendfsync everysec` | fire-and-forget | ≤1 s on master crash; replica lag on failover | lowest latency | +| Durable on the master | `--appendfsync always` | — | 0 on master crash (disk-bound) | fsync per write | +| **Zero-RPO across nodes** | `--appendfsync always` | `WAIT 1 ` after the write | 0 even if the master's disk is lost | fsync + one replica round-trip | + +`WAIT numreplicas timeout` blocks until `numreplicas` replicas have ACKed the +write (replicas ACK on a ~1 s cadence, so a `WAIT` timeout below ~1 s may return +before an idle replica reports in — size the timeout accordingly, or keep the +write stream busy). It reports the count that ACKed; it never rolls the write back. + +**Read scaling:** replicas are `--shards 1` and read-only. Add *more replicas* for +read throughput and DR breadth — you cannot add shards to a replica. Route reads +to replicas and writes to the master at the client/proxy layer. + +**Monitoring:** on the master, `INFO replication` lists each replica's `offset` +and `lag` (bytes behind); alert on sustained lag growth. On a replica, +`master_link_status:up` confirms a live stream — treat `down` as an outage even if +TCP is connected (it means the PSYNC handshake has not completed). + +Full setup and failover: [clustering & replication guide](clustering.md#replication). + ## Pub/sub fan-out Moon's subscriber delivery path **coalesces automatically** — when a publish burst queues From 3856b8ce76c016d4941192aaedd47ba462f4936c Mon Sep 17 00:00:00 2001 From: Tin Dang Date: Wed, 15 Jul 2026 14:57:44 +0700 Subject: [PATCH 3/5] docs(replication): add read/write splitting subsection (no built-in proxy) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Moon ships no read/write-splitting proxy: each node is a single server and a v0.7 replica is read-only (writes → -READONLY). Document how to serve one logical endpoint anyway — client-side split (with per-library replica-read option names: lettuce ReadFrom.REPLICA_PREFERRED, redis-py, ioredis scaleReads, go-redis ReadOnly) and a command-aware external RESP proxy (Envoy redis_proxy) — plus the key caveats: a plain L4 LB can't split by command, writes must stay on the master for WAIT durability, and reads must re-point on promotion/failover. Refs #65. author: Tin Dang --- CHANGELOG.md | 8 +++++--- docs/guides/clustering.md | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a518a21f..6f5a9705 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -44,9 +44,11 @@ replica walkthrough (it still showed the pre-GA topology — single-shard leader multi-shard replica — the reverse of the shipped shape) and its outdated "WS/MQ not replicated yet" note (all six planes replicate as of Wave B). Added the `WAIT`-durability ladder, replica promotion (`REPLICAOF NO ONE`), `INFO replication` -monitoring, and the replica-TTL caveat. New **Replication** section in -`configuration.md` and a **Replication durability** section in `guides/tuning.md` -(RPO/latency trade-offs, read-scaling, lag alerting). +monitoring, a **Read/write splitting** subsection (Moon has no built-in R/W-split +proxy — client-side vs command-aware external-proxy patterns), and the replica-TTL +caveat. New **Replication** section in `configuration.md` and a **Replication +durability** section in `guides/tuning.md` (RPO/latency trade-offs, read-scaling, +lag alerting). ### Fixed — `segment_plane_scan` missed v0.6.0's nested-Command plane framing, risking WS/MQ/temporal data loss on upgrade (task #69) diff --git a/docs/guides/clustering.md b/docs/guides/clustering.md index d6088ba6..7205a265 100644 --- a/docs/guides/clustering.md +++ b/docs/guides/clustering.md @@ -95,6 +95,43 @@ redis-cli -p 6380 REPLICAOF NO ONE # replica stops applying and becomes a writ Repoint surviving replicas at the new master with `REPLICAOF `. There is no automatic replication failover outside cluster mode (see below). +### Read/write splitting + +Moon has **no built-in read/write-splitting proxy** — each node is a single server, +and a replica is read-only. Sending a write to a replica returns `-READONLY`. To +serve writes from the master and reads from replicas behind one logical service, +split at the client or with an external Redis-aware proxy. A plain L4/TCP load +balancer **cannot** do this — it can't see commands, so it can't tell a read from a +write; it only helps for failover. + +**Client-side (recommended, no extra hop).** Open one connection to the master for +writes and one (or a pool) to the replicas for reads; most clients have a built-in +replica-read mode: + +```bash +# writes → master +redis-cli -p 6379 SET session:42 '{"user":"alice"}' +# reads → replica +redis-cli -p 6380 GET session:42 +``` + +- **lettuce (Java):** `ReadFrom.REPLICA_PREFERRED` +- **redis-py:** `Redis(..., )` with a replica connection, or `RedisCluster(read_from_replicas=True)` +- **ioredis (Node):** a `scaleReads: "slave"` cluster client, or separate clients +- **go-redis:** `ClusterOptions{ ReadOnly: true, RouteRandomly: true }`, or explicit master/replica clients + +Keep write-path connections pointed at the master so `WAIT`-based cross-node +durability still works — `WAIT` only counts ACKs for writes issued on the master. + +**External proxy (one endpoint).** Front the pair with a **command-aware** RESP proxy +that routes by command flag (writes → master, reads → replica pool): e.g. Envoy's +`redis_proxy` filter with a read policy, or a purpose-built RESP router. The trade-off +is an extra network hop and another component to operate and fail over. + +**Failover note.** However you split, reads must re-point when a replica is promoted +(`REPLICAOF NO ONE`) or a master is lost — client libraries with topology awareness +or the proxy's health checks handle this; a static split does not. + ### Replication features - **All six data planes** — KV, vector/text index, graph, WS, MQ, temporal (v0.7 GA) From f71b3f4d9328ab7462eaf8081fbda8164b8117b1 Mon Sep 17 00:00:00 2001 From: Tin Dang Date: Wed, 15 Jul 2026 16:14:13 +0700 Subject: [PATCH 4/5] docs(replication): fix WAIT-connection, replica-durability, and TTL-drift accuracy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review of the v0.7.0 replication docs surfaced four accuracy issues; three fixed, one skipped as contradicting the code. - WAIT example (clustering guide): the SET and WAIT ran as two separate `redis-cli` invocations, i.e. two connections. WAIT only counts ACKs for writes issued on its OWN connection, so as written it would observe no pending write. Rewritten to a single interactive session (heredoc). - Zero-RPO durability (clustering + configuration + tuning): a replica ACKs a write when it APPLIES it, not when it fsyncs — so a replica running `everysec`/`no` can satisfy WAIT and still lose the write on its own crash. True zero-RPO needs `--appendfsync always` on BOTH nodes. Added the replica fsync to the startup example, a replica row to the configuration table, and a replica column + note to the tuning RPO ladder. This aligns the docs with the guide's own "appendfsync always on both sides" statement. - Replica TTL drift (clustering warning): relative-TTL expiry on a replica shifts primarily because the countdown starts at APPLY time, so replication/apply delay moves the expiry moment even with perfectly synced clocks. Reworded to lead with apply delay; clock skew retained as an additional offset. Skipped: a suggestion to rename the full-resync payload to a Moon-specific "RRDSHARD" format. The payload is a genuine merged Redis-format RDB (`write_rdb_merged`, master.rs:281/451/483) with Moon's extra planes carried as RDB AUX fields — there is no RRDSHARD format, so the rename would introduce a false claim. author: Tin Dang --- docs/configuration.md | 3 ++- docs/guides/clustering.md | 28 ++++++++++++++++++++++------ docs/guides/tuning.md | 15 ++++++++++----- 3 files changed, 34 insertions(+), 12 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 19557478..84a7fb62 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -69,7 +69,8 @@ topology and durability of the pair: | `--shards N` | master | Multi-core writer; the master merges all shards into one exactly-once replication feed | | `--shards 1` | replica | **Required** — replicas are single-shard; scale reads by adding replicas | | `--appendonly yes` | both | Persist the AOF so a restarted node recovers before re-syncing | -| `--appendfsync always` | master | RPO 0 on the master; pair with `WAIT N` for zero-RPO cross-node durability | +| `--appendfsync always` | master | RPO 0 on the master; pair with `WAIT N` for cross-node durability | +| `--appendfsync always` | replica | **Required for zero-RPO** — a replica ACKs on apply, not on fsync, so it must persist durably or a `WAIT`-acked write can still be lost if the replica crashes | Replicas are read-only (`slave_read_only:1`; writes return `-READONLY`). `WAIT numreplicas timeout` reports real replica ACKs. Full setup, `WAIT` durability, diff --git a/docs/guides/clustering.md b/docs/guides/clustering.md index 7205a265..ce0a848e 100644 --- a/docs/guides/clustering.md +++ b/docs/guides/clustering.md @@ -50,9 +50,15 @@ startup flag; operators script it after the replica is up (e.g. via an init hook #### 2. Start the replica (must be --shards 1) ```bash -./target/release/moon --port 6380 --shards 1 --appendonly yes +./target/release/moon --port 6380 --shards 1 --appendonly yes --appendfsync always ``` +`--appendfsync always` on the replica is what makes a `WAIT`-acknowledged write +survive a **replica** crash: the replica ACKs when it *applies* a write, not when it +fsyncs, so without durable replica persistence a replica can ACK and then lose the +write on its own crash. Drop it to `everysec` only if the replica is read-scaling/DR +and you accept ≤1 s of replica-side loss. + #### 3. Attach the replica to the master ```bash @@ -77,9 +83,17 @@ every write issued on the connection (replicas send `REPLCONF ACK` on a ~1 s cadence). Combine it with `appendfsync always` on both sides for a **zero-RPO, cross-node durable write**: +`WAIT` counts ACKs only for writes issued on **its own connection**, so the `SET` +and the `WAIT` must run on the *same* connection — two separate `redis-cli` +invocations open two connections, and the `WAIT` would see no pending write. Use one +interactive session (or pipe both commands into a single `redis-cli`): + ```bash -redis-cli -p 6379 SET k v -redis-cli -p 6379 WAIT 1 1000 # returns the number of replicas that ACKed within 1000 ms +redis-cli -p 6379 <<'EOF' +SET k v +WAIT 1 1000 +EOF +# WAIT returns the number of replicas that ACKed the SET within 1000 ms ``` On the master, `INFO replication` lists each replica's `offset` and `lag`; on the @@ -144,9 +158,11 @@ or the proxy's health checks handle this; a static split does not. !!! warning "Replica TTL semantics (v0.7)" Relative-expire commands (`EXPIRE`, `SETEX`, `PEXPIRE`, `GETEX` with a relative TTL) replicate verbatim rather than being rewritten to absolute `PEXPIREAT`, and - replicas run their own active-expiry cycle. Under normal clock sync keys expire - correctly on both sides, but master/replica clock skew can shift a relative-TTL - key's expiry moment by up to that skew. For exact cross-node expiry parity, set + replicas run their own active-expiry cycle. Because the replica's countdown starts + when it **applies** the command — not when the master ran it — replication/apply + delay shifts a relative-TTL key's expiry moment by that delay **even with perfectly + synchronized clocks** (the replica holds the key slightly longer). Master/replica + clock skew adds a further offset on top. For exact cross-node expiry parity, set absolute deadlines with `PEXPIREAT`. Absolute-rewrite + role-gated expiry land in v0.7.1. diff --git a/docs/guides/tuning.md b/docs/guides/tuning.md index 7a7bed20..8bb1729a 100644 --- a/docs/guides/tuning.md +++ b/docs/guides/tuning.md @@ -163,11 +163,16 @@ AOF durability protects a **single node** against a crash. Replication (v0.7 GA) adds a **second node** so a write survives losing the master's disk entirely. The two combine on a latency/RPO ladder — pick the rung your workload needs: -| Goal | Master | Client | RPO | Cost | -|------|--------|--------|-----|------| -| Fast, replica for read-scaling/DR | `--appendfsync everysec` | fire-and-forget | ≤1 s on master crash; replica lag on failover | lowest latency | -| Durable on the master | `--appendfsync always` | — | 0 on master crash (disk-bound) | fsync per write | -| **Zero-RPO across nodes** | `--appendfsync always` | `WAIT 1 ` after the write | 0 even if the master's disk is lost | fsync + one replica round-trip | +| Goal | Master | Replica | Client | RPO | Cost | +|------|--------|---------|--------|-----|------| +| Fast, replica for read-scaling/DR | `--appendfsync everysec` | `--appendfsync everysec` | fire-and-forget | ≤1 s on master crash; replica lag on failover | lowest latency | +| Durable on the master | `--appendfsync always` | — | — | 0 on master crash (disk-bound) | fsync per write | +| **Zero-RPO across nodes** | `--appendfsync always` | `--appendfsync always` | `WAIT 1 ` after the write | 0 even if the master's disk is lost | fsync (both nodes) + one replica round-trip | + +The replica column matters for the zero-RPO rung: a replica ACKs a write when it +**applies** it, not when it fsyncs, so a replica running `everysec` (or `no`) can ACK +a write — satisfying `WAIT` — and then lose it on its own crash. Zero-RPO requires +`--appendfsync always` on **both** nodes. `WAIT numreplicas timeout` blocks until `numreplicas` replicas have ACKed the write (replicas ACK on a ~1 s cadence, so a `WAIT` timeout below ~1 s may return From af82a4e7fac4bfb8fc72df75da94975c1da89a77 Mon Sep 17 00:00:00 2001 From: Tin Dang Date: Wed, 15 Jul 2026 16:32:10 +0700 Subject: [PATCH 5/5] =?UTF-8?q?docs(replication):=20correct=20read/write-s?= =?UTF-8?q?plit=20guidance=20=E2=80=94=20staleness=20+=20standalone=20vs?= =?UTF-8?q?=20cluster?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second review pass on the read/write-splitting subsection; both findings valid. - Read-after-write staleness (finding 1): the client-side example did a master SET immediately followed by a replica GET of the same key — the exact eventually-consistent race — with no caveat. Added a warning admonition: async replication means a replica read right after a master write can return a stale value or a miss; route read-after-write and session-critical reads to the master, staleness-tolerant reads to replicas, and gate read-your-writes on a WAIT + slave_repl_offset catch-up check. - Standalone vs Cluster conflation (finding 2): the client bullets cited Cluster-only routing constructs (redis-py RedisCluster(read_from_replicas), ioredis scaleReads, go-redis ClusterOptions) inside the STANDALONE master/replica section. Replaced them with standalone equivalents (separate master/replica clients; lettuce MasterReplica) and moved the cluster-client modes into an explicit "Cluster mode only" note. - Failover note: clarified that REPLICAOF NO ONE does not notify clients and health checks alone only detect liveness (not which node is the new master), so repointing needs Moon's Cluster mode or an external orchestrator (k8s/systemd). Explicitly stated Moon does not implement Sentinel — matches the documented non-goal in redis-compat.md / PRODUCTION-CONTRACT.md / ROADMAP.md (do NOT reference Sentinel as a Moon coordinator). author: Tin Dang --- docs/guides/clustering.md | 39 +++++++++++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/docs/guides/clustering.md b/docs/guides/clustering.md index ce0a848e..21a6d330 100644 --- a/docs/guides/clustering.md +++ b/docs/guides/clustering.md @@ -125,14 +125,31 @@ replica-read mode: ```bash # writes → master redis-cli -p 6379 SET session:42 '{"user":"alice"}' -# reads → replica +# reads → replica (eventually consistent — may lag; see the caveat below) redis-cli -p 6380 GET session:42 ``` -- **lettuce (Java):** `ReadFrom.REPLICA_PREFERRED` -- **redis-py:** `Redis(..., )` with a replica connection, or `RedisCluster(read_from_replicas=True)` -- **ioredis (Node):** a `scaleReads: "slave"` cluster client, or separate clients -- **go-redis:** `ClusterOptions{ ReadOnly: true, RouteRandomly: true }`, or explicit master/replica clients +!!! warning "Replica reads are eventually consistent" + Replication is **asynchronous**, so a `GET` on a replica issued right after a + `SET` on the master can return the **old value or a miss** until the write + streams across (the example above is exactly that race). Route + **read-after-write** and **session-critical** reads to the *master*; send only + staleness-tolerant reads (caches, analytics, browse traffic) to replicas. If you + must read your own writes from a replica, gate the read on catch-up: `WAIT 1 + ` on the master, then confirm the replica's `slave_repl_offset` (from + `INFO replication`) has reached the master's `master_repl_offset`. + +Most clients expose a **non-cluster** replica-read mode, or you can simply hold +separate master and replica clients: + +- **lettuce (Java):** a `MasterReplica` connection with `ReadFrom.REPLICA_PREFERRED` +- **redis-py:** a dedicated replica `Redis(host=, port=6380)` client for reads +- **ioredis (Node):** separate `Redis` clients for the master and the replica +- **go-redis:** explicit master and replica `redis.NewClient(...)` instances + +The cluster-client read-routing modes — redis-py `RedisCluster(read_from_replicas=True)`, +ioredis `scaleReads: "slave"`, go-redis `ClusterOptions{ReadOnly, RouteRandomly}` — +apply only in **[Cluster mode](#cluster-mode)**, not to a standalone master/replica pair. Keep write-path connections pointed at the master so `WAIT`-based cross-node durability still works — `WAIT` only counts ACKs for writes issued on the master. @@ -142,9 +159,15 @@ that routes by command flag (writes → master, reads → replica pool): e.g. En `redis_proxy` filter with a read policy, or a purpose-built RESP router. The trade-off is an extra network hop and another component to operate and fail over. -**Failover note.** However you split, reads must re-point when a replica is promoted -(`REPLICAOF NO ONE`) or a master is lost — client libraries with topology awareness -or the proxy's health checks handle this; a static split does not. +**Failover note.** However you split, reads and writes must re-point when a replica +is promoted (`REPLICAOF NO ONE`) or a master is lost. In a standalone master/replica +pair there is **no built-in topology discovery**: `REPLICAOF NO ONE` promotes a +replica but does not notify clients, and health checks alone only detect +liveness — they cannot tell a client *which* node is the new master. Repointing +therefore requires **Moon's [Cluster mode](#cluster-mode)** (automatic promotion + +`MOVED` redirection) or an **external orchestrator** (k8s/systemd health-managed +endpoints) that rewrites the client's target; a static split does not self-heal. +Moon does not implement the Redis Sentinel protocol. ### Replication features