-
Notifications
You must be signed in to change notification settings - Fork 0
release(v0.7.0): Replication GA for multi-shard masters #333
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
4938ecd
4f2ea61
3856b8c
f71b3f4
af82a4e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,10 @@ | ||
| # Releases | ||
|
|
||
| ## v0.7.0 — 2026-07-15 | ||
| milestones: v0.7.0 "Replication GA for multi-shard masters" (soak-gated tag; v0.6.1 hardening folded in) | ||
| waivers: none — the v0.6.0 `shardslice-migration` waiver (was: expires 2026-08-01) is **retired**; lock-free cross-shard read work is now tracked as an open GA gap (`XSHARD-READ-01`, ROADMAP R4), not a time-boxed waiver. | ||
|
Comment on lines
+3
to
+5
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win Synchronize the waiver status with the production-contract ledger. This release entry says the 🤖 Prompt for AI Agents |
||
| evidence: Real async replication with WAIT/ACK across all six data planes (KV, vector/text, graph, WS, MQ, temporal); multi-shard master → single-shard streaming replica; every plane crash-durable under kill -9. Plus full v0.6.1 hardening (WAL v3 storage-kernel M1–M4) and a supply-chain CI gate (cargo audit + cargo deny). Release gate — 24h replication kill-9 soak (REPL-SOAK-01), zero acked-write loss: `SOAK-PASS duration=86400s cycles=114 acked=82044 inflight=7 master_kills=57 replica_kills=57` (2026-07-15; run dir moon-dev:~/moon-soak/runs/20260714-141946; RC main e2d87893; 114 alternating master/replica kill -9 cycles, 82,044 WAIT-acked writes preserved). Known limitation: streaming replica is single-shard only. Replica relative-TTL semantics: verbatim replication (no PEXPIREAT rewrite) + role-agnostic active expiry — absolute-rewrite lands in v0.7.1 (task #71b). | ||
|
|
||
| ## v0.6.0 — 2026-07-08 | ||
| milestones: v0.6.0-release (PR #249, six workstreams) | ||
| waivers: shardslice-migration (expires 2026-08-01; retirement planned with v0.7 L4 cross-shard-read work) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,42 +30,164 @@ Moon implements PSYNC2-compatible replication with per-shard WAL streaming and p | |
| - `WAIT N timeout` reflects real replica ACKs (1s `REPLCONF ACK` cadence). | ||
| - `master_link_status` in `INFO replication` reflects the handshake state — use it to detect a failed REPLICAOF. | ||
| - `CLIENT LIST TYPE replica` has no predicate yet; returns all clients. | ||
| - WS.\*/MQ.\* planes are **not replicated yet** (the master logs one warning | ||
| when a replica is attached); vector/text/graph planes replicate fully. | ||
| - **All six data planes replicate** (v0.7 GA): KV, vector/text index, graph, | ||
| workspace (WS.\*), message-queue (MQ.\*), and temporal. A full resync ships | ||
| each plane's snapshot; the live stream carries every plane's effect records. | ||
|
|
||
| ### Set up a replica | ||
|
|
||
| ### Start the leader (must be --shards 1 in v0.1.x) | ||
| **Topology (v0.7):** the **master** runs any `--shards N` (multi-core writer); each | ||
| **replica** runs `--shards 1`. Scale reads by adding replicas, not replica shards. | ||
| Replication is initiated at runtime with the `REPLICAOF` command — there is no | ||
| startup flag; operators script it after the replica is up (e.g. via an init hook). | ||
|
|
||
| #### 1. Start the master (any shard count) | ||
|
|
||
| ```bash | ||
| ./target/release/moon --port 6379 --shards 1 | ||
| ./target/release/moon --port 6379 --shards 4 --appendonly yes --appendfsync always | ||
| ``` | ||
|
|
||
| ### Start the replica (any shard count) | ||
| #### 2. Start the replica (must be --shards 1) | ||
|
|
||
| ```bash | ||
| ./target/release/moon --port 6380 --shards 4 | ||
| ./target/release/moon --port 6380 --shards 1 --appendonly yes --appendfsync always | ||
| ``` | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| ### Connect the replica to the leader | ||
| `--appendfsync always` on the replica is what makes a `WAIT`-acknowledged write | ||
| survive a **replica** crash: the replica ACKs when it *applies* a write, not when it | ||
| fsyncs, so without durable replica persistence a replica can ACK and then lose the | ||
| write on its own crash. Drop it to `everysec` only if the replica is read-scaling/DR | ||
| and you accept ≤1 s of replica-side loss. | ||
|
|
||
| #### 3. Attach the replica to the master | ||
|
|
||
| ```bash | ||
| redis-cli -p 6380 REPLICAOF 127.0.0.1 6379 | ||
| ``` | ||
|
|
||
| ### Verify link status | ||
| The replica performs a full resync (one merged Redis-format RDB across all planes), | ||
| then applies the live stream. Replicas are **read-only**: writes return | ||
| `-READONLY` (`INFO replication` reports `slave_read_only:1`). | ||
|
Comment on lines
+68
to
+70
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win Do not imply standard Redis replica interoperability. The supplied architecture contract describes the full-resync payload as Also applies to: 100-103 🤖 Prompt for AI Agents |
||
|
|
||
| #### 4. Verify the link | ||
|
|
||
| ```bash | ||
| redis-cli -p 6380 INFO replication | grep master_link_status | ||
| # Expect: master_link_status:up | ||
| # Expect: master_link_status:up (anything else = handshake not complete) | ||
| ``` | ||
|
|
||
| ### Acknowledged writes with WAIT | ||
|
|
||
| `WAIT numreplicas timeout` blocks until at least `numreplicas` replicas have ACKed | ||
| every write issued on the connection (replicas send `REPLCONF ACK` on a ~1 s | ||
| cadence). Combine it with `appendfsync always` on both sides for a **zero-RPO, | ||
| cross-node durable write**: | ||
|
|
||
| `WAIT` counts ACKs only for writes issued on **its own connection**, so the `SET` | ||
| and the `WAIT` must run on the *same* connection — two separate `redis-cli` | ||
| invocations open two connections, and the `WAIT` would see no pending write. Use one | ||
| interactive session (or pipe both commands into a single `redis-cli`): | ||
|
|
||
| ```bash | ||
| redis-cli -p 6379 <<'EOF' | ||
| SET k v | ||
| WAIT 1 1000 | ||
| EOF | ||
| # WAIT returns the number of replicas that ACKed the SET within 1000 ms | ||
| ``` | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| On the master, `INFO replication` lists each replica's `offset` and `lag`; on the | ||
| replica it reports `slave_repl_offset`. See the [tuning guide](tuning.md#replication-durability) | ||
| for the durability/latency trade-offs. | ||
|
|
||
| ### Promote a replica (failover) | ||
|
|
||
| ```bash | ||
| redis-cli -p 6380 REPLICAOF NO ONE # replica stops applying and becomes a writable master | ||
| ``` | ||
|
|
||
| Repoint surviving replicas at the new master with `REPLICAOF <new-host> <new-port>`. | ||
| There is no automatic replication failover outside cluster mode (see below). | ||
|
|
||
| ### Read/write splitting | ||
|
|
||
| Moon has **no built-in read/write-splitting proxy** — each node is a single server, | ||
| and a replica is read-only. Sending a write to a replica returns `-READONLY`. To | ||
| serve writes from the master and reads from replicas behind one logical service, | ||
| split at the client or with an external Redis-aware proxy. A plain L4/TCP load | ||
| balancer **cannot** do this — it can't see commands, so it can't tell a read from a | ||
| write; it only helps for failover. | ||
|
|
||
| **Client-side (recommended, no extra hop).** Open one connection to the master for | ||
| writes and one (or a pool) to the replicas for reads; most clients have a built-in | ||
| replica-read mode: | ||
|
|
||
| ```bash | ||
| # writes → master | ||
| redis-cli -p 6379 SET session:42 '{"user":"alice"}' | ||
| # reads → replica (eventually consistent — may lag; see the caveat below) | ||
| redis-cli -p 6380 GET session:42 | ||
| ``` | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| !!! warning "Replica reads are eventually consistent" | ||
| Replication is **asynchronous**, so a `GET` on a replica issued right after a | ||
| `SET` on the master can return the **old value or a miss** until the write | ||
| streams across (the example above is exactly that race). Route | ||
| **read-after-write** and **session-critical** reads to the *master*; send only | ||
| staleness-tolerant reads (caches, analytics, browse traffic) to replicas. If you | ||
| must read your own writes from a replica, gate the read on catch-up: `WAIT 1 | ||
| <timeout>` on the master, then confirm the replica's `slave_repl_offset` (from | ||
| `INFO replication`) has reached the master's `master_repl_offset`. | ||
|
|
||
| Most clients expose a **non-cluster** replica-read mode, or you can simply hold | ||
| separate master and replica clients: | ||
|
|
||
| - **lettuce (Java):** a `MasterReplica` connection with `ReadFrom.REPLICA_PREFERRED` | ||
| - **redis-py:** a dedicated replica `Redis(host=<replica>, port=6380)` client for reads | ||
| - **ioredis (Node):** separate `Redis` clients for the master and the replica | ||
| - **go-redis:** explicit master and replica `redis.NewClient(...)` instances | ||
|
|
||
| The cluster-client read-routing modes — redis-py `RedisCluster(read_from_replicas=True)`, | ||
| ioredis `scaleReads: "slave"`, go-redis `ClusterOptions{ReadOnly, RouteRandomly}` — | ||
| apply only in **[Cluster mode](#cluster-mode)**, not to a standalone master/replica pair. | ||
|
|
||
| Keep write-path connections pointed at the master so `WAIT`-based cross-node | ||
| durability still works — `WAIT` only counts ACKs for writes issued on the master. | ||
|
|
||
| **External proxy (one endpoint).** Front the pair with a **command-aware** RESP proxy | ||
| that routes by command flag (writes → master, reads → replica pool): e.g. Envoy's | ||
| `redis_proxy` filter with a read policy, or a purpose-built RESP router. The trade-off | ||
| is an extra network hop and another component to operate and fail over. | ||
|
|
||
| **Failover note.** However you split, reads and writes must re-point when a replica | ||
| is promoted (`REPLICAOF NO ONE`) or a master is lost. In a standalone master/replica | ||
| pair there is **no built-in topology discovery**: `REPLICAOF NO ONE` promotes a | ||
| replica but does not notify clients, and health checks alone only detect | ||
| liveness — they cannot tell a client *which* node is the new master. Repointing | ||
| therefore requires **Moon's [Cluster mode](#cluster-mode)** (automatic promotion + | ||
| `MOVED` redirection) or an **external orchestrator** (k8s/systemd health-managed | ||
| endpoints) that rewrites the client's target; a static split does not self-heal. | ||
| Moon does not implement the Redis Sentinel protocol. | ||
|
|
||
| ### Replication features | ||
|
|
||
| - **All six data planes** — KV, vector/text index, graph, WS, MQ, temporal (v0.7 GA) | ||
| - **PSYNC2 protocol** — compatible with Redis replication clients | ||
| - **Per-shard WAL streaming** — each shard streams its own WAL independently (once connected) | ||
| - **Partial resync** — reconnecting replicas resume from where they left off via the replication backlog | ||
| - **Lazy backlog** — the replication backlog is only allocated when the first replica handshake begins (REPLCONF), saving ~12 MB baseline memory | ||
| - **Per-shard WAL streaming** — each master shard streams its own WAL; a multi-shard master merges them into one exactly-once feed | ||
| - **Partial resync** — single-shard masters resume reconnecting replicas from the backlog window; multi-shard masters answer every reconnect with a full resync | ||
| - **Lazy backlog** — allocated only when the first replica handshake begins (REPLCONF), saving ~12 MB baseline memory | ||
| - **Validated** — 24 h continuous-load kill-9 soak (alternating master/replica restarts), zero loss of any WAIT-acknowledged write | ||
|
|
||
| !!! warning "Replica TTL semantics (v0.7)" | ||
| Relative-expire commands (`EXPIRE`, `SETEX`, `PEXPIRE`, `GETEX` with a relative | ||
| TTL) replicate verbatim rather than being rewritten to absolute `PEXPIREAT`, and | ||
| replicas run their own active-expiry cycle. Because the replica's countdown starts | ||
| when it **applies** the command — not when the master ran it — replication/apply | ||
| delay shifts a relative-TTL key's expiry moment by that delay **even with perfectly | ||
| synchronized clocks** (the replica holds the key slightly longer). Master/replica | ||
| clock skew adds a further offset on top. For exact cross-node expiry parity, set | ||
| absolute deadlines with `PEXPIREAT`. Absolute-rewrite + role-gated expiry land in | ||
| v0.7.1. | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| ## Cluster mode | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
The soak is documented as a tag gate, but the supplied release workflow does not enforce it for v0.7.0.
CHANGELOG.md#L22-L25: either wire REPL-SOAK-01 into an enforcing release step or describe it as completed/manual evidence.docs/PRODUCTION-CONTRACT.md#L129-L129: remove the claim that the row gates the v0.7.0 tag unless the workflow is changed to block that tag.📍 Affects 2 files
CHANGELOG.md#L22-L25(this comment)docs/PRODUCTION-CONTRACT.md#L129-L129🤖 Prompt for AI Agents