From 52eeba1e88516159cb2dc3b7ece79bf85221f7a2 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 7 Jul 2026 09:05:39 +0000 Subject: [PATCH 1/2] docs: add NATS traffic-estimation sizing model (nats-traffic-estimation.md) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rework docs/nats-traffic-estimation.md into a current, code-verified sizing model for a single site's NATS/JetStream traffic and storage. Inventory (§2): - §2.1 JetStream streams corrected to pkg/stream/stream.go + pkg/subject: ROOMS, PUSH_NOTIFICATION, HR, MIGRATION_OPLOG subjects/names; OUTBOX + flat INBOX federation (outbox-worker forwards to remote INBOX); parallel BOT_* pipeline - §2.2 server->client delivery subjects surveyed (RoomEvent/UserRoomEvent, SubscriptionUpdate, RoomKeyUpdate, PresenceState); stale stream.msg / metadata / notification subjects removed - §2.3 request/reply inventory surveyed across all services (client-facing by service + server-to-server), incl. presence R/R and lifecycle publishes Model (§4-§10): - Params: M=2.1M (incl user->botDM), M_room=2.0M, F=100, R_sub=10, R_hist=150, R_room=100, R_member=20, C_pres=10, R_presq=5, N=3, f_fed=20% (federated share of member changes -> OUTBOX), plus bot params; 4M split = 2.1M human + 1.9M bot - §6 single-connection baseline: JetStream (incl bot + OUTBOX) ~35M ops/ ~32 GB, core delivery ~448M/~278 GB, client + server-to-server R/R; TOTAL ~0.49B/day, ~0.46 TB/day - OUTBOX carries only the cross-site subset of member changes (R_member x U x f_fed); INBOX = local search feed + federated inflow - §7 multi-connection (D scaling), §8 migration phase (standalone), §9 storage (~8 GB; CANONICAL streams at 1-day TTL), §10 per-fab summary, §11 caveats All volumes are capacity-planning estimates, parameterized for re-runs. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01H7c7aaLBkrM3zDiyT67GL8 --- docs/nats-traffic-estimation.md | 482 ++++++++++++++++++++++---------- 1 file changed, 335 insertions(+), 147 deletions(-) diff --git a/docs/nats-traffic-estimation.md b/docs/nats-traffic-estimation.md index dc429075b..350a0f3f5 100644 --- a/docs/nats-traffic-estimation.md +++ b/docs/nats-traffic-estimation.md @@ -22,56 +22,146 @@ Each site runs its own NATS server, so these figures are per-site. > essentially alone; the **steady-state phase** (all other streams, §6–§7, §9) begins > only after migration completes. The two do not overlap and are never summed. > -> **Pending streams:** `HRSYNC_{siteID}`, `PUSH_NOTIFICATIONS_{siteID}`, and -> `MIGRATION_OPLOG_{siteID}` are not yet merged — revisit when the real specs land. +> **Stream inventory** — the streams below match `pkg/stream/stream.go` (canonical +> names/subjects in §2.1); their **traffic volumes** remain estimates pending production +> telemetry. **Cross-site federation** uses a per-site `OUTBOX_{siteID}` +> (`chat.outbox.{siteID}.>`) whose consumer outbox-worker forwards each event to the +> destination's flat `INBOX_{siteID}` (`chat.inbox.{siteID}.>`). **Bot traffic** runs on a +> parallel `BOT_*` pipeline. Both federation and the bot pipeline are folded into the +> steady-state tables (§6.1/§9); the 4M/day total = 2.1M human + 1.9M bot (§4 split note). ## 2. Stream & Endpoint Inventory ### 2.1 JetStream streams -| Stream | Subject | Producer | Consumer(s) | In estimate? | -|--------|---------|----------|-------------|:---:| +Stream names/subjects below reflect the current code (`pkg/stream/stream.go` schema + +`pkg/subject` builders), plus the new-design **`OUTBOX_{siteID}` + flat `INBOX_{siteID}`** +federation and the parallel **`BOT_*`** bot pipeline — both folded into the steady-state +estimate (§6.1/§9). `{siteID}` is the local site; `HR_` is keyed by the central site's ID. + +| Stream | Subject(s) | Producer | Consumer(s) | In estimate? | +|--------|------------|----------|-------------|:---:| | `MESSAGES_{siteID}` | `chat.user.*.room.*.{siteID}.msg.>` | Client | message-gatekeeper | ✅ | | `MESSAGES_CANONICAL_{siteID}` | `chat.msg.canonical.{siteID}.>` | message-gatekeeper, room-worker (sys msgs) | message-worker, broadcast-worker, notification-worker, search-sync-worker | ✅ | -| `ROOMS_{siteID}` | `chat.user.*.request.room.*.{siteID}.member.>` | room-service | room-worker | ✅ | -| `INBOX_{siteID}` | `chat.inbox.{siteID}.*` + `…aggregate.>` | room-worker (local member events), remote OUTBOX (sourced) | inbox-worker, search-sync-worker | ✅ | -| `PUSH_NOTIFICATIONS_{siteID}` | `chat.server.notification.push.{siteID}.>` | notification-worker *(spec pending)* | push-gateway worker → APNs/FCM *(spec pending)* | ✅ | -| `HRSYNC_{siteID}` | `hr.sync.>` | HR sync source *(spec pending)* | hr-sync consumer *(spec pending)* | ✅ | -| `MIGRATION_OPLOG_{siteID}` | `migration.oplog.>` | Migration source *(spec pending)* | migration applier (1) *(spec pending)* | ⏱ §8 phase | -| `OUTBOX_{siteID}` | `outbox.{siteID}.to.{destSiteID}.{eventType}` | room-worker, room-service, message-worker | Remote site INBOX | ❌ excluded per scope | +| `ROOMS_{siteID}` | `chat.room.canonical.{siteID}.>` | room-service | room-worker | ✅ | +| `INBOX_{siteID}` | `chat.inbox.{siteID}.>` | remote sites' `OUTBOX` (sourced), same-site services (local search feed) | inbox-worker, search-sync-worker | ✅ | +| `OUTBOX_{siteID}` | `chat.outbox.{siteID}.>` | room-worker, room-service, message-worker, user-service (cross-site publishers) | outbox-worker → forwards to remote `INBOX_{destSiteID}` | ✅ | +| `PUSH_NOTIFICATION_{siteID}` | `chat.server.notification.push.{siteID}.>` | notification-worker | push-gateway worker → APNs/FCM | ✅ | +| `HR_{centralSiteID}` | `chat.hr.{centralSiteID}.>` | hr-syncer (central site) | search-sync-worker (every fab) | ✅ | +| `SYSTEM_{siteID}` | `chat.system.{siteID}.>` | system services (admin/ops events) | system consumers | ✅ *(negligible, ~10/day)* | +| `MIGRATION_OPLOG_{siteID}` | `chat.oplog.{siteID}.>` | oplog-connector | migration applier (1) | ⏱ §8 phase | +| `BOT_MESSAGES_CANONICAL_{siteID}` | `chat.bot.msg.canonical.{siteID}.>` | Bot Msg Handler | Shared Sync Worker, Bot Broadcast Worker, Bot Notification Worker | ✅ | +| `BOT_PUSH_NOTIFICATION_{siteID}` | `chat.bot.server.notification.push.{siteID}.>` | Bot Notification Worker | Bot Push Notification | ✅ | +| `BOT_PLATFORM_{siteID}` | `chat.bot.event.{siteID}.>` | Broadcast Worker | Bot Webhook Worker | ✅ | + +**Bot streams (parallel pipeline).** Bot message traffic is split off the normal +(human) message flow onto a parallel `BOT_*` pipeline so it can be scaled, throttled, and +observed independently: + +- `BOT_MESSAGES_CANONICAL_{siteID}` (`chat.bot.msg.canonical.{siteID}.>`) — the bot-side + analogue of `MESSAGES_CANONICAL`. **Bot Msg Handler** publishes validated bot messages; + **Shared Sync Worker**, **Bot Broadcast Worker**, and **Bot Notification Worker** consume. +- `BOT_PUSH_NOTIFICATION_{siteID}` (`chat.bot.server.notification.push.{siteID}.>`) — bot + mobile-push lane. **Bot Notification Worker** publishes; **Bot Push Notification** forwards + to APNs/FCM. +- `BOT_PLATFORM_{siteID}` (`chat.bot.event.{siteID}.>`) — outbound platform/webhook events. + **Broadcast Worker** publishes; **Bot Webhook Worker** delivers to external bot platforms. + +Bot traffic is sized alongside the human pipeline in §6.1/§9 (params in §4); the 4M/day +total = 2.1M human + 1.9M bot. Payload sizes are in §3. + +**Federation via OUTBOX → INBOX.** Cross-site federation uses a per-site `OUTBOX_{siteID}` +stream (`chat.outbox.{siteID}.>`): a service at the origin site publishes each cross-site +event to its **local** OUTBOX, and its consumer **outbox-worker** forwards each event to +the destination's `INBOX_{destSiteID}` (`chat.inbox.{siteID}.>`), where inbox-worker +consumes and applies it to the local DB. `INBOX` carries no `internal`/`external` lane +split — its single flat subject holds both the federated inflow and the same-site +search-indexing feed. ### 2.2 Core NATS delivery subjects (server → client) -| Subject | Publisher | Purpose | -|---------|-----------|---------| -| `chat.room.{roomID}.stream.msg` (+ DM `chat.user.{a}.stream.msg`) | broadcast-worker | Message delivery (incl. member-change sys messages) | -| `chat.room.{roomID}.event.metadata.update` | broadcast-worker | Room metadata (`lastMessageAt`) — emitted per message | -| `chat.user.{account}.notification` | notification-worker | Desktop banner (mentions/DMs) | -| `chat.user.{account}.event.subscription.update` | room-worker, inbox-worker | Room added/removed | -| `chat.user.{account}.event.presence` | Presence (future) / client heartbeat | Online/offline/away | +Fire-and-forget events clients subscribe to for real-time updates. Publishers verified +against the services (broadcast-worker, room-worker, user-presence-service); every subject +is built by a `pkg/subject` builder (named in parentheses). + +| Subject (builder) | Publisher | Purpose | +|-------------------|-----------|---------| +| `chat.room.{roomID}.event` (`RoomEvent`) | broadcast-worker (message CRUD, reactions, pins, thread metadata, member changes); room-worker (room rename) | Channel/group room event delivery — the single combined event stream for a room | +| `chat.user.{account}.event.room` (`UserRoomEvent`) | broadcast-worker | Per-user delivery for DM / BotDM rooms and thread-subscription events | +| `chat.user.{account}.event.subscription.update` (`SubscriptionUpdate`) | room-worker | Subscription added / removed (room join / leave). inbox-worker deliberately does **not** re-publish — the origin site's room-worker already did | +| `chat.user.{account}.event.room.key` (`RoomKeyUpdate`) | room-worker (via `pkg/roomkeysender`) | Encrypted room-key material pushed to members | +| `chat.user.presence.state.{account}` (`PresenceState`) | user-presence-service | Presence state broadcast — online / away / offline / manual override | + +> **Stale subjects removed (survey, 2026-07).** Prior drafts listed +> `chat.room.{roomID}.stream.msg`, `chat.user.{a}.stream.msg`, and +> `chat.room.{roomID}.event.metadata.update`. Those builders (`RoomMsgStream`, +> `UserMsgStream`, `RoomMetadataUpdate`) are no longer referenced by any service — +> message and per-message metadata delivery were consolidated onto the single combined +> `chat.room.{roomID}.event` / `chat.user.{account}.event.room` event. Presence delivery +> also moved from the placeholder `chat.user.{account}.event.presence` to +> `chat.user.presence.state.{account}`. User notifications are **not** a core delivery +> subject: they are handled by **notification-worker** via the push lane +> (`PUSH_NOTIFICATION_{siteID}`, §2.1) → APNs/FCM, so the former +> `chat.user.{account}.notification` (`Notification`) subject is dropped here. The traffic +> model (§6.2) sizes message + metadata as **one** combined `chat.room.{roomID}.event` +> delivery (~1.3KB), matching the consolidated event. ### 2.3 Request/Reply endpoints -| Group | Subjects | Heaviest response | -|-------|----------|-------------------| -| Subscription R/R | `…subscription.{getCurrent,getRooms,getChannels,getApps,count}` | `subscriptionListResp{Subscriptions[]}` | -| Message history R/R | `…msg.{history,next,surrounding,thread,get,edit,delete}` | `LoadHistoryResponse{Messages[]}` | -| Room R/R | `…rooms.create`, `…room.{siteID}.info.batch`, `…member.list`, `…key.ensure` | `RoomsInfoBatchResponse` / `ListRoomMembersResponse` | -| Search R/R | `…search.{siteID}.{messages,rooms,apps,users}` | `SearchMessagesResponse` | +Synchronous NATS request/reply (client publishes to `…request.…`, awaits an `_INBOX.>` +reply). Surveyed from the service registrations (`pkg/natsrouter` `Register`/`RegisterNoBody` ++ replying `QueueSubscribe`); mirrors `docs/client-api/request-reply.md`. **Bold** responses +are the heavy ones (return arrays/lists). The presence connection-lifecycle ops +(hello/ping/activity/bye) are `RegisterVoid` **publishes** (no reply); they're grouped into +the Presence R/R row below for completeness and flagged as publishes. + +**Client-facing** (`chat.user.{account}.request.…`), by owning service — these are the four +groups sized in §6.3: + +| Service (§6.3 group) | Subject families (after the `…request.` prefix) | Heaviest response | +|----------------------|--------------------------------------------------|-------------------| +| history-service (Message history R/R) | `room.{roomID}.{siteID}.msg.{history,next,surrounding,get,get.ids,edit,delete,pin,unpin,pinned.list,react,thread,thread.parent}` (13) | **`LoadHistoryResponse{Messages[]}`** (also next / surrounding / get.ids / thread / thread.parent / pinned.list) | +| user-service (Subscription R/R) | `user.{siteID}.{me,status.getByName,profile.getByName,status.set,subscription.list,subscription.getChannels,subscription.getDM,subscription.getByRoomID,subscription.count,subscription.setAppSubscription,apps.list,thread.list,thread.unread.summary}` (13) | **`PagedSubscriptionListResponse{Subscriptions[]}`** (list / getChannels); `ThreadListResponse`, `AppsListResponse` | +| room-service (Room R/R) | `room.{roomID}.{siteID}.{member.add,member.remove,member.role-update,member.list,member.statuses,subscription.mentionable,message.read,message.thread.read,message.read-receipt,mute.toggle,favorite.toggle,room.rename,key.get,app.tabs,app.cmd-menu,teams.call,teams.meeting}` · `room.{siteID}.create` · `orgs.{orgID}.{siteID}.members` · `teams.{siteID}.call.user` (20) | **`ListRoomMembersResponse`** / `ListOrgMembersResponse` / `MentionableSubscriptionsResponse` | +| search-service (Search R/R) | `search.{siteID}.{messages,rooms,apps,users}` (4) | **`SearchMessagesResponse`** | +| user-presence-service (Presence R/R) | `presence.{siteID}.manual.set` (R/R) · `chat.user.presence.{siteID}.query.batch` (R/R) · `event.presence.{siteID}.{hello,ping,activity,bye}` (publish¹) (6) | **`PresenceQueryResponse`** (batch) | + +**Server-to-server** (`chat.server.request.…` / peers; not client-facing, not sized in §6.3): + +| Subject | Owner | Response | +|---------|-------|----------| +| `chat.server.request.room.{siteID}.info.batch` | room-service | **`RoomsInfoBatchResponse`** | +| `chat.server.request.room.{siteID}.thread.info.batch` | room-service | **`ThreadRoomInfoBatchResponse`** | +| `chat.server.request.room.{siteID}.key.ensure` | room-service | `RoomKeyEnsureResponse` | +| `chat.server.request.room.{siteID}.restricted` | room-service | `StatusWithRequestReply` (admin) | +| `chat.server.request.room.{siteID}.create.dm` | room-worker | `SyncCreateDMReply` | +| `chat.server.request.thread.{siteID}.subscription.list` | history-service | **`ThreadSubscriptionListResponse`** | +| `chat.server.request.presence.{siteID}.query.batch` | user-presence-service | **`PresenceQueryResponse`** (peer leaf) | +| `chat.migration.internal.{siteID}.msg.{edit,delete}` | history-service | `MigrationAck` (migration-phase only) | + +¹ `hello`/`ping`/`activity`/`bye` are client→server publishes (fire-and-forget, no reply) +on `chat.user.{account}.event.presence.{siteID}.{op}` — connection up / keepalive / +active-inactive / tab-close. Each ~1/user/day (= `C_hc`); grouped in the Presence R/R row +for inventory completeness, delivered to watchers in §6.2. ## 3. Payload Size Reference | Category | Streams / Endpoints | avg | max | |----------|---------------------|-----|-----| | Message JetStream | MESSAGES, MESSAGES_CANONICAL | 500B–1.5KB | ~20KB | -| Push notification | PUSH_NOTIFICATIONS | ~0.8KB | ~15KB | -| HR sync | HRSYNC (≈ `model.User`: ~12 fields) | ~0.5KB | ~1KB | +| Push notification | PUSH_NOTIFICATION | ~0.8KB | ~15KB | +| HR sync | HR (≈ `model.User`: ~12 fields) | ~0.5KB | ~1KB | | Migration oplog | MIGRATION_OPLOG | ~130KB | — | | Room JetStream | ROOMS | 200–400B | ~5KB | -| Federation | INBOX | 200–400B | ~5KB | +| Federation | INBOX, OUTBOX | 200–400B | ~5KB | +| Bot message JetStream | BOT_MESSAGES_CANONICAL | 500B–1.5KB | ~20KB | +| Bot push notification | BOT_PUSH_NOTIFICATION | ~0.8KB | ~15KB | +| Bot platform event | BOT_PLATFORM | 200B–1KB | ~5KB | | Message R/R | history, search-messages | 15–50KB | 100KB+ | | Room R/R | RoomsInfoBatch, CreateRoom, member.list | 2–20KB | ~65KB | | Subscription R/R | subscription.get*, member-list | 100–300KB | ~700KB | +| Presence R/R | manual.set, query.batch (batch of ~P states) | 0.2–3KB | ~5KB | +| Presence publish | hello / ping / activity / bye (client → server) | ~50–200B | ~0.5KB | ## 4. Parameters (locked inputs) @@ -80,23 +170,37 @@ Each site runs its own NATS server, so these figures are per-site. | Users | U | 20,789 | | Subjects subscribed per connection | S | 100 (user + room) | | Presence subjects watched per connection | P | **20** | -| Messages per day (human + bot) | M | 4,000,000 | +| Human messages per day (pipeline) | M | 2,100,000 *(user↔room 2.0M + user→bot 0.1M via broadcast-worker→BOT_PLATFORM). 4M total = 2.1M human + 1.9M bot.* | +| Human messages fanned ×F to rooms | M_room | 2,000,000 *(user↔room only; botDM/mention go to BOT_PLATFORM, no ×F)* | | Recipients per room (fan-out) | F | 100 | | Subscription R/R ops per day per connection | R_sub | **10** | | Message-history R/R ops per day per connection | R_hist | **150** | -| Room R/R ops per day per user | R_room | **250** | -| Member add/remove ops per day per user *(member-change subset of R_room)* | R_member | **50** | +| Room R/R ops per day per user | R_room | **100** | +| Member add/remove ops per day per user *(member-change subset of R_room)* | R_member | **20** | | Search ops per day per user | R_search | ~5 | -| Presence status changes per day per user | C_pres | ~20 | -| Presence health-check ops per day per user (ping/pong/hello/bye) | C_hc | 4 | -| Push notifications per day | M_push | 4,000,000 | +| Presence status changes per day per user | C_pres | ~10 | +| Presence health-check ops per day per user (hello/ping/activity/bye) | C_hc | 4 | +| Presence batch-query (initial-state) ops per day per user | R_presq | ~5 | +| Cross-site fan-out (remote sites per federated query) | N | ~3 | +| Federated fraction of member changes (→ OUTBOX / INBOX inflow) | f_fed | ~20% | +| Push notifications per day (human, = M) | M_push | 2,100,000 | | HR sync records per daily run (burst @ 100 msg/s) | M_hr | 40,000 | | Migration oplog QPS (sustained 24/7, 130KB payload, 1 consumer) | Q_mig | 200 | +| Bot-originated messages per day (→ BOT_MESSAGES_CANONICAL) | M_bot | 1,900,000 | +| Bot push notifications per day (= bot canonical input) | M_bot_push | 1,900,000 | +| User→bot events per day (→ BOT_PLATFORM, forwarded by broadcast-worker) | M_bot_platform | 100,000 | +| Bot fan-out per message (room delivery, core-NATS) | F_bot | 100 | | Peak factor (business-hours concentration) | k | ~4× | -> **R_member** is the member-change slice of the 250 room ops/day: `R_member = 50` drives -> the ROOMS stream + sys-message fan-out, and the remaining ~200 are read-only Room R/R. -> Lines tagged *(member-driven)* scale linearly with it. +> **R_member** is the member-change slice of the 100 room ops/day: `R_member = 20` drives +> the ROOMS + INBOX streams + sys-message fan-out; **OUTBOX** (and the symmetric INBOX +> federated inflow) carries only the cross-site subset `× f_fed`. The remaining ~80 room ops +> are read-only Room R/R. Lines tagged *(member-driven)* scale linearly with R_member. + +> **4M/day traffic split** (bot pipeline folded into §6.1/§9): user↔room 2.0M and user→bot +> 0.1M flow through `MESSAGES`→`MESSAGES_CANONICAL` (M = 2.1M; broadcast-worker forwards the +> 0.1M user→bot subset to `BOT_PLATFORM`, no ×F). Bot-originated 1.9M enters via Bot Msg +> Handler → `BOT_MESSAGES_CANONICAL`. Total = 2.1M human + 1.9M bot = 4.0M. ## 5. Methodology — ingress vs. fan-out @@ -105,16 +209,16 @@ to subscribers). Each message flows: ``` Client ─pub→ MESSAGES ─→ gatekeeper ─pub→ CANONICAL ─→ 4 consumers - └─ broadcast-worker ─pub→ chat.room.{id}.stream.msg ──→ ×F members - └─ broadcast-worker ─pub→ chat.room.{id}.event.metadata.update ──→ ×F members - └─ notification-worker ─pub→ chat.user.{a}.notification + └─ broadcast-worker ─pub→ chat.room.{id}.event (+DM chat.user.{a}.event.room) ──→ ×F members + └─ broadcast-worker ─pub→ room metadata (lastMessageAt) ──→ ×F members + └─ notification-worker ─pub→ PUSH_NOTIFICATION → push gateway → APNs/FCM ``` Per message: **~7 JetStream hops** (2 stores + 5 consumer deliveries, persisted) plus **~2×F core deliveries** (room stream + metadata). The broadcast-worker publishes once per subject; the multiplication happens at the NATS delivery layer. -`PUSH_NOTIFICATIONS` and `HRSYNC` are **terminal JetStream streams** — a single +`PUSH_NOTIFICATION` and `HR` are **terminal JetStream streams** — a single worker consumes and forwards out-of-band (APNs/FCM, HR DB), so they incur no ×F NATS fan-out. @@ -127,57 +231,122 @@ counts publishes + consumer deliveries (JetStream), deliveries (core), or reques | Stream | Driver | Pub/day | Deliveries/day | Ops/day | avg msg/s | Payload | Bytes/day | |--------|--------|--------:|---------------:|--------:|----------:|---------|----------:| -| `MESSAGES` | M (client → gatekeeper) | 4M | 4M | 8M | 93 | 1KB | 8 GB | -| `MESSAGES_CANONICAL` | (M + member sys) × (1 pub + 4 consumers) | 5M | 20M | 25M | 289 | 1KB | 25 GB | -| `ROOMS` *(member-driven)* | R_member × U | 1.04M | 1.04M | 2.08M | 24 | 0.4KB | 0.8 GB | -| `INBOX` *(member-driven)* | local member events × 2 consumers | 1.04M | 2.08M | 3.12M | 36 | 0.3KB | 0.9 GB | -| `PUSH_NOTIFICATIONS` | M_push × (1 pub + 1 consumer) | 4M | 4M | 8M | 93 | 0.8KB | 6.4 GB | -| `HRSYNC` | M_hr × (1 pub + 1 consumer); 100 msg/s burst | 40K | 40K | 80K | ~1 (200/s burst) | 0.5KB | 0.04 GB | -| **JetStream subtotal** | | | | **~46M** | **~536** | | **~41 GB** | - -`MESSAGES_CANONICAL` pub = 4M user messages + ~1.04M member-change system messages. -`INBOX` carries local member add/remove events (room-worker publishes to the local INBOX -regardless of site); federated cross-site inflow is excluded with OUTBOX. -`MIGRATION_OPLOG` is **excluded here** — it is a separate ~2-month phase covered on its -own in §8 and must not be summed with steady-state. +| `MESSAGES` | M (client → gatekeeper) | 2.1M | 2.1M | 4.2M | 49 | 1KB | 4.2 GB | +| `MESSAGES_CANONICAL` | (M + member sys) × (1 pub + 4 consumers) | 2.52M | 10.1M | 12.6M | 146 | 1KB | 12.6 GB | +| `ROOMS` *(member-driven)* | R_member × U | 0.42M | 0.42M | 0.83M | 10 | 0.4KB | 0.33 GB | +| `INBOX` *(member-driven)* | (local feed R_member×U + federated inflow ×f_fed) × 2 consumers | 0.50M | 1.0M | 1.5M | 17 | 0.3KB | 0.45 GB | +| `OUTBOX` *(member-driven)* | R_member × U × f_fed (1 pub + outbox-worker) | 0.08M | 0.08M | 0.17M | 2 | 0.3KB | 0.05 GB | +| `PUSH_NOTIFICATION` | M_push × (1 pub + 1 consumer) | 2.1M | 2.1M | 4.2M | 49 | 0.8KB | 3.4 GB | +| `HR` | M_hr × (1 pub + 1 consumer); 100 msg/s burst | 40K | 40K | 80K | ~1 (200/s burst) | 0.5KB | 0.04 GB | +| `SYSTEM` | system/admin events (~10/day) | 10 | 10 | 20 | ~0 | — | ~0 GB | +| `BOT_MESSAGES_CANONICAL` | M_bot × (1 pub + 3 consumers) | 1.9M | 5.7M | 7.6M | 88 | 1KB | 7.6 GB | +| `BOT_PUSH_NOTIFICATION` | M_bot_push × (1 pub + 1 consumer) | 1.9M | 1.9M | 3.8M | 44 | 0.8KB | 3.0 GB | +| `BOT_PLATFORM` | M_bot_platform × (1 pub + Bot Webhook Worker) | 0.1M | 0.1M | 0.2M | 2 | 1KB | 0.2 GB | +| **JetStream subtotal** | | | | **~35M** | **~408** | | **~32 GB** | + +`MESSAGES_CANONICAL` pub = 2.1M human messages + ~0.42M member-change system messages. +**OUTBOX** carries only the **cross-site subset** of member changes (`f_fed ≈ 20%` — those +touching a federated room); its lone consumer **outbox-worker** forwards each to the +destination site's `INBOX`. **INBOX** carries the local-origin search feed (all member +changes, `R_member × U`) **plus** the federated inflow from remote outbox-workers (`×f_fed`), +both on `chat.inbox.{siteID}.>` (×2 consumers: inbox-worker applies, search-sync-worker +indexes). Bot streams (`BOT_*`) are the parallel bot +pipeline (4M split in §4); bot room fan-out is core-NATS (§6.2 note), not shown here. +`MIGRATION_OPLOG` is **excluded** — separate ~2-month phase (§8). ### 6.2 Core delivery subjects (server → client, fanned out ×F or ×P) | Subject | Driver | Pub/day | Deliveries/day | avg msg/s | Payload | Bytes/day | |---------|--------|--------:|---------------:|----------:|---------|----------:| -| `chat.room.*.stream.msg` (+DM) | M × F | 4M | 400M | 4,630 | 1KB | 400 GB | -| `chat.room.*.event.metadata.update` | M × F | 4M | 400M | 4,630 | 0.3KB | 120 GB | -| member-change broadcast (sys msg/event) *(member-driven)* | R_member × U × F | 1.04M | 104M | 1,204 | 0.4KB | 42 GB | -| `chat.user.*.notification` | M × ~10% | 0.4M | 0.4M | 5 | 1KB | 0.4 GB | -| `chat.user.*.event.subscription.update` *(member-driven)* | R_member × U × ~2 | 1.04M | 2.08M | 24 | 0.5KB | 1 GB | -| `chat.user.*.event.presence` | U × C_pres × P | 0.42M | 8.3M | 96 | 0.15KB | 1.3 GB | -| `chat.user.*.event.presence` (health-check ping/pong/hello/bye) | U × C_hc × P | 0.083M | 1.66M | 19 | ~0.05KB | 0.08 GB | +| `RoomEvent`/`UserRoomEvent` — message + `lastMessageAt`, **human + bot** (`chat.room.*.event` +DM `chat.user.*.event.room`) | (M_room + M_bot) × F | 3.9M | 390M | 4,514 | ~1.3KB | 507 GB | +| member-change — sys msg on `chat.room.*.event`, member event on `chat.room.*.event.member` *(member-driven)* | R_member × U × F | 0.42M | 41.6M | 481 | 0.4KB | 17 GB | +| `chat.user.*.event.subscription.update` (`SubscriptionUpdate`) | R_room × U (read/mute/favorite; per-user) | 2.08M | 2.08M | 24 | 0.5KB | 1 GB | +| `chat.user.*.event.room.key` (`RoomKeyUpdate`) *(member-driven)* | R_member × U × F (key rotation → members) | 0.42M | 42M | 481 | 0.2KB | 8.3 GB | +| `chat.user.presence.state.*` (`PresenceState`, status broadcast) | U × C_pres × P | 0.21M | 4.16M | 48 | 0.15KB | 0.6 GB | +| `chat.user.*.event.presence.{siteID}.{hello,ping,activity,bye}` — **client→server** (ingress; §2.3), watcher fan-out modeled here | U × C_hc × P | 0.083M | 1.66M | 19 | ~0.05KB | 0.08 GB | | `chat.room.*.event.typing` | active room only | — | — | — | — | *(ignored)* | -| **Core delivery subtotal** | | | **~917M** | **~10,610** | | **~565 GB** | - -Presence delivery dropped sharply (was 42M) because P went 100 → 20. The health-check -row is **client-published** (each connection publishes ping/pong/hello/bye to its own -presence subject); each op fans out to the user's P watchers like a status change, but -with a tiny ~50B payload so it adds negligible bytes. +| **Core delivery subtotal** | | | **~481M** | **~5,570** | | **~534 GB** | + +**Reconciling with §2.2 — the message/metadata/member-change rows.** + +- **message + metadata** are **one** delivery: the `chat.room.{roomID}.event` (`RoomEvent`) / + `chat.user.{account}.event.room` (`UserRoomEvent`) event carries both the message body and + the room's `lastMessageAt` (for sidebar reordering) — there is **no dedicated metadata + subject** (the old `RoomMetadataUpdate` / `…event.metadata.update` builder is unused). Covers + **both human and bot** messages: the Bot Broadcast Worker delivers bot-originated messages + (`M_bot`) to room members on the same event, so the driver is `(M_room + M_bot) × F` — one + `×F` row at ~1.3KB (~1KB body + ~0.3KB metadata). +- **member-change** spans **two** subjects. The *system message* (`members_added` / + `member_removed`) is a real `model.Message` published to `chat.msg.canonical.{siteID}.created` + (MESSAGES_CANONICAL, so it also rides the §6.1 CANONICAL member-sys pub) and delivered by + broadcast-worker on `chat.room.{roomID}.event` / `chat.user.{account}.event.room` — the + same path as a normal message. The *member event* (`member_added` / `member_removed`) is + published **directly** by room-worker to `chat.room.{roomID}.event.member` + (`RoomMemberEvent`), which clients subscribe to. That `.event.member` subject is + server→client but **not yet in §2.2** (survey gap). The single `×F` here is a lower bound — + sys-msg and member-event each fan out to ~F members. + +- **`SubscriptionUpdate`** (`chat.user.*.event.subscription.update`) is **per-user**, not a + room fan-out (×1, to the affected account's own connections). It fires on member add/remove + **and** on that user's own `read` / `mute` / `favorite` / `role` / `rename` actions — so it + scales with **`R_room`** (the state-mutating room ops), *not* `R_member`. `R_room × U` here + is an upper bound: read-only room RPCs (member.list, statuses, mentionable) don't fire it. +- **`RoomKeyUpdate`** (`chat.user.*.event.room.key`) is **member-driven**: `R_member` is + exactly the add/remove ops, and every one rotates the room key and pushes a new per-member + key event — so it fans out `≈ R_member × U × F`, ~42M deliveries/day at ~0.2KB. Upper bound + (an *add* pushes only to new members, not all F; assumes E2E rooms). + +The presence health-check row is a **client→server publish** (`RegisterVoid`, §2.3) modeled +here for its optional watcher fan-out, not a true server→client subject. + +Bot room delivery (Bot Broadcast Worker → members, `M_bot × F` ≈ ~190M deliveries/day) is +now **included** in the message row above — bots post to rooms and fan out to members on the +same `RoomEvent`/`UserRoomEvent` as human messages. + +Presence status delivery = ~4.16M/day (`C_pres = 10` changes × `P = 20` watchers), on +`presence.state.*`. The health-check row is **client-published** (each connection publishes +hello/ping/activity/bye to its own presence subject); each op is modeled as fanning out to +the user's P watchers like a status change, but with a tiny ~50B payload it adds negligible +bytes (and if these are server-only keepalives with no fan-out, the delivery count drops +~20× — §11). ### 6.3 Request/Reply endpoints -| Endpoint group | Driver | Req/day | req/s | Resp payload | Bytes/day | -|----------------|--------|--------:|------:|-------------|----------:| -| Subscription R/R | R_sub × U | 0.21M | 2.4 | 150KB | 31 GB | -| **Message history R/R** | R_hist × U | 3.12M | 36 | 30KB | **94 GB** | -| Room R/R | R_room × U | 5.20M | 60 | 10KB | 52 GB | -| Search R/R | R_search × U | 0.10M | 1.2 | 30KB | 3 GB | -| **R/R subtotal** | | **~8.6M** | **~100** | | **~180 GB** | +Client-facing (`chat.user.{account}.request.…`, scales with D). Groups per §2.3. + +| Endpoint group (service) | Driver | Req/day | req/s | Resp payload | Bytes/day | +|--------------------------|--------|--------:|------:|-------------|----------:| +| Subscription R/R (user-service) | R_sub × U | 0.21M | 2.4 | 150KB | 31 GB | +| **Message history R/R (history-service)** | R_hist × U | 3.12M | 36 | 30KB | **94 GB** | +| Room R/R (room-service) | R_room × U | 2.08M | 24 | 10KB | 21 GB | +| Search R/R (search-service) | R_search × U | 0.10M | 1.2 | 30KB | 3 GB | +| Presence R/R (user-presence-service) | (R_presq + 1) × U | 0.12M | 1.4 | 3KB | 0.3 GB | +| **Client R/R subtotal** | | **~5.6M** | **~65** | | **~149 GB** | + +Server-to-server (`chat.server.request.…`, **server-side — flat with D**, derived from the +ops that trigger each; see §2.3). Kept out of the client subtotal so §7's ×D scaling +doesn't inflate it. + +| Endpoint | Driver | Req/day | req/s | Resp payload | Bytes/day | +|----------|--------|--------:|------:|-------------|----------:| +| room `info.batch` enrichment | ≈ R_sub × U (1 per subscription-list) | 0.21M | 2.4 | 10KB | 2.1 GB | +| `key.ensure` / `create.dm` | ≈ R_member × U (member-change) | 0.42M | 5 | 0.3KB | 0.1 GB | +| `thread.subscription.list` (cross-site) | thread.list × N | 0.12M | 1.4 | 8KB | 1.0 GB | +| presence `query.batch` peer (cross-site) | query.batch × N | 0.31M | 3.6 | 3KB | 0.9 GB | +| **Server-to-server subtotal** | | **~1.1M** | **~12** | | **~4 GB** | ### 6.4 Totals | Layer | Ops or deliveries/day | avg msg/s | Bytes/day | |-------|----------------------:|----------:|----------:| -| JetStream streams | ~46M | ~536 | ~41 GB | -| Core delivery subjects | ~917M | ~10,610 | ~565 GB | -| R/R (req + resp) | ~17M | ~200 | ~180 GB | -| **TOTAL (steady-state)** | **~0.98B/day** | **~11,340/s avg · ~45,400/s peak** | **~0.79 TB/day** | +| JetStream streams (incl. `BOT_*`, server-side/flat) | ~35M | ~408 | ~32 GB | +| Core delivery subjects (scales with D) | ~481M | ~5,570 | ~534 GB | +| Client R/R (req + resp, scales with D) | ~11M | ~130 | ~149 GB | +| **TOTAL (steady-state, D=1)** | **~0.53B/day** | **~6,100/s avg · ~24,400/s peak** | **~0.72 TB/day** | + +Plus **server-to-server R/R** (§6.3) — server-side, **flat with D** — ~2.1M ops/day (req + +resp), ~25/s, ~4 GB/day per site. Like the JetStream layer it does not scale with D (§7); +the ×D total in §7 scales only Core delivery + Client R/R. Excludes `MIGRATION_OPLOG` (separate phase — §8). @@ -193,18 +362,21 @@ At the single-connection baseline: 20,789 connections × (S=100 + P=20) = ### 6.6 Bottlenecks (steady-state) -- **Message + metadata delivery = ~520 GB/day & ~800M deliveries/day (~9,300/s)** — now - the dominant term for *both* bandwidth (~66% of bytes) and message rate, driven by - F = 100. -- **Message history R/R = ~94 GB/day** — the largest R/R term now that R_sub dropped to - 10/day; Room R/R (~52 GB) is next. -- **Member-change fan-out = ~104M deliveries/day** — driven by `R_member = 50/day/user` - (sys-message + member-event broadcast ×F). +- **Message delivery = ~507 GB/day & ~390M deliveries/day (~4,514/s)** — **human + bot** + messages fanned ×F (one combined event with metadata); the dominant term for *both* + bandwidth (~71% of bytes) and message rate, driven by F = 100 on (M_room + M_bot) = 3.9M. +- **Message history R/R = ~94 GB/day** — the largest R/R term now that R_room dropped to + 100/day (Room R/R ~21 GB). R/R is ~32% of steady-state bytes. +- **Member-change fan-out = ~42M deliveries/day** — driven by `R_member = 20/day/user` + (sys-message + member-event broadcast ×F); **room-key rotation** adds another ~42M/day + (every member op rotates → ×F), and OUTBOX/INBOX federation ~1.7M ops/day. ### 6.7 Optimization levers -- **Coalesce metadata updates** (e.g. max 1/sec/room) → halves the message-delivery term, - now the top bandwidth lever (removes the second `M × F` row). +- **Message fan-out (×F) is the top bandwidth term** (~507 GB/day, human + bot). Metadata already rides + the message event (no separate delivery to coalesce), so the remaining levers are reducing + effective fan-out (e.g. don't deliver to idle/very-large rooms in real time) or trimming + the combined payload. - **Keep subscription-list payloads lean** — at R_sub = 10/day the endpoint is no longer dominant, but its 150KB response still makes each call expensive on reconnect bursts. @@ -218,9 +390,9 @@ subscriber that re-subscribes and fetches its own state. | Scales linearly with D | Stays flat (independent of D) | |------------------------|-------------------------------| -| All core deliveries (message, metadata, member-change, presence, notification) | Message ingress (user sends from one client) | -| Subscription, history, room, search R/R | JetStream pipeline & terminal streams (server-side) | -| Connections & subscription interests | `MIGRATION_OPLOG`, `PUSH`, `HRSYNC` (server-side, not client-facing) | +| All core deliveries (message, metadata, member-change, presence) | Message ingress (user sends from one client) | +| Subscription, history, room, search, **presence** R/R (client-facing) | JetStream pipeline & terminal streams (server-side) | +| Connections & subscription interests | `MIGRATION_OPLOG`, `PUSH`, `HR`, **server-to-server R/R** (§6.3, server-side, not client-facing) | Rule of thumb: **ingress and server-side processing are flat; delivery/egress, R/R, and connection state scale with D.** Effective fan-out becomes `F × D = 500` per message. @@ -229,11 +401,12 @@ connection state scale with D.** Effective fan-out becomes `F × D = 500` per me | Layer | D=1 deliveries/day | D=5 deliveries/day | D=1 bytes/day | D=5 bytes/day | |-------|-------------------:|-------------------:|--------------:|--------------:| -| JetStream streams | ~46M | ~46M *(flat)* | ~41 GB | ~41 GB | -| Core delivery | ~917M | ~4,585M | ~565 GB | ~2,825 GB | -| R/R (req+resp) | ~17M | ~86M | ~180 GB | ~900 GB | -| **TOTAL (steady-state)** | **~0.98B** | **~4.72B** | **~0.79 TB** | **~3.77 TB** | -| **avg / peak msg/s** | ~11.3k / ~45k | **~54.6k / ~218k** | | | +| JetStream streams (incl. bot) | ~35M | ~35M *(flat)* | ~32 GB | ~32 GB | +| Core delivery | ~481M | ~2,405M | ~534 GB | ~2,670 GB | +| Client R/R (req+resp) | ~11M | ~56M | ~149 GB | ~745 GB | +| Server-to-server R/R | ~2.1M | ~2.1M *(flat)* | ~4 GB | ~4 GB | +| **TOTAL (steady-state)** | **~0.53B** | **~2.50B** | **~0.72 TB** | **~3.45 TB** | +| **avg / peak msg/s** | ~6.1k / ~25k | **~29k / ~116k** | | | Connection state at D=5: **~104k connections** × (100 + 20) = **~12.5M subscription interests**. Excludes `MIGRATION_OPLOG` (server-side, does not scale with D — §8). @@ -242,9 +415,9 @@ interests**. Excludes `MIGRATION_OPLOG` (server-side, does not scale with D — - Traffic scales **~linearly with D** (~5×), because the only flat term (ingress + terminal streams) is a tiny fraction of the total. -- **Message + metadata delivery dominates at D=5 (~2.8 TB/day, ~75% of bytes)** — the - per-message fan-out, multiplied across every connected device. Coalescing metadata - (§6.7) is the highest-value optimization. +- **Message delivery dominates at D=5 (~2.5 TB/day, ~73% of bytes)** — the per-message + fan-out (human + bot, message + metadata in one event), multiplied across every connected + device. Reducing effective fan-out (§6.7) is the highest-value lever. - **Reconnect storms multiply by D**: a restart triggers up to `U × D ≈ 104k` simultaneous subscription-list fetches (150KB each) — jitter/rate-limit to avoid a thundering herd. @@ -264,7 +437,7 @@ TTL 8 hr. | Flow | Rate | Per day | Bytes/s | Bytes/day | |------|-----:|--------:|--------:|----------:| -| Publish (`migration.oplog.>`) | 200 msg/s | 17.28M | 26 MB/s | ~2.25 TB | +| Publish (`chat.oplog.{siteID}.>`) | 200 msg/s | 17.28M | 26 MB/s | ~2.25 TB | | Consumer delivery (×1 applier) | 200 msg/s | 17.28M | 26 MB/s | ~2.25 TB | | **Total** | **400 msg/s** | **34.56M** | **~52 MB/s** | **~4.49 TB/day** | @@ -296,31 +469,38 @@ JetStream storage at steady state ≈ `publish-rate × retention (TTL) × payloa | Stream | TTL | Pub/day | Payload | Retained msgs | Logical storage | |--------|-----|--------:|---------|--------------:|----------------:| -| `MESSAGES` | 8 hr | 4M | 1KB | 1.33M | 1.3 GB | -| `PUSH_NOTIFICATIONS` | 8 hr | 4M | 0.8KB | 1.33M | 1.1 GB | -| `HRSYNC` | 8 hr | 40K (daily burst) | 0.5KB | 40K | 0.02 GB | -| `MESSAGES_CANONICAL` | 7 day | 5M | 1KB | 35M | 35 GB | -| `INBOX` | 7 day | 1.04M | 0.3KB | 7.3M | 2.2 GB | -| `ROOMS` | 1 day | 1.04M | 0.4KB | 1.04M | 0.4 GB | -| **TOTAL (logical)** | | | | | **~40 GB** | +| `MESSAGES` | 8 hr | 2.1M | 1KB | 0.70M | 0.7 GB | +| `PUSH_NOTIFICATION` | 8 hr | 2.1M | 0.8KB | 0.70M | 0.6 GB | +| `HR` | 8 hr | 40K (daily burst) | 0.5KB | 40K | 0.02 GB | +| `SYSTEM` | 8 hr | 10 | — | negligible | ~0 GB | +| `MESSAGES_CANONICAL` | 1 day | 2.52M | 1KB | 2.52M | 2.5 GB | +| `INBOX` | 7 day | 0.50M | 0.3KB | 3.5M | 1.1 GB | +| `OUTBOX` | 7 day | 0.08M | 0.3KB | 0.59M | 0.2 GB | +| `ROOMS` | 1 day | 0.42M | 0.4KB | 0.42M | 0.2 GB | +| `BOT_MESSAGES_CANONICAL` | 1 day | 1.9M | 1KB | 1.9M | 1.9 GB | +| `BOT_PUSH_NOTIFICATION` | 8 hr | 1.9M | 0.8KB | 0.63M | 0.5 GB | +| `BOT_PLATFORM` | 1 day | 0.1M | 1KB | 0.1M | 0.1 GB | +| **TOTAL (logical)** | | | | | **~8 GB** | Excludes `MIGRATION_OPLOG` storage (separate phase — §8). Notes: -- `MESSAGES_CANONICAL` dominates steady-state storage (7-day retention × full message - bodies). It is the canonical source of truth, so retention is intentional — but it's - the first place to look for disk pressure. -- `HRSYNC` is a once-daily 40K burst; with an 8 hr TTL the whole batch is retained for +- `MESSAGES_CANONICAL` (2.5 GB) and `BOT_MESSAGES_CANONICAL` (1.9 GB) are the canonical + sources of truth; at **1-day** retention their footprint is modest. `INBOX` (1.1 GB, 7-day) + is next; the steady-state storage total is ~8 GB (1-day canonical retention keeps it low). +- `HR` is a once-daily 40K burst; with an 8 hr TTL the whole batch is retained for ~8 hr then expires. Peak ingest is 100 msg/s during the ~7-minute burst. - For TTL < 1 day, retained ≈ `(pub/day) × (TTL_hours / 24)`. ## 10. Per-Fab Traffic Summary Each fab is an independent site (its own NATS). The steady-state model (§6) decomposes -into a **per-message** component (delivery fan-out — ~71% of bytes, ~85% of deliveries) -and a **per-user** component (R/R, presence, member events — ~29% of bytes), so each -fab's total is `per_user × Users + per_message × Msg/day`. Fab 1 reproduces §6.4 exactly; -the rest scale from the same per-unit rates (all other parameters — F, P, R_sub, +into a **per-message** component (delivery fan-out — ~71% of bytes, ~74% of deliveries) +and a **per-user** component (R/R, presence, member events — ~40% of bytes); since message +volume itself scales with users, each fab's total scales ~linearly with its user count. +`Msg/day` below is the **human pipeline** volume (2.1M for Fab 1); the bot pipeline (§6.1) +and server-to-server R/R are included in Deliveries/Traffic and also scale ∝ users. Fab 1 +reproduces §6.4 exactly; the rest scale by user count (all other parameters — F, P, R_sub, R_member, etc. — held equal across fabs). Figures are **steady-state, single connection per user (D=1)**, and **exclude @@ -329,45 +509,53 @@ site independently. Peak ≈ 4× avg. | Fab | Users | Msg/day | Deliveries/day | avg msg/s | peak msg/s | Traffic/day | avg MB/s | |-----|------:|--------:|---------------:|----------:|-----------:|------------:|---------:| -| Fab 1 | 20,789 | 4.00M | ~980M | 11,340 | 45,400 | 0.79 TB | 9.1 | -| Fab 2 | 12,150 | 2.33M | ~571M | 6,610 | 26,400 | 0.46 TB | 5.3 | -| Fab 3 | 2,922 | 0.56M | ~137M | 1,590 | 6,360 | 0.11 TB | 1.3 | -| Fab 4 | 2,078 | 0.39M | ~96M | 1,110 | 4,440 | 0.08 TB | 0.9 | -| Fab 5 | 17,061 | 3.38M | ~825M | 9,550 | 38,200 | 0.66 TB | 7.6 | -| Fab 6 | 4,138 | 0.79M | ~194M | 2,240 | 8,970 | 0.16 TB | 1.8 | -| Fab 7 | 2,199 | 0.42M | ~103M | 1,190 | 4,770 | 0.08 TB | 1.0 | -| Fab 8 | 3,244 | 0.62M | ~152M | 1,760 | 7,040 | 0.12 TB | 1.4 | -| Fab 9 | 4,492 | 0.86M | ~211M | 2,440 | 9,760 | 0.17 TB | 2.0 | -| Fab 10 | 4,754 | 0.90M | ~221M | 2,560 | 10,240 | 0.18 TB | 2.1 | -| Fab 11 | 5,537 | 1.00M | ~247M | 2,860 | 11,450 | 0.20 TB | 2.3 | -| Fab 12 | 4,356 | 0.83M | ~204M | 2,360 | 9,430 | 0.16 TB | 1.9 | -| Fab 13 | 2,227 | 0.42M | ~103M | 1,190 | 4,780 | 0.08 TB | 1.0 | -| Fab 14 | 5,215 | 1.00M | ~245M | 2,840 | 11,350 | 0.20 TB | 2.3 | - -Per-fab byte split holds at the §6.4 ratio for every site: **core delivery ~72%**, R/R -~23%, JetStream streams ~5%. For multi-device (D), scale Deliveries/day, Traffic/day, and -MB/s by the rule in §7 (≈ ×D); `MIGRATION_OPLOG` per fab is per §8 and independent of D. +| Fab 1 | 20,789 | 2.10M | ~529M | 6,130 | 24,520 | 0.72 TB | 8.3 | +| Fab 2 | 12,150 | 1.23M | ~309M | 3,580 | 14,330 | 0.42 TB | 4.9 | +| Fab 3 | 2,922 | 0.30M | ~74M | 860 | 3,450 | 0.10 TB | 1.2 | +| Fab 4 | 2,078 | 0.21M | ~53M | 610 | 2,450 | 0.07 TB | 0.8 | +| Fab 5 | 17,061 | 1.72M | ~435M | 5,030 | 20,120 | 0.59 TB | 6.8 | +| Fab 6 | 4,138 | 0.42M | ~105M | 1,220 | 4,880 | 0.14 TB | 1.7 | +| Fab 7 | 2,199 | 0.22M | ~56M | 650 | 2,590 | 0.08 TB | 0.9 | +| Fab 8 | 3,244 | 0.33M | ~83M | 960 | 3,830 | 0.11 TB | 1.3 | +| Fab 9 | 4,492 | 0.45M | ~114M | 1,320 | 5,300 | 0.16 TB | 1.8 | +| Fab 10 | 4,754 | 0.48M | ~121M | 1,400 | 5,610 | 0.16 TB | 1.9 | +| Fab 11 | 5,537 | 0.56M | ~141M | 1,630 | 6,530 | 0.19 TB | 2.2 | +| Fab 12 | 4,356 | 0.44M | ~111M | 1,280 | 5,140 | 0.15 TB | 1.7 | +| Fab 13 | 2,227 | 0.22M | ~57M | 660 | 2,630 | 0.08 TB | 0.9 | +| Fab 14 | 5,215 | 0.53M | ~133M | 1,540 | 6,150 | 0.18 TB | 2.1 | + +Per-fab byte split holds at the §6.4 ratio for every site: **core delivery ~75%**, R/R +~21%, JetStream streams (incl. bot) ~4%. For multi-device (D), scale Deliveries/day, +Traffic/day, and MB/s by the rule in §7 (≈ ×D); `MIGRATION_OPLOG` per fab is per §8 and +independent of D. ## 11. Caveats -- **R_member = 50/day/user** is the member-change slice of the 250 room ops (§4); all - *(member-driven)* lines scale with it. -- **Presence is not implemented** ("future"). The estimate assumes ~20 event-driven - status changes/user/day; a heartbeat design would explode this. -- **Presence health-check** (ping/pong/hello/bye, C_hc = 4/user/day) is modeled as - publishing to `chat.user.{account}.event.presence` and fanning out to the user's P - watchers (~1.66M deliveries/day, negligible bytes). If these are server-only keepalives - with no watcher fan-out, the delivery count drops ~20× (to ~83k/day). -- **OUTBOX and federated INBOX inflow are excluded** from this estimate per scope; INBOX - here reflects local member events only. -- **PUSH_NOTIFICATIONS / HRSYNC / MIGRATION_OPLOG are pre-merge** — revisit when the - real specs land. +- **R_member = 20/day/user** is the member-change slice of the 100 room ops (§4); all + *(member-driven)* lines (ROOMS, OUTBOX, INBOX, member fan-out) scale with it. +- **Presence** assumes ~10 event-driven status changes/user/day (`C_pres`); a heartbeat + design would explode this. +- **Presence health-check** (hello/ping/activity/bye, C_hc = 4/user/day) is modeled as + publishing to `chat.user.{account}.event.presence.{siteID}.{op}` and fanning out to the + user's P watchers (~1.66M deliveries/day, negligible bytes). If these are server-only + keepalives with no watcher fan-out, the delivery count drops ~20× (to ~83k/day). +- **Cross-site federation is now modeled** (§6.1/§9): a per-site `OUTBOX_{siteID}` + (`chat.outbox.{siteID}.>`) whose consumer **outbox-worker** forwards each event to the + destination's `INBOX_{siteID}` (`chat.inbox.{siteID}.>`). OUTBOX carries only the cross-site + subset of member changes (`≈ R_member × U × f_fed`, f_fed ~20%); the symmetric INBOX inflow + scales the same. Volumes are estimates pending telemetry. +- **PUSH_NOTIFICATION / HR / BOT_* / MIGRATION_OPLOG streams are defined in + `pkg/stream/stream.go`** (names/subjects per §2.1); their **traffic volumes** here remain + estimates — revisit against production telemetry. - **MIGRATION_OPLOG is a separate ~2-month phase (§8)** — reported standalone and never - summed with steady-state. At ~4.5 TB/day traffic and ~0.75 TB storage it still exceeds - the entire steady-state load (~0.79 TB/day) while running; isolate it and reclaim the + summed with steady-state. At ~4.5 TB/day traffic and ~0.75 TB storage it dwarfs + the entire steady-state load (~0.72 TB/day) while running; isolate it and reclaim the capacity after cutover. -- **Message + metadata delivery is the dominant steady-state cost** (~66% of bytes, - ~9,300 msg/s) — driven by F = 100. With R_sub now 10/day, subscription R/R is a minor - term; coalescing per-message metadata (§6.7) is the highest-value lever. +- **Message delivery is the dominant steady-state cost** (~71% of bytes, ~4,514 msg/s) — + **human + bot** messages in one combined event, fanned ×F = 100 on (M_room + M_bot) = 3.9M. + Reducing effective fan-out (§6.7) is the highest-value lever. +- **Bot pipeline (`BOT_*`) is folded into the steady-state tables** (§6.1 traffic, §9 + storage, §4 params) — the 4M total = 2.1M human + 1.9M bot (§4 split note). Bot room + fan-out is core-NATS (~190M deliveries/day, §6.2 note) and not yet in the bytes. - **Peak factor k ≈ 4** assumes 80% of traffic in a 10-hour window with 2× intra-window peaking. Figures are first-order; validate against production telemetry. From 6d3fd5fa9ba69056801a3f71034ec993a6b102fe Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 8 Jul 2026 02:53:45 +0000 Subject: [PATCH 2/2] =?UTF-8?q?docs(nats-traffic):=20raise=20bot/user?= =?UTF-8?q?=E2=86=92bot=20volumes;=20key=20rotation=20on=20remove=20only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Volume adjustments (total messages 4.0M → 4.5M): - BOT_PLATFORM (user→bot) 0.1M → 0.5M, so M (MESSAGES/MESSAGES_CANONICAL) 2.1M → 2.5M; M_push set to M_room (2.0M) since user→bot forwards to BOT_PLATFORM and doesn't push a human - BOT_MESSAGES_CANONICAL / BOT_PUSH_NOTIFICATION 1.9M → 2.0M - room.key: rotation only on member-remove (add just hands the current key to the new member), so R_member × U × F ÷ 2 (~42M → ~21M deliveries) Recomputed §4, §6.1 (JS subtotal ~39M/~36 GB), §6.2 (message row (M_room+M_bot)×F = 400M/520 GB; core ~470M/~543 GB), §6.4 (~0.52B/day, ~0.73 TB), §6.6, §7.2/§7.3, §9 (~9 GB), §10 (all 14 fabs), §11. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01H7c7aaLBkrM3zDiyT67GL8 --- docs/nats-traffic-estimation.md | 138 ++++++++++++++++---------------- 1 file changed, 69 insertions(+), 69 deletions(-) diff --git a/docs/nats-traffic-estimation.md b/docs/nats-traffic-estimation.md index 350a0f3f5..9b7c7ad82 100644 --- a/docs/nats-traffic-estimation.md +++ b/docs/nats-traffic-estimation.md @@ -28,7 +28,7 @@ Each site runs its own NATS server, so these figures are per-site. > (`chat.outbox.{siteID}.>`) whose consumer outbox-worker forwards each event to the > destination's flat `INBOX_{siteID}` (`chat.inbox.{siteID}.>`). **Bot traffic** runs on a > parallel `BOT_*` pipeline. Both federation and the bot pipeline are folded into the -> steady-state tables (§6.1/§9); the 4M/day total = 2.1M human + 1.9M bot (§4 split note). +> steady-state tables (§6.1/§9); the 4.5M/day total = 2.5M human + 2.0M bot (§4 split note). ## 2. Stream & Endpoint Inventory @@ -68,7 +68,7 @@ observed independently: **Broadcast Worker** publishes; **Bot Webhook Worker** delivers to external bot platforms. Bot traffic is sized alongside the human pipeline in §6.1/§9 (params in §4); the 4M/day -total = 2.1M human + 1.9M bot. Payload sizes are in §3. +total = 2.5M human + 2.0M bot. Payload sizes are in §3. **Federation via OUTBOX → INBOX.** Cross-site federation uses a per-site `OUTBOX_{siteID}` stream (`chat.outbox.{siteID}.>`): a service at the origin site publishes each cross-site @@ -170,7 +170,7 @@ for inventory completeness, delivered to watchers in §6.2. | Users | U | 20,789 | | Subjects subscribed per connection | S | 100 (user + room) | | Presence subjects watched per connection | P | **20** | -| Human messages per day (pipeline) | M | 2,100,000 *(user↔room 2.0M + user→bot 0.1M via broadcast-worker→BOT_PLATFORM). 4M total = 2.1M human + 1.9M bot.* | +| Human messages per day (pipeline) | M | 2,500,000 *(user↔room 2.0M + user→bot 0.5M via broadcast-worker→BOT_PLATFORM). 4.5M total = 2.5M human + 2.0M bot.* | | Human messages fanned ×F to rooms | M_room | 2,000,000 *(user↔room only; botDM/mention go to BOT_PLATFORM, no ×F)* | | Recipients per room (fan-out) | F | 100 | | Subscription R/R ops per day per connection | R_sub | **10** | @@ -183,12 +183,12 @@ for inventory completeness, delivered to watchers in §6.2. | Presence batch-query (initial-state) ops per day per user | R_presq | ~5 | | Cross-site fan-out (remote sites per federated query) | N | ~3 | | Federated fraction of member changes (→ OUTBOX / INBOX inflow) | f_fed | ~20% | -| Push notifications per day (human, = M) | M_push | 2,100,000 | +| Push notifications per day (human room msgs, = M_room) | M_push | 2,000,000 | | HR sync records per daily run (burst @ 100 msg/s) | M_hr | 40,000 | | Migration oplog QPS (sustained 24/7, 130KB payload, 1 consumer) | Q_mig | 200 | -| Bot-originated messages per day (→ BOT_MESSAGES_CANONICAL) | M_bot | 1,900,000 | -| Bot push notifications per day (= bot canonical input) | M_bot_push | 1,900,000 | -| User→bot events per day (→ BOT_PLATFORM, forwarded by broadcast-worker) | M_bot_platform | 100,000 | +| Bot-originated messages per day (→ BOT_MESSAGES_CANONICAL) | M_bot | 2,000,000 | +| Bot push notifications per day (= bot canonical input) | M_bot_push | 2,000,000 | +| User→bot events per day (→ BOT_PLATFORM, forwarded by broadcast-worker) | M_bot_platform | 500,000 | | Bot fan-out per message (room delivery, core-NATS) | F_bot | 100 | | Peak factor (business-hours concentration) | k | ~4× | @@ -197,10 +197,10 @@ for inventory completeness, delivered to watchers in §6.2. > federated inflow) carries only the cross-site subset `× f_fed`. The remaining ~80 room ops > are read-only Room R/R. Lines tagged *(member-driven)* scale linearly with R_member. -> **4M/day traffic split** (bot pipeline folded into §6.1/§9): user↔room 2.0M and user→bot -> 0.1M flow through `MESSAGES`→`MESSAGES_CANONICAL` (M = 2.1M; broadcast-worker forwards the -> 0.1M user→bot subset to `BOT_PLATFORM`, no ×F). Bot-originated 1.9M enters via Bot Msg -> Handler → `BOT_MESSAGES_CANONICAL`. Total = 2.1M human + 1.9M bot = 4.0M. +> **4.5M/day traffic split** (bot pipeline folded into §6.1/§9): user↔room 2.0M and user→bot +> 0.5M flow through `MESSAGES`→`MESSAGES_CANONICAL` (M = 2.5M; broadcast-worker forwards the +> 0.5M user→bot subset to `BOT_PLATFORM`, no ×F). Bot-originated 2.0M enters via Bot Msg +> Handler → `BOT_MESSAGES_CANONICAL`. Total = 2.5M human + 2.0M bot = 4.5M. ## 5. Methodology — ingress vs. fan-out @@ -231,20 +231,20 @@ counts publishes + consumer deliveries (JetStream), deliveries (core), or reques | Stream | Driver | Pub/day | Deliveries/day | Ops/day | avg msg/s | Payload | Bytes/day | |--------|--------|--------:|---------------:|--------:|----------:|---------|----------:| -| `MESSAGES` | M (client → gatekeeper) | 2.1M | 2.1M | 4.2M | 49 | 1KB | 4.2 GB | -| `MESSAGES_CANONICAL` | (M + member sys) × (1 pub + 4 consumers) | 2.52M | 10.1M | 12.6M | 146 | 1KB | 12.6 GB | +| `MESSAGES` | M (client → gatekeeper) | 2.5M | 2.5M | 5M | 58 | 1KB | 5 GB | +| `MESSAGES_CANONICAL` | (M + member sys) × (1 pub + 4 consumers) | 2.92M | 11.7M | 14.6M | 169 | 1KB | 14.6 GB | | `ROOMS` *(member-driven)* | R_member × U | 0.42M | 0.42M | 0.83M | 10 | 0.4KB | 0.33 GB | | `INBOX` *(member-driven)* | (local feed R_member×U + federated inflow ×f_fed) × 2 consumers | 0.50M | 1.0M | 1.5M | 17 | 0.3KB | 0.45 GB | | `OUTBOX` *(member-driven)* | R_member × U × f_fed (1 pub + outbox-worker) | 0.08M | 0.08M | 0.17M | 2 | 0.3KB | 0.05 GB | -| `PUSH_NOTIFICATION` | M_push × (1 pub + 1 consumer) | 2.1M | 2.1M | 4.2M | 49 | 0.8KB | 3.4 GB | +| `PUSH_NOTIFICATION` | M_push × (1 pub + 1 consumer) | 2M | 2M | 4M | 46 | 0.8KB | 3.2 GB | | `HR` | M_hr × (1 pub + 1 consumer); 100 msg/s burst | 40K | 40K | 80K | ~1 (200/s burst) | 0.5KB | 0.04 GB | | `SYSTEM` | system/admin events (~10/day) | 10 | 10 | 20 | ~0 | — | ~0 GB | -| `BOT_MESSAGES_CANONICAL` | M_bot × (1 pub + 3 consumers) | 1.9M | 5.7M | 7.6M | 88 | 1KB | 7.6 GB | -| `BOT_PUSH_NOTIFICATION` | M_bot_push × (1 pub + 1 consumer) | 1.9M | 1.9M | 3.8M | 44 | 0.8KB | 3.0 GB | -| `BOT_PLATFORM` | M_bot_platform × (1 pub + Bot Webhook Worker) | 0.1M | 0.1M | 0.2M | 2 | 1KB | 0.2 GB | -| **JetStream subtotal** | | | | **~35M** | **~408** | | **~32 GB** | +| `BOT_MESSAGES_CANONICAL` | M_bot × (1 pub + 3 consumers) | 2M | 6M | 8M | 93 | 1KB | 8 GB | +| `BOT_PUSH_NOTIFICATION` | M_bot_push × (1 pub + 1 consumer) | 2M | 2M | 4M | 46 | 0.8KB | 3.2 GB | +| `BOT_PLATFORM` | M_bot_platform × (1 pub + Bot Webhook Worker) | 0.5M | 0.5M | 1M | 12 | 1KB | 1 GB | +| **JetStream subtotal** | | | | **~39M** | **~454** | | **~36 GB** | -`MESSAGES_CANONICAL` pub = 2.1M human messages + ~0.42M member-change system messages. +`MESSAGES_CANONICAL` pub = 2.5M human messages + ~0.42M member-change system messages. **OUTBOX** carries only the **cross-site subset** of member changes (`f_fed ≈ 20%` — those touching a federated room); its lone consumer **outbox-worker** forwards each to the destination site's `INBOX`. **INBOX** carries the local-origin search feed (all member @@ -258,14 +258,14 @@ pipeline (4M split in §4); bot room fan-out is core-NATS (§6.2 note), not show | Subject | Driver | Pub/day | Deliveries/day | avg msg/s | Payload | Bytes/day | |---------|--------|--------:|---------------:|----------:|---------|----------:| -| `RoomEvent`/`UserRoomEvent` — message + `lastMessageAt`, **human + bot** (`chat.room.*.event` +DM `chat.user.*.event.room`) | (M_room + M_bot) × F | 3.9M | 390M | 4,514 | ~1.3KB | 507 GB | +| `RoomEvent`/`UserRoomEvent` — message + `lastMessageAt`, **human + bot** (`chat.room.*.event` +DM `chat.user.*.event.room`) | (M_room + M_bot) × F | 4.0M | 400M | 4,630 | ~1.3KB | 520 GB | | member-change — sys msg on `chat.room.*.event`, member event on `chat.room.*.event.member` *(member-driven)* | R_member × U × F | 0.42M | 41.6M | 481 | 0.4KB | 17 GB | | `chat.user.*.event.subscription.update` (`SubscriptionUpdate`) | R_room × U (read/mute/favorite; per-user) | 2.08M | 2.08M | 24 | 0.5KB | 1 GB | -| `chat.user.*.event.room.key` (`RoomKeyUpdate`) *(member-driven)* | R_member × U × F (key rotation → members) | 0.42M | 42M | 481 | 0.2KB | 8.3 GB | +| `chat.user.*.event.room.key` (`RoomKeyUpdate`) *(member-driven)* | R_member × U × F ÷ 2 (rotation on member-remove only) | 0.21M | 21M | 241 | 0.2KB | 4.2 GB | | `chat.user.presence.state.*` (`PresenceState`, status broadcast) | U × C_pres × P | 0.21M | 4.16M | 48 | 0.15KB | 0.6 GB | | `chat.user.*.event.presence.{siteID}.{hello,ping,activity,bye}` — **client→server** (ingress; §2.3), watcher fan-out modeled here | U × C_hc × P | 0.083M | 1.66M | 19 | ~0.05KB | 0.08 GB | | `chat.room.*.event.typing` | active room only | — | — | — | — | *(ignored)* | -| **Core delivery subtotal** | | | **~481M** | **~5,570** | | **~534 GB** | +| **Core delivery subtotal** | | | **~470M** | **~5,440** | | **~543 GB** | **Reconciling with §2.2 — the message/metadata/member-change rows.** @@ -291,15 +291,15 @@ pipeline (4M split in §4); bot room fan-out is core-NATS (§6.2 note), not show **and** on that user's own `read` / `mute` / `favorite` / `role` / `rename` actions — so it scales with **`R_room`** (the state-mutating room ops), *not* `R_member`. `R_room × U` here is an upper bound: read-only room RPCs (member.list, statuses, mentionable) don't fire it. -- **`RoomKeyUpdate`** (`chat.user.*.event.room.key`) is **member-driven**: `R_member` is - exactly the add/remove ops, and every one rotates the room key and pushes a new per-member - key event — so it fans out `≈ R_member × U × F`, ~42M deliveries/day at ~0.2KB. Upper bound - (an *add* pushes only to new members, not all F; assumes E2E rooms). +- **`RoomKeyUpdate`** (`chat.user.*.event.room.key`) is **member-driven**: only a member + **remove** rotates the room key (an add just hands the current key to the new member), so + ~half of the `R_member` ops trigger a full ×F rotation — `≈ R_member × U × F ÷ 2`, ~21M + deliveries/day at ~0.2KB (E2E rooms). The presence health-check row is a **client→server publish** (`RegisterVoid`, §2.3) modeled here for its optional watcher fan-out, not a true server→client subject. -Bot room delivery (Bot Broadcast Worker → members, `M_bot × F` ≈ ~190M deliveries/day) is +Bot room delivery (Bot Broadcast Worker → members, `M_bot × F` ≈ ~200M deliveries/day) is now **included** in the message row above — bots post to rooms and fan out to members on the same `RoomEvent`/`UserRoomEvent` as human messages. @@ -339,10 +339,10 @@ doesn't inflate it. | Layer | Ops or deliveries/day | avg msg/s | Bytes/day | |-------|----------------------:|----------:|----------:| -| JetStream streams (incl. `BOT_*`, server-side/flat) | ~35M | ~408 | ~32 GB | -| Core delivery subjects (scales with D) | ~481M | ~5,570 | ~534 GB | +| JetStream streams (incl. `BOT_*`, server-side/flat) | ~39M | ~454 | ~36 GB | +| Core delivery subjects (scales with D) | ~470M | ~5,440 | ~543 GB | | Client R/R (req + resp, scales with D) | ~11M | ~130 | ~149 GB | -| **TOTAL (steady-state, D=1)** | **~0.53B/day** | **~6,100/s avg · ~24,400/s peak** | **~0.72 TB/day** | +| **TOTAL (steady-state, D=1)** | **~0.52B/day** | **~6,020/s avg · ~24,100/s peak** | **~0.73 TB/day** | Plus **server-to-server R/R** (§6.3) — server-side, **flat with D** — ~2.1M ops/day (req + resp), ~25/s, ~4 GB/day per site. Like the JetStream layer it does not scale with D (§7); @@ -362,14 +362,14 @@ At the single-connection baseline: 20,789 connections × (S=100 + P=20) = ### 6.6 Bottlenecks (steady-state) -- **Message delivery = ~507 GB/day & ~390M deliveries/day (~4,514/s)** — **human + bot** +- **Message delivery = ~520 GB/day & ~400M deliveries/day (~4,630/s)** — **human + bot** messages fanned ×F (one combined event with metadata); the dominant term for *both* - bandwidth (~71% of bytes) and message rate, driven by F = 100 on (M_room + M_bot) = 3.9M. + bandwidth (~71% of bytes) and message rate, driven by F = 100 on (M_room + M_bot) = 4.0M. - **Message history R/R = ~94 GB/day** — the largest R/R term now that R_room dropped to 100/day (Room R/R ~21 GB). R/R is ~32% of steady-state bytes. - **Member-change fan-out = ~42M deliveries/day** — driven by `R_member = 20/day/user` - (sys-message + member-event broadcast ×F); **room-key rotation** adds another ~42M/day - (every member op rotates → ×F), and OUTBOX/INBOX federation ~1.7M ops/day. + (sys-message + member-event broadcast ×F); **room-key rotation** adds another ~21M/day + (member-remove only → ×F), and OUTBOX/INBOX federation ~1.7M ops/day. ### 6.7 Optimization levers @@ -401,12 +401,12 @@ connection state scale with D.** Effective fan-out becomes `F × D = 500` per me | Layer | D=1 deliveries/day | D=5 deliveries/day | D=1 bytes/day | D=5 bytes/day | |-------|-------------------:|-------------------:|--------------:|--------------:| -| JetStream streams (incl. bot) | ~35M | ~35M *(flat)* | ~32 GB | ~32 GB | -| Core delivery | ~481M | ~2,405M | ~534 GB | ~2,670 GB | +| JetStream streams (incl. bot) | ~39M | ~39M *(flat)* | ~36 GB | ~36 GB | +| Core delivery | ~470M | ~2,350M | ~543 GB | ~2,715 GB | | Client R/R (req+resp) | ~11M | ~56M | ~149 GB | ~745 GB | | Server-to-server R/R | ~2.1M | ~2.1M *(flat)* | ~4 GB | ~4 GB | -| **TOTAL (steady-state)** | **~0.53B** | **~2.50B** | **~0.72 TB** | **~3.45 TB** | -| **avg / peak msg/s** | ~6.1k / ~25k | **~29k / ~116k** | | | +| **TOTAL (steady-state)** | **~0.52B** | **~2.45B** | **~0.73 TB** | **~3.50 TB** | +| **avg / peak msg/s** | ~6.0k / ~24k | **~28k / ~113k** | | | Connection state at D=5: **~104k connections** × (100 + 20) = **~12.5M subscription interests**. Excludes `MIGRATION_OPLOG` (server-side, does not scale with D — §8). @@ -415,7 +415,7 @@ interests**. Excludes `MIGRATION_OPLOG` (server-side, does not scale with D — - Traffic scales **~linearly with D** (~5×), because the only flat term (ingress + terminal streams) is a tiny fraction of the total. -- **Message delivery dominates at D=5 (~2.5 TB/day, ~73% of bytes)** — the per-message +- **Message delivery dominates at D=5 (~2.5 TB/day, ~74% of bytes)** — the per-message fan-out (human + bot, message + metadata in one event), multiplied across every connected device. Reducing effective fan-out (§6.7) is the highest-value lever. - **Reconnect storms multiply by D**: a restart triggers up to `U × D ≈ 104k` @@ -469,25 +469,25 @@ JetStream storage at steady state ≈ `publish-rate × retention (TTL) × payloa | Stream | TTL | Pub/day | Payload | Retained msgs | Logical storage | |--------|-----|--------:|---------|--------------:|----------------:| -| `MESSAGES` | 8 hr | 2.1M | 1KB | 0.70M | 0.7 GB | -| `PUSH_NOTIFICATION` | 8 hr | 2.1M | 0.8KB | 0.70M | 0.6 GB | +| `MESSAGES` | 8 hr | 2.5M | 1KB | 0.83M | 0.8 GB | +| `PUSH_NOTIFICATION` | 8 hr | 2.0M | 0.8KB | 0.67M | 0.5 GB | | `HR` | 8 hr | 40K (daily burst) | 0.5KB | 40K | 0.02 GB | | `SYSTEM` | 8 hr | 10 | — | negligible | ~0 GB | -| `MESSAGES_CANONICAL` | 1 day | 2.52M | 1KB | 2.52M | 2.5 GB | +| `MESSAGES_CANONICAL` | 1 day | 2.92M | 1KB | 2.92M | 2.9 GB | | `INBOX` | 7 day | 0.50M | 0.3KB | 3.5M | 1.1 GB | | `OUTBOX` | 7 day | 0.08M | 0.3KB | 0.59M | 0.2 GB | | `ROOMS` | 1 day | 0.42M | 0.4KB | 0.42M | 0.2 GB | -| `BOT_MESSAGES_CANONICAL` | 1 day | 1.9M | 1KB | 1.9M | 1.9 GB | -| `BOT_PUSH_NOTIFICATION` | 8 hr | 1.9M | 0.8KB | 0.63M | 0.5 GB | -| `BOT_PLATFORM` | 1 day | 0.1M | 1KB | 0.1M | 0.1 GB | -| **TOTAL (logical)** | | | | | **~8 GB** | +| `BOT_MESSAGES_CANONICAL` | 1 day | 2.0M | 1KB | 2.0M | 2.0 GB | +| `BOT_PUSH_NOTIFICATION` | 8 hr | 2.0M | 0.8KB | 0.67M | 0.5 GB | +| `BOT_PLATFORM` | 1 day | 0.5M | 1KB | 0.5M | 0.5 GB | +| **TOTAL (logical)** | | | | | **~9 GB** | Excludes `MIGRATION_OPLOG` storage (separate phase — §8). Notes: -- `MESSAGES_CANONICAL` (2.5 GB) and `BOT_MESSAGES_CANONICAL` (1.9 GB) are the canonical +- `MESSAGES_CANONICAL` (2.9 GB) and `BOT_MESSAGES_CANONICAL` (2.0 GB) are the canonical sources of truth; at **1-day** retention their footprint is modest. `INBOX` (1.1 GB, 7-day) - is next; the steady-state storage total is ~8 GB (1-day canonical retention keeps it low). + is next; the steady-state storage total is ~9 GB (1-day canonical retention keeps it low). - `HR` is a once-daily 40K burst; with an 8 hr TTL the whole batch is retained for ~8 hr then expires. Peak ingest is 100 msg/s during the ~7-minute burst. - For TTL < 1 day, retained ≈ `(pub/day) × (TTL_hours / 24)`. @@ -495,10 +495,10 @@ Notes: ## 10. Per-Fab Traffic Summary Each fab is an independent site (its own NATS). The steady-state model (§6) decomposes -into a **per-message** component (delivery fan-out — ~71% of bytes, ~74% of deliveries) +into a **per-message** component (delivery fan-out — ~71% of bytes, ~77% of deliveries) and a **per-user** component (R/R, presence, member events — ~40% of bytes); since message volume itself scales with users, each fab's total scales ~linearly with its user count. -`Msg/day` below is the **human pipeline** volume (2.1M for Fab 1); the bot pipeline (§6.1) +`Msg/day` below is the **human pipeline** volume (2.5M for Fab 1); the bot pipeline (§6.1) and server-to-server R/R are included in Deliveries/Traffic and also scale ∝ users. Fab 1 reproduces §6.4 exactly; the rest scale by user count (all other parameters — F, P, R_sub, R_member, etc. — held equal across fabs). @@ -509,23 +509,23 @@ site independently. Peak ≈ 4× avg. | Fab | Users | Msg/day | Deliveries/day | avg msg/s | peak msg/s | Traffic/day | avg MB/s | |-----|------:|--------:|---------------:|----------:|-----------:|------------:|---------:| -| Fab 1 | 20,789 | 2.10M | ~529M | 6,130 | 24,520 | 0.72 TB | 8.3 | -| Fab 2 | 12,150 | 1.23M | ~309M | 3,580 | 14,330 | 0.42 TB | 4.9 | -| Fab 3 | 2,922 | 0.30M | ~74M | 860 | 3,450 | 0.10 TB | 1.2 | -| Fab 4 | 2,078 | 0.21M | ~53M | 610 | 2,450 | 0.07 TB | 0.8 | -| Fab 5 | 17,061 | 1.72M | ~435M | 5,030 | 20,120 | 0.59 TB | 6.8 | -| Fab 6 | 4,138 | 0.42M | ~105M | 1,220 | 4,880 | 0.14 TB | 1.7 | -| Fab 7 | 2,199 | 0.22M | ~56M | 650 | 2,590 | 0.08 TB | 0.9 | -| Fab 8 | 3,244 | 0.33M | ~83M | 960 | 3,830 | 0.11 TB | 1.3 | -| Fab 9 | 4,492 | 0.45M | ~114M | 1,320 | 5,300 | 0.16 TB | 1.8 | -| Fab 10 | 4,754 | 0.48M | ~121M | 1,400 | 5,610 | 0.16 TB | 1.9 | -| Fab 11 | 5,537 | 0.56M | ~141M | 1,630 | 6,530 | 0.19 TB | 2.2 | -| Fab 12 | 4,356 | 0.44M | ~111M | 1,280 | 5,140 | 0.15 TB | 1.7 | -| Fab 13 | 2,227 | 0.22M | ~57M | 660 | 2,630 | 0.08 TB | 0.9 | -| Fab 14 | 5,215 | 0.53M | ~133M | 1,540 | 6,150 | 0.18 TB | 2.1 | +| Fab 1 | 20,789 | 2.50M | ~523M | 6,050 | 24,210 | 0.73 TB | 8.5 | +| Fab 2 | 12,150 | 1.46M | ~306M | 3,540 | 14,150 | 0.43 TB | 4.9 | +| Fab 3 | 2,922 | 0.35M | ~73M | 850 | 3,400 | 0.10 TB | 1.2 | +| Fab 4 | 2,078 | 0.25M | ~52M | 600 | 2,420 | 0.07 TB | 0.8 | +| Fab 5 | 17,061 | 2.05M | ~429M | 4,970 | 19,870 | 0.60 TB | 7.0 | +| Fab 6 | 4,138 | 0.50M | ~104M | 1,200 | 4,820 | 0.15 TB | 1.7 | +| Fab 7 | 2,199 | 0.26M | ~55M | 640 | 2,560 | 0.08 TB | 0.9 | +| Fab 8 | 3,244 | 0.39M | ~82M | 940 | 3,780 | 0.11 TB | 1.3 | +| Fab 9 | 4,492 | 0.54M | ~113M | 1,310 | 5,230 | 0.16 TB | 1.8 | +| Fab 10 | 4,754 | 0.57M | ~120M | 1,380 | 5,540 | 0.17 TB | 1.9 | +| Fab 11 | 5,537 | 0.67M | ~139M | 1,610 | 6,450 | 0.19 TB | 2.3 | +| Fab 12 | 4,356 | 0.52M | ~110M | 1,270 | 5,070 | 0.15 TB | 1.8 | +| Fab 13 | 2,227 | 0.27M | ~56M | 650 | 2,590 | 0.08 TB | 0.9 | +| Fab 14 | 5,215 | 0.63M | ~131M | 1,520 | 6,070 | 0.18 TB | 2.1 | Per-fab byte split holds at the §6.4 ratio for every site: **core delivery ~75%**, R/R -~21%, JetStream streams (incl. bot) ~4%. For multi-device (D), scale Deliveries/day, +~21%, JetStream streams (incl. bot) ~5%. For multi-device (D), scale Deliveries/day, Traffic/day, and MB/s by the rule in §7 (≈ ×D); `MIGRATION_OPLOG` per fab is per §8 and independent of D. @@ -549,13 +549,13 @@ independent of D. estimates — revisit against production telemetry. - **MIGRATION_OPLOG is a separate ~2-month phase (§8)** — reported standalone and never summed with steady-state. At ~4.5 TB/day traffic and ~0.75 TB storage it dwarfs - the entire steady-state load (~0.72 TB/day) while running; isolate it and reclaim the + the entire steady-state load (~0.73 TB/day) while running; isolate it and reclaim the capacity after cutover. -- **Message delivery is the dominant steady-state cost** (~71% of bytes, ~4,514 msg/s) — - **human + bot** messages in one combined event, fanned ×F = 100 on (M_room + M_bot) = 3.9M. +- **Message delivery is the dominant steady-state cost** (~71% of bytes, ~4,630 msg/s) — + **human + bot** messages in one combined event, fanned ×F = 100 on (M_room + M_bot) = 4.0M. Reducing effective fan-out (§6.7) is the highest-value lever. - **Bot pipeline (`BOT_*`) is folded into the steady-state tables** (§6.1 traffic, §9 - storage, §4 params) — the 4M total = 2.1M human + 1.9M bot (§4 split note). Bot room + storage, §4 params) — the 4.5M total = 2.5M human + 2.0M bot (§4 split note). Bot room fan-out is core-NATS (~190M deliveries/day, §6.2 note) and not yet in the bytes. - **Peak factor k ≈ 4** assumes 80% of traffic in a 10-hour window with 2× intra-window peaking. Figures are first-order; validate against production telemetry.