Skip to content

feat(server): c1M P1 follow-ups — TLS task-parking (47→26 KB), registration handoff, ECANCELED spin fix - #424

Merged
TinDang97 merged 3 commits into
mainfrom
feat/c1m-followups
Jul 29, 2026
Merged

feat(server): c1M P1 follow-ups — TLS task-parking (47→26 KB), registration handoff, ECANCELED spin fix#424
TinDang97 merged 3 commits into
mainfrom
feat/c1m-followups

Conversation

@TinDang97

@TinDang97 TinDang97 commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

c1M campaign round 6 — the three follow-ups documented in round 5 (PR #422) and the v0.8.3 release, plus a serious latent bug each of them flushed out. Two commits:

1. fix(server): registration handoff across park/wake + migrated-conn parking

  • Registration handoff (closes the round-5 review's Low): the data-wake previously deregistered then re-registered the client across a task-scheduling boundary. Fixing it exposed a worse bug: the fresh registration silently reset the connection's CLIENT SETNAME and age on every wake (register() inserts name: None; only the SETNAME dispatch path sets it). The watcher now hands its RegistryGuard through to the resumed handler (ParkArgs::kept_registration + new client_registry::live_handle()); the entry — name, connected-at, kill state, counters — persists unbroken. Red/green: resumed_connection_keeps_registry_identity (RED before: name= empty, age=0 after wake).
  • Migrated-conn parking: the Linux migrated-spawn site routes ParkIdle like the primary accept site.

2. feat(server): TLS task-exit parking — idle TLS 47.0 → 26.0 KB/conn (−45 %)

  • Vendored (moon-patch style, no new unsafe): Stream::io_ref() (raw-transport readiness passthrough) + Stream::task_park_safe() (park veto while ANY TLS-stack state can't be signalled by fd readability: wrapper bytes or pending EOF/error status via new is_drained(), decrypted plaintext, received close_notify, pending output; a partial deframer record is park-safe). Wrapper moon-patch tests 7/7.
  • moon: TLS sets SUPPORTS_TASK_PARK=true; ParkWatchable trait lets the watcher await readable(false) on plain TCP or io_ref().readable(false) for TLS; watcher/resume helpers genericized; TLS accept site routes ParkIdle.

The spin bug (found by E11's first ON leg; affects plain TCP too)

The idle-park arms treated every read error as the sweep's cancel. With task parking, a dead connection whose error leaves the fd permanently readable — a TLS client FIN without close_notify surfaces as a read error (not Ok(0)), same for a plain-TCP RST — spun park→wake→park at 100 % CPU forever: the first E11 run ended with 3 000 CLOSE_WAIT connections and a pinned shard. The arms now match monoio's exact cancel error (raw_os_error == 125 on both drivers: io_uring surfaces kernel -ECANCELED, the legacy driver hardcodes 125): only a sweep cancel downshifts/parks; real errors tear down promptly. Small-N repro post-fix: fds released, 0 CLOSE_WAIT, 0 % CPU. Regression tests: tls_fin_while_parked_tears_down_promptly, rst_while_parked_tears_down_promptly.

Measured (E11, moon-dev VM, 3 000 idle TLS conns, shards=2, same-binary flag A/B)

Leg TLSPARK_OFF (P4b) TLSPARK_ON
Parked/idle 47.0 KB/conn 26.0 KB/conn (−45 %)
Wake sweep 0.13 s, bad=0 0.08 s, bad=0
Idle again after sweep 48.4 31.4 (re-parks)
After close RSS + fds return RSS + fds return

Plain-park E10 sanity re-run: unchanged at 3.28 KB/conn. Remaining TLS floor ≈ rustls session (~15–20 KB, not shrinkable via public API).

Gates

  • Parity suites green on kqueue AND io_uring: plain 5/5 (incl. new identity + RST tests), TLS 3/3 (incl. new park/kill + FIN tests)
  • VM monoio lib 4466 pass; VM tokio lib 3628 pass
  • fmt; clippy --all-targets both matrices 0 lints; vendor wrapper tests 7/7

refs: .planning/rfcs/c1m-connection-plane.md (round-6 section), tmp/C10K-REVIEW.md round-6 appendix, PR #422 follow-ups

Summary by CodeRabbit

  • New Features
    • Idle TCP and TLS connections can now pause more efficiently while preserving client identity, metadata, and controls across wake cycles.
    • Migrated connections use the same idle parking behavior.
  • Bug Fixes
    • Prevented CPU spinning when parked connections encounter cancellation or readiness errors.
    • Parked connections now shut down promptly when clients disconnect or reset connections.
    • TLS connections remain responsive, visible, and killable after being parked.
  • Tests
    • Added coverage for TCP and TLS parking, waking, registry persistence, and clean shutdown behavior.

…rking (c1M P1 follow-ups)

Two follow-ups from the round-5 task-exit-parking review:

1. Registration handoff (closes the review's Low finding, and a worse
   bug found while fixing it): waking a parked connection deregistered
   then re-registered the client across a task-scheduling boundary.
   Beyond the transient CLIENT LIST invisibility / KILL-returns-0
   window, the fresh registration silently RESET the connection's
   CLIENT SETNAME and age (register() inserts name: None; only the
   SETNAME dispatch path ever sets it). The watcher now passes the held
   RegistryGuard through spawn_resumed_parked_conn into the handler
   (ParkArgs::kept_registration); the handler reuses the live entry via
   the new client_registry::live_handle() instead of re-registering.
   The entry — name, connected_at, kill state, TOTAL_CLIENTS/shard
   counters — persists unbroken across any number of park/wake cycles;
   there is no deregistered window at all.
   The can_park bool param becomes ParkArgs {can_park,
   kept_registration} with NO_PARK/PARK consts at the six monoio call
   sites.

2. Migrated-spawn site routes ParkIdle: a connection that migrated
   shards (Linux) and then idled past --conn-park-secs now parks like a
   primary-accept connection instead of holding its handler task
   forever. Mechanical mirror of the reviewed plain-TCP routing (same
   watcher, same close-accounting transfer). No dedicated migration
   integration harness exists (migration is affinity-driven,
   Linux-only); covered by compile gates + the shared watcher paths.

Red/green: new tests/parked_idle_parity.rs test
resumed_connection_keeps_registry_identity (SETNAME → 2× park/data-wake
cycles → name+id survive, CLIENT LIST holds exactly victim+control) was
RED before (name= empty, age=0 after wake), GREEN after.

Gates: parked parity 4/4 on kqueue AND io_uring (VM); TLS parity green;
VM monoio lib 4466 pass; VM tokio lib 3628 pass; fmt; clippy
--all-targets both matrices 0 lints.

refs: PR #422 review Low, .planning/rfcs/c1m-connection-plane.md
author: Tin Dang
…1M P1-TLS)

Extends round-5 task-exit parking to TLS connections, closing the last
c1M follow-up. No new unsafe; vendored additions are moon-patch style.

Vendored (monoio-rustls + monoio-io-wrapper):
- Stream::io_ref() — raw-transport reference so the parked watcher can
  await readable(false) on the underlying fd through the TLS wrapper.
- Stream::task_park_safe() — park-safety veto: refuses while the TLS
  stack holds ANYTHING the raw fd's readability cannot signal (wrapper
  buffer bytes or pending EOF/error status via the new
  ReadBuffer/WriteBuffer::is_drained(), decrypted plaintext, a received
  close_notify, or pending session output). A partial record in the
  deframer is park-safe: completing it requires more socket bytes.
- SafeRead/SafeWrite::is_drained() with moon-patch unit tests (7/7).

moon:
- IdleParkRead: TLS sets SUPPORTS_TASK_PARK=true; new per-park
  task_park_safe() hook (plain TCP: always true) appended LAST in the
  parkable predicate.
- conn_accept: ParkWatchable trait (park_readable()) — TcpStream awaits
  its own readable(false), TLS awaits io_ref().readable(false); watcher +
  resume helpers genericized over the stream; the monoio-TLS accept site
  routes ParkIdle like the plain site.

Spin bug found by E11 and fixed (affects plain TCP too):
- The idle-park arms treated EVERY read error as the sweep cancel. With
  task parking, a dead connection whose error leaves the fd permanently
  readable (TLS client FIN without close_notify => read ERROR; plain RST)
  spun park→wake→park at 100% CPU forever — E11's first ON leg ended
  with 3000 CLOSE_WAIT conns and a pinned shard. The arms now match
  monoio's exact cancel error (raw os 125 on BOTH drivers — uring kernel
  -ECANCELED, legacy hardcodes 125): only the sweep cancel
  downshifts/parks, real errors tear down promptly (idle_park::
  is_sweep_cancel). Small-N repro: fds released, 0 CLOSE_WAIT, 0% CPU.

Measured (E11, moon-dev VM, 3000 idle TLS conns, shards=2, same-binary
flag A/B, tmp/c10k/e11_tls_park_rss.sh):
- TLSPARK_OFF (P4b downshift only): 47.0 KB/conn idle
- TLSPARK_ON: 26.0 KB/conn idle (−45%); re-parks at 31.4 after a full
  wake sweep; sweeps bad=0 both legs; RSS + fds fully return on close
- Remaining TLS floor ≈ rustls session (~15-20 KB, not shrinkable via
  public API) + registry entry + kernel socket

Tests:
- tls_idle_downshift_parity: +tls_parked_connection_serves_traffic_and_
  stays_killable (3 park/wake cycles incl. 100-pipeline on ONE session;
  CLIENT LIST shows the parked conn WITH its name via registration
  handoff; CLIENT KILL closes it), +tls_fin_while_parked_tears_down_
  promptly (the spin regression, TLS leg)
- parked_idle_parity: +rst_while_parked_tears_down_promptly (spin
  regression, plain-TCP RST leg)
- All parity suites green on kqueue AND io_uring (plain 5/5, TLS 3/3)

Gates: VM monoio+tokio lib suites pass; E10 plain-park sanity re-run
green; fmt; clippy --all-targets both matrices 0 lints; vendor wrapper
moon-patch tests 7/7.

refs: .planning/rfcs/c1m-connection-plane.md (P1-TLS follow-up)
author: Tin Dang
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@TinDang97, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 40 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 99050eef-67f8-4cb6-a785-8375fa8c8eca

📥 Commits

Reviewing files that changed from the base of the PR and between 99f4b87 and 5ba8ccd.

📒 Files selected for processing (4)
  • src/server/conn/core.rs
  • src/server/conn/handler_monoio/mod.rs
  • tests/parked_idle_parity.rs
  • vendor/monoio-io-wrapper/src/safe_io.rs
📝 Walkthrough

Walkthrough

Idle task-exit parking now supports TLS and migrated monoio connections, preserves client registry identity across wake cycles, distinguishes cancellation from real I/O errors, and adds readiness checks plus integration coverage for wake, kill, RST, and FIN behavior.

Changes

Idle task-exit parking

Layer / File(s) Summary
TLS parking safety and drained-state contracts
vendor/monoio-io-wrapper/src/*, vendor/monoio-rustls/src/stream.rs, src/server/conn/handler_monoio/idle_park.rs
IO buffers expose drained-state checks, while TLS streams expose transport access and veto task parking when buffered or pending TLS state cannot be readiness-woken safely.
Handler parking state and error handling
src/client_registry.rs, src/server/conn/handler_monoio/mod.rs, src/server/conn/handler_monoio/idle_park.rs
ParkArgs carries parking opt-in and reusable registry guards; resumed handlers reuse live client state, and only the exact sweep-cancellation error triggers downshift behavior.
Accept, wake, and migration routing
src/shard/conn_accept.rs
Plain TCP, TLS, and migrated connections route ParkIdle results to generic parked watchers, which retain registry guards through wake and resume.
Behavioral validation and release notes
tests/parked_idle_parity.rs, tests/tls_idle_downshift_parity.rs, CHANGELOG.md
Tests cover identity preservation, prompt teardown, repeated TLS park/wake cycles, TLS killability, and migrated parking; the changelog records the behavior changes.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant MonoioHandler
  participant ParkedIdleWatcher
  participant ParkWatchable
  participant ResumedHandler
  participant ClientRegistry
  MonoioHandler->>ParkedIdleWatcher: return ParkIdle
  ParkedIdleWatcher->>ParkWatchable: await park_readable()
  ParkWatchable-->>ParkedIdleWatcher: stream readable
  ParkedIdleWatcher->>ResumedHandler: resume with RegistryGuard
  ResumedHandler->>ClientRegistry: reuse live registration
Loading

Possibly related PRs

  • pilotspace/moon#421: Modifies the same monoio idle-parking and client-registry code paths.
  • pilotspace/moon#422: Implements the related task-exit parking mechanism, which this change extends to TLS and registry preservation.

Suggested reviewers: pilotspacex-byte

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly covers the main server follow-ups: TLS task-parking, registration handoff, and the ECANCELED spin fix.
Description check ✅ Passed The description has a strong Summary, Performance Impact, and Notes content, but it omits the template's checklist section.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/c1m-followups

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
tests/tls_idle_downshift_parity.rs (1)

141-165: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

--conn-park-secs help text is now stale.

src/config.rs lines 307-312 documents the flag as applying to "plain-TCP monoio connections only — TLS and the tokio runtime keep the buffer-downshift behavior". These tests (and the changelog) now rely on TLS parking, so the user-facing help text contradicts actual behavior. Please update that doc comment in the same PR.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/tls_idle_downshift_parity.rs` around lines 141 - 165, Update the
`--conn-park-secs` documentation in the configuration definition in
`src/config.rs` to describe its current behavior for TLS connections as well as
plain-TCP connections, removing the stale claim that TLS and tokio retain the
old behavior. Keep the help text consistent with the TLS parking exercised by
`spawn_moon_tls_park`.
🧹 Nitpick comments (4)
tests/parked_idle_parity.rs (2)

203-257: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Spawned moon children are only killed on the success path. All four new tests end with child.kill(); child.wait(); as plain statements, so any earlier assertion failure leaks the server process and its reserved port for the remainder of the test binary. Wrapping the child in a Drop guard (as tests/txn_kv_wiring.rs lines 972-975 does) fixes every site.

  • tests/parked_idle_parity.rs#L203-L257: hold the child in a kill-on-drop guard in resumed_connection_keeps_registry_identity, and do the same in rst_while_parked_tears_down_promptly.
  • tests/tls_idle_downshift_parity.rs#L244-L352: same guard in tls_parked_connection_serves_traffic_and_stays_killable and in tls_fin_while_parked_tears_down_promptly.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/parked_idle_parity.rs` around lines 203 - 257, The four new tests leak
spawned moon processes when assertions fail because cleanup only runs on the
success path. In tests/parked_idle_parity.rs:203-257, update
resumed_connection_keeps_registry_identity and
rst_while_parked_tears_down_promptly; in
tests/tls_idle_downshift_parity.rs:244-352, update
tls_parked_connection_serves_traffic_and_stays_killable and
tls_fin_while_parked_tears_down_promptly to store each child in the existing
kill-on-drop guard pattern, ensuring cleanup occurs on both success and failure.

302-310: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Poll instead of a single 2 s sleep.

Teardown timing depends on the shard sweep cadence and CI load; a fixed sleep makes this a latent flake (and a slow test when it passes). Retrying CLIENT LIST until name=rstvictim disappears, with a deadline, is both faster and stable. Same pattern applies to the TLS FIN test.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/parked_idle_parity.rs` around lines 302 - 310, Replace the fixed
2-second sleep in the parked-connection teardown test around command_reply and
the rstvictim assertion with deadline-based polling of CLIENT LIST, retrying
until name=rstvictim disappears and failing when the deadline expires. Apply the
same polling pattern to the TLS FIN test, preserving the existing assertion
diagnostics and connection cleanup behavior.
tests/tls_idle_downshift_parity.rs (2)

339-348: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Ok(n) panic arm can trip on residual TLS records.

The read is on the raw socket, so any bytes still in flight from the server side (a TLS alert, or a record the test's read_exact_deadline left buffered in the kernel) fail the test with "expected close" even though the close did happen. Draining until Ok(0)/Err with a deadline would assert the intended property — that the fd closes promptly — without depending on nothing being pending.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/tls_idle_downshift_parity.rs` around lines 339 - 348, Update the
raw-socket read assertion in the parked TLS victim test to drain repeatedly
until it observes EOF or an error, using a bounded deadline or the existing
10-second timeout. Remove the panic on successful nonzero reads so residual TLS
records are consumed, while still failing if the socket does not close promptly.

299-316: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract the bulk-reply reader.

This exact read-until-complete-bulk loop is repeated verbatim at lines 398-413, and tests/parked_idle_parity.rs already has an equivalent command_reply helper. A single shared helper in tests/common/mod.rs would remove three copies.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/tls_idle_downshift_parity.rs` around lines 299 - 316, Extract the
repeated bulk-reply reading loop into a shared helper in tests/common/mod.rs,
following the existing command_reply pattern from tests/parked_idle_parity.rs.
Update the current reader and the duplicate at the later CLIENT LIST call in
tls_idle_downshift_parity.rs to use the helper, preserving the existing
completion and error behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/shard/conn_accept.rs`:
- Around line 1122-1127: Update the parked-wake handling around ParkArgs and the
HijackForPsync arm so resumed plain-TCP streams enter the same inline PSYNC flow
used by initial connections instead of logging unsupported and closing. Preserve
the parked registration, and make any TLS limitation explicit while keeping
valid plain replicas connected.

In `@tests/parked_idle_parity.rs`:
- Around line 283-299: Add an accurate // SAFETY: comment immediately above the
libc::setsockopt call within the existing unsafe block, documenting why the raw
file descriptor, linger pointer, and socket option arguments are valid. Keep the
SO_LINGER(0) setup and assertion unchanged.

In `@vendor/monoio-io-wrapper/src/safe_io.rs`:
- Around line 224-234: The library safety predicates SafeRead::is_drained and
SafeWrite::is_drained must not panic when their buffers are absent. Replace the
buffer expect-based access with logic that returns false for a missing buffer
while preserving the existing drained-state checks for present buffers.

---

Outside diff comments:
In `@tests/tls_idle_downshift_parity.rs`:
- Around line 141-165: Update the `--conn-park-secs` documentation in the
configuration definition in `src/config.rs` to describe its current behavior for
TLS connections as well as plain-TCP connections, removing the stale claim that
TLS and tokio retain the old behavior. Keep the help text consistent with the
TLS parking exercised by `spawn_moon_tls_park`.

---

Nitpick comments:
In `@tests/parked_idle_parity.rs`:
- Around line 203-257: The four new tests leak spawned moon processes when
assertions fail because cleanup only runs on the success path. In
tests/parked_idle_parity.rs:203-257, update
resumed_connection_keeps_registry_identity and
rst_while_parked_tears_down_promptly; in
tests/tls_idle_downshift_parity.rs:244-352, update
tls_parked_connection_serves_traffic_and_stays_killable and
tls_fin_while_parked_tears_down_promptly to store each child in the existing
kill-on-drop guard pattern, ensuring cleanup occurs on both success and failure.
- Around line 302-310: Replace the fixed 2-second sleep in the parked-connection
teardown test around command_reply and the rstvictim assertion with
deadline-based polling of CLIENT LIST, retrying until name=rstvictim disappears
and failing when the deadline expires. Apply the same polling pattern to the TLS
FIN test, preserving the existing assertion diagnostics and connection cleanup
behavior.

In `@tests/tls_idle_downshift_parity.rs`:
- Around line 339-348: Update the raw-socket read assertion in the parked TLS
victim test to drain repeatedly until it observes EOF or an error, using a
bounded deadline or the existing 10-second timeout. Remove the panic on
successful nonzero reads so residual TLS records are consumed, while still
failing if the socket does not close promptly.
- Around line 299-316: Extract the repeated bulk-reply reading loop into a
shared helper in tests/common/mod.rs, following the existing command_reply
pattern from tests/parked_idle_parity.rs. Update the current reader and the
duplicate at the later CLIENT LIST call in tls_idle_downshift_parity.rs to use
the helper, preserving the existing completion and error behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1c68a072-e5a5-46ba-8c52-c154ca6774b2

📥 Commits

Reviewing files that changed from the base of the PR and between 4a924ab and 99f4b87.

📒 Files selected for processing (10)
  • CHANGELOG.md
  • src/client_registry.rs
  • src/server/conn/handler_monoio/idle_park.rs
  • src/server/conn/handler_monoio/mod.rs
  • src/shard/conn_accept.rs
  • tests/parked_idle_parity.rs
  • tests/tls_idle_downshift_parity.rs
  • vendor/monoio-io-wrapper/src/lib.rs
  • vendor/monoio-io-wrapper/src/safe_io.rs
  • vendor/monoio-rustls/src/stream.rs

Comment thread src/shard/conn_accept.rs
Comment thread tests/parked_idle_parity.rs
Comment thread vendor/monoio-io-wrapper/src/safe_io.rs
…omment, expect-free is_drained

CodeRabbit review of PR #424, all three findings addressed:

1. (Major) PSYNC-after-park closed the replica connection: connections
   that have issued REPLCONF (replica mid-handshake, PSYNC next) are now
   permanently excluded from task-parking via a sticky
   ConnectionState::saw_replconf flag checked in the parkable predicate —
   the unsupported HijackForPsync warn+close arm on the resumed path is
   now unreachable for real replicas. (A manual PSYNC with no prior
   REPLCONF after 60s of idle still lands on the loud warn+close and the
   replica's reconnect loop recovers — documented.)
2. (Minor) SO_LINGER unsafe block in rst_while_parked test gained its
   required // SAFETY: comment.
3. (Minor) Vendored SafeRead/SafeWrite::is_drained() are expect-free:
   a transiently-absent buffer answers false ("don't park" is always the
   safe verdict) instead of panicking.

Gates: parity suites green kqueue (plain 5/5, TLS 3/3); vendor wrapper
7/7; fmt; clippy --all-targets both matrices 0 lints.

refs: PR #424 review threads
author: Tin Dang
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Monoio: TLS task parking, registry handoff on wake, and ECANCELED spin fix

✨ Enhancement 🐞 Bug fix 🧪 Tests 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Enable task-exit idle parking for TLS connections with safe raw-fd readiness watching.
• Preserve client registry identity across park/wake by handing registration into resumed handlers.
• Fix parked-connection CPU spin by distinguishing monoio sweep-cancel (ECANCELED) from real read
 errors.
Diagram

graph TD
A["Accept / migrate"] --> B["Conn handler"] --> C{"Idle & parkable?"} --> D["ParkIdle"] --> E["Idle watcher"] --> F["Resumed handler"]
B --> R[("Client registry")]
E --> R
B --> V["TLS park veto"]
E --> H["Raw-fd readiness"]
B --> X["Close / teardown"]

subgraph Legend
  direction LR
  _svc["Component"] ~~~ _dec{"Decision"} ~~~ _db[("Registry")]
end
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Avoid TLS task-exit parking (keep in-task downshift only)
  • ➕ No vendored wrapper changes; simpler operational risk profile
  • ➕ Less chance of subtle readiness/buffering mismatches in TLS stack
  • ➖ Keeps higher per-idle-TLS memory footprint and scheduler load
  • ➖ Does not align TLS with plain TCP parking behavior
2. Upstream monoio-rustls/monoio-io-wrapper APIs instead of vendoring
  • ➕ Reduces long-term maintenance burden of moon patches
  • ➕ Allows broader review/testing by upstream community
  • ➖ Upstream latency may block the campaign timeline
  • ➖ May require API design compromises vs. the narrowly-scoped needs here
3. Model sweep-cancel explicitly via monoio-specific error type/flag (if available)
  • ➕ More self-documenting than raw errno matching
  • ➕ Potentially more portable if cancel representation changes
  • ➖ May not exist across both monoio drivers; current code documents why raw 125 is consistent
  • ➖ Risk of accidentally reintroducing the spin if matching is too broad

Recommendation: The PR’s approach is appropriate for the stated goals: TLS task parking is enabled with an explicit, conservative park-safety veto (buffer/state drained) and a minimal raw-fd readiness hook, and the ECANCELED spin fix correctly narrows the cancel match to monoio’s exact construction. If this functionality is intended to persist, consider upstreaming the io_ref()/task_park_safe()/is_drained pieces after rollout to reduce vendor drift; otherwise, the current vendored patches are a pragmatic, low-unsafe change.

Files changed (10) +732 / -76

Enhancement (5) +290 / -73
client_registry.rsAdd live_handle() to reuse existing client registration +8/-0

Add live_handle() to reuse existing client registration

• Introduces client_registry::live_handle(id) to fetch the existing Arc<ClientLiveState> for an already-registered client. Used by resumed parked connections to avoid deregister/re-register and preserve name/connected_at/counters continuity.

src/client_registry.rs

mod.rsIntroduce ParkArgs and hand off registry guard across park/wake +92/-28

Introduce ParkArgs and hand off registry guard across park/wake

• Replaces the single can_park flag with ParkArgs { can_park, kept_registration } and constants for call sites. On resumed connections, reuses the existing registration via kept_registration + client_registry::live_handle() to preserve CLIENT LIST identity (name, age, kill state) and avoid counter churn. Adds TLS stream-side task_park_safe() veto into the park predicate and breaks immediately on real read errors (non-ECANCELED).

src/server/conn/handler_monoio/mod.rs

conn_accept.rsGenericize parked watcher for TCP/TLS and route ParkIdle for TLS+migrated conns +136/-45

Genericize parked watcher for TCP/TLS and route ParkIdle for TLS+migrated conns

• Adds ParkWatchable abstraction so the parked watcher can await readability on either TcpStream or TLS via io_ref().readable(false). Changes parked watcher/resume helpers to be generic over the stream type and passes RegistryGuard into the resumed handler (registration handoff). Routes ParkIdle for TLS accept and migrated-connection spawn paths so both can task-park and resume correctly.

src/shard/conn_accept.rs

lib.rsExpose is_drained() for read/write buffers to support task-park safety +23/-0

Expose is_drained() for read/write buffers to support task-park safety

• Adds ReadBuffer::is_drained and WriteBuffer::is_drained that conservatively return false for unsafe_io buffers. These APIs allow higher-level TLS stream logic to determine whether internal buffering or deferred status would make raw-fd readiness insufficient for waking.

vendor/monoio-io-wrapper/src/lib.rs

stream.rsAdd io_ref() and task_park_safe() to enable safe TLS task parking +31/-0

Add io_ref() and task_park_safe() to enable safe TLS task parking

• Adds Stream::io_ref() to expose the underlying transport for readiness watching. Implements Stream::task_park_safe() to veto parking if wrapper buffers aren’t drained, session wants write, plaintext is pending, peer has closed, or TLS state is invalid—ensuring raw-fd readability is a complete wake signal when parked.

vendor/monoio-rustls/src/stream.rs

Bug fix (1) +36 / -3
idle_park.rsAdd sweep-cancel detection and TLS task-park capability hooks +36/-3

Add sweep-cancel detection and TLS task-park capability hooks

• Adds is_sweep_cancel() (raw_os_error == 125) so parked-capable paths distinguish sweep cancel from real socket/TLS errors, preventing park→wake→park CPU spins. Extends IdleParkRead with task_park_safe() (default true) and enables SUPPORTS_TASK_PARK + TLS override calling the vendored task_park_safe().

src/server/conn/handler_monoio/idle_park.rs

Tests (3) +366 / -0
parked_idle_parity.rsAdd regression tests for registry identity and RST teardown while parked +113/-0

Add regression tests for registry identity and RST teardown while parked

• Adds resumed_connection_keeps_registry_identity to ensure CLIENT SETNAME and client id persist across multiple park/wake cycles without double-counting in CLIENT LIST. Adds rst_while_parked_tears_down_promptly (unix) to ensure an RST during parked state tears down cleanly and does not leave a spinning watcher/handler loop.

tests/parked_idle_parity.rs

tls_idle_downshift_parity.rsAdd TLS task-parking parity and FIN-without-close_notify teardown tests +190/-0

Add TLS task-parking parity and FIN-without-close_notify teardown tests

• Adds spawn helper to set --conn-park-secs for TLS runs. Introduces tls_parked_connection_serves_traffic_and_stays_killable to validate repeated park/wake cycles over the same TLS session, including pipelining and killability/visibility via a plain TCP control connection. Adds tls_fin_while_parked_tears_down_promptly to ensure unexpected EOF (FIN without close_notify) does not spin and is removed from CLIENT LIST.

tests/tls_idle_downshift_parity.rs

safe_io.rsImplement is_drained() and add unit tests for drain semantics +63/-0

Implement is_drained() and add unit tests for drain semantics

• Adds SafeRead::is_drained and SafeWrite::is_drained checks that account for both buffered bytes and deferred EOF/error status. Adds tests ensuring is_drained flips appropriately when bytes or deferred statuses are pending, preventing incorrect task parking on raw readiness.

vendor/monoio-io-wrapper/src/safe_io.rs

Documentation (1) +40 / -0
CHANGELOG.mdDocument TLS task parking, registry handoff, and ECANCELED spin fix +40/-0

Document TLS task parking, registry handoff, and ECANCELED spin fix

• Adds Unreleased changelog entries describing TLS task-exit parking behavior, the parked-connection spin fix by matching ECANCELED precisely, and registry identity preservation across park/wake. Also notes migrated connections now park via the migrated spawn path.

CHANGELOG.md

@qodo-code-review

qodo-code-review Bot commented Jul 29, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (0)

Context used
✅ Compliance rules (platform): 55 rules

Grey Divider


Action required

1. setsockopt unsafe lacks SAFETY ✓ Resolved 📘 Rule violation ≡ Correctness
Description
A new unsafe block calling libc::setsockopt was added without the required adjacent // SAFETY:
invariants comment. This violates the repo unsafe policy and increases the risk of undocumented
UB/FFI preconditions slipping in unnoticed.
Code

tests/parked_idle_parity.rs[R289-297]

+        let rc = unsafe {
+            libc::setsockopt(
+                victim.as_raw_fd(),
+                libc::SOL_SOCKET,
+                libc::SO_LINGER,
+                &linger as *const _ as *const libc::c_void,
+                std::mem::size_of::<libc::linger>() as libc::socklen_t,
+            )
+        };
Relevance

●●● Strong

Similar unsafe-in-tests SAFETY-comment requests were at least partially accepted (libc::kill)
indicating enforcement of UNSAFE_POLICY.md.

PR-#65

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
UNSAFE_POLICY.md requires every unsafe block to have an adjacent // SAFETY: comment
documenting preconditions and why violating them could cause UB. The added `unsafe {
libc::setsockopt(...) } block in tests/parked_idle_parity.rs` has no such SAFETY comment.

Rule 297369: Enforce unsafe code usage against UNSAFE_POLICY.md
UNSAFE_POLICY.md[14-26]
tests/parked_idle_parity.rs[282-299]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
An `unsafe` block was introduced without the required `// SAFETY:` comment describing invariants and why the call is sound.

## Issue Context
`UNSAFE_POLICY.md` requires every `unsafe` block to include a `// SAFETY:` comment with explicit preconditions/invariants.

## Fix Focus Areas
- tests/parked_idle_parity.rs[282-299]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. TLS park ignores new writes 🐞 Bug ☼ Reliability
Description
Stream::task_park_safe() checks session.wants_write() only before process_new_packets(), but
packet processing can make wants_write() become true; the function can still return true and allow
task-exit parking with pending TLS output. Because the parked watcher only awaits raw-fd
readability, the connection can stall indefinitely waiting for a write that no task will perform.
Code

vendor/monoio-rustls/src/stream.rs[R103-113]

+    pub fn task_park_safe(&mut self) -> bool {
+        if !self.r_buffer.is_drained() || !self.w_buffer.is_drained() || self.session.wants_write()
+        {
+            return false;
+        }
+        match self.session.process_new_packets() {
+            Ok(state) => state.plaintext_bytes_to_read() == 0 && !state.peer_has_closed(),
+            // Corrupt/unexpected TLS state: don't park; the next read
+            // surfaces the error and tears down cleanly.
+            Err(_) => false,
+        }
Relevance

●● Moderate

No clear historical evidence found for this specific rustls wants_write/process_new_packets
task-parking ordering issue.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
task_park_safe() only checks wants_write() before calling process_new_packets() and does not
verify whether packet processing introduced new pending output. The same file’s handshake() loop
repeatedly checks wants_write() around reads, demonstrating that processing incoming packets can
make the session require writes after reads/packet processing.

vendor/monoio-rustls/src/stream.rs[92-114]
vendor/monoio-rustls/src/stream.rs[187-219]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`vendor/monoio-rustls::Stream::task_park_safe()` can return `true` (allowing task-exit parking) even if `process_new_packets()` causes `self.session.wants_write()` to become `true`. This can leave the TLS connection parked behind a read-readiness watcher while the TLS state machine is waiting to write, stalling the session.

## Issue Context
The new task-exit parking feature relies on raw-fd readability as the only wake signal while parked. Any pending TLS output after deciding to park is unsafe because there is no handler task left to flush it.

## Fix Focus Areas
- vendor/monoio-rustls/src/stream.rs[92-114]

## Suggested change
After calling `process_new_packets()`, include a second `!self.session.wants_write()` check in the success path, e.g.:
- `Ok(state) => state.plaintext_bytes_to_read() == 0 && !state.peer_has_closed() && !self.session.wants_write()`

(Optionally keep the early `wants_write()` check as the fast path.)

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

3. SafeRead::is_drained() uses expect ✓ Resolved 📘 Rule violation ☼ Reliability
Description
New production-library code uses .expect(...), which can panic and violates the rule disallowing
unwrap()/expect() outside tests. This can cause unexpected process termination if the invariant
is ever violated.
Code

vendor/monoio-io-wrapper/src/safe_io.rs[R224-234]

+    /// moon patch (c1M P1-TLS): true when the buffer holds no bytes AND no
+    /// deferred status (EOF/error) is pending delivery. Task-parking on raw
+    /// fd readability is only correct when nothing is waiting here — a
+    /// buffered byte or a pending EOF would never make the fd readable.
+    pub fn is_drained(&self) -> bool {
+        self.buffer
+            .as_ref()
+            .expect("buffer ref expected")
+            .is_empty()
+            && matches!(self.status, ReadStatus::Ok)
+    }
Relevance

●●● Strong

Team has accepted removing unwrap/expect in non-test code previously (e.g., replace expect/unwrap
with guarded patterns).

PR-#66
PR-#174

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
PR Compliance ID 302080 disallows .unwrap()/.expect() in non-test library code, and both
SafeRead::is_drained() and SafeWrite::is_drained() (library code) introduce `.expect("buffer ref
expected")`, which can panic in production builds if the expected buffer reference is missing.

Rule 302080: Disallow unwrap() and expect() in non-test Rust library code
vendor/monoio-io-wrapper/src/safe_io.rs[224-234]
vendor/monoio-io-wrapper/src/safe_io.rs[339-347]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`SafeRead::is_drained()` and `SafeWrite::is_drained()` use `.expect("buffer ref expected")`, which violates the no-`expect`/no-`unwrap` rule for non-test library code.

## Issue Context
This code is not under `tests/` and is part of a library crate; panicking via `expect()` is disallowed by PR Compliance ID 302080, and using `expect()` here can terminate the process unexpectedly if the assumed invariant is violated.

## Fix Focus Areas
- vendor/monoio-io-wrapper/src/safe_io.rs[224-234]
- vendor/monoio-io-wrapper/src/safe_io.rs[339-347]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

To customize comments, go to the Qodo configuration screen, or learn more in the docs.

Qodo Logo

Comment thread tests/parked_idle_parity.rs
Comment thread vendor/monoio-io-wrapper/src/safe_io.rs
Comment on lines +103 to +113
pub fn task_park_safe(&mut self) -> bool {
if !self.r_buffer.is_drained() || !self.w_buffer.is_drained() || self.session.wants_write()
{
return false;
}
match self.session.process_new_packets() {
Ok(state) => state.plaintext_bytes_to_read() == 0 && !state.peer_has_closed(),
// Corrupt/unexpected TLS state: don't park; the next read
// surfaces the error and tears down cleanly.
Err(_) => false,
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

3. Tls park ignores new writes 🐞 Bug ☼ Reliability

Stream::task_park_safe() checks session.wants_write() only before process_new_packets(), but
packet processing can make wants_write() become true; the function can still return true and allow
task-exit parking with pending TLS output. Because the parked watcher only awaits raw-fd
readability, the connection can stall indefinitely waiting for a write that no task will perform.
Agent Prompt
## Issue description
`vendor/monoio-rustls::Stream::task_park_safe()` can return `true` (allowing task-exit parking) even if `process_new_packets()` causes `self.session.wants_write()` to become `true`. This can leave the TLS connection parked behind a read-readiness watcher while the TLS state machine is waiting to write, stalling the session.

## Issue Context
The new task-exit parking feature relies on raw-fd readability as the only wake signal while parked. Any pending TLS output after deciding to park is unsafe because there is no handler task left to flush it.

## Fix Focus Areas
- vendor/monoio-rustls/src/stream.rs[92-114]

## Suggested change
After calling `process_new_packets()`, include a second `!self.session.wants_write()` check in the success path, e.g.:
- `Ok(state) => state.plaintext_bytes_to_read() == 0 && !state.peer_has_closed() && !self.session.wants_write()`

(Optionally keep the early `wants_write()` check as the fast path.)

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@TinDang97
TinDang97 merged commit fd4bea4 into main Jul 29, 2026
24 checks passed
@TinDang97
TinDang97 deleted the feat/c1m-followups branch July 29, 2026 16:50
TinDang97 added a commit that referenced this pull request Jul 29, 2026
…ve v0.8.3 hazard) (#425)

Patch release rolling up PR #424 (c1M round 6).

Ships the fix for a bug LIVE in v0.8.3: task-exit parking defaults on,
and the idle-park arms treated every read error as the sweep cancel — a
client dying with an RST while parked (crashed client, NAT reset, LB
probe) left the fd permanently readable and spun the connection through
park→wake→park at 100% shard CPU, socket stuck in CLOSE_WAIT. The arms
now match monoio's exact ECANCELED (raw os 125, both drivers); real
errors tear down promptly; FIN/RST-while-parked regression-tested.

Also: TLS task-exit parking (idle TLS 47.0 → 26.0 KB/conn, −45%, no new
unsafe — vendored io_ref() readiness passthrough + task_park_safe()
veto); registration handoff across park/wake (no deregistered window;
fixes the wake silently resetting CLIENT SETNAME/age); migrated conns
park; REPLCONF'd conns excluded from parking (PSYNC-after-park
unreachable for real replicas).

Validation: parity suites green kqueue + io_uring (plain 5/5, TLS 3/3);
VM monoio 4466 + tokio 3628 lib tests; E10 plain-park sanity unchanged
(3.28 KB/conn); spin repro post-fix clean. Release gate: crash-matrix
nightly + ITERS=20 soak dispatched on the RC (ritual held), green
before tag.

Rolls CHANGELOG [Unreleased] into [0.8.4], bumps Cargo.toml/lock,
adds the RELEASES.md row, updates the README milestone table.

author: Tin Dang
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant