fix: don't park the ctrl_lock holder on StartConditions notifications under peer churn

[godardt]

Closes #2581

What happens

In peer mode with gossip autoconnect, the routing layer can deadlock permanently
when sessions open and close while other peers are connecting. A long-lived
publisher then stalls forever in put().wait() (the symptom reported in #2581),
and every other operation that touches routing (declares, accepts, OAM handling,
session close) wedges behind it. In debug builds the same race can instead
poison the routing ctrl_lock and turn every rx callback into a PoisonError
panic cascade from demux.rs.

Root cause

Two call sites notify StartConditions that a peer connector has terminated by
running terminate_peer_connector_zid through ZRuntime::Net.block_in_place(..):

• hat::peer::interests::route_declare_final (initial-interest finalization),
  reached under the routing ctrl_lock via Face::send_declare
• the tail of gossip::link_states, reached under the routing ctrl_lock via
  OAM handling in DeMux::handle_message

block_in_place parks the calling thread until the future completes, so in both
cases a thread holding ctrl_lock goes to sleep waiting on the
StartConditions::peer_connectors tokio mutex. That closes a cycle, captured
live with gdb thread apply all bt on a hung process:

1. An rx thread handles a DeclareFinal, takes ctrl_lock in
   Face::send_declare, and parks in
   block_in_place(terminate_peer_connector_zid(..)), waiting on the
   StartConditions mutex.
2. The Net runtime's only worker (worker_threads = 1 by default) runs a gossip
   autoconnect task (link_states -> connect_peer ->
   Gateway::new_transport_unicast) that blocks synchronously on the same
   ctrl_lock.
3. With the Net worker wedged, tasks queued on the StartConditions mutex are
   never polled again, so the mutex never frees, the rx thread in (1) never
   wakes, and ctrl_lock is never released.

Fix

Spawn the notification on the Net runtime instead of blocking on it.
terminate_peer_connector_zid returns nothing and its only effect is to mark
the connector terminated and possibly notify_one() a waiting open(), so
deferring it by one task-schedule is semantically equivalent. The call already
raced with add_peer_connector_zid across threads (terminate-before-add inserts
an already-terminated entry), so the reordering introduces no new interleavings.
With no thread parked while holding ctrl_lock, the cycle cannot form.

Validation

Reproduced with a stress setup that repeatedly opens ~20 short-lived peer
sessions (declaring queryables and issuing queries) against a router while a
publisher streams at 5 kHz, all sessions gossip-linked. Unpatched, it deadlocked
within ~23 iterations on a 24-core Linux machine; patched, 150 iterations across
2/4/8/24-CPU affinities plus repeated full-suite runs completed with zero hangs
and zero panics. The reproducer from #2581 (z_p2p_declare_final_stress)
exercises the same churn pattern.

---

🏷️ Label-Based Checklist

Based on the labels applied to this PR, please complete these additional requirements:

Labels: bug

🐛 Bug Fix Requirements

Since this PR is labeled as a bug fix, please ensure:

• Root cause documented - Explain what caused the bug in the PR description
• Reproduction test added - Test that fails on main branch without the fix
• Test passes with fix - The reproduction test passes with your changes
• Regression prevention - Test will catch if this bug reoccurs in the future
• Fix is minimal - Changes are focused only on fixing the bug
• Related bugs checked - Verified no similar bugs exist in related code

Why this matters: Bugs without tests often reoccur.

Instructions:

1. Check off items as you complete them (change - [ ] to - [x])
2. The PR checklist CI will verify these are completed

This checklist updates automatically when labels change, but preserves your checked boxes.

[Mensch88]

Independent Reproduction (C API) & Safety Argument for Ordering Change

We identified this issue from the other end: our test suite experienced hangs lasting up to 300 seconds, initially reported against zenoh-c as zenoh-c#1308 before we localized the root cause. Tracing the source code revealed the same block_in_place under ctrl_lock that this PR addresses, which subsequently led us to #2581 and this PR.

Below are our measurements and a safety analysis regarding the ordering change, which could be the primary concern for merging.

Different Victim Path, Same Defect

The reproducer in #2581 stalls a publisher within route_data while waiting for the table read lock. Our case blocks an application declare operation:

#2  std::sys::sync::mutex::futex::Mutex::lock_contended ()
#3  <...Face as ...Primitives>::send_declare ()
#4  zenoh::api::key_expr::KeyExprWireDeclaration::new ()
#5  zenoh::api::session::Session::declare_nonwild_prefix ()
#6  z_declare_queryable ()

Here, rx-1 parks while holding the lock, consistent with the mechanism described in this PR:

#1  tokio::runtime::park::Inner::park ()
#2  zenoh_runtime::ZRuntime::block_in_place::{{closure}} ()
#3  ...hat::peer::interests...::route_declare_final ()
#4  <...Face as ...Primitives>::send_declare ()

We observed identical behavior via z_declare_subscriber. Consequently, this fix addresses three distinct victim paths across two separate reproduction cases.

Measurement Data

We ran ctest -j 4 --repeat until-fail:10 over 95 test cases. Each case spawns a short-lived mode: "peer" process with a per-test timeout of 300 seconds.

• Hardware: 12-core x86-64
• Environment: zenoh-c 1.9.0 built from source against zenoh release/1.9.0

Library Version	Sweeps	Hangs	Total Wall Time
Unpatched	3	Run 1: 2/95, Run 2: 1/95, Run 3: 3/95	348 / 381 / 377 s
PR b898eae Cherry-pick	3	0, 0, 0 (95/95)	117 / 119 / 119 s
Reverted to Unpatched*	1	2/95	385 s

*The unpatched library was swapped back in on the same machine immediately after the patched runs. The immediate return of hangs confirms the variable is the library itself, not a transient improvement in system state. The variance in total wall time is driven by whether the 300s timeout triggers.

The Switch to spawn looks safe

The asynchronous termination introduced by this PR requires a safety argument beyond measurement data. We analyzed the consumers of the notification signal. In zenoh/src/net/runtime/orchestrator.rs, there is exactly one waiter in start_peer():

if wait_scouting
    && (scouting || !peers.is_empty())
    && tokio::time::timeout(delay, self.state.start_conditions.notified())
        .await
        .is_err()
    && !peers.is_empty()
{
    tracing::warn!("Scouting delay elapsed before start conditions are met.");
}
Ok(())

Analysis:

1. Bounded Wait: The wait is bounded by the scouting delay.
2. Graceful Degradation: The timeout branch only logs a warning and returns Ok(()). A delayed notification degrades to "waited the full scouting delay," not a stall or error.
3. No Lost Wakeups: Notify::notify_one stores a permit if no task is currently waiting. Therefore, a spawned task completing before notified() is called does not result in a lost wakeup.

Trade-off:
The critical failure mode (deadlock) and the new worst-case scenario (warning + bounded wait) both occur under the same condition: a saturated Net runtime. Before this PR, saturation leads to permanent deadlock; after this PR, it results in a bounded wait and a log entry.

Backport Note

Commit b898eae cherry-picks cleanly onto release/1.9.0 with no conflicts. The resulting zenoh-c 1.9.0 builds and links without changes. There are no technical barriers to backporting this fix to the 1.9.x branch if desired.

Limitation of Evidence

Prior to discovering this PR, we independently attempted to fix the issue by modifying only the site in hat/peer/interests.rs. This alone was sufficient to eliminate our deadlocks; we had initially missed the call in gossip.rs. Therefore, our reproducer does not exercise the gossip path and cannot serve as evidence for the validity of the changes in that specific module.

[otamachan]

Independent reproduction of this deadlock from a different workload (ROS 2 / rmw_zenoh), with a full thread apply all bt that shows every participant of the cycle, and an A/B of this PR against a control build.

Standalone reproducer: https://gist.github.com/otamachan/c43030eca21246d56151312fceb91d80

Setup

ROS 2 Jazzy, rmw_zenoh_cpp 0.2.10, zenoh-c 05bd370 (zenoh 2687c513). All sessions are mode: "peer" with gossip autoconnect (autoconnect: { peer: ["router", "peer"] }, to_peer: "greater-zid") against a single router, everything on one Linux host.

A router plus 40 resident peer sessions, then repeated waves of 30 new peer sessions joining the existing mesh, all pinned to 4 CPUs. Each "peer" is just a static_transform_publisher, i.e. Session::open() plus one publisher.

In ROS terms the symptom is that rmw_init() never returns for a freshly started node: the process exists but never becomes a node, prints nothing at all, and only SIGKILL removes it (it has SIGINT/SIGTERM in SigCgt but cannot service them). On a robot this silently disabled a sensor transform for a whole run.

Backtrace

Five threads, one cycle. ctrl_lock is TablesLock::ctrl_lock.

thread	what it is doing
rx-0	holds ctrl_lock + tables write, DeMux::handle_message -> p2p_peer::HatCode::handle_oam -> ZRuntime::block_in_place -> tokio::runtime::park::Inner::park
net-0	connect_peer -> open_transport_unicast -> open_link -> notify_new_transport_unicast -> RuntimeTransportEventHandler::new_unicast -> Mutex::lock_contended (ctrl_lock)
acc-0	accept_link -> notify_new_transport_unicast -> RuntimeTransportEventHandler::new_unicast -> Mutex::lock_contended (ctrl_lock)
rx-3	DeMux::handle_message -> Face::send_declare -> Mutex::lock_contended (ctrl_lock)
main	rmw_init -> z_liveliness_get -> LivelinessGetBuilder::wait -> Face::send_interest -> Mutex::lock_contended (ctrl_lock)

Full log is gdb_hang.log in the gist. This is exactly the cycle described in the PR body, including the worker_threads = 1 step.

Two things that may be worth adding to the description:

• The Face::send_interest frame shows the cycle also traps session open, not just long-lived publishers. Any Session::open() that issues a liveliness query after the routing tables are wedged blocks forever. That is what makes this fatal for ROS 2 - the node never finishes initialising, so it never shows up in the graph and cannot be recovered in-process.
• Scouting task may deadlock #2409 explains why the parked block_on never wakes even though nothing visibly holds the StartConditions mutex: tokio's Mutex can refuse the acquisition when a waiter already has an assigned permit.

A/B

The gist also documents how to build a patched libzenohc.so and swap it into a ROS install with LD_LIBRARY_PATH, so the numbers below can be checked independently.

30 rounds each, one round = 30 fresh processes.

build	rounds with a hang
stock zenoh-c 05bd370	9 / 30
locally built, unpatched (control)	9 / 30
locally built + this PR	0 / 30

The control build rules out toolchain differences. Our production build also carries one unrelated local patch to the transmission pipeline; with that included the numbers are 8 / 30 before and 0 / 30 after this PR.

Raising the Net worker count does not avoid it, so the single-worker default is what makes it easy to hit rather than what makes it possible:

ZENOH_RUNTIME	rounds with a hang
default	9 / 30
(net: (worker_threads: 4))	9 / 30
(net: .., acc: .., rx: (worker_threads: 4))	7 / 30

Happy to run anything else against the reproducer if it helps move this along.

[JEnoch]

We just merged an alternative fix to this PR: #2779
For those of you with reproducers, could you please check if it works for you ?

I'm now preparing a bump of rmw_zenoh with 1.10.1 + #2779