fix: make StartConditions sync, drop block_in_place under ctrl_lock - #2779
Open
YuanYuYuan wants to merge 1 commit into
Open
fix: make StartConditions sync, drop block_in_place under ctrl_lock#2779YuanYuYuan wants to merge 1 commit into
YuanYuYuan wants to merge 1 commit into
Conversation
The RX thread handles OAM/Declare messages holding ctrl_lock and the tables write lock, then parks in ZRuntime::Net.block_in_place(...) waiting on the StartConditions tokio mutex (two sites: the gossip link_states tail and the peer hat's route_declare_final). The single Net worker meanwhile blocks on ctrl_lock in autoconnect's transport setup, and tokio's fair mutex handoff can strand the permit in a Net task that never gets polled again -- deadlocking the whole session. Every StartConditions critical section is a pure Vec operation that never awaits, so its tokio::sync::Mutex becomes std::sync::Mutex, all its methods go sync, and both block_in_place sites are deleted outright. Ported onto current main from a fix independently developed and field-validated by Guillaume Doisy (Dexory), adapting it past the eclipse-zenoh#2096 hat rename (p2p_peer -> peer) and the resulting route_declare_final signature change. Original work, full credit preserved via authorship. See PR body for the upstream discussion this consolidates.
YuanYuYuan
marked this pull request as ready for review
September 9, 2026 11:02
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2779 +/- ##
==========================================
+ Coverage 74.72% 74.76% +0.04%
==========================================
Files 419 419
Lines 63955 63934 -21
==========================================
+ Hits 47790 47803 +13
+ Misses 16165 16131 -34 ☔ View full report in Codecov by Harness. |
|
ECA validated |
Contributor
Author
JEnoch
approved these changes
Sep 10, 2026
JEnoch
left a comment
Member
There was a problem hiding this comment.
LGTM but I would like also @OlivierHecart to review, as he's the architect of this code part.
OlivierHecart
approved these changes
Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2581. Under peer churn, the RX thread can deadlock the whole session.
The RX thread holds
ctrl_lock. It callsblock_in_placeto wait onStartConditions, which is atokio::sync::Mutex. At the same time, the Net runtime's single worker thread tries to acquire that samectrl_lock, fromRouter::new_transport_unicastduring gossip autoconnect. The worker cannot run. So the task that would releaseStartConditionsnever runs either. The session freezes.This PR is a companion to #2637. #2637 fixes the same root cause with a different mechanism. See "Relationship to #2637" below.
The deadlock
ctrl_lock. Callsblock_in_placeonStartConditions. Two call sites:gossip.rs'slink_statestail, and the peer hat'sroute_declare_final.ctrl_lockfromRouter::new_transport_unicastduring gossip autoconnect.StartConditions, because the one worker that could run is stuck onctrl_lock. The session wedges: puts freeze, transport leases die.The fix
Every
StartConditionscritical section is a plainVecpush or drain. None of them await anything. Sopeer_connectorsbecomes astd::sync::Mutexinstead of atokio::sync::Mutex. Every method onStartConditionsbecomes synchronous. Bothblock_in_placecall sites are deleted, not rescheduled.Relationship to #2637
#2637 takes a different approach at the same two sites: spawn the notification as a task on
ZRuntime::Net, instead of blocking on it. That removes the RX thread from this specific cycle. But the spawned task still needsZRuntime::Net's one worker to run. Nothing in that fix adds a second worker, and nothing removes the single-worker precondition that caused the deadlock. If that worker is ever blocked elsewhere, the spawned task queues indefinitely. If a future caller blocks synchronously on the task's result, the same deadlock returns, with one extra hop.This PR removes the scheduling dependency instead. There is no
.awaitleft insideStartConditionsfor anything to be rescheduled onto.@otamachan's independent reproduction on #2637 goes through
rmw_zenoh_cppand ROS 2 directly, with a full five-thread backtrace. It confirms both #2637's approach and this PR's approach target the same twoblock_in_placesites. Worth reading alongside this PR.Credit
This PR ports a fix independently developed and field-validated by Guillaume Doisy (Dexory):
botsandus/zenoh@ff6cc2bc3. Authorship is preserved on the commit (Author: Guillaume Doisy <guillaume@dexory.com>). Their original commit message, quoted for the record:This PR ports that fix onto current
main. It adapts it past the#2096hat rename (p2p_peertopeer) and the resultingroute_declare_finalsignature change. No other logic changed.Testing
cargo fmt --check -p zenoh: cleancargo clippy -p zenoh --lib -- -D warnings: clean. Five pre-existing warnings remain in unrelated files, untouched by this diff.cargo test -p zenoh --lib net::runtime::orchestrator: 3 of 3 pass