Skip to content

Fix deadlock in IPC named lock access tests (#2030) - #2032

Open
elharo wants to merge 1 commit into
masterfrom
fix/ipc-flaky-test-deadlock
Open

Fix deadlock in IPC named lock access tests (#2030)#2032
elharo wants to merge 1 commit into
masterfrom
fix/ipc-flaky-test-deadlock

Conversation

@elharo

@elharo elharo commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Closes #2030

Problem

org.eclipse.aether.named.ipc.IpcNamedLockFactoryIT.exclusiveAccess is flaky on Jenkins, timing out after 25 s with the main thread stuck in t1.join().

The root cause is a deadlock in the Access handshake shared by sharedAccess, exclusiveAccess and mixedAccess. When both competing threads fail to acquire the lock within the 100 ms acquisition timeout, both threads count down the loser latch and then block forever on winner.await(), because the winners latch is never counted down. The test then hangs until the JUnit @timeout fires.

This happens under load when the in-process IPC server thread is starved for more than 100 ms, so neither acquisition completes in time. Earlier fixes only raised the overall test timeout (5 s to 15 s to 25 s), which converts the deadlock into a longer hang without fixing it.

Fix

  • Raise the lock acquisition timeout in Access from 100 ms to 1 s so at least one thread reliably wins even on a loaded machine. Mutual exclusion is still verified: the loser is queued behind the winner on the server and cannot be granted while the winner holds, so its failure is still caused by the winner holding the lock.
  • Bound the winner/loser handshake awaits and the test-side joins and latch awaits, and assert the expected outcome. In the (now unlikely) case both threads lose, the test fails fast with a clear message instead of hanging for the full @timeout.

Verified by running IpcNamedLockFactoryIT 21 consecutive times (15 normal + 6 under simulated CPU load), all passing.

The shared/exclusive/mixed access tests in NamedLockFactoryTestSupport
deadlock when both competing threads fail to acquire the lock within the
100 ms acquisition timeout (e.g. the IPC server thread being starved on a
loaded CI machine): both threads count down the loser latch and then
block forever on winner.await(), so the winners latch is never released
and the test hangs until @timeout fires.

Fix by giving the lock acquisition a 1 s timeout (the loser still fails
while the winner holds, so mutual exclusion is still verified) and by
bounding the winner/loser handshake and the test-side joins/awaits so
the test fails fast with a clear message instead of hanging.

Closes #2030

@gnodet gnodet left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well-diagnosed fix for the real root cause — both threads failing the 100ms lock acquisition under CPU starvation, leaving both stuck in unbounded winner.await(). The two-pronged approach (increasing acquisition timeout to 1s + bounding all internal awaits/joins with timeouts and assertions) is the right fix over the previous band-aid @Timeout increases.

The PR description is excellent — clearly documents the root cause, explains why previous fixes were insufficient, and provides verification evidence.

One observation: NamedLockFactoryAdapterTestSupport.java has the same unbounded join()/await() pattern with a 100ms acquisition timeout (ADAPTER_TIME = 100L). It may be susceptible to the same deadlock — worth a follow-up pass.

This review was generated by an AI agent (Claude Code) and may contain inaccuracies. Please verify all suggestions before applying.

On behalf of gnodet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

org.eclipse.aether.named.ipc.IpcNamedLockFactoryIT flaky

2 participants