Skip to content

Fix Windows shutdown deadlock and leak with the Ruy backend#2076

Open
timo9378 wants to merge 1 commit into
OpenNMT:masterfrom
timo9378:fix-windows-ruy-shutdown-deadlock
Open

Fix Windows shutdown deadlock and leak with the Ruy backend#2076
timo9378 wants to merge 1 commit into
OpenNMT:masterfrom
timo9378:fix-windows-ruy-shutdown-deadlock

Conversation

@timo9378

Copy link
Copy Markdown

Hi, thanks for CTranslate2. I ran into a Windows-only shutdown hang and tracked it down
to the Ruy backend, so here's a fix.

Creating and destroying a Translator (or Generator/Whisper) with the Ruy
backend on Windows hangs the process on shutdown. The culprit is the per-thread
ruy::Context in src/cpu/backend.cc:

ruy::Context *get_ruy_context() {
  static thread_local ruy::Context context;
  return &context;
}

It's constructed lazily on a worker thread the first time that thread runs a GEMM,
and its destructor joins Ruy's internal thread pool. Because it's thread_local, that
destructor runs while the worker thread is exiting, under the Windows loader lock, and
the join deadlocks. ThreadPool::~ThreadPool in src/thread_pool.cc then blocks
forever in worker->join().

Sequence when a model is destroyed:

  1. ~ReplicaPool~ThreadPool closes the queue and calls worker->join()
  2. the worker's run() loop returns and the thread begins to terminate
  3. the thread's thread_local destructors run, so ~ruy::Context joins the Ruy threads
  4. that join, on a thread that is already exiting, deadlocks

This only happens on CPU. On CUDA the GEMM never calls get_ruy_context(), so the
worker thread has nothing to join and exits cleanly. That matches the reports that GPU
builds don't hang while CPU builds do.

The fix is to destroy the context from ReplicaWorker::finalize() instead. finalize()
runs on the worker thread inside run(), before the thread exits, so the join happens
in a normal context (no loader lock) and completes. It already calls
destroy_context(device) for this kind of per-thread cleanup (freeing curand state on
CUDA today), so the Ruy cleanup goes in the same place:

  • backend.cc: make the context a resettable heap pointer and add clear_ruy_context()
  • backend.h: declare it
  • devices.cc: call cpu::clear_ruy_context() from destroy_context() for Device::CPU

Everything stays under the existing CT2_WITH_RUY / CT2_WITH_CUDA guards, so other
backends and platforms are untouched.

Besides fixing the hang, this frees the context instead of leaking it. RSS after each
of 5 load/translate/unload cycles (Windows, MSVC 14.44, x64, int8 NLLB-200 600M, Ruy):

cycle 1..5 (MB)
bypassing the model's destructor (workaround used downstream) 885, 1755, 2624, 3496, 4371
heap-leaking just the ruy::Context 268, 520, 772, 1024, 1276
this PR 16, 17, 18, 18, 19

The ~250 MB/cycle in the middle row is Ruy's prepacked cache, which the leak keeps
alive; finalize-clear releases it. Translation output is unchanged and identical across
repeated calls, so rebuilding the context after clearing it is fine.

Found via jkawamoto/ctranslate2-rs#64, where the hang was worked around by skipping the
model's destructor entirely (avoiding the hang but leaking the model). With this change
the destructor can run.

Happy to adjust the approach if you'd prefer the cleanup somewhere else. Thanks!

The per-thread ruy::Context in src/cpu/backend.cc is a static thread_local
whose destructor joins Ruy's internal thread pool. On Windows that join runs
while the worker thread is exiting (under the loader lock) and deadlocks
ThreadPool shutdown; ThreadPool::~ThreadPool then blocks forever in
worker->join().

Destroy the context explicitly from ReplicaWorker::finalize() via
destroy_context() instead. finalize() runs on the worker thread in a normal
context (not thread exit), so the join completes and the context is freed
rather than leaked.

Refs jkawamoto/ctranslate2-rs#64
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant