Fix Windows shutdown deadlock and leak with the Ruy backend#2076
Open
timo9378 wants to merge 1 commit into
Open
Fix Windows shutdown deadlock and leak with the Ruy backend#2076timo9378 wants to merge 1 commit into
timo9378 wants to merge 1 commit into
Conversation
The per-thread ruy::Context in src/cpu/backend.cc is a static thread_local whose destructor joins Ruy's internal thread pool. On Windows that join runs while the worker thread is exiting (under the loader lock) and deadlocks ThreadPool shutdown; ThreadPool::~ThreadPool then blocks forever in worker->join(). Destroy the context explicitly from ReplicaWorker::finalize() via destroy_context() instead. finalize() runs on the worker thread in a normal context (not thread exit), so the join completes and the context is freed rather than leaked. Refs jkawamoto/ctranslate2-rs#64
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi, thanks for CTranslate2. I ran into a Windows-only shutdown hang and tracked it down
to the Ruy backend, so here's a fix.
Creating and destroying a
Translator(orGenerator/Whisper) with the Ruybackend on Windows hangs the process on shutdown. The culprit is the per-thread
ruy::Contextinsrc/cpu/backend.cc:It's constructed lazily on a worker thread the first time that thread runs a GEMM,
and its destructor joins Ruy's internal thread pool. Because it's
thread_local, thatdestructor runs while the worker thread is exiting, under the Windows loader lock, and
the join deadlocks.
ThreadPool::~ThreadPoolinsrc/thread_pool.ccthen blocksforever in
worker->join().Sequence when a model is destroyed:
~ReplicaPool→~ThreadPoolcloses the queue and callsworker->join()run()loop returns and the thread begins to terminatethread_localdestructors run, so~ruy::Contextjoins the Ruy threadsThis only happens on CPU. On CUDA the GEMM never calls
get_ruy_context(), so theworker thread has nothing to join and exits cleanly. That matches the reports that GPU
builds don't hang while CPU builds do.
The fix is to destroy the context from
ReplicaWorker::finalize()instead.finalize()runs on the worker thread inside
run(), before the thread exits, so the join happensin a normal context (no loader lock) and completes. It already calls
destroy_context(device)for this kind of per-thread cleanup (freeing curand state onCUDA today), so the Ruy cleanup goes in the same place:
backend.cc: make the context a resettable heap pointer and addclear_ruy_context()backend.h: declare itdevices.cc: callcpu::clear_ruy_context()fromdestroy_context()forDevice::CPUEverything stays under the existing
CT2_WITH_RUY/CT2_WITH_CUDAguards, so otherbackends and platforms are untouched.
Besides fixing the hang, this frees the context instead of leaking it. RSS after each
of 5 load/translate/unload cycles (Windows, MSVC 14.44, x64, int8 NLLB-200 600M, Ruy):
ruy::ContextThe ~250 MB/cycle in the middle row is Ruy's prepacked cache, which the leak keeps
alive; finalize-clear releases it. Translation output is unchanged and identical across
repeated calls, so rebuilding the context after clearing it is fine.
Found via jkawamoto/ctranslate2-rs#64, where the hang was worked around by skipping the
model's destructor entirely (avoiding the hang but leaking the model). With this change
the destructor can run.
Happy to adjust the approach if you'd prefer the cleanup somewhere else. Thanks!