Propagate BaseException raised in pool workers - #452
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes Pool worker behavior so that tasks raising BaseException (e.g., BaseException, SystemExit, KeyboardInterrupt) are marshalled back to the parent process and re-raised by ApplyResult.get(), instead of causing the worker to exit and the parent to raise a misleading WorkerLostError (closes #427).
Changes:
- Catch
BaseException(not justException) around task execution in pool workers so failures are returned asExceptionInfo. - Add a unit test asserting that a
BaseExceptionraised in a worker is re-raised byresult.get(). - Document the behavioral change in
CHANGES.txt.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
billiard/pool.py |
Catch BaseException in the worker task execution path so exceptions are marshalled instead of killing the worker. |
t/unit/test_pool.py |
Adds a regression test to ensure BaseException propagates to the caller via ApplyResult.get(). |
CHANGES.txt |
Notes the unreleased behavior change and references the fixing issue. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
A task raising BaseException (e.g. plain BaseException, SystemExit, KeyboardInterrupt) was not caught by the worker's result marshalling (except Exception), so the worker exited prematurely and the caller got a confusing WorkerLostError instead of the exception. Catch BaseException so any exception raised by a task is propagated back to the caller. Closes celery#427
chjnett
force-pushed
the
fix/propagate-base-exception
branch
from
August 16, 2026 16:37
c677609 to
856940e
Compare
auvipy
approved these changes
Aug 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #427.
A task that raises a
BaseException(plainBaseException,SystemExit,KeyboardInterrupt, …) was not caught by the worker's result marshalling —the worker only caught
Exception— so the exception propagated out of theworker, the worker exited prematurely, and the caller got a confusing
WorkerLostErrorinstead of the exception.Change:
billiard/pool.py: the worker's task-execution catch is nowexcept BaseException, so any exception a task raises is marshalled back as aresult and re-raised by
ApplyResult.get()instead of killing the worker.Test:
test_base_exception_propagates, which submits a task raisingBaseExceptionand assertsresult.get()raises it.Ran
pytest t/unit/test_pool.pylocally — all 7 tests pass (including thenew one); without the fix the new test fails. The
ExceptionInfomachineryalready handles any exception type (it rebuilds the original exception when
unpickling), so no further changes were needed on the parent side.