Skip to content

Propagate BaseException raised in pool workers - #452

Merged
auvipy merged 1 commit into
celery:mainfrom
chjnett:fix/propagate-base-exception
Aug 16, 2026
Merged

Propagate BaseException raised in pool workers#452
auvipy merged 1 commit into
celery:mainfrom
chjnett:fix/propagate-base-exception

Conversation

@chjnett

@chjnett chjnett commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Closes #427.

A task that raises a BaseException (plain BaseException, SystemExit,
KeyboardInterrupt, …) was not caught by the worker's result marshalling —
the worker only caught Exception — so the exception propagated out of the
worker, the worker exited prematurely, and the caller got a confusing
WorkerLostError instead of the exception.

Change:

  • billiard/pool.py: the worker's task-execution catch is now except BaseException, so any exception a task raises is marshalled back as a
    result and re-raised by ApplyResult.get() instead of killing the worker.

Test:

  • Added test_base_exception_propagates, which submits a task raising
    BaseException and asserts result.get() raises it.

Ran pytest t/unit/test_pool.py locally — all 7 tests pass (including the
new one); without the fix the new test fails. The ExceptionInfo machinery
already handles any exception type (it rebuilds the original exception when
unpickling), so no further changes were needed on the parent side.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes Pool worker behavior so that tasks raising BaseException (e.g., BaseException, SystemExit, KeyboardInterrupt) are marshalled back to the parent process and re-raised by ApplyResult.get(), instead of causing the worker to exit and the parent to raise a misleading WorkerLostError (closes #427).

Changes:

  • Catch BaseException (not just Exception) around task execution in pool workers so failures are returned as ExceptionInfo.
  • Add a unit test asserting that a BaseException raised in a worker is re-raised by result.get().
  • Document the behavioral change in CHANGES.txt.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
billiard/pool.py Catch BaseException in the worker task execution path so exceptions are marshalled instead of killing the worker.
t/unit/test_pool.py Adds a regression test to ensure BaseException propagates to the caller via ApplyResult.get().
CHANGES.txt Notes the unreleased behavior change and references the fixing issue.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread t/unit/test_pool.py Outdated
A task raising BaseException (e.g. plain BaseException, SystemExit,
KeyboardInterrupt) was not caught by the worker's result marshalling
(except Exception), so the worker exited prematurely and the caller got
a confusing WorkerLostError instead of the exception. Catch BaseException
so any exception raised by a task is propagated back to the caller.

Closes celery#427
@chjnett
chjnett force-pushed the fix/propagate-base-exception branch from c677609 to 856940e Compare August 16, 2026 16:37
@auvipy
auvipy merged commit bd2f803 into celery:main Aug 16, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Raising BaseException unexpectedly results in WorkerLostError in Pool

3 participants