Skip to content

Propagate MemoryError from pool callbacks - #451

Merged
auvipy merged 1 commit into
celery:mainfrom
sahilmathur254:fix/10462-propagate-memoryerror-callback
Aug 13, 2026
Merged

Propagate MemoryError from pool callbacks#451
auvipy merged 1 commit into
celery:mainfrom
sahilmathur254:fix/10462-propagate-memoryerror-callback

Conversation

@sahilmathur254

Copy link
Copy Markdown
Contributor

Summary

  • Propagate MemoryError raised by pool callbacks instead of logging and
    swallowing it.
  • Add regression coverage through the ApplyResult._set() callback path.

On Python 3, MemoryError is an Exception, so the generic callback handler
currently catches it. Celery's request callback expects this exception to escape,
and swallowing it can leave an acks_late message unacknowledged. The explicit
MemoryError branch restores propagation without changing how other callback
exceptions or callbacks_propagate are handled.

Related to celery/celery#10462. A Celery regression around Request.on_failure
remains a follow-up after the billiard behavior is available.

Validation

All tests were run in an isolated Python 3.12 Docker container.

  • Before the fix, the new regression failed with DID NOT RAISE MemoryError and
    logged the callback exception.
  • python -m pytest t/unit/test_pool.py -q: 6 passed.
  • python -m pytest t/unit -q: 39 passed, 51 skipped.
  • python -m compileall -q billiard t/unit: passed.
  • git diff --check: passed.

AI disclosure

This PR includes AI-assisted code understanding to speed up the process; the
implementation and tests were updated and reviewed manually with AI assistance.

@sahilmathur254
sahilmathur254 marked this pull request as ready for review August 13, 2026 16:46
@auvipy
auvipy requested review from auvipy and a lite review from Copilot August 13, 2026 17:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates billiard.pool.ApplyResult callback execution so that MemoryError raised inside pool callbacks is propagated instead of being logged and swallowed, matching Celery’s expectation on Python 3 and avoiding situations where acks_late tasks can remain unacknowledged.

Changes:

  • Update ApplyResult.safe_apply_callback() to explicitly re-raise MemoryError (while preserving existing callbacks_propagate behavior for other exception types).
  • Add a unit regression test that exercises the ApplyResult._set() → callback path and asserts MemoryError propagation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
billiard/pool.py Ensures MemoryError from ApplyResult callbacks escapes instead of being logged/swallowed.
t/unit/test_pool.py Adds regression coverage proving MemoryError propagates through ApplyResult._set() callback execution.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@auvipy
auvipy merged commit 001b9f0 into celery:main Aug 13, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants