Skip to content

fix: prevent silent RecurringTask death and swallowed timeouts in AutoscaledPool#2009

Draft
vdusek wants to merge 1 commit into
masterfrom
fix/recurring-task-silent-failure
Draft

fix: prevent silent RecurringTask death and swallowed timeouts in AutoscaledPool#2009
vdusek wants to merge 1 commit into
masterfrom
fix/recurring-task-silent-failure

Conversation

@vdusek

@vdusek vdusek commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Problem

  • RecurringTask dies silently. RecurringTask ran its function bare in the loop, so the first exception permanently killed the underlying asyncio task. Nothing awaited the task until stop() (which suppresses exceptions), so state persistence, autoscaling snapshots, statistics logging, and browser cleanup could die silently mid-run.
  • AutoscaledPool swallows timeouts. AutoscaledPool._worker_task caught every asyncio.TimeoutError and logged "Task timed out after not set seconds". Since _TASK_TIMEOUT is never set, wait_for(timeout=None) can't time out itself, so the only timeouts it ever caught were real errors escaping the task function (e.g. session fetch or mark-as-handled timeouts). Swallowing them left the request stuck in progress and run() hung forever.

Changes

  • RecurringTask. The wrapper now logs the exception and keeps the recurrence going.
  • AutoscaledPool. The pool now re-raises when no task timeout is configured, so such errors propagate and stop the pool, matching how all other exceptions already behave.

Tests

Both fixes are covered by regression tests that failed before the change.

@vdusek vdusek added t-tooling Issues with this label are in the ownership of the tooling team. adhoc Ad-hoc unplanned task added during the sprint. labels Jul 2, 2026
@vdusek vdusek self-assigned this Jul 2, 2026
@codecov

codecov Bot commented Jul 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.36%. Comparing base (7780e78) to head (f527258).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
src/crawlee/_autoscaling/autoscaled_pool.py 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2009      +/-   ##
==========================================
+ Coverage   93.35%   93.36%   +0.01%     
==========================================
  Files         179      179              
  Lines       12482    12486       +4     
==========================================
+ Hits        11652    11658       +6     
+ Misses        830      828       -2     
Flag Coverage Δ
unit 93.36% <88.88%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

adhoc Ad-hoc unplanned task added during the sprint. t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants