Harden batch collections: cancellation, reattach, and error reporting - #317
Harden batch collections: cancellation, reattach, and error reporting#317JyotinderSingh wants to merge 1 commit into
Conversation
Four defects in kinetic.collections surfaced by a docs audit. cancel() now stops the whole collection. It sets a flag the submission loop reads, so inputs queued behind a bounded max_concurrent are dropped instead of launched, and cancelled indices are marked not-retryable so retries > 0 no longer resubmits a cancelled job when cancellation turns its status NOT_FOUND. A job registered while cancel() was mid-snapshot is cancelled by the submission loop, which closes that race. attach_batch() decides completeness from the child indices the manifest names, not from how many handles loaded. results(cleanup=True) deletes a child's whole GCS prefix including handle.json, so a later reattach could not rebuild those children and blocked wait()/results() forever behind a poll thread with no timeout. A named child whose handle is gone is now terminal and reported by the new BatchHandle.unavailable_children, and poll_timeout defaults to 30 minutes. _all_accounted_for follows the same rule: once submission is complete the jobs list is frozen, so as_completed() no longer waits on slots that can never hold a job. BatchError.failures holds only JobHandles, so the documented "for job in e.failures: job.job_id" no longer raises AttributeError. Inputs that fail at submission time are reported through the new BatchError.submission_failures, and they raise BatchError rather than being reachable only from the handle. map(max_concurrent=None, retries=0, fail_fast=True) hands back a handle immediately again. Once every input is launched, fail_fast alone leaves the loop nothing to act on, so it no longer polls, and the calling-thread decision mirrors that predicate. Adds 23 tests across the four areas, each verified to fail against the previous behaviour, and updates the batched-jobs guide and API reference.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #317 +/- ##
=======================================
Coverage ? 36.38%
=======================================
Files ? 83
Lines ? 15511
Branches ? 0
=======================================
Hits ? 5644
Misses ? 9867
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request enhances the robustness of batched job execution, cancellation, and reattachment in Kinetic. Key changes include separating submission-time errors from runtime failures via a new submission_failures attribute, ensuring failures only contains valid JobHandles, and improving cancel() to drop queued inputs and prevent retries of cancelled jobs. Additionally, attach_batch() now handles cleaned-up children through unavailable_children and introduces a default 30-minute timeout for polling partially submitted manifests to prevent infinite blocking. Comprehensive unit tests have been added to cover these scenarios. I have no further feedback to provide as the implementation is robust, well-tested, and adheres to the repository's design guidelines.
2 participants
Description
Fixes four defects in
kinetic.collectionscancel()now stops the whole collection. It sets a flag the submission loop reads, so inputs queued behind a boundedmax_concurrentare dropped instead of launched, and cancelled indices are marked not-retryable soretries > 0no longer resubmits a cancelled job when cancellation turns its statusNOT_FOUND. A job registered whilecancel()was mid-snapshot is cancelled by the submission loop, which closes that race.attach_batch()decides completeness from the child indices the manifest names, not from how many handles loaded.results(cleanup=True)deletes a child's whole GCS prefix includinghandle.json, so a later reattach could not rebuild those children and blockedwait()/results()forever behind a poll thread with no timeout. A named child whose handle is gone is now terminal and reported by the new BatchHandle.unavailable_children, and poll_timeout defaults to 30 minutes._all_accounted_forfollows the same rule: once submission is complete the jobs list is frozen, so as_completed() no longer waits on slots that can never hold a job.BatchError.failuresholds onlyJobHandles, so the documented "for job in e.failures: job.job_id" no longer raisesAttributeError. Inputs that fail at submission time are reported through the newBatchError.submission_failures, and they raiseBatchErrorrather than being reachable only from the handle.map(max_concurrent=None, retries=0, fail_fast=True)hands back a handle immediately again. Once every input is launched,fail_fastalone leaves the loop nothing to act on, so it no longer polls, and the calling-thread decision mirrors that predicate.Contributor Agreement