Swartzn/feat/add xtreemstore rst - #312
Conversation
407202d to
fa556f3
Compare
swartzn
changed the title
iamjoemccormick
left a comment
iamjoemccormick
left a comment
There was a problem hiding this comment.
Dropping a note I finished with my initial pass through this.
a2a3f5e to
4e1d63a
Compare
634f0aa to
cce8e6e
Compare
iamjoemccormick
left a comment
iamjoemccormick
left a comment
There was a problem hiding this comment.
Submitting review feedback on the first two commits (up to "add xtreemstore provider").
| executeAfter := rst.GetJobExecuteAfter(j.Get()) | ||
| j.Segments = make([]*Segment, 0, len(workRequests)) | ||
| for _, wr := range workRequests { | ||
| if executeAfter != nil { | ||
| wr.SetExecuteAfter(executeAfter) | ||
| } |
There was a problem hiding this comment.
todo: unless I'm missing something doesn't this still do what we were trying to avoid? Generating an timestamp on one node (Remote) that is then passed to the Sync nodes which who's times may not be in sync?
I would prefer instead we make this a DelayExecution field on the job request that uses the protobuf duration type that is propagated to a DelayExecution field on the work requests.
I would also propose this is set by the GenerateWorkRequests method, if appropriate for that RST client type. It doesn't feel right to set this one field in GenerateSubmission().
There was a problem hiding this comment.
I saw this but was distracted by the other comments and forgot to come back to it. Anyway, this required adding a DelayExecution field to both beeremote.JobRequest and flex.WorkRequest which allows the sync node to convert the delay to ExecuteAfter in SubmitWorkRequest. GenerateWorkRequests is still called by remote.
See e85cf75
| zap.Bool("hasWorkResult", entry.WorkResult != nil), | ||
| zap.Bool("hasStatus", status != nil), | ||
| ) | ||
| m.scheduler.AddRescheduleWorkToken(submissionId, time.Time{}) |
There was a problem hiding this comment.
question: so when this happens we still add a token because when the journal is replayed later this WR will still get picked up and presumably the invalid result sent back to Remote?
Edit: I see this was added with the "update job state to running..." commit. Was this a bug? If so it'd be worth a mention in the commit message.
There was a problem hiding this comment.
Yes, the work needs to be processed in order to complete/update remote.
It was a bug. I missed adding m.scheduler.AddRescheduleWorkToken(submissionId, time.Time{}) the first time. Showed up in my testing.
iamjoemccormick
left a comment
iamjoemccormick
left a comment
There was a problem hiding this comment.
Posting review comments for the remaining commits.
| if _, err := w.beeRemoteClient.UpdateWorkRequest(work.ctx, result.Work); err != nil { | ||
| log.Warn("unable to update remote job status to running; continuing work request without retrying", zap.Error(err)) | ||
| } |
There was a problem hiding this comment.
suggestion(blocking): we discussed this on Slack, but adding a note here so we don't loose track.
Sync intentionally never informed Remote when a work request shifted to "running" to reduce load on Remote and the jobs DB. I would prefer we keep it that way, as for the most part Remote never needs to know about this unless a user checks the job status.
What we could do is implement this issue https://github.com/ThinkParQ/bee-remote/issues/14 so if the user runs remote status or remote job list Remote will reach out and refresh the job and work request statuses. The GetJobsRequest message already has a UpdateWorkResults field that was intended to control if only the latest results known to Remote are returned, or if it also probes the Sync nodes.
We could either set UpdateWorkResults by default for CTL commands, or add a new --refresh-results flag for this. My thinking is for the most part users only care to know once jobs reach a terminal state, the intermediate states aren't that interesting unless you're trying to debug specific job issues.
cce8e6e to
b01a2c0
Compare
2ba6ab2 to
9d97563
Compare
iamjoemccormick
left a comment
iamjoemccormick
left a comment
There was a problem hiding this comment.
Just a handful of remaining observations, my comment on GenerateSubmission() is the main blocker.
| } | ||
|
|
||
| } else { | ||
| workRequests = rst.RecreateWorkRequests(j.Get(), j.GetSegments()) |
There was a problem hiding this comment.
todo: the RecreateWorkRequests() path was updated to handle SetDelayExecution(), but I think we also need to set this in the GenerateWorkRequests() path?
I'm actually not sure if the RecreateWorkRequests() path in GenerateSubmission() is ever executed anymore, elsewhere just calls RecreateWorkRequests() directly.
| if client.IncludeInBulkRequest(walkCtx, jobRequest) { | ||
| rstId := jobRequest.GetRemoteStorageTarget() | ||
|
|
||
| bulkRequestStatesMu.Lock() |
There was a problem hiding this comment.
suggestion(non-blocking): to avoid lock contention in a hot path consider making this a RWMutex as after the initial population of bulkRequestStates the map is only read.
Assisted-by: Claude:claude-opus-4-6
|
|
||
| type SubmitBulkRequestFn func(ctx context.Context) | ||
| type EmitBulkRequestFn func(ctx context.Context, request *beeremote.JobRequest) | ||
| type AppendBulkRequestFn func(ctx context.Context, request *beeremote.JobRequest) |
There was a problem hiding this comment.
suggestion(non-blocking): we should mention AppendBulkRequestsFn is called under a lock so it should be fast. For example of a provider did I/O in append it could really slow everything down.
Assisted-by: Claude:claude-opus-4-6
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…xtension hooks Introduce an s3ApiClient interface backed by a default AWS SDK implementation.
* Add IncludeInBulkRequest and OpenBulkOperation to the Provider interface, along with the clientBulkOperation type * Give MockClient a working bulk-operation implementation for tests, and stub the new methods on S3Client and JobBuilderClient.
8c941da to
d23cac0
Compare
| return | ||
| } | ||
|
|
||
| // getBuilderResults generates a status message based on the builder submission counters. |
There was a problem hiding this comment.
Here were some generated examples,
| Builder / SchedulerResult | State | Status message |
|---|---|---|
submitted=250clean, mid-walkReschedule=true |
RESCHEDULED |
waiting for builder job to continue; 250 job request(s) submitted |
submitted=100, errors=5Reschedule=true |
RESCHEDULED |
waiting for builder job to continue; builder cancelled: 100 job request(s) submitted; 5 submitted with errors |
submitted=42clean, abortedErr=MarkBuilderCancelled("job builder request was aborted: context canceled") |
CANCELLED |
job builder failed to complete: job builder request was aborted: context canceled; 42 job request(s) submitted |
submitted=10, errors=3Err=MarkBuilderFailed("job builder request was aborted: unable to abort bulk operation: connection refused") |
FAILED |
job builder failed to complete: job builder request was aborted: unable to abort bulk operation: connection refused; builder cancelled: 10 job request(s) submitted; 3 submitted with errors |
submitted=7, alreadyExist=2Err="unexpected nil bulk result"unclassified |
FAILED |
job builder returned unclassified error: unexpected nil bulk result; 7 job request(s) submitted; 2 already exist |
submitted=180, alreadyComplete=6, notAllowed=2, errors=4Err=nil, Reschedule=false |
CANCELLED |
completed with errors; builder cancelled: 180 job request(s) submitted; 6 already complete; 2 not allowed; 4 submitted with errors |
submitted=500clean runErr=nil, Reschedule=false |
COMPLETED |
completed successfully; 500 job request(s) submitted |
alreadyComplete=1000nothing newErr=nil, Reschedule=false |
COMPLETED |
completed successfully; 1000 already complete |
notAllowed=50all conflictedErr=nil, Reschedule=false |
CANCELLED |
completed with errors; builder cancelled: 50 not allowed |
submitted=999, alreadyOffloaded=1Err=nil, Reschedule=false |
COMPLETED |
completed successfully; 999 job request(s) submitted; 1 already offloaded |
0 matcheslocal path /data/empty-dirmid-walkReschedule=true |
RESCHEDULED |
waiting for builder job to continue; builder cancelled: no matches found in local path: /data/empty-dir |
0 matchesremote path s3://bucket/incoming/Err=nil, Reschedule=false |
CANCELLED |
completed with errors; builder cancelled: no matches found in remote path: s3://bucket/incoming/ |
0 matchesdownload fallback to local walk /local/mirrorErr=nil, Reschedule=false |
CANCELLED |
completed with errors; builder cancelled: walked local path since --remote-path was not provided; No matches found in path: /local/mirror |
0 jobs countedaborted immediatelylocal path /data/xErr=MarkBuilderCancelled("job builder request was aborted: context canceled") |
CANCELLED |
job builder failed to complete: job builder request was aborted: context canceled; builder cancelled: no matches found in local path: /data/x |
d23cac0 to
6d4e8a0
Compare
Rework the job builder's request pipeline so remote targets that can batch many paths into a single
remote-side request can opt in, and fix job/work cancellation so it no longer corrupts local file state.
* Split the monolithic builder into focused files:
* builderjobrequest.go manages path processing and request generation.
* buildercontroller.go manages the path and bulk operation walks to request processing pipeline.
* builderbulk.go manages bulk operations. Implements what is needed to support
IncludeInBulkRequest, OpenBulkOperation, and clientBulkOperation so clients can create, add
paths, execute, resume, and cancel bulk operations.
* Replace GetLockedInfo with GetPathState which provides path state information and optionally takes
the file access lock. This primarily removes the rstIds handling from GetLockedInfo and the need
for a JobRequestCfg which was specifically needed by builder.
* Have PrepareFileStateForWorkRequests return an undo() closure and stop zeroing/truncating files in
preparation for downloads which are now only enlarged if needed. This leaves original contents intact
so a cancelled or failed job can restore it.
* Update S3Client's download complete function to finalize the file state on success or when the
job was aborted. To ensure accurate messaging, work parts now track whether they started to
distinguish between modified or not files.
* Sync workers now derives accurate job submission counts from remote's SubmitJobRequest responses.
So, the statuses: ALREADY_COMPLETE, ALREADY_OFFLOADED, EXISTING, NOT_ALLOWED, and
FAILED_PRECONDITION, are accurately reflected in job statuses.
* Cancelled or failed bulk operations will need to be cleaned so builder jobs can now be marked as
FAILED so clients can properly cleanup local and remote resources.
* Fix stale resume tokens when a cutoff lands right after a nested subdirectory's files, which caused already-sent paths to be resent on resume. * Remove directories from maxFiles (previously maxPaths) counter. * Prevent directory paths from being returned as resumeTokens.
The work journal previously passed entries by value instead of by reference so the update for ExecuteAfter performed on a copy and never updated the journal itself.
…alization * Replace ExecuteAfter.IsZero() heuristic with explicit state based dispatch on work state. Entries with running or completed states left by a crashed node are now replayed through AddRescheduleWorkToken. * Log unknown states context instead of silently misbehaving. * Startup log also includes replayed and unrecoverable
Allow remote to create job entries when SubmitJobRequest recieves a failed-precondition requests without a valid rstId. This allows a builder job request with a configured invalid rstId to not be silently discarded and notify the user of the situation.
Previously any call to SetFileRstPattern issued a SetFilePatternRequest even if the requested RST IDs and cooldown already matched the current config. Compare against the current values (ignoring RST ID order) and only send the request when something actually changed.
* Add running state for work requests * Update builder job with it's work result status message so users can see the builder job updates.
* Add parallelism to builder job submission on the sync workers. * Add CancelBulkOperationRequest to rst Provider so sync workers can notify clients when their bulk operation requests failed to be submitted. * Improve xtreemstore test execute function.
* Fix missing stub file creation apply-undo step in rst.PlanFileStateForWorkRequest() when file doesn't exist or is already synced. * Add pathState argument to rst.PlanFileStateForWorkRequest() apply function so steps can update pathState if needed based on plan alterations. * Fix s3 client bug where a builder-created job for a missing remote object treats causes the s3 job to be treated as though lockedInfo was not already determined. * Update s3 client GenerateWorkRequests() to use common rst helpers.
* Change rst clientBulkOperation's CancelBulkOperationRequest function to ExcludeRequestFromBulkOperation. * Remove clientBulkOperation's Resume function. * Fix processing paths from bulk operation so it can only generate one job request configuration. * Improve and clean xtreemstore * Add sync worker saturation metric so builder jobs can gauge whether to continue or reschedule. * Add parallelism to sync worker builder job submission processing.
* Remove residual tests for bulkStreamPathResultMultiplexer * Rename jobBuilderBulkOperationsManager to bulkOperationRegistry * Simplify requestBuildController * Remove external handlers and handles result merges internally * Add support for using the controller multiple times Fix dropped pathIssue in ProcessFromSource
…unsupported operations
CheckEntry has been unused since the access-state lock in GetLockedInfo replaced its read/write-session check.
342cf7d to
52b9071
Compare
* Add sync workersaturation to requestBuilderController to make requestBuildController's waitForSubmissionCapacity dynamic based on processed work throughput. * Add support for failing individual bulk operation managers so source and other bulk operations can finish. * Clean up bulk operation registry (formerly known as bulkOperationsManager). * Improve PlanFileStateForWorkRequests so apply/undo functionality * All added steps are given a chance to execute but with previous errors/sentinels forwarded as arguments. * Re-integrate setting file rst configuration into PlanFileStateForWorkRequests. * Fix S3Client * Job request preparation to properly use PlanFileStateForWorkRequests. * Move restoring a downloaded file's mtime till after reducing the file to the correct size. * Fix xtreemstore bulk retrieve Cancel so it properly closes the walk channel.
52b9071 to
4bf837e
Compare
2 participants
What does this PR do / why do we need it?
Required for all PRs.
Related Issue(s)
Required when applicable.
Where should the reviewer(s) start reviewing this?
Only required for larger PRs when this may not be immediately obvious.
Are there any specific topics we should discuss before merging?
Not required.
What are the next steps after this PR?
Not required.
Checklist before merging:
Required for all PRs.
When creating a PR these are items to keep in mind that cannot be checked by GitHub actions:
For more details refer to the Go coding standards and the pull request process.