Skip to content

refactor: extract global coordination into invariant-coordinator crate#12

Merged
JeancarloBarrios merged 1 commit into
mainfrom
feat/scheduler-coordinator-split
May 30, 2026
Merged

refactor: extract global coordination into invariant-coordinator crate#12
JeancarloBarrios merged 1 commit into
mainfrom
feat/scheduler-coordinator-split

Conversation

@JeancarloBarrios

Copy link
Copy Markdown
Contributor

What

Slice 0 of the local-scheduler build: split the local node scheduling layer from the global durable-ownership layer along a hard crate boundary.

  • New invariant-coordinator crate (WIP) — owns leases, fencing epochs, the lease-laden Job aggregate, and WorkerId. This is the global "who may run this job" layer, parked here until it's fully designed.
  • invariant-scheduler is now lease-freedomain/ is just SchedulerTime + a DomainError trimmed to local variants. It does not depend on the coordinator crate (work flows down, not up).
  • No shared-kernel crate — each crate owns its value types and translates at the boundary (deliberate; locked design decision).
  • PLAN.md — the slice-by-slice implementation plan for the local layer (Slices 1–5 follow).

Why

The old Job jammed global lease/fencing concerns together with local scheduling. The crate boundary is the compiler-enforced wall that keeps the two bounded contexts from leaking into each other.

Verification

  • cargo build
  • cargo test ✅ — 27 coordinator tests pass unchanged, scheduler tests green
  • cargo clippy --all-targets -- -D warnings
  • cargo fmt --all --check
  • Reviewed via a 3-lens reflection loop (Rust idiom / DDD-boundary / build-hygiene) — converged with zero actionable findings.

Scope

Mechanical split only. The lease-free local Job, JobOrder, ReadyQueue, ReadyClock, slot/admission boundaries, and the run loop are future slices per PLAN.md.

@coderabbitai

coderabbitai Bot commented May 30, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Summary by CodeRabbit

  • Chores
    • Added two new internal modules for job coordination and scheduling infrastructure.
    • Implemented domain models supporting job lifecycle management, including claim, renewal, completion, and release operations.
    • Added lease management with expiration validation and renewal capabilities.
    • Introduced worker identification and comprehensive error handling for domain-specific validation scenarios.

Walkthrough

This PR introduces two new Rust crates to the workspace: invariant-coordinator for global lease-based job coordination and invariant-scheduler for local worker-side job execution. The coordinator crate defines a comprehensive domain model with error handling, job lifecycle state machine, lease validation, and credential-based ownership semantics. The scheduler crate establishes initial scaffolding with its own domain layer focusing on local scheduling concerns.

Changes

Invariant Coordinator - Global Lease Coordination

Layer / File(s) Summary
Crate scaffolding and workspace integration
Cargo.toml, crates/invariant-coordinator/Cargo.toml, crates/invariant-coordinator/src/lib.rs
Workspace members list expanded to include the coordinator crate; coordinator manifest and lib.rs established with public domain module export.
Error contracts and foundation value types
crates/invariant-coordinator/src/domain/error.rs, crates/invariant-coordinator/src/domain/worker.rs, crates/invariant-coordinator/src/domain/time.rs
DomainError enum with leasing and job failure variants; Display and Error trait implementations. WorkerId and SchedulerTime value types with non-empty validation, overflow handling, and epoch-relative constructors.
Job lease model with credential validation
crates/invariant-coordinator/src/domain/lease.rs
LeaseToken, LeaseEpoch, LeaseDuration value types with validation. LeaseCredential bundles holder/token/epoch. JobLease aggregate computes expiry, validates credentials, and supports renewal with extended expiry while preserving token and epoch.
Job aggregate and state machine
crates/invariant-coordinator/src/domain/job.rs
JobId, TargetRef, AttemptNumber, JobStatus value types and enum with terminal-state detection. Job aggregate tracks lease, status, and attempt count; provides claim/renew_lease/complete/release lifecycle methods with state transition enforcement, lease ownership validation, and epoch management.
Domain module structure and exports
crates/invariant-coordinator/src/domain/mod.rs
Domain module declares error, job, lease, time, and worker submodules; re-exports key public types for cohesive domain::* API surface.

Invariant Scheduler - Local Worker Scheduling

Layer / File(s) Summary
Crate scaffolding and module structure
crates/invariant-scheduler/Cargo.toml, crates/invariant-scheduler/src/lib.rs
Scheduler crate manifest and lib.rs with public domain module export; establishes foundation for local execution layer.
Scheduler domain: errors and time
crates/invariant-scheduler/src/domain/error.rs, crates/invariant-scheduler/src/domain/time.rs, crates/invariant-scheduler/src/domain/mod.rs
DomainError enum with scheduler-specific variants (EmptyJobId, EmptyTargetRef, TimeOverflow, AttemptOverflow, JobTerminal, QueueFull); Display and Error trait implementations. SchedulerTime value type with epoch-based constructors and checked-addition overflow handling. Domain module structure and re-exports.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested reviewers

  • PJEstrada

🐰 Two new crates hop into the pen,
Coordination and scheduling threads tied tight,
Leases with epochs dance through the domain,
State machines hop forward with all their might,
The coordinator keeps the flock in sight! 🎯

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 30.93% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title 'refactor: extract global coordination into invariant-coordinator crate' accurately describes the main change: creating a new crate to separate global coordination concerns from local scheduling.
Description check ✅ Passed The description clearly explains the changes: split of scheduling layers, new invariant-coordinator crate ownership, scheduler simplification, and verification steps. It is directly related to the changeset.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/scheduler-coordinator-split

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/invariant-coordinator/src/domain/lease.rs`:
- Around line 184-185: The renew method currently assigns self.renewed_at before
computing self.expires_at, causing self.renewed_at to be mutated even if
checked_add_duration overflows; to fix, first compute the new expiration with
now.checked_add_duration(duration.as_duration())? and only if that returns Some
(or Ok) assign both self.renewed_at = now and self.expires_at = new_expires,
otherwise return the error without mutating fields; update the renew
implementation to perform the checked_add_duration call and validation prior to
any mutation of renewed_at/expires_at.

In `@crates/invariant-scheduler/PLAN.md`:
- Line 5: Replace the local filesystem reference
`~/.claude/.../memory/invariant-scheduler-local-design.md` in PLAN.md with a
repository-accessible link or embedded decision info: either point to a
checked-in ADR/decision file (e.g., docs/decisions/invariant-scheduler-*.md)
using a relative path, or list the included decision IDs (1–11) and their
summaries inline so CI/contributors can resolve them without relying on a
home-directory file; update the Locked decisions line accordingly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 65588f38-2a8d-4cde-aa94-4f6be84f2b5d

📥 Commits

Reviewing files that changed from the base of the PR and between 044ee25 and ab77dd1.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (15)
  • Cargo.toml
  • crates/invariant-coordinator/Cargo.toml
  • crates/invariant-coordinator/src/domain/error.rs
  • crates/invariant-coordinator/src/domain/job.rs
  • crates/invariant-coordinator/src/domain/lease.rs
  • crates/invariant-coordinator/src/domain/mod.rs
  • crates/invariant-coordinator/src/domain/time.rs
  • crates/invariant-coordinator/src/domain/worker.rs
  • crates/invariant-coordinator/src/lib.rs
  • crates/invariant-scheduler/Cargo.toml
  • crates/invariant-scheduler/PLAN.md
  • crates/invariant-scheduler/src/domain/error.rs
  • crates/invariant-scheduler/src/domain/mod.rs
  • crates/invariant-scheduler/src/domain/time.rs
  • crates/invariant-scheduler/src/lib.rs

Comment on lines +184 to +185
self.renewed_at = now;
self.expires_at = now.checked_add_duration(duration.as_duration())?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Make renew state updates atomic on overflow failure.

renewed_at is assigned before checked_add_duration. If checked_add_duration overflows, renew returns Err with renewed_at already mutated, leaving inconsistent state after a failed renewal.

Suggested fix
     pub fn renew(
         &mut self,
         credential: &LeaseCredential,
         now: SchedulerTime,
         duration: LeaseDuration,
     ) -> Result<(), DomainError> {
         self.require_current(credential, now)?;
-        self.renewed_at = now;
-        self.expires_at = now.checked_add_duration(duration.as_duration())?;
+        let new_expires_at = now.checked_add_duration(duration.as_duration())?;
+        self.renewed_at = now;
+        self.expires_at = new_expires_at;
         Ok(())
     }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
self.renewed_at = now;
self.expires_at = now.checked_add_duration(duration.as_duration())?;
pub fn renew(
&mut self,
credential: &LeaseCredential,
now: SchedulerTime,
duration: LeaseDuration,
) -> Result<(), DomainError> {
self.require_current(credential, now)?;
let new_expires_at = now.checked_add_duration(duration.as_duration())?;
self.renewed_at = now;
self.expires_at = new_expires_at;
Ok(())
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/invariant-coordinator/src/domain/lease.rs` around lines 184 - 185, The
renew method currently assigns self.renewed_at before computing self.expires_at,
causing self.renewed_at to be mutated even if checked_add_duration overflows; to
fix, first compute the new expiration with
now.checked_add_duration(duration.as_duration())? and only if that returns Some
(or Ok) assign both self.renewed_at = now and self.expires_at = new_expires,
otherwise return the error without mutating fields; update the renew
implementation to perform the checked_add_duration call and validation prior to
any mutation of renewed_at/expires_at.

Comment thread crates/invariant-scheduler/PLAN.md Outdated

Status: ready to implement · Author hands-on (you implement; this is the spec to work from)
Research backing: `research/local-layer.md`
Locked decisions: `~/.claude/.../memory/invariant-scheduler-local-design.md` (decisions 1–11)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Replace local filesystem reference with a repo-accessible source.

Line 5 points to ~/.claude/..., which other contributors/CI cannot resolve. Please link to a checked-in ADR/decision doc (or embed the decision IDs here) so the plan remains reproducible.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/invariant-scheduler/PLAN.md` at line 5, Replace the local filesystem
reference `~/.claude/.../memory/invariant-scheduler-local-design.md` in PLAN.md
with a repository-accessible link or embedded decision info: either point to a
checked-in ADR/decision file (e.g., docs/decisions/invariant-scheduler-*.md)
using a relative path, or list the included decision IDs (1–11) and their
summaries inline so CI/contributors can resolve them without relying on a
home-directory file; update the Locked decisions line accordingly.

Split the local node scheduling layer from the global durable-ownership
layer along a hard crate boundary (bounded-context wall).

- Add `invariant-coordinator` (WIP): owns leases, fencing epochs, the
  lease-laden Job aggregate, and WorkerId — the global "who may run this"
  layer. Parked here until the global layer is fully designed.
- Make `invariant-scheduler` lease-free: domain/ is now SchedulerTime +
  a DomainError trimmed to local variants. No upward dependency on the
  coordinator crate (work flows down, not up).
- No shared-kernel crate: each crate owns its value types and translates
  at the boundary.

All 27 coordinator tests pass unchanged; scheduler tests green.
@JeancarloBarrios
JeancarloBarrios force-pushed the feat/scheduler-coordinator-split branch from ab77dd1 to 50afe5b Compare May 30, 2026 00:26

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
crates/invariant-coordinator/src/domain/lease.rs (1)

183-186: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Compute the new expiry before mutating renewed_at.

renew assigns self.renewed_at = now before calling checked_add_duration. On overflow the method returns Err with renewed_at already advanced while expires_at is unchanged, leaving the lease in an inconsistent state after a failed renewal.

Suggested fix
         self.require_current(credential, now)?;
-        self.renewed_at = now;
-        self.expires_at = now.checked_add_duration(duration.as_duration())?;
+        let new_expires_at = now.checked_add_duration(duration.as_duration())?;
+        self.renewed_at = now;
+        self.expires_at = new_expires_at;
         Ok(())
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/invariant-coordinator/src/domain/lease.rs` around lines 183 - 186, The
renew implementation mutates self.renewed_at before computing expires_at,
causing an inconsistent state if checked_add_duration overflows; change renew
(the method containing self.require_current, self.renewed_at, self.expires_at)
to first compute new_expires = now.checked_add_duration(duration.as_duration())?
and only after that succeeds assign self.renewed_at = now and self.expires_at =
new_expires, preserving atomicity on error.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/invariant-coordinator/src/domain/job.rs`:
- Around line 134-138: The code in
crates/invariant-coordinator/src/domain/job.rs uses a let-chain (`if let
Some(current) = &self.current_lease && !current.is_expired_at(now)`) which
requires Rust 1.88+, so make the MSRV explicit: add rust-version = "1.88" either
to the root workspace Cargo.toml under a [workspace.package] table (to apply to
all members) or add rust-version = "1.88" to each package Cargo.toml (including
the invariant-coordinator crate) so older toolchains won't fail to compile.

---

Duplicate comments:
In `@crates/invariant-coordinator/src/domain/lease.rs`:
- Around line 183-186: The renew implementation mutates self.renewed_at before
computing expires_at, causing an inconsistent state if checked_add_duration
overflows; change renew (the method containing self.require_current,
self.renewed_at, self.expires_at) to first compute new_expires =
now.checked_add_duration(duration.as_duration())? and only after that succeeds
assign self.renewed_at = now and self.expires_at = new_expires, preserving
atomicity on error.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 15368ea0-c831-4c65-b242-6cdbc8c6d30b

📥 Commits

Reviewing files that changed from the base of the PR and between ab77dd1 and 50afe5b.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (14)
  • Cargo.toml
  • crates/invariant-coordinator/Cargo.toml
  • crates/invariant-coordinator/src/domain/error.rs
  • crates/invariant-coordinator/src/domain/job.rs
  • crates/invariant-coordinator/src/domain/lease.rs
  • crates/invariant-coordinator/src/domain/mod.rs
  • crates/invariant-coordinator/src/domain/time.rs
  • crates/invariant-coordinator/src/domain/worker.rs
  • crates/invariant-coordinator/src/lib.rs
  • crates/invariant-scheduler/Cargo.toml
  • crates/invariant-scheduler/src/domain/error.rs
  • crates/invariant-scheduler/src/domain/mod.rs
  • crates/invariant-scheduler/src/domain/time.rs
  • crates/invariant-scheduler/src/lib.rs

Comment on lines +134 to +138
if let Some(current) = &self.current_lease
&& !current.is_expired_at(now)
{
return Err(DomainError::JobAlreadyLeased);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Inspect coordinator crate manifest and workspace for edition / MSRV
fd -t f 'Cargo.toml' | while read -r f; do
  echo "==== $f ===="
  rg -n 'edition|rust-version|\[workspace\.package\]|\[package\]' "$f"
done

Repository: ybird-labs/invariant

Length of output: 473


🏁 Script executed:

#!/bin/bash
# Find any rust-version/MSRV pins across the workspace Cargo.toml files
fd -t f 'Cargo.toml' | while read -r f; do
  echo "==== $f ===="
  rg -n 'rust-version' "$f" || true
done

Repository: ybird-labs/invariant

Length of output: 318


🌐 Web query:

Rust let chains stabilized version if let ... && condition

💡 Result:

As of May 30, 2026, "let chains" are a stabilized feature in the Rust language, introduced in Rust 1.88 [1][2]. This feature allows the use of let expressions alongside boolean conditions within if and while statements, connected by the && operator [2][3]. Key details regarding this feature include: - Edition Requirement: Let chains are stabilized specifically for the 2024 edition and later editions [1][4]. They are not available in older editions (2015, 2018, 2021) [1][5]. - Functionality: You can intermix let patterns (which may be refutable) with standard boolean expressions [2][3]. The expression evaluates from left to right; if a let pattern match fails or a boolean expression evaluates to false, the subsequent operands are not evaluated (short-circuiting) [5][6]. - Syntax Constraints: If any condition operand is a let pattern, the || operator cannot be used directly in the chain due to ambiguity [5][6]. If a logical OR is required, parentheses must be used (e.g., if let Some(x) = foo && (cond_a || cond_b) { ... }) [5][6]. - Usage: This feature significantly reduces code nesting by allowing flat chains of conditions rather than deeply nested if let blocks [3]. It is also supported in match guards [4][3].

Citations:


Confirm MSRV for let chains (if let ... && ...)
The code at crates/invariant-coordinator/src/domain/job.rs uses if let Some(current) = &self.current_lease && !current.is_expired_at(now), which requires Rust 1.88+ (let-chains; works with edition = "2024"). All workspace crates use edition = "2024", but no rust-version/MSRV is pinned in the workspace manifests, so older toolchains could fail to compile. Consider adding rust-version = "1.88" to make the MSRV explicit.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/invariant-coordinator/src/domain/job.rs` around lines 134 - 138, The
code in crates/invariant-coordinator/src/domain/job.rs uses a let-chain (`if let
Some(current) = &self.current_lease && !current.is_expired_at(now)`) which
requires Rust 1.88+, so make the MSRV explicit: add rust-version = "1.88" either
to the root workspace Cargo.toml under a [workspace.package] table (to apply to
all members) or add rust-version = "1.88" to each package Cargo.toml (including
the invariant-coordinator crate) so older toolchains won't fail to compile.

@JeancarloBarrios
JeancarloBarrios merged commit 239ce96 into main May 30, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant