From 6a030c515f20d0d5cd83bd3d15334cb58d0aac7f Mon Sep 17 00:00:00 2001 From: HyeockJinKim Date: Sat, 18 Jul 2026 17:46:30 +0900 Subject: [PATCH 1/2] docs(BA-6926): amend BEP-1055 with scope-local job_priority for preemption Preemption compared the global scheduler priority across all owners, so a user could raise their own priority to evict another owner's sessions. This amendment introduces a separate scope-local axis, job_priority, as the sole victim-comparison key, restricted to the same owner (v1: user scope). - 2.4 Priority Model: three-axis split (scheduler priority / job_priority / fair-share), user vs future project scope, BEP-1014 within-owner caveat - 3.b: victim filter uses job_priority within the same user_uuid; drop the preemptible_priority threshold, keep preemption_order - 3.a: snapshot carries job_priority + user_uuid (scope-owner key) - Keypair resource policy max_priority caps the global priority - Open Questions: cross-tenant preemption as a future admin cross-scope axis Co-Authored-By: Claude Opus 4.8 (1M context) --- ...BEP-1055-preemption-scheduler-mechanics.md | 53 ++++++++++++++++--- 1 file changed, 45 insertions(+), 8 deletions(-) diff --git a/proposals/BEP-1055-preemption-scheduler-mechanics.md b/proposals/BEP-1055-preemption-scheduler-mechanics.md index 7d9dffe0db2..e4baff237a1 100644 --- a/proposals/BEP-1055-preemption-scheduler-mechanics.md +++ b/proposals/BEP-1055-preemption-scheduler-mechanics.md @@ -12,6 +12,7 @@ Implemented-Version: ## Related Issues - Epic **BA-3056**, this BEP **BA-6692** (blocks **BA-3058**, the implementation) +- Amended by **BA-6926** — scope-local `job_priority` for preemption (sections 2.4, 3.a, 3.b) - Upstream spec (motivation): **[BEP-1014](BEP-1014-preemption-of-low-priority-sessions.md)** — referenced, not expanded - Prerequisites (done): BA-3057 (RG config), BA-4912 (`is_preemptible`). Out of scope: BA-4913 (Not Planned) @@ -32,6 +33,9 @@ For each area, separate **✅ what already exists** from **➕ what to add**. | ✅ | On session creation, the user sets `priority` (0..100) and `is_preemptible` (default True) — `services/session/actions/{create_from_params,enqueue_session}` | | ✅ | An admin sets `preemptible_priority`, `order`, `mode` via the RG config — scaling group API | | ➕ | Expose `preemption.enabled` (preemption on/off toggle) and `preemption_min_runtime` (the anti-thrashing knob in section 3) in the RG config | +| ➕ | On session creation, the user also sets `job_priority` — the session's **scope-local** preemption priority (used only for preemption; see 2.4). Distinct from the global scheduler `priority` | +| ➕ | Keypair (user) resource policy gains `max_priority` — caps the `priority` a user may declare on session creation (2.4) | +| ➕ | Remove admin `preemptible_priority` from the RG config — unused once victims are chosen by relative `job_priority` (3.b) | > Preemption is never invoked directly by a user; the scheduler decides it automatically, so **there is no new trigger API**. @@ -45,6 +49,9 @@ For each area, separate **✅ what already exists** from **➕ what to add**. | ➕ | New `PreemptionConfig.enabled: bool` — preemption on/off toggle, **default False (opt-in)** | | ➕ | New `SessionStatus.PREEMPTED` — the **single state** for a confirmed victim; after kernel cleanup it branches by mode to **PENDING (reschedule) or TERMINATED (terminate)** (3.(c)) | | ➕ | New `preemption_min_runtime` config field (RG opts) | +| ➕ | New `SessionRow.job_priority` (default 10) — scope-local preemption priority, compared only within the same scope-owner (2.4, 3.b) | +| ➕ | New keypair resource-policy field `max_priority` — per-user cap on the global `priority` (2.4) | +| ➕ | Remove `PreemptionConfig.preemptible_priority` — the absolute RG threshold is unnecessary once victims are chosen by relative `job_priority` (3.b) | ### 2.3 Sokovan (scheduler behavior) @@ -60,6 +67,31 @@ For each area, separate **✅ what already exists** from **➕ what to add**. > Because termination is async, the provisioner cannot terminate synchronously, so **preemption is inherently multi-tick**. +### 2.4 Priority Model — `priority` vs `job_priority` + +Preemption is decided by a **new axis, `job_priority`**, not the scheduler `priority`. Three axes answer three different questions and apply at different stages: + +| Axis | What | Set by | Stage | This amendment | +|------|------|--------|-------|----------------| +| Scheduler `priority` (existing) | System-wide order of the pending queue | User, up to the keypair policy `max_priority` | **First — scheduling** | Kept; excluded from preemption | +| `job_priority` (new) | A user's ranking **among their own jobs**, within their scope | User, at session creation | **After scheduling — preemption** | Introduced; the **only** key for victim comparison | +| Fair-share | A difference in scheduling *method* (sequencer) | — | Scheduling (alternative) | **Unrelated** — untouched | + +**How it plays out:** the scheduler orders the queue by `priority` first; when the front pending session cannot be placed, preemption evicts **that owner's own** running sessions with a lower `job_priority` — "my urgent job bumps my background job." Sessions of a different owner never enter the comparison. + +**Scope of `job_priority`:** + +| Scope | Owner key | v1 | +|-------|-----------|----| +| User | `session.user_uuid` | ✅ victims restricted to the pending session's `user_uuid` | +| Project | `session.group_id` | ➖ **future** — needs a "project session" concept (a session owned by a project, not a user) | + +`job_priority` values are **only comparable within one owner** (no global meaning). v1 fixes the owner key to `user_uuid`; wider scopes reuse the same column and only change the owner key used for comparison. + +> The keypair resource policy `max_priority` bounds the global `priority` a user may declare on session creation; `job_priority` needs no such cap since it only reorders the user's own jobs. + +> **Caveat — v1's within-owner rule vs BEP-1014's motivation.** BEP-1014's original driver was *cross-tenant* preemption (evicting a *different* owner's low-priority work on a shared cluster). v1 restricts victims to the **same user**, so the case it actually opens is the narrow, safe one: "my urgent job preempts my own background job." This is intentional — cross-tenant preemption needs a policy for who may evict whose work, which must rest on an **admin-managed cross-scope axis**, not a user-declared `job_priority`. Cross-tenant is therefore left to the future axis in Open Questions. + ## 3. Implementation Design **Core flow:** pending placement fails → **planner selects victims** → **controller marks PREEMPTED and branches by mode** → (next tick) once resources free, normal allocation places the pending session. @@ -68,7 +100,7 @@ For each area, separate **✅ what already exists** from **➕ what to add**. Since allocation is per-agent, load the running sessions' **preemption-relevant data grouped by agent** into the snapshot: -- priority, is_preemptible, occupied slots, agent binding, created-at (for `order` and min-runtime), cluster_mode (for multi-node victim handling), session_type and private flag (candidate exclusion). +- `job_priority` (victim comparison key), `user_uuid` (scope-owner key for the same-owner filter; `group_id` added when project scope opens), is_preemptible, occupied slots, agent binding, created-at (for `order` and min-runtime), cluster_mode (for multi-node victim handling), session_type and private flag (candidate exclusion). - The repository fetches `RUNNING` sessions with per-agent occupied slots. (Exact types and field signatures are an implementation detail.) ### (b) Provisioner — preemption planner (read + propose) @@ -81,12 +113,12 @@ On allocation failure the provisioner calls the planner. The planner **only read |-----------|------| | `status == RUNNING` | Excludes PREEMPTED/TERMINATING so in-flight victims are not re-picked | | `is_preemptible` and not private/SFTP | Per-session opt-out + infrastructure sessions excluded | -| `priority <= preemptible_priority` | RG config threshold | -| `priority < pending.priority` (strict) | **Equal priority is never preempted** | +| `victim.user_uuid == pending.user_uuid` | **Same scope-owner (v1: user scope). `job_priority` is only comparable within one owner** | +| `victim.job_priority < pending.job_priority` (strict) | **Preempt by scope-local `job_priority`, not the global `priority`. Equal `job_priority` is never preempted** | | `now - started_at >= preemption_min_runtime` | Anti-thrashing. **Default 0 (disabled)** | **Victim selection** (per-agent): -- On an agent where the pending session could fit, sort candidates by **priority ascending, then `order` (oldest/newest)** and accumulate. +- On an agent where the pending session could fit, sort candidates by **`job_priority` ascending, then `order` (oldest/newest)** and accumulate. `preemption_order` is retained (only the `preemptible_priority` threshold is removed). - Preempt **only when fully satisfied**: `free(A) + sum(victim slots) >= requested` (no partial preemption, avoiding wasted kills and livelock). - **v1 trigger = single-node pending only.** Multi-node victims are allowed but **evicted atomically across all kernels** (no partial gang preemption). Multi-node (cross-agent) triggers are out of scope. - Keep a selected-victim set within a tick so one session is not double-counted for two pending sessions. @@ -98,7 +130,7 @@ The schedule pass (handler) **only judges and proposes**; it does not mutate vic Then **a new preemption lifecycle handler targeting `PREEMPTED` cleans up kernels and decides the destination by `preemption.mode`**: - **terminate**: hand off to the existing termination path → `TERMINATED`. Reason `PREEMPTED_BY_SCHEDULER`. -- **reschedule**: **re-enqueue the same session as `PENDING`** (session id/config/priority preserved, Slurm REQUEUE). Suited to batch jobs with checkpointing; **interactive sessions lose in-container state (accepted, documented).** +- **reschedule**: **re-enqueue the same session as `PENDING`** (session id/config/priority/`job_priority` preserved, Slurm REQUEUE). Suited to batch jobs with checkpointing; **interactive sessions lose in-container state (accepted, documented).** ``` victim selected -> PREEMPTED (kernel cleanup) @@ -117,14 +149,19 @@ There is no separate `RESCHEDULING` state. `PREEMPTED` covers the kernel-cleanup | Enable toggle | New RG `PreemptionConfig.enabled`, default False (opt-in). No preemption when off | | Trigger | Preempt when normal allocation fails AND preemption is on AND a fully-satisfying victim set exists | | Eviction path | Schedule pass only proposes; **the (injected) controller marks victims `PREEMPTED`**; a new preemption handler branches by mode | -| State model | Single new state `PREEMPTED` → terminate: TERMINATED / reschedule: PENDING (REQUEUE, id/priority preserved). No separate RESCHEDULING | +| State model | Single new state `PREEMPTED` → terminate: TERMINATED / reschedule: PENDING (REQUEUE, id/priority/`job_priority` preserved). No separate RESCHEDULING | +| Preemption axis | Preempt by the new **`job_priority`** (scope-local), not the global scheduler `priority`; victims chosen by relative comparison within the same owner | +| Priority scope | v1 = user scope only (`victim.user_uuid == pending.user_uuid`); project scope is future (needs a "project session") | +| Config change | Drop RG `preemptible_priority` (absolute threshold unneeded); keep `preemption_order` (oldest/newest tie-break) | +| Priority cap | Keypair resource policy `max_priority` caps the global `priority` a user may declare; `job_priority` needs no cap (self-scoped) | | Scope | v1 single-node trigger + atomic multi-node victim eviction, no partial preemption | -| Anti-thrashing | Preempt only on full satisfaction, never preempt equal priority, `preemption_min_runtime` (default 0). Slurm's `PreemptExemptTime` is likewise no-exemption when unset (-1 equals 0) | +| Anti-thrashing | Preempt only on full satisfaction, never preempt equal `job_priority`, `preemption_min_runtime` (default 0). Slurm's `PreemptExemptTime` is likewise no-exemption when unset (-1 equals 0) | | Pipeline | provisioner planner (read + propose) + controller initiates eviction | -| Out of scope | BA-4913 (resource-policy priority validation) | +| Out of scope | Broad BA-4913 resource-policy priority validation (Not Planned). Only the keypair `max_priority` cap for the global `priority` is in scope (2.4) | ## Open Questions +- Cross-tenant (cross-scope) preemption — v1's same-owner rule cannot evict *another* owner's session. Evicting a different tenant's low-priority work belongs to a separate **admin-managed cross-scope (project) axis**, gated on the "project session" concept, and must not be conflated with the user-declared `job_priority`. - Multi-node (cross-agent) trigger preemption — out of scope for v1, a follow-up BEP/Epic. - Strict reservation of freed resources (soft in v1) — decide the need in a follow-up. From bb67e495e6a84cb0f4d303cc22cbf4eab8ba257a Mon Sep 17 00:00:00 2001 From: HyeockJinKim Date: Sat, 18 Jul 2026 17:46:57 +0900 Subject: [PATCH 2/2] changelog: add news fragment for PR #12939 Co-Authored-By: Claude Opus 4.8 (1M context) --- changes/12939.doc.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 changes/12939.doc.md diff --git a/changes/12939.doc.md b/changes/12939.doc.md new file mode 100644 index 00000000000..cc2a22dda88 --- /dev/null +++ b/changes/12939.doc.md @@ -0,0 +1 @@ +Amend BEP-1055 to preempt on a scope-local job_priority axis (restricted to the same owner) instead of the global scheduler priority.