Restore live conformer jobs by name on restart, not the fossil 'conformer{i}'

[alongd]

Motivation

Restarting a project that had conformer jobs in flight crashes with an unhelpful KeyError: '':

  File "arc/scheduler.py", line 3508, in get_completed_incore_jobs
    job = self.job_dict[label][job_type][job_name]
KeyError: ''

The cause is a naming mismatch between the two places running_jobs is populated.

The live path (run_job) appends f'{job_type}_{conformer}', e.g. conf_opt_0:

self.running_jobs[label].append(f'{job_type}_{conformer}')

The restart path (restore_running_jobs) instead appends a fossil conformer{i} name that no
consumer of running_jobs accepts any more:

elif 'conformer' in job_description:
    self.running_jobs[spc_label].append(f'conformer{job_description["conformer"]}')

get_i_from_job_name only recognises the CONFORMER_JOB_TYPES prefixes (conf_opt, conf_sp)
and tsg, so it returns None for conformer0. The caller then falls through to the
generic-job-name branch, which derives a job type by stripping the last _-delimited field —
'conformer0'.split('_')[:-1] is [], so job_type becomes '' and the lookup raises
KeyError: ''.

The effect is that a restart cannot reconnect to running conformer jobs. It is not specific to any
adapter or server; any project restarted while conformer jobs are queued or running hits it.

Changes

• arc/scheduler.py — restore_running_jobs now emits the same '{job_type}_{conformer}' name
  the live path uses, so the restored entry is one get_i_from_job_name can parse.
• arc/checks/common.py — corrected the get_i_from_job_name docstring, which cited
  'conformer12' as an example input. That form has not been accepted since CONFORMER_JOB_TYPES
  was introduced, and the stale example is what makes the defect above read as intentional.
• arc/scheduler_test.py — regression test test_restore_running_jobs_conformer_reconnects.

Verification

The test is red/green against this change, and the red reproduces the production error exactly:

• with arc/scheduler.py reverted to main: FAILED ... KeyError: '' at arc/scheduler.py:3508
• with the fix: 1 passed

arc/scheduler_test.py is otherwise unaffected (44 passed alongside it locally; the remaining
local errors are a pre-existing environment issue, present identically on main).

Context

Found while diagnosing a ~4 hour outage of a long-running rate campaign, where the orchestrator
died on restart and the traceback gave no indication that conformer jobs were the trigger.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.61%. Comparing base (7ae26d3) to head (a692ce4).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1015      +/-   ##
==========================================
+ Coverage   64.46%   64.61%   +0.15%     
==========================================
  Files         119      119              
  Lines       39636    39639       +3     
  Branches    10276    10277       +1     
==========================================
+ Hits        25550    25612      +62     
+ Misses      11102    11036      -66     
- Partials     2984     2991       +7     

Flag	Coverage Δ
functionaltests	64.61% <ø> (+0.15%)	⬆️
unittests	64.61% <ø> (+0.15%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Fixes ARC restart behavior so in-flight conformer jobs are reconnected correctly by restoring conformer job names in the same format used during normal execution, preventing the post-restart KeyError: '' crash in get_completed_incore_jobs.

Changes:

• Align restore_running_jobs() conformer entries in running_jobs with the live '{job_type}_{i}' naming convention.
• Update get_i_from_job_name() docstring to reflect the accepted conformer job name formats.
• Add a regression test covering restart + sweep for conformer job reconnection.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
arc/scheduler.py	Restore conformer job names on restart using the live '{job_type}_{i}' format.
arc/scheduler_test.py	Add a regression test to ensure restart reconnection prevents the scheduling sweep crash.
arc/checks/common.py	Correct get_i_from_job_name docstring examples to match actual accepted formats.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.