feat(#6526): surface runtime/model and make the runtime selectable at setup and per run - #6534
Conversation
…Tel, metrics, and docs Print Runtime in the run plan block next to Model/Effort so users can see which runtime was selected and from which config source. Add a footer line to the terminal agent-status comment showing runtime, model (requested vs reported when they differ), effort, and cost. Emit the same metadata as a ::notice:: GHA annotation. Add fullsend.runtime attribute to the agent OTel span alongside gen_ai.request.model. Add requested_runtime, requested_model, and override_source fields to metrics.json so a FULLSEND_PI_MODEL override is visible after the fact. Docs: new docs/cli/run.md in the CLI sidebar, a "Choose a Runtime" Getting Started page between Getting Inference and Configuring GitHub, Runtimes moved next to Getting Started in the sidebar, and a "Selecting and overriding" subsection with capability table in docs/runtimes.md. Closes #6526
Site previewPreview: https://101b1a23-site.fullsend-ai.workers.dev Commit: |
|
🤖 Review · Commit: |
…the CLI `fullsend run --runtime/--model/--effort` and the runtime-neutral FULLSEND_RUNTIME, FULLSEND_MODEL, FULLSEND_EFFORT and FULLSEND_FALLBACK_MODELS are overrides of the values config.yaml and the composed harness already carry. The CLI resolves them once (flag > env > config/harness > default), validates them the way the config/harness would, prints the source in the plan block and on stderr, records the effective value as requested_model/override_source in metrics.json, and hands the result to the runtime. Runtimes do not read these variables; FULLSEND_PI_MODEL stays as an alias of FULLSEND_MODEL for pi runs. RunParams gains FallbackModels; Claude Code passes it as --fallback-model (ordered chain for overloaded/retired models). Assisted-by: Claude (code) Signed-off-by: Wayne Sun <gsun@redhat.com>
…nt fallback The pi runtime no longer reads FULLSEND_PI_MODEL itself (the CLI resolves the model once); it still applies FULLSEND_PI_PROVIDER to bare ids. The run command exports GOOGLE_CLOUD_LOCATION from CLOUD_ML_REGION: pi's built-in google-vertex provider requires GOOGLE_APPLICATION_CREDENTIALS, GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION (no fallback), and the fleet only exports the region under the name the Anthropic-on-Vertex extension reads. With it, Gemini on Vertex is a model name (google-vertex/gemini-2.5-flash) on the same credentials and egress. FULLSEND_RUNTIME=pi is exported into the sandbox and APPEND_SYSTEM.md ends with a runtime note saying no sub-agent tool exists and how a skill should proceed, so pr-review/retro take their single-context path on purpose instead of recording failed dispatches. Fallback models are reported as unsupported on pi rather than silently dropped (#6527). Assisted-by: Claude (code) Signed-off-by: Wayne Sun <gsun@redhat.com>
`fullsend github setup <owner/repo>` asks once which agent runtime the per-repo config should select when --runtime is not given — only on an interactive terminal (Enter or EOF keep the default, so CI, pipes and --dry-run never block) — and warns when pi is chosen that the sandbox image must carry pi. The setup PR body gains a "Runtime" section naming the choice and the three ways to change it (the runtime key, re-running setup with --runtime, a per-run --runtime), and the rendered .fullsend/config.yaml header explains the key and the per-run precedence (flag, then FULLSEND_RUNTIME, then the file). Assisted-by: Claude (code) Signed-off-by: Wayne Sun <gsun@redhat.com>
The fleet manifest can now carry the agent runtime: `defaults.runtime` and a per-entry `runtime` (entry wins; "none" stops the chain like the other string fields), validated against the same set the per-repo config accepts. `repos set-default defaults.runtime <claude|pi>` edits it, `repos install --runtime` records it on the entries the command adds, and batch install writes the resolved value as the repo's `runtime:` through the existing InstallConfig.Runtime path — closing the gap where docs/cli/repos.md already described a defaults.runtime mapping the manifest could not express. Assisted-by: Claude (code) Signed-off-by: Wayne Sun <gsun@redhat.com>
…emini on Vertex Document the flag > env > config/harness > default order for runtime, model, effort and fallback models, the FULLSEND_* variables and their CI passthrough, repos.yaml's runtime keys and repos install --runtime, the setup prompt, and that Gemini on Vertex is selected by model name on the same credentials. Assisted-by: Claude (code) Signed-off-by: Wayne Sun <gsun@redhat.com>
|
🤖 Review · Commit: |
…into CI runs The per-run override variables (FULLSEND_RUNTIME, FULLSEND_MODEL, FULLSEND_EFFORT, FULLSEND_FALLBACK_MODELS, FULLSEND_PI_PROVIDER) are runner-process environment that nothing in CI could set: harness env.runner only reaches pre/post scripts, and harness model: cannot carry a provider/id. Every stage's "Setup agent environment" step now hands the calling repository's Actions variables (toJSON(vars)) to setup-agent-env.sh, which exports the allowlisted keys — a role-prefixed variable (TRIAGE_FULLSEND_MODEL) winning over the plain one — after checking each value is a single line of model-id/runtime-name characters. fullsend validates the values themselves at run start, so a repo can switch one role to another model or provider with a repository variable instead of a pull request, and the run plan, status comment and metrics.json show where the value came from. Assisted-by: Claude (code) Signed-off-by: Wayne Sun <gsun@redhat.com>
|
🤖 Review · Commit: |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
|
🤖 Review · Commit: |
…passthrough, runtime_source - Getting Started no longer shows the deprecated org-config `defaults.runtime` snippet; `repos.yaml` `defaults.runtime` is the fleet-wide way. - `github setup --runtime` help says dummy is for behaviour-test installs only; the interactive prompt and its retry warning list only claude and pi (the per-repo config still validates every value on write). - `setup-agent-env.sh` also forwards the legacy FULLSEND_PI_MODEL repository variable (the CLI honours it as an alias on pi), and records why the whole `vars` map is passed rather than individual keys. - metrics.json gains `runtime_source` for parity with `override_source`. - docs/runtimes.md: knob lists and the capability table describe the CLI overrides, the Claude fallback chain, cross-provider Vertex, and point the sub-agent follow-up at #6527. Assisted-by: Claude (fix), Grok (review) Signed-off-by: Wayne Sun <gsun@redhat.com>
54faea0 to
d251359
Compare
|
🤖 Review · Commit: |
waynesun09
left a comment
There was a problem hiding this comment.
Taken over from the code agent: S0 (code agent) + S1 (runtime at setup, repos.yaml runtime, CLI/env overrides with flag > env > config precedence, pi Gemini-on-Vertex env, CI passthrough). Two Grok review rounds (round 2 clean at MEDIUM+), Gemini-on-Vertex smoke run verified the plumbing end-to-end, all unit/lint checks green. functional-tests failure on this head was the pre-existing OpenShell provider race (#6523), rerun in progress. Approving.
Verified on the PR's site preview: the sidebar placed the page correctly but the index's numbered reading order still skipped it. Assisted-by: Claude (fix) Signed-off-by: Wayne Sun <gsun@redhat.com>
waynesun09
left a comment
There was a problem hiding this comment.
Re-approving on 429f047 (docs-only: Getting Started reading order, verified on the site preview with playwright).
|
🤖 Review · Commit: |
…ct in the next step The Getting Started runtime page now leads with Claude Code as the stable, long-running default and marks pi as enablement-phase/experimental, and explains that nothing is selected on that page — the choice is made in the next step (Configuring GitHub) by the setup prompt or --runtime, with config/repos.yaml/per-run overrides documented as later options. The CLI prompt uses the same stable-vs-experimental wording and keeps Enter = claude. Assisted-by: Claude (fix) Signed-off-by: Wayne Sun <gsun@redhat.com>
waynesun09
left a comment
There was a problem hiding this comment.
Re-approving on f1e5309 (docs + prompt wording: Claude Code stable default, pi experimental; selection happens in Configuring GitHub).
|
🤖 Review · Commit: |
…atus comments The ::notice:: footer was emitted inside the status-notifier block, so runs without status comments (eval harness, local CI runs) never printed it — found by checking the functional-test logs on this PR. It now fires once after the final metrics.json write whenever GITHUB_ACTIONS is set; the notifier footer shares the same RunInfo builder. Assisted-by: Claude (fix) Signed-off-by: Wayne Sun <gsun@redhat.com>
waynesun09
left a comment
There was a problem hiding this comment.
Re-approving on 6026539: the run-info ::notice:: now fires at run end regardless of status comments (the functional-test logs showed it missing without them).
|
🤖 Finished Review · ✅ Success · Started 5:23 PM UTC · Completed 5:56 PM UTC Commit: |
|
🤖 Finished Retro · ✅ Success · Started 5:43 PM UTC · Completed 5:57 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Cost: $6.12 |
|
Review skipped — this PR is already merged. The Posted by fullsend post-review check |
Retro: PR #6534 — Surface runtime/model and make runtime selectableTimeline
Code agent qualityThe code agent performed well. It followed the Review agent: 10 cancelled runs, zero completed reviewsThe This is strong evidence for several well-tracked existing issues:
Cancelled review comments still posting8 "Review cancelled" comments were posted to the PR timeline, adding noise. #2470 ("Suppress status comments for cancelled review runs") was closed on 2026-06-22. The continued appearance of these comments two months later warrants investigation — either the suppression has a gap in coverage for No spurious second code dispatchThe second code agent run (32654084446 at 17:12) was initially suspected as a duplicate dispatch for #6526 triggered by human activity. Investigation confirmed it was a legitimate code run for an unrelated issue #6535, correctly triggered by a fresh Autonomy readinessNot assessable for this PR — the review agent never completed a review, so there is no agent-vs-human review delta to analyze. The human reviewer (waynesun09) self-reviewed their own code with external tooling (Grok review rounds, Gemini-on-Vertex smoke test). ConclusionThe code agent delivered solid work. The systemic issues — review thrashing on rapid pushes, stuck post-merge review runs, and noisy cancelled-review comments — are all well-tracked by existing issues. No new proposals are warranted; the evidence from this PR reinforces the priority of resolving the debounce and post-merge cleanup issues. |
Summary
Surfaces the selected runtime and requested model in every place users look, as the first step toward making the multi-runtime choice usable (#6526).
Code changes (Go):
Runtime: <name> (from <source>)next to Model/Effort by resolving the runtime before the plan printRuntime: <name> · Model: <requested → reported> · Effort: <level> · Cost: $<total>(arrow only when requested differs from reported; unknown fields omitted)::notice::annotation — emits the same footer format as a GHA annotation for script consumersfullsend.runtimeattribute to the agent span alongsidegen_ai.request.modelrequested_runtime,requested_model, andoverride_sourcefields so aFULLSEND_PI_MODELoverride is visible after the factDocumentation:
docs/cli/run.mdCLI reference page added to the/cli/sidebardocs/runtimes.md--runtimecross-reference added to Configuring GitHubScope extension (taken over after the code agent's first commit)
Implements #6526 items 6–11 — choosing the runtime at setup and overriding it per run — as separate commits on top of the code agent's S0 commit:
fullsend github setup— TTY prompt for the runtime when--runtimeis not given (Enter/EOF →claude;--dry-runand non-TTY never prompt); "Runtime" section in the setup-PR body; comment aboveruntime:in the rendered.fullsend/config.yaml; warning whenpiis chosen and the target image cannot be confirmed to carry pi.fullsend repos—runtimeinrepos.yaml(per entry +defaults.runtime,nonestops the chain),repos install --runtime,repos set-default defaults.runtime. (Arepos statusruntime column would need a per-repo config fetch — left to Runtime parity backlog: deferred runtime/model UX items (pi Agent tool, per-role models, per-run overrides, model switching, other clouds) #6527.)fullsend run --runtime/--model/--effort;FULLSEND_RUNTIME,FULLSEND_MODEL,FULLSEND_EFFORT,FULLSEND_FALLBACK_MODELS(precedence flag > env > config/harness > default, printed with source;FULLSEND_PI_MODEL/FULLSEND_PI_PROVIDERkept as aliases). Claude →--model/--effort/--fallback-model; pi → alias table,provider/idpassthrough.requested_modelnow records the value after overrides.GOOGLE_CLOUD_LOCATIONexported fromCLOUD_ML_REGIONso pi's built-ingoogle-vertex(Gemini) provider resolves with the same Vertex credentials;FULLSEND_RUNTIMEexported into the sandbox; system-prompt note making the no-sub-agent fallback deliberate.${{ toJSON(vars) }}asFULLSEND_REPO_VARS;setup-agent-env.shexports the allowlistedFULLSEND_RUNTIME/FULLSEND_MODEL/FULLSEND_EFFORT/FULLSEND_FALLBACK_MODELS/FULLSEND_PI_PROVIDER(role-prefixed wins over plain; single-line[A-Za-z0-9._/@:,-]values only). Reusable workflows read the caller repo'svars, so no shim change is needed. Bash test insetup-agent-env-test.sh(make script-test).Out of scope (tracked in #6527): per-role
models:/providers:schema, slash-command arguments, alias normalisation in the requested→reported display, the piAgenttool.Review rounds (takeover)
1e4abb2): two CRITICALs — the org-configdefaults.runtimesnippet in Getting Started (removed; org mode is deprecated) and "github setup --runtimepersists unvalidated/dummy" (false positive for garbage:perRepoCfg.Validate()rejects anything outsideValidRuntimes();dummymust stay accepted because the behaviour suite installs with--runtime dummy). Applied: help text marksdummyas behaviour-test only; the interactive prompt offers and accepts onlyclaude/pi;FULLSEND_PI_MODELadded to the CI passthrough allowlist; stale knob/capability/issue references indocs/runtimes.mdfixed;runtime_sourceadded tometrics.json.toJSON(vars)in the passthrough was kept deliberately: the custom-harness matrix job only knows its role at runtime and GitHub expressions cannot upper-case it to buildROLE_FULLSEND_MODEL;varsare the caller repo's non-secret variables already in the workflow's scope; only allowlisted, charset-checked keys leavesetup-agent-env.sh.54faea0): no MEDIUM+; dispositions above independently confirmed. Two LOWs closed ind78c840: the prompt now rejects a typeddummy, andruntime_sourceis documented. Commits after54faea0are content-equivalent apart from that gate and doc lines (plus a commit-subject reword for commit-lint).Deliberate residual:
dummyremains a valid runtime for--runtime dummy,FULLSEND_RUNTIME=dummyand the repository-variable passthrough — it is inValidRuntimes()for behaviour-test installs and is not offered interactively.Testing
BuildRunInfoFooter, completion body,agentSpanEndAttrs/finalizeAgentSpan; docs-link lint + lychee.TestResolveRunOverrides_Precedence(flag > env > config;FULLSEND_PI_MODELalias only on pi and belowFULLSEND_MODEL),TestResolveRunOverrides_InvalidRuntime,TestResolveBackend_OverrideWinsOverConfig,TestModelOverrideSource,TestWithSource;TestPromptRuntime(non-TTY never prompts; Enter/EOF keep default; invalid → retry);TestRuntimeSection; pi:TestTranslatePiModel(runtime ignoresFULLSEND_PI_MODEL),TestBuildPiRunCommand_Basic(GOOGLE_CLOUD_LOCATION,FULLSEND_RUNTIMEexports), bootstrap test asserts the runtime note; repos:TestManifest_RuntimeResolvesAndValidates,TestSetDefault_Runtime; scaffold content test;setup-agent-env-test.sh(10 cases incl. metacharacter/newline rejection).go test ./internal/{cli,runtime,repos,config,harness,statuscomment,scaffold}/pass locally (the twoTestDummyRuntime_*cases fail on this host onmaintoo — environment); pre-commit on all changed files clean exceptlint-staged (web)which needsnpxon the host (config.ts untouched by the takeover commits; thewebCI job covers it).fullsend run triage --runtime pi --model google-vertex/gemini-2.5-flash --effort medium): plan block showsModel: google-vertex/gemini-2.5-flash (from --model flag)/Effort: medium (from --effort flag)/Runtime: pi (from --runtime flag); stderrruntime: selected "pi" from --runtime flagandmodel: requested … from --model flag;metrics.jsonruntime: pi,requested_runtime: pi,requested_model: google-vertex/gemini-2.5-flash,model: gemini-2.5-flash,override_source: --model flag; thegoogle-vertexprovider resolved auth and region from the same WIF/ADC file as Claude-on-Vertex (no credential/region errors), 2 iterations, $0.04. The run did not reach a schema-valid result for model-behaviour reasons outside this PR: Gemini printed the triage JSON as chat text instead of writingagent-result.json, then hit the pre-existingssrf_pretoolfalse positive onsed 's|https://github.com/||'(Claude hits it too and routes around it). Both noted on Runtime parity backlog: deferred runtime/model UX items (pi Agent tool, per-role models, per-run overrides, model switching, other clouds) #6527.github setup --runtime pire-run on an installed repo yields a config-only update PR — not exercised in this PR (noted in Surface the selected runtime/model where users look, then make the runtime selectable at setup and per run #6526).Closes #6526
Post-script verification
agent/6526-surface-runtime-model)5b23b4f04e044413ebe3adcabcec6ae9ed781723..HEAD)