Skip to content

feat(#6526): surface runtime/model and make the runtime selectable at setup and per run - #6534

Merged
waynesun09 merged 12 commits into
mainfrom
agent/6526-surface-runtime-model
Aug 23, 2026
Merged

feat(#6526): surface runtime/model and make the runtime selectable at setup and per run#6534
waynesun09 merged 12 commits into
mainfrom
agent/6526-surface-runtime-model

Conversation

@fullsend-ai-coder

@fullsend-ai-coder fullsend-ai-coder Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Surfaces the selected runtime and requested model in every place users look, as the first step toward making the multi-runtime choice usable (#6526).

Code changes (Go):

  • Run plan block — prints Runtime: <name> (from <source>) next to Model/Effort by resolving the runtime before the plan print
  • Agent-status comment — adds a footer line to the terminal status comment: Runtime: <name> · Model: <requested → reported> · Effort: <level> · Cost: $<total> (arrow only when requested differs from reported; unknown fields omitted)
  • ::notice:: annotation — emits the same footer format as a GHA annotation for script consumers
  • OTel — adds fullsend.runtime attribute to the agent span alongside gen_ai.request.model
  • metrics.json — adds requested_runtime, requested_model, and override_source fields so a FULLSEND_PI_MODEL override is visible after the fact

Documentation:

  • New docs/cli/run.md CLI reference page added to the /cli/ sidebar
  • New "Choose a Runtime" Getting Started page between Getting Inference and Configuring GitHub
  • Runtimes moved from Concepts to a standalone sidebar entry next to Getting Started
  • "Selecting and overriding" subsection with precedence list and runtime capability table added to docs/runtimes.md
  • One-sentence --runtime cross-reference added to Configuring GitHub

Scope extension (taken over after the code agent's first commit)

Implements #6526 items 6–11 — choosing the runtime at setup and overriding it per run — as separate commits on top of the code agent's S0 commit:

  • fullsend github setup — TTY prompt for the runtime when --runtime is not given (Enter/EOF → claude; --dry-run and non-TTY never prompt); "Runtime" section in the setup-PR body; comment above runtime: in the rendered .fullsend/config.yaml; warning when pi is chosen and the target image cannot be confirmed to carry pi.
  • fullsend reposruntime in repos.yaml (per entry + defaults.runtime, none stops the chain), repos install --runtime, repos set-default defaults.runtime. (A repos status runtime column would need a per-repo config fetch — left to Runtime parity backlog: deferred runtime/model UX items (pi Agent tool, per-role models, per-run overrides, model switching, other clouds) #6527.)
  • Overrides of the same valuefullsend run --runtime/--model/--effort; FULLSEND_RUNTIME, FULLSEND_MODEL, FULLSEND_EFFORT, FULLSEND_FALLBACK_MODELS (precedence flag > env > config/harness > default, printed with source; FULLSEND_PI_MODEL/FULLSEND_PI_PROVIDER kept as aliases). Claude → --model/--effort/--fallback-model; pi → alias table, provider/id passthrough. requested_model now records the value after overrides.
  • pi run envGOOGLE_CLOUD_LOCATION exported from CLOUD_ML_REGION so pi's built-in google-vertex (Gemini) provider resolves with the same Vertex credentials; FULLSEND_RUNTIME exported into the sandbox; system-prompt note making the no-sub-agent fallback deliberate.
  • CI passthrough (human-only commit, workflow file) — every stage's "Setup agent environment" step passes ${{ toJSON(vars) }} as FULLSEND_REPO_VARS; setup-agent-env.sh exports the allowlisted FULLSEND_RUNTIME / FULLSEND_MODEL / FULLSEND_EFFORT / FULLSEND_FALLBACK_MODELS / FULLSEND_PI_PROVIDER (role-prefixed wins over plain; single-line [A-Za-z0-9._/@:,-] values only). Reusable workflows read the caller repo's vars, so no shim change is needed. Bash test in setup-agent-env-test.sh (make script-test).

Out of scope (tracked in #6527): per-role models:/providers: schema, slash-command arguments, alias normalisation in the requested→reported display, the pi Agent tool.

Review rounds (takeover)

  • Round 1 (Grok, full diff at 1e4abb2): two CRITICALs — the org-config defaults.runtime snippet in Getting Started (removed; org mode is deprecated) and "github setup --runtime persists unvalidated/dummy" (false positive for garbage: perRepoCfg.Validate() rejects anything outside ValidRuntimes(); dummy must stay accepted because the behaviour suite installs with --runtime dummy). Applied: help text marks dummy as behaviour-test only; the interactive prompt offers and accepts only claude/pi; FULLSEND_PI_MODEL added to the CI passthrough allowlist; stale knob/capability/issue references in docs/runtimes.md fixed; runtime_source added to metrics.json. toJSON(vars) in the passthrough was kept deliberately: the custom-harness matrix job only knows its role at runtime and GitHub expressions cannot upper-case it to build ROLE_FULLSEND_MODEL; vars are the caller repo's non-secret variables already in the workflow's scope; only allowlisted, charset-checked keys leave setup-agent-env.sh.
  • Round 2 (Grok, fix diff + full-diff regression pass at 54faea0): no MEDIUM+; dispositions above independently confirmed. Two LOWs closed in d78c840: the prompt now rejects a typed dummy, and runtime_source is documented. Commits after 54faea0 are content-equivalent apart from that gate and doc lines (plus a commit-subject reword for commit-lint).

Deliberate residual: dummy remains a valid runtime for --runtime dummy, FULLSEND_RUNTIME=dummy and the repository-variable passthrough — it is in ValidRuntimes() for behaviour-test installs and is not offered interactively.

Testing

  • Code agent's commit: unit tests for BuildRunInfoFooter, completion body, agentSpanEndAttrs/finalizeAgentSpan; docs-link lint + lychee.
  • Takeover commits: TestResolveRunOverrides_Precedence (flag > env > config; FULLSEND_PI_MODEL alias only on pi and below FULLSEND_MODEL), TestResolveRunOverrides_InvalidRuntime, TestResolveBackend_OverrideWinsOverConfig, TestModelOverrideSource, TestWithSource; TestPromptRuntime (non-TTY never prompts; Enter/EOF keep default; invalid → retry); TestRuntimeSection; pi: TestTranslatePiModel (runtime ignores FULLSEND_PI_MODEL), TestBuildPiRunCommand_Basic (GOOGLE_CLOUD_LOCATION, FULLSEND_RUNTIME exports), bootstrap test asserts the runtime note; repos: TestManifest_RuntimeResolvesAndValidates, TestSetDefault_Runtime; scaffold content test; setup-agent-env-test.sh (10 cases incl. metacharacter/newline rejection).
  • go test ./internal/{cli,runtime,repos,config,harness,statuscomment,scaffold}/ pass locally (the two TestDummyRuntime_* cases fail on this host on main too — environment); pre-commit on all changed files clean except lint-staged (web) which needs npx on the host (config.ts untouched by the takeover commits; the web CI job covers it).
  • Local Gemini-on-Vertex smoke run on this branch (mac-mini, fullsend run triage --runtime pi --model google-vertex/gemini-2.5-flash --effort medium): plan block shows Model: google-vertex/gemini-2.5-flash (from --model flag) / Effort: medium (from --effort flag) / Runtime: pi (from --runtime flag); stderr runtime: selected "pi" from --runtime flag and model: requested … from --model flag; metrics.json runtime: pi, requested_runtime: pi, requested_model: google-vertex/gemini-2.5-flash, model: gemini-2.5-flash, override_source: --model flag; the google-vertex provider resolved auth and region from the same WIF/ADC file as Claude-on-Vertex (no credential/region errors), 2 iterations, $0.04. The run did not reach a schema-valid result for model-behaviour reasons outside this PR: Gemini printed the triage JSON as chat text instead of writing agent-result.json, then hit the pre-existing ssrf_pretool false positive on sed 's|https://github.com/||' (Claude hits it too and routes around it). Both noted on Runtime parity backlog: deferred runtime/model UX items (pi Agent tool, per-role models, per-run overrides, model switching, other clouds) #6527.
  • github setup --runtime pi re-run on an installed repo yields a config-only update PR — not exercised in this PR (noted in Surface the selected runtime/model where users look, then make the runtime selectable at setup and per run #6526).

Closes #6526

Post-script verification

  • Branch is not main/master (agent/6526-surface-runtime-model)
  • Secret scan passed (gitleaks — 5b23b4f04e044413ebe3adcabcec6ae9ed781723..HEAD)
  • PR body secret scan passed (gitleaks — no-git)

…Tel, metrics, and docs

Print Runtime in the run plan block next to Model/Effort so users can
see which runtime was selected and from which config source. Add a
footer line to the terminal agent-status comment showing runtime,
model (requested vs reported when they differ), effort, and cost.
Emit the same metadata as a ::notice:: GHA annotation.

Add fullsend.runtime attribute to the agent OTel span alongside
gen_ai.request.model. Add requested_runtime, requested_model, and
override_source fields to metrics.json so a FULLSEND_PI_MODEL
override is visible after the fact.

Docs: new docs/cli/run.md in the CLI sidebar, a "Choose a Runtime"
Getting Started page between Getting Inference and Configuring GitHub,
Runtimes moved next to Getting Started in the sidebar, and a
"Selecting and overriding" subsection with capability table in
docs/runtimes.md.

Closes #6526
@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown

Site preview

Preview: https://101b1a23-site.fullsend-ai.workers.dev

Commit: 6026539203e211325d358ca5257cc6b0f446ebd8

@fullsend-ai-review

fullsend-ai-review Bot commented Aug 23, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 4:08 PM UTC · Ended 4:23 PM UTC

Commit: 314c667 · View workflow run →

@waynesun09 waynesun09 changed the title feat(#6526): surface runtime and model in run plan, status comment, OTel, metrics, and docs feat(#6526): surface runtime/model and make the runtime selectable at setup and per run Aug 23, 2026
…the CLI

`fullsend run --runtime/--model/--effort` and the runtime-neutral
FULLSEND_RUNTIME, FULLSEND_MODEL, FULLSEND_EFFORT and
FULLSEND_FALLBACK_MODELS are overrides of the values config.yaml and the
composed harness already carry. The CLI resolves them once (flag > env >
config/harness > default), validates them the way the config/harness
would, prints the source in the plan block and on stderr, records the
effective value as requested_model/override_source in metrics.json, and
hands the result to the runtime. Runtimes do not read these variables;
FULLSEND_PI_MODEL stays as an alias of FULLSEND_MODEL for pi runs.

RunParams gains FallbackModels; Claude Code passes it as
--fallback-model (ordered chain for overloaded/retired models).

Assisted-by: Claude (code)
Signed-off-by: Wayne Sun <gsun@redhat.com>
…nt fallback

The pi runtime no longer reads FULLSEND_PI_MODEL itself (the CLI resolves
the model once); it still applies FULLSEND_PI_PROVIDER to bare ids.

The run command exports GOOGLE_CLOUD_LOCATION from CLOUD_ML_REGION: pi's
built-in google-vertex provider requires GOOGLE_APPLICATION_CREDENTIALS,
GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION (no fallback), and the
fleet only exports the region under the name the Anthropic-on-Vertex
extension reads. With it, Gemini on Vertex is a model name
(google-vertex/gemini-2.5-flash) on the same credentials and egress.

FULLSEND_RUNTIME=pi is exported into the sandbox and APPEND_SYSTEM.md
ends with a runtime note saying no sub-agent tool exists and how a skill
should proceed, so pr-review/retro take their single-context path on
purpose instead of recording failed dispatches. Fallback models are
reported as unsupported on pi rather than silently dropped (#6527).

Assisted-by: Claude (code)
Signed-off-by: Wayne Sun <gsun@redhat.com>
`fullsend github setup <owner/repo>` asks once which agent runtime the
per-repo config should select when --runtime is not given — only on an
interactive terminal (Enter or EOF keep the default, so CI, pipes and
--dry-run never block) — and warns when pi is chosen that the sandbox
image must carry pi. The setup PR body gains a "Runtime" section naming
the choice and the three ways to change it (the runtime key, re-running
setup with --runtime, a per-run --runtime), and the rendered
.fullsend/config.yaml header explains the key and the per-run precedence
(flag, then FULLSEND_RUNTIME, then the file).

Assisted-by: Claude (code)
Signed-off-by: Wayne Sun <gsun@redhat.com>
The fleet manifest can now carry the agent runtime: `defaults.runtime`
and a per-entry `runtime` (entry wins; "none" stops the chain like the
other string fields), validated against the same set the per-repo
config accepts. `repos set-default defaults.runtime <claude|pi>` edits
it, `repos install --runtime` records it on the entries the command
adds, and batch install writes the resolved value as the repo's
`runtime:` through the existing InstallConfig.Runtime path — closing
the gap where docs/cli/repos.md already described a defaults.runtime
mapping the manifest could not express.

Assisted-by: Claude (code)
Signed-off-by: Wayne Sun <gsun@redhat.com>
…emini on Vertex

Document the flag > env > config/harness > default order for runtime, model, effort and fallback models, the FULLSEND_* variables and their CI passthrough, repos.yaml's runtime keys and repos install --runtime, the setup prompt, and that Gemini on Vertex is selected by model name on the same credentials.

Assisted-by: Claude (code)
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Aug 23, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 4:25 PM UTC · Ended 4:26 PM UTC

Commit: 68afee4 · View workflow run →

…into CI runs

The per-run override variables (FULLSEND_RUNTIME, FULLSEND_MODEL,
FULLSEND_EFFORT, FULLSEND_FALLBACK_MODELS, FULLSEND_PI_PROVIDER) are
runner-process environment that nothing in CI could set: harness
env.runner only reaches pre/post scripts, and harness model: cannot
carry a provider/id. Every stage's "Setup agent environment" step now
hands the calling repository's Actions variables (toJSON(vars)) to
setup-agent-env.sh, which exports the allowlisted keys — a role-prefixed
variable (TRIAGE_FULLSEND_MODEL) winning over the plain one — after
checking each value is a single line of model-id/runtime-name characters.
fullsend validates the values themselves at run start, so a repo can
switch one role to another model or provider with a repository variable
instead of a pull request, and the run plan, status comment and
metrics.json show where the value came from.

Assisted-by: Claude (code)
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Aug 23, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 4:28 PM UTC · Ended 4:45 PM UTC

Commit: 1e4abb2 · View workflow run →

@codecov

codecov Bot commented Aug 23, 2026

Copy link
Copy Markdown

@fullsend-ai-review

fullsend-ai-review Bot commented Aug 23, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 4:46 PM UTC · Ended 4:47 PM UTC

Commit: 54faea0 · View workflow run →

…passthrough, runtime_source

- Getting Started no longer shows the deprecated org-config `defaults.runtime`
  snippet; `repos.yaml` `defaults.runtime` is the fleet-wide way.
- `github setup --runtime` help says dummy is for behaviour-test installs
  only; the interactive prompt and its retry warning list only claude and
  pi (the per-repo config still validates every value on write).
- `setup-agent-env.sh` also forwards the legacy FULLSEND_PI_MODEL
  repository variable (the CLI honours it as an alias on pi), and records
  why the whole `vars` map is passed rather than individual keys.
- metrics.json gains `runtime_source` for parity with `override_source`.
- docs/runtimes.md: knob lists and the capability table describe the CLI
  overrides, the Claude fallback chain, cross-provider Vertex, and point
  the sub-agent follow-up at #6527.

Assisted-by: Claude (fix), Grok (review)
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Aug 23, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 4:54 PM UTC · Ended 5:10 PM UTC

Commit: d78c840 · View workflow run →

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taken over from the code agent: S0 (code agent) + S1 (runtime at setup, repos.yaml runtime, CLI/env overrides with flag > env > config precedence, pi Gemini-on-Vertex env, CI passthrough). Two Grok review rounds (round 2 clean at MEDIUM+), Gemini-on-Vertex smoke run verified the plumbing end-to-end, all unit/lint checks green. functional-tests failure on this head was the pre-existing OpenShell provider race (#6523), rerun in progress. Approving.

Verified on the PR's site preview: the sidebar placed the page correctly but the index's numbered reading order still skipped it.

Assisted-by: Claude (fix)
Signed-off-by: Wayne Sun <gsun@redhat.com>

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving on 429f047 (docs-only: Getting Started reading order, verified on the site preview with playwright).

@fullsend-ai-review

fullsend-ai-review Bot commented Aug 23, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 5:12 PM UTC · Ended 5:13 PM UTC

Commit: 429f047 · View workflow run →

…ct in the next step

The Getting Started runtime page now leads with Claude Code as the stable, long-running default and marks pi as enablement-phase/experimental, and explains that nothing is selected on that page — the choice is made in the next step (Configuring GitHub) by the setup prompt or --runtime, with config/repos.yaml/per-run overrides documented as later options. The CLI prompt uses the same stable-vs-experimental wording and keeps Enter = claude.

Assisted-by: Claude (fix)
Signed-off-by: Wayne Sun <gsun@redhat.com>

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving on f1e5309 (docs + prompt wording: Claude Code stable default, pi experimental; selection happens in Configuring GitHub).

@fullsend-ai-review

fullsend-ai-review Bot commented Aug 23, 2026

Copy link
Copy Markdown

🤖 Review · ⚠️ Cancelled · Started 5:15 PM UTC · Ended 5:21 PM UTC

Commit: f1e5309 · View workflow run →

…atus comments

The ::notice:: footer was emitted inside the status-notifier block, so runs without status comments (eval harness, local CI runs) never printed it — found by checking the functional-test logs on this PR. It now fires once after the final metrics.json write whenever GITHUB_ACTIONS is set; the notifier footer shares the same RunInfo builder.

Assisted-by: Claude (fix)
Signed-off-by: Wayne Sun <gsun@redhat.com>

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving on 6026539: the run-info ::notice:: now fires at run end regardless of status comments (the functional-test logs showed it missing without them).

@fullsend-ai-review

fullsend-ai-review Bot commented Aug 23, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 5:23 PM UTC · Completed 5:56 PM UTC

Commit: 6026539 · View workflow run →

@waynesun09
waynesun09 added this pull request to the merge queue Aug 23, 2026
Merged via the queue into main with commit 24f01cd Aug 23, 2026
35 of 39 checks passed
@waynesun09
waynesun09 deleted the agent/6526-surface-runtime-model branch August 23, 2026 17:41
@fullsend-ai-retro

fullsend-ai-retro Bot commented Aug 23, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 5:43 PM UTC · Completed 5:57 PM UTC

Commit: 6026539 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Cost: $6.12

@fullsend-ai-review

Copy link
Copy Markdown

Review skipped — this PR is already merged.

The /fs-review command only reviews open PRs/MRs.

Posted by fullsend post-review check

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #6534 — Surface runtime/model and make runtime selectable

Timeline

  1. 15:36Issue #6526 opened by waynesun09 (11 scope items: surface runtime/model info + make runtime selectable).
  2. 15:44/fs-code dispatched with instructions for items 1–5 only (surface runtime in run plan, status comment, OTel, metrics.json, docs).
  3. 15:47–16:06 — Code agent (run 32649472524) completed items 1–5 in ~19 min, $13.36, 163 turns. Opened PR feat(#6526): surface runtime/model and make the runtime selectable at setup and per run #6534 with commit 314c667.
  4. 16:06–17:21 — waynesun09 pushed 11 additional commits implementing items 6–11 (setup prompt, repos.yaml runtime, CLI overrides, pi env, CI passthrough). Each push triggered and cancelled a review run.
  5. 17:21 — Final review run (32654562857) started on commit 6026539.
  6. 17:41 — PR merged with 4 human approvals from waynesun09 (self-review). Review agent still in progress.
  7. 17:42+ — Review run still in_progress ~31+ min after merge.

Code agent quality

The code agent performed well. It followed the code-implementation skill procedure methodically, read repo conventions, implemented all 5 dispatched scope items, wrote comprehensive tests, ran verification (go vet, secret scan, lint-docs-links), and fixed a VitePress absolute-path lint failure on first encounter. Single iteration, no retries, output validation passed first time. Agents repo: fullsend-ai/agents@888a02e88b9f (main).

Review agent: 10 cancelled runs, zero completed reviews

The cancel-in-progress concurrency group correctly cancelled stale review runs as new commits landed, but the rapid push cadence (every 2–18 minutes) meant no review ever completed. ~75 minutes of runner time was consumed across 10 cancelled runs with zero review output. The PR was merged without any completed automated review.

This is strong evidence for several well-tracked existing issues:

  • #1418 / #4960 / #4069 — Debounce or deduplicate review runs on rapid successive pushes. This PR had 10 cancelled runs in 75 minutes, matching the exact pattern these issues describe.
  • #2388 — Cancel in-flight review when PR merges. Run 32654562857 was still in_progress 31+ minutes after merge with no sign of self-termination. This is a concrete, recent instance of the stuck-run scenario.
  • #6040 — Review dispatch job hangs after merged PR. The final review run's behavior matches this pattern.
  • #4509 / #2866 — Post informational review post-merge / gate merge on review completion. The PR merged with zero completed automated review.

Cancelled review comments still posting

8 "Review cancelled" comments were posted to the PR timeline, adding noise. #2470 ("Suppress status comments for cancelled review runs") was closed on 2026-06-22. The continued appearance of these comments two months later warrants investigation — either the suppression has a gap in coverage for cancel-in-progress cancellations, or there is a regression.

No spurious second code dispatch

The second code agent run (32654084446 at 17:12) was initially suspected as a duplicate dispatch for #6526 triggered by human activity. Investigation confirmed it was a legitimate code run for an unrelated issue #6535, correctly triggered by a fresh ready-to-code label.

Autonomy readiness

Not assessable for this PR — the review agent never completed a review, so there is no agent-vs-human review delta to analyze. The human reviewer (waynesun09) self-reviewed their own code with external tooling (Grok review rounds, Gemini-on-Vertex smoke test).

Conclusion

The code agent delivered solid work. The systemic issues — review thrashing on rapid pushes, stuck post-merge review runs, and noisy cancelled-review comments — are all well-tracked by existing issues. No new proposals are warranted; the evidence from this PR reinforces the priority of resolving the debounce and post-merge cleanup issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-review Agent PR ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Surface the selected runtime/model where users look, then make the runtime selectable at setup and per run

1 participant