Feat/generic rca agent plugin v3 - #2
Open
Dave3130 wants to merge 56 commits into
Open
Conversation
…EADME Identity-only .claude-plugin/plugin.json; root .mcp.json wires the bstack MCP server (stdio); config/rca.config.json centralizes all formerly-hardcoded product/infra values (no kubectl/chitragupta/bifrost literals); /rca-build command parses build id + mode and hands off to the skill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port the obs-tfa-rca loop decoupled: ai-tfa-coordinator drives tfaRcaTurn to a terminal RCA (turn-cap, one-thread, soft-PENDING, digest-not-dump) with the gather mechanism routed by capability (no kubectl/chitragupta/bifrost literals). lib/routing.mjs classifies each ask skip/gather/gap against the config registry + capability manifest; the gap action is the only mode fork (auto=unavailable, interactive=ask-user). references/evidence-routing.md carries the digest format and size caps verbatim. Adds sibling pre-seed one-turn-confirm hook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SKILL.md orchestrator spec: mandatory GitHub intake ('I don't have one' → RCA-only;
headless missing-input fail-fast), discovery via listTestIds(failed,
includeFailureDetail), then cluster/pre-compute/fan-out/report steps.
lib/csv-state.mjs is the resumable WAL spine — seed (idempotent, terminal-
preserving), claim/heartbeat/flip, reaper, pendingRows — with timestamps injected
(workflow-sandbox-safe) and an RFC4180 codec for multiline RCA fields.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…otocol lib/signature.mjs computes signature = normalize(category|error|file) off the U1 discovery payload (folds timestamps/uuids/hex/line:col/numbers), groups rows by signature, picks a deterministic representative (non-flaky, then smallest id), and leaves signal-less rows as their own singletons. references/clustering.md documents the O(causes) protocol: representative runs the full loop; siblings pre-seed a one-turn confirm against their own logs with a fall-back-to-own-loop safeguard (never blindly inherit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
buildManifest enumerates the client's discovered capabilities once into
capability→{available,via}, declared to the user + TFA so no evidence is asked
for that the client provably can't get. lib/evidence-cache.mjs computes the
last-green→this-build delta once and caches by (repo,range,evidenceType) — fresh
per-run Map, no module globals (multi-tenant-safe) — with resolveBaseline for the
never-green fallback. Routes the same grounded window into every coordinator.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
workflows/rca-batch.mjs orchestrates the batch in auto mode: a pipeline over clusters dispatches ai-tfa-coordinator agents — representative full loop → siblings one-turn-confirm, no barrier between stages — with a structured RCA schema. Sandbox-correct: does no state I/O itself (orchestrator passes the clustered work-list + manifest + pre-computed build evidence via args; each coordinator agent persists its own CSV row eagerly). Gap → 'unavailable' back to TFA, no user prompt. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
references/interactive-mode.md specifies the orchestrator loop: spawn ai-tfa-coordinator subagents 5 at a time; a subagent cannot pause to prompt the user, so on an evidence gap it ends early with a GAP_OUTPUT carrying resume handles (threadId+turnId); the orchestrator asks A1, then re-dispatches with resume= and the answer. Same coordinator as auto — only the gap action differs. Compact blocks not transcripts (lean main context); partial-first; auto-first/ escalate-the-residue noted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
references/github-evidence.md specifies exactly what each github ask needs (diff-since-baseline, PRs-in-window touching the failing path, blame, deploy timing) and the discovery order GitHub MCP → gh → degrade — no shipped forensics harness. Adds the adversarial falsification protocol (path overlap / deploy-state guard / direction) so only verdict:supported suspects enter related_prs; ruled-out suspects stay as disconfirming evidence. Coordinator runs it for product_code/ deploy/ci asks, reusing the pre-computed build evidence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… in U4) lib/coverage.mjs derives a per-row evidence-coverage band — TFA confidence capped by coverage (full keeps it, partial→medium, thin→low) so a RESOLVED built with evidence unavailable reads as lower confidence BECAUSE of the gap. lib/report.mjs renders the CSV to markdown: status counts + per-test table + coverage caveats, degrading missing fields to 'not available' and never crashing on an empty/partial batch. report-format.md documents the stamp, layout, and the startup reaper resume path. Blast-radius digest explicitly deferred. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…harness lib/loop.mjs (runRcaLoop) is an executable mirror of the coordinator loop — status branching, ask routing, gap resolution, turn-cap, one-thread, soft-PENDING — driven by an injected submit(). It doubles as the D5 sequential thin-client harness. tests/conformance.test.mjs replays recorded tfaRcaTurn transcripts (resolved/blocked/pending/turn-cap fixtures) and proves: rca capture, test_logs skip, soft-PENDING no-re-poll, turn-cap never submits a 7th turn, and the degraded (no-capability auto) path still reaches a valid terminal RCA — same loop, same result. 48 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…, skip turn-cap gather Code-review fixes (suggested, non-blocking): - pending-resume removed from TERMINAL_STATES → soft-PENDING rows are now re-claimable, listed by pendingRows, and skipped by the reaper (they cleared in_flight), so the retained threadId/turnId actually drive an in-session resume instead of being stranded as a permanent non-terminal terminal. - flip() now rejects a missing/non-terminal rca_done without mutating, so a partial flip can't clear the claim yet leave the row pending (duplicate-RCA clobber). - loop checks the turn-cap BEFORE gathering, so evidence on the never-submitted final turn isn't gathered for nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mirror the slack-mcp-plugin parallel-manifest pattern over one shared skills/ tree: add .cursor-plugin/plugin.json + .cursor-mcp.json (stdio bstack, Cursor dialect) and codex-mcp.example.toml (~/.codex/config.toml block). INTEGRATION.md documents per-host wiring — MCP config, Agent-Skills skill/agent discovery (.cursor/skills, .agents/skills, .codex/agents), the Add-to-Cursor deeplink and codex mcp add — and the one Claude-only piece (auto-mode dynamic workflow), which Cursor/Codex fill with interactive subagents or the sequential lib/loop.mjs harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
automation/: failing API cases for the rcaChat / is_mcp_driven feature (pins the jsonb 42601, setTestRca data-gate, and ownership-lockout regressions), the JUnit report (build-rca-failures.xml), and upload.sh. Uploaded as project 'RCA Feature Fencing' / build 'VRT Build' (build awswxm0t5ve7vbjnspfna4xbvjwxn92u2lwv5fw2). skills/k8s-rengg-tfa/ + bin/k8s-context.sh: the discovered k8s evidence capability — read-only obs-api context (pod health, deployed image, error-level logs, events) from the rengg-tfa namespace, error-level grep + secret redaction. Wired as the k8s discoveryHint in rca.config.json. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both commands/rca-build.md (/tfa-rca:rca-build) and skills/rca-build/SKILL.md (name: rca-build) existed in the plugin — the slash invocation resolved ambiguously and returned a bare 'loaded' stub, forcing the agent to hunt for and re-read SKILL.md. Remove the redundant command (the skill is already slash-invocable and parses build id/mode/PRs in Step 0) and tighten the skill description (393 -> ~270 chars) so it registers cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…is present Auto mode is autonomous (no mid-run user input), but Step 1 said the intake was 'mandatory, both modes', so the agent prompted and STALLED waiting for intake answers even with mode=auto. Rewrite Step 1 per mode: auto gathers intake from invocation args + cheap inference (gh repo view, current branch), records any missing field as 'I don't have one' (RCA-only), shows a one-line FYI, and continues immediately — never prompting. Only interactive mode asks A1, once, upfront. Headless unchanged (build id required, rest default to none). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Plugin agents register under <plugin>:<name>, so the auto workflow's
agent({agentType:'ai-tfa-coordinator'}) failed with 'agent type not found'.
Use 'tfa-rca:ai-tfa-coordinator' in rca-batch.mjs (both representative + sibling
dispatch) and fix the interactive-mode / agent-doc dispatch examples to match.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e_all) The prior fix only updated the sibling call (different indentation); the representative agent() at the top of the pipeline still used the bare 'ai-tfa-coordinator' and would fail identically. Both now use the namespaced type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- skills/rca-build/ -> skills/factory/ (frontmatter name: factory); no command file with the same name (collision lesson, see 2af22ef) - delete the mode concept: no mode arg, no auto|interactive fork, drop references/interactive-mode.md - restructure to ONE GATE before execution: Part A = connector discovery + cheap probe validation into a connector -> valid|invalid|absent manifest (gaps recorded, never blockers); Part B = intake resolved by assumption, at most ONE consolidated question at gate close (headless never asks). After gate close the run never asks the user again. - no local report: drop references/report-format.md and config paths.reportFile; finish = glimpse table + triggerRcaReport + Test Observability UI link - PRODUCT_BUG mandate in github-evidence.md: application-bug RCA must carry culprit PR link(s) or explicitly state what was searched - server emits only NEEDS_INFO|RESOLVED: drop BLOCKED from references Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… local report
- lib/loop.mjs: RESOLVED now carries {glimpse:{root_cause,failure_type,
related_prs}, viewRca} — no full rca payload; drop BLOCKED (server
emits only NEEDS_INFO|RESOLVED, plus soft-PENDING); drop the
resolveGap injection — gaps always degrade to an 'unavailable' block
(auto-only, no prompt path)
- delete lib/report.mjs and .rca/rca-report*.md — the plugin never
renders a local RCA report; new lib/glimpse.mjs renders the terse
end-of-run glimpse table (testRunId → cluster → status → confidence
one-liner) from the CSV spine
- fixtures: resolved.json/pending.json to the trimmed shapes; drop
blocked.json; conformance + coverage tests updated (49 green)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- ai-tfa-coordinator: drop the mode input and the interactive gap-return
(GAP_OUTPUT); gaps always degrade to 'unavailable' — never a prompt.
Classify only RESOLVED|PENDING|NEEDS_INFO (BLOCKED gone); consume the
trimmed RESOLVED glimpse {root_cause, failure_type, related_prs} +
viewRca and pass it through verbatim; RCA_OUTPUT gains failure_type +
view_rca, drops possible_fix/BLOCKED
- mandatory culprit-PR hunt for PRODUCT_BUG / application bugs: feed PR
link(s) to TFA so dashboard related_prs populates; no PR by turn cap →
explicit 'no culprit PR identified after <searched>' + recorded gap
- workflows/rca-batch.mjs: single autonomous path (no mode arg), schema
matches the new RCA_OUTPUT, PRODUCT_BUG mandate in the shared prompt
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…CKED residue - README: /factory usage, the one-gate two-part flow (validated connector manifest + assumption-first intake, at most one consolidated question), autonomous execution, glimpse + triggerRcaReport + Test Observability UI link as the only outputs - INTEGRATION: factory skill naming, triggerRcaReport in the MCP core row, no-local-report finish on every host - automation/README: demo invocation is now /factory <build-id> - lib/routing.mjs comments: gaps degrade to 'unavailable', no resolveGap Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…c-generator skill Revert only the rename from the single-gate rework (name collision with the user-level /factory skill); all auto-only, single-gate, glimpse, and PRODUCT_BUG-mandate behavior stays. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eDir) Two fixes to the state spine's location: 1. The build id is now IN the filename (rca-state.<buildId>.csv) — previously a fixed .rca/rca-state.csv meant a run on build B would collide with and 'resume' into build A's state. 2. Default directory is OS temp (<tmpdir>/bstack-rca/), not the invoking workspace — a background harness must not pollute the repo it runs from. paths.stateDir remains an override for retained artifacts (e.g. CI). Resume-safety is preserved per build (same id -> same path). Hostile ids are sanitized. 4 new tests; 53/53 green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…les/, scripts/ - k8s-rengg-tfa is self-contained: bin/k8s-context.sh -> skills/k8s-rengg-tfa/ scripts/ (carries the pending redaction hardening for auth headers) - rca-build gains templates/ (suspect-packet, evidence-block, gate-summary) as the canonical fillable formats — references now point at them instead of embedding drift-prone copies — and examples/sample-run.md (fictional, matches the recorded-turn fixtures): gate summary, answered NEEDS_INFO turn with supported + ruled-out suspects, terminal glimpse output Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ai_report/aitfa tabs The viewReport/viewRca destination is the build's AI-TFA report: <base>/builds/<buildUuid>?tab=ai_report&subTab=aitfa on automation.browserstack.com (designer-confirmed; was an observability.browserstack.com assumption). Fixture now carries the guidance string the bridge actually emits on RESOLVED turns — closes the fixture-drift review finding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Environment-specific harness removed; k8s stays a generic gate-probed capability (discoveryHints now empty, like logs/metrics). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…runtime the user has The harness no longer assumes Kubernetes. 'infra' is the capability (runtime state of the service); k8s/EKS, ECS, docker, Nomad, VMs, PM2 are all just possible connectors behind it — the gate probes what actually exists and records the kind in the manifest (via). The k8s evidenceType remains a routing alias into infra so TFA asks spelled 'k8s' keep working. kubectl is one probe on a menu, not the definition of the capability. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…; note prod default base URL Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…beta.1 Pinned across all three client wirings (.mcp.json, .cursor-mcp.json, codex example) + INTEGRATION.md. Codex example env aligned with the prod-default base URL (staging override optional). Version verified present on the public npm registry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…erence The automation/ demo (failing rcaChat API cases + upload.sh, named observability-api as 'the product repo') was env-specific bait: the gate latched onto the only repo named in a workspace doc and carried its PRs into the manifest — even for a build whose failures were self-healing tests, wholly unrelated to observability-api. - Delete automation/ (tests, upload.sh, README, fixture xml). - Gate Part B now REQUIRES product-repo corroboration against the failure signatures; a doc-sourced repo is a weak hint, never an assumption — if it doesn't match the failures it's recorded as a gap and culprit-PR hunts report 'no culprit PR identified' rather than blaming an unrelated repo. - README 'Demo run' section replaced with a generic 'Run' section (no seeded build, no hardcoded repo). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…interactive) Previously the gate silently recorded 'product repo: gap' whenever corroboration failed — which kills the mandatory culprit-PR hunt without telling the human sitting at the terminal. Now, when the repo can't be corroborated against the failure signatures AND no PRs were supplied, the product repo is treated as non-assumable AND load-bearing and earns the single consolidated gate question (interactive only): 'Failures look like <domain>; which repo owns that code? (reply none → RCA without culprit-PR attribution).' Headless still records the gap and proceeds RCA-only. Preserves both invariants: never blame the wrong repo, and never silently cripple culprit-PR hunting when a human could answer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…etail in Claude The end-of-run glimpse was leaking exactly what's supposed to live only on the TRA UI: per-test root causes, culprit/related PRs, cluster breakdowns, confidence rationales. renderGlimpse now emits just 'RCA analysis complete — build <id>' + a status-count line (N tests · R resolved · P pending · F failed); the caller adds the 'Full report on the Test Observability UI: <link>' line. Step 6 and the example updated to match; a test asserts no testRunId/cluster/root_cause/PR ever appears in the output. Humans open the link for the why. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A soft PENDING is not an agent verdict. tfaRcaTurn abandons its own in-call
poll at POLL_MAX_WAIT_MS (90s) and returns `{status:"PENDING", turnId}` while
the TFA agent keeps working server-side. The loop treated that as terminal, so
any turn finalizing past 90s was reported PENDING even though the agent had
already committed to an answer.
Observed on test run 3840238857 (build xclakkm...): turn 1 ran 10:09:47 ->
10:11:31 (104s) and finalized NEEDS_INFO with 4 asks + 2 questions, webhooking
the result to o11y. The client saw only PENDING and stopped. Reading the same
turnId with getTfaTurnResult returns that NEEDS_INFO verbatim.
On PENDING the loop now READS that turnId (getTfaTurnResult) on a bounded
budget until a real agent status lands, and only then routes asks and submits
the next message:
- lib/loop.mjs: drainSoftPending() + injected `readTurn`/`sleep`. Reads never
consume turnCap (a drain re-reads the same turn, it is not a new turn) and
the spent turnId is dropped so follow-ups ride threadId alone. A failed read
is not a verdict. Never submits onto a turn still in flight, which would
stack two turns on one thread.
- config: softPendingDrain { maxWaitMs 600000, intervalMs 5000, maxReads 40 }.
Budget spent -> still ends PENDING (pending-resume row), so a wedged turn
cannot hang the batch. Old behaviour is now the floor, not the default.
- No getTfaTurnResult tool on the client -> end PENDING immediately as before,
never busy-wait through tfaRcaTurn resubmits.
- Also handle BLOCKED as terminal. It carries no asks, so classifying it as
NEEDS_INFO resubmitted empty messages to the turn cap. Reported as PENDING
with note "blocked"; giving it its own status in RCA_OUTPUT is a follow-up.
- Contract updated in the three places that encode it: the coordinator agent
(principle 5, loop step 2, hard limits, tools), the workflow's shared prompt,
and SKILL.md (resume + hard rules).
Tests: 6 new conformance tests over a recorded 3840238857 drain fixture --
drains before the next submit, reads the same turnId, bounded budget, transient
read failure, BLOCKED terminal, and the no-tool floor. 58/58 pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Strips product-specific artifacts that leaked into the generic plugin from the nl2steps latency-instrumentation runs: the hardcoded "at least 5 kubectl calls" busywork floor and the hardcoded VictoriaLogs Grafana datasource UID in the shared coordinator prompt, plus de-hardcoded remaining nl2steps/o11y/tcm example references in SKILL.md and ai-tfa-coordinator.md down to neutral placeholders. Restores concurrency to 20 (the profiling-backed fleet ceiling) instead of the untested 50. Also gitignores .DS_Store/*.code-workspace to keep local editor artifacts out of the repo. The generic "mandatory connector sweep" mechanism itself (a connector skill may declare a compulsory check, routed by capability) is kept — only the instrumentation-specific numbers/UID were testing artifacts.
Removes the "proactive, never ask-gated" mandatory connector sweep mechanism (Operating Principle 0 + loop step 0.5 in ai-tfa-coordinator.md, the shared-prompt reminder in rca-batch.mjs, and the mandatory_checks field in the RCA_OUTPUT schema/contract and SKILL.md's closing bullet). Forcing evidence gathering into turn 1 regardless of whether TFA actually asked for it bloats every coordinator's first message and burns tokens for evidence that may not be needed. Evidence gathering is now purely reactive to TFA's own NEEDS_INFO asks, routed by capability as before.
… all coordinators Each dispatched ai-tfa-coordinator previously re-ran its own turn-1 PR-window search and kubectl/VictoriaLogs sweep independently, even though that evidence is a property of the build, not of any one test. Add lib/evidence-file.mjs (a build-scoped JSON artifact, keyed by repo and workload, mirroring csv-state.mjs's path convention) so the orchestrator gathers this once and every representative/sibling reads it before falling back to a live call. Validated against a real build: two re-run tests landed the same root causes (including the same culprit PR) at 69-70% fewer tokens and 90-94% fewer tool calls than the original per-coordinator sweep. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Self-imposed plugin budget, well under the tfaRcaTurn tool's actual 5000-char hard cap. Retunes the per-block digest caps (SUMMARY, SNIPPET) proportionally so real turn-1 messages fit the new budget instead of relying on ad-hoc truncation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d file The pre-fetch file was read-only from a coordinator's side — when a live gather filled a gap or went deeper than the pre-fetch had (a full diff instead of a summary, a PR the pre-fetch never named), that extra work died with the coordinator instead of benefiting its own siblings or other clusters sharing the same repo/workload. Add mergeGithubEvidence/mergeLogsEvidence (read-modify-write, dedupe PRs by number, union clusterIds, never drops what a patch doesn't mention) and wire Operating Principle 0 + the culprit-PR mandate + the routing step in ai-tfa-coordinator.md to call them after any live gather. Confirmed on a real prior run (vrt_345): a representative's own deep PR-diff investigation was exactly the kind of work its sibling had no way to reuse under the read-only version of this file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The write-back added in 70ef859 had every coordinator read-modify-writing one shared JSON file. Measured under 8 concurrent writers with a realistic read -> work -> write window, that design lost 28 of 40 updates (70%). Replace it with a layout where contention is structurally impossible instead of merely unlikely: the orchestrator solely owns the base file, and each coordinator writes only its own shard at rca-evidence.<buildId>.contrib/<testRunId>.json. No two processes ever open the same file for writing. readEvidenceFile folds base + all shards into one view (sorted order; real evidence beats a recorded gap; PRs unioned by number), and readBaseFile keeps the orchestrator's own write path from absorbing shard content back into base. Same 8-writer test against the new layout: 0 of 40 lost. A 12-process run hammering one repo key kept all 480 writes. A corrupt/half-written shard is skipped rather than breaking every subsequent read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two measured sources of duplicate work on a real 10-test build: 1. `gh` was 37% of all coordinator tool calls (151), and 46 were byte-identical commands re-run by different coordinators — one BStackAutomation spec fetched 12 times, a frontend component 5 times. None of these fit the evidence file's schema (deployState / prsInWindow), so the write-back added earlier could not share them. Add a build-scoped memo cache keyed by the CALL rather than by evidence shape, so duplication is caught regardless of what the call was for. `bin/cached-exec.mjs` wraps a shell fetch transparently (same stdout, same exit code, executes only on a miss); `bin/cached-mcp.mjs` does check-then- store for read-only MCP queries. One file per call key + atomic rename, so concurrent writers cannot collide or produce a torn read. Failures are never cached (a transient rate-limit must not become a permanent answer), stateful tools (tfaRcaTurn/getTfaTurnResult/triggerRcaReport) are refused, secrets are redacted before anything reaches disk, and the runner uses execFile with a hand-rolled tokenizer so no shell ever interprets an argument. 2. Drain reads plus their sleeps were 23% of all coordinator tool calls. The drain treated "TFA is still thinking" and "the TFA run hard-failed" identically, spending the full 40-read/10-min budget on turns that had already died — the four tests that wedged this way were the four slowest in the batch. Distinguish the two: stop after `maxErrorReads` (default 3) CONSECUTIVE hard failures, note it as `tfa-error`, keep the row resumable. A single good read clears the streak, so flaky-but-recovering reads still land. Measured: cross-writer cache hit served in 0.185s vs 1.131s for the live fetch. 108 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All three on-disk artifacts live under a world-readable OS temp dir and carry sensitive content: the tool cache holds raw gh/kubectl output (private repo source, internal hostnames, log bodies), the evidence file holds PR detail and app-log digests, and the state CSV holds root causes and culprit PRs. Default umask left them 0644 — readable by any local user. Create directories 0700 and write files 0600. The file mode is the load-bearing control: a pre-existing directory keeps its own permissions (silently chmod'ing a caller-supplied stateDir would be presumptuous), but entries are 0600 either way, and a traversable cache dir only exposes opaque hash filenames. Note this hardens storage, not the redaction: redact() is best-effort pattern matching over token-shaped strings and was never sufficient on its own to make these files safe to leave world-readable. Verified on disk for both a fresh dir (0700/0600) and a pre-existing 0755 dir (files still 0600). 111 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dispatched three coordinators concurrently on a live build. They surfaced issues no unit test had, worst first: 1. CORRECTNESS: an empty `prsInWindow` with `gap: null` is byte-identical whether the PR search ran and found nothing or was never populated. Seen live — the file asserted 0 PRs for a repo that actually had 21 (the loader's search had silently returned empty), which would let a coordinator report "no culprit PR identified" with false confidence. Add an explicit `prsSearched` flag (sticky across contributors), a `hasTrustworthyPrList` helper, and a `coverage.reposWithUntrustedPrList` signal. Repo-level coverage semantics are unchanged — trustworthiness is reported alongside, not folded into, covered/gapped. 2. The runnable-guard scanned the raw command string, so it refused legitimate read-only calls: `;` inside a jq expression, `&` inside a quoted URL. Check whole argv TOKENS instead — post-tokenization a quoted metacharacter is inside an argument (harmless, we execFile) while a real operator is its own token. Verified both previously-refused commands now run. 3. execFileSync BOTH inherits and captures stderr, so relaying err.stderr printed failures three times. Capture only, relay once — wrapper stderr is now byte-identical to the command's own, plus one banner line. 4. Empty results were cached, making a sticky invisible negative. Not stored. 5. Nested single quotes made `--jq '...'` unpassable inside a quoted command argument. Accept `-` to read the command from stdin. 6. File mode applies on create only, so files left by a pre-hardening run kept 0644 forever. chmod explicitly on overwrite too. Also documented two traps that are usage, not bugs: `2>&1 | jq` merges the stderr banner into the pipe (the banner is correctly on stderr — one agent misreported this as a stdout bug; measured to confirm), and `cached-mcp get`'s exit code is lost behind a pipe. 118 tests pass, including regression tests for each fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaying 223 recorded gh/kubectl calls from prior coordinator runs through the wrapper showed it was nearly useless in practice: only 25 calls were accepted and ZERO were hits. The cache was fine; the guard was wrong about what real traffic looks like. Three causes, each measured: - 89% of real calls embed the fetch inside a pipeline (`gh api X | jq .y`), which the guard refused outright. Now pipelines are parsed into a fetch plus a chain of pure text filters (jq/grep/head/... allowlisted), each run via execFile with no shell anywhere. Only the FETCH is keyed, so several agents filtering one fetch differently share a single network call. - `2>&1` appeared on 134 of 223 calls — agents add it reflexively because gh is chatty. It says nothing about what to fetch, and the wrapper captures stderr separately anyway, so it is normalized away instead of refused. Refusing it alone drove acceptance to zero. - `>/dev/null` was classified as "looks mutating", which is both wrong and misleading. Redirection is now separate from mutation, with an accurate message; genuine file redirects are still refused. Result on the same recorded traffic: 123/223 accepted, 32 duplicate fetches eliminated, 26% hit rate. The rest are `for repo in ...` shell loops, which legitimately need decomposing into one fetch per call — that is also better for cache granularity. Verified end-to-end that `gh api X 2>&1 | jq .a` and `... | jq .b` now share one cached fetch, and that chaining, non-filter binaries and file redirects are still refused. 123 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dispatched six coordinators concurrently against a warm cache. Five of six resolved; the swarm found three real defects, one of them data-destroying. 1. DATA LOSS: redaction consumed the rest of the LINE after a secret-ish key. GitHub's file API returns SINGLE-LINE JSON whose download_url always carries `?token=…`, so a 214,074-byte response was cached as 816 bytes with the content field silently gone. Every private-repo file fetch through the cache was corrupted, and a coordinator hit exactly the predicted failure: it could not verify a line-number-specific hypothesis and nearly falsified a claim that was in fact TRUE (TestRunService.java:1747 really is an unguarded `.get(0)`). Redaction is now bounded by the first structural delimiter. Two already-corrupted entries were purged from the live cache. 2. Neither tokenize nor splitPipeline handled backslash escapes, so `\"` read as a closing quote. Two coordinators hit the two symptoms: jq string equality arrived mangled (`select(.filename==\a/b.json\)`), and a `|` in a jq regex alternation was split as a shell pipe, refusing the command. One fix, both symptoms — escapes are now POSIX-style (literal everywhere except inside single quotes). 3. Two findings corroborated independently by multiple coordinators, now in the contract: a drain error kills the TURN, not the THREAD — resubmitting on the same threadId resolves immediately, whereas the old reading ended PENDING and discarded a resolvable test; and the turn-2 wedge correlates with message size (~1400/~1350-char submits failed, a ~940-char retry landed) against a configured 1000-char cap. Cache went 33 -> 63 entries over the swarm with 7 cross-agent hits, all seven writer shards present and no collisions. 128 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…abulary `flip` accepted only the lowercase CSV states and returned a bare `false` for anything else — including `RESOLVED`, which is the exact value the RCA_OUTPUT contract tells every coordinator to emit. Callers that didn't check the return saw no difference between success and a no-op, so their rows stayed `pending` and looked un-run. This was not theoretical: a six-coordinator swarm appeared to "skip" the CSV flip entirely. A later coordinator diagnosed the real cause — they had almost certainly called flip with `RESOLVED` and been silently refused. Field names had the same trap: `thread_id`/`status` were dropped without comment by the COLUMNS guard, since the columns are `threadId`/`rca_done`. Accept and normalize the output-block vocabulary (RESOLVED->resolved, PENDING->pending-resume, case-insensitive) plus the field aliases, and make genuine rejections and dropped keys warn loudly instead of failing silently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ce and hit counts Three findings from a warm-cache verification run. 1. RETRACTION. The contract told coordinators to keep turn messages under 1000 chars to avoid the TFA wedge, on the strength of a two-point correlation (~1400/~1350-char submits failing where a ~940-char retry landed). A later run refuted it: a 240-char message wedged exactly as a 1500-char one did. The cap stays — a tight digest is the contract — but it is no longer presented as a wedge cure, and a wedge must not be read as evidence the message was too long. The reliable response remains resubmit on the same thread. 2. Contributions were invisible to the obvious read. Coordinators are handed the base path and naturally `cat` it, which shows base only — one agent reported "2 repos" when the folded view had 5, including an 11-PR observability-api entry a sibling had contributed. Shards are what make concurrent write-back safe, so add `bin/evidence-show.mjs` to print the merged view (with --summary flagging repos whose PR list was never actually searched) rather than give up the layout. 3. Hit-rate was unmeasurable in normal use: callers silence tool chatter with `2>/dev/null`, which also discards the banner the metric depends on. Banners now tee to TOOLCACHE_LOG when set, so stderr can be suppressed and hits still counted. Warm-cache result that motivated this: cross-agent hits confirmed — one coordinator re-filtered a sibling's cached 76KB PR diff three ways for free, another reused two cached diffs it never fetched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The MCP cache had stored zero entries across every live run. The cause is structural: an agent's get -> call -> put costs three tool calls on a miss to save one later, so skipping it is correct for a one-off query, and every coordinator did. Pre-seeding inverts the economics. The orchestrator already runs these log sweeps in Step 4, so depositing each digest under the key a coordinator would compute makes the agent's `get` a single call that usually hits. Verified: after seeding four VictoriaLogs digests, a coordinator get returns HIT, and arg-order canonicalization means a differently-ordered query still matches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1. `flip` silently dropped `view_rca` and `turns_used` — both are in the RCA_OUTPUT contract but had no CSV column, and `view_rca` is the dashboard link the entire run exists to produce. Added as columns. 2. An MCP cache HIT requires reproducing the args exactly, and canonicalization normalizes key ORDER, not content — so a coordinator that guesses the logql/window/limit triple misses. One run burned four probe calls guessing to save two, making the cache a net loss on that pass. Added `cached-mcp <build> list`, which prints every cached query with its exact args to copy: one call instead of guessing. 3. MCP banners were not tee'd to TOOLCACHE_LOG, so shell and MCP hits could not be totalled from one file. Both now log there. Verification-run result for the record: 22 tool calls vs 81 on the same test cold (73% fewer), but the saving came overwhelmingly from the shared evidence file, not the call-level caches (1 shell hit of 8; MCP roughly a wash after the failed probes). The caches only pay on repeated identical lookups, and that run's second turn demanded a code path nobody had walked — a low hit rate there is the correct outcome, not a cache failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…sely Both from a live run on a second build (ObservabilityApiLaneSuite, 189 failures across 9 clusters). - `evidence-show --prs [repo]` prints mergedAt | #num | title plus the build's started_at. That table is the highest-value falsification per byte: on this run it disqualified 11 of 22 candidate PRs on the window guard alone, before a single diff was fetched. Previously a coordinator had to pipe --repo's raw JSON through an ad-hoc node one-liner to build it. - `flip(testRunId, fields)` — dropping the leading csvPath — used to bind an object to testRunId, read a nonexistent CSV, and return a bare `false` that a caller mistook for success. It now says exactly what went wrong and shows the correct signature. Cache behaviour on this build, worth recording: the shell cache was COLD, yet the representative still logged 4 hits of 7 fetches — concurrently dispatched siblings populated it for each other, including the single most expensive call (`gh pr diff 16450`). And `cached-mcp list` alone avoided two grafana calls: the seeded digests were self-sufficient, so no `get` was needed at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ve absence A live run on a second build found the pre-fetch structurally blind to its own build's cause. An upstream outage ran 06:12:00-06:15:51 while the build's finished_at was 06:12:21, so a sweep scoped strictly to started_at..finished_at saw 21 seconds of a 4-minute outage. Step 4 now pads to started_at-2m .. finished_at+10m and labels findings inside/outside the strict window — without licensing an arbitrary window, which is the opposite failure (matching a coincidence in unrelated traffic). Two query mechanics added, both of which cost real calls in that run: - `direction` defaults to newest-first, so a limited query returns the END of the range. Verifying this finding required direction:"forward" — my own backward queries kept returning traffic clustered at each window's tail, which looked like the gap was absent when it was simply out of view. - Absence needs a control: a zero-result query is indistinguishable from a bad selector. Prove the logger was alive in the same window with a query you expect to be non-empty before treating silence as evidence. Verification note on the finding itself: the gap is real (zero /ext/v1 06:12:00->06:15:51, resuming with a 500 on /ext/v1/badge/build). Two secondary details in the coordinator's report did not hold — traffic resumes at 06:15:51 not 06:16:30, and badge traffic was not absent until 06:48. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ng a negative Both from a live run on a second build. - `TFA agent run failed` (the wedge, size-independent, fix = resubmit on the same thread) and `turn expired or not found` (observed on a ~2000-char over-cap submit) are different failures. The latter's text names a thread/turn problem, so a coordinator reads it as a wedge and resubmits unchanged instead of shortening. Named both explicitly. - A coordinator nearly concluded a manifest lacked an entry when the fetch had been cut at ~64KB; its own `wc -l` control caught it (1042 lines vs 1518). Investigated and the tool cache is NOT the cause — verified 104KB and 214KB files round-trip intact, and the only truncation is past 256KB with an explicit marker — so this was surrounding tool plumbing. The hazard is real regardless of layer, and a silent truncation converts "grep found nothing" into a false negative, so: size-check a large fetch before treating an absent match as evidence of absence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.