What happened
PR #6476 raised max_turns from 30 to 35 across all 4 triage eval cases to accommodate the issue-labels skill activation from fullsend-ai/agents#917. The code agent chose a 5-turn increase. During human review, ralphbean approved but requested a followup to increase the budget further to 40, indicating the 5-turn increase was insufficient headroom.
What could go better
The code agent's reasoning was sound — it cited the eval docs' "start generous and tighten" guidance — but its interpretation of "generous" was too conservative. A 5-turn buffer over the observed failure point leaves little margin for normal variance in LLM turn counts, especially since all 4 cases now invoke the issue-labels skill on every run. The human reviewer's preference for 40 (a 10-turn increase from the original 30) reflects operational experience with eval non-determinism. Confidence: high — this is a direct human request with clear rationale.
Proposed change
Update max_turns from 35 to 40 in all four triage eval case annotation files:
eval/triage/cases/001-bug-url-encoding/annotations.yamleval/triage/cases/002-needs-info-vague-crash/annotations.yamleval/triage/cases/003-feature-request/annotations.yamleval/triage/cases/004-duplicate-issue/annotations.yaml
This is consistent with the eval docs' guidance to start generous and tighten once a stable baseline is established.
Validation criteria
The max_turns judge should score 100% (4/4) on the next 10 consecutive functional-tests runs on main where all 4 triage eval cases complete without crashes. The additional 5-turn buffer (35 → 40) should eliminate any remaining near-budget runs without masking genuine turn-count regressions.
Generated by retro agent from #6476
What happened
PR #6476 raised
max_turnsfrom 30 to 35 across all 4 triage eval cases to accommodate theissue-labelsskill activation fromfullsend-ai/agents#917. The code agent chose a 5-turn increase. During human review, ralphbean approved but requested a followup to increase the budget further to 40, indicating the 5-turn increase was insufficient headroom.What could go better
The code agent's reasoning was sound — it cited the eval docs' "start generous and tighten" guidance — but its interpretation of "generous" was too conservative. A 5-turn buffer over the observed failure point leaves little margin for normal variance in LLM turn counts, especially since all 4 cases now invoke the
issue-labelsskill on every run. The human reviewer's preference for 40 (a 10-turn increase from the original 30) reflects operational experience with eval non-determinism. Confidence: high — this is a direct human request with clear rationale.Proposed change
Update
max_turnsfrom 35 to 40 in all four triage eval case annotation files:eval/triage/cases/001-bug-url-encoding/annotations.yamleval/triage/cases/002-needs-info-vague-crash/annotations.yamleval/triage/cases/003-feature-request/annotations.yamleval/triage/cases/004-duplicate-issue/annotations.yamlThis is consistent with the eval docs' guidance to start generous and tighten once a stable baseline is established.
Validation criteria
The
max_turnsjudge should score 100% (4/4) on the next 10 consecutive functional-tests runs on main where all 4 triage eval cases complete without crashes. The additional 5-turn buffer (35 → 40) should eliminate any remaining near-budget runs without masking genuine turn-count regressions.Generated by retro agent from #6476