experiment: evaluate direct, RLM, and Codex todo agents

[mpscholten]

Experiment — do not merge

This PR records an exploratory comparison between direct agent-cli, an in-process recursive-language-model (RLM) mode, and Codex. It is intentionally left as a draft and is not proposed for merge. The implementation and transcripts are useful as an experimental artifact.

What was built

• reproducible Haskell todo-server coding eval
• deterministic grading of GHC 9.10, MVar state, JSON schema, and GET/POST/DELETE behavior
• wall-clock and provider-reported token accounting
• in-process RLM workers exposed through rlmQuery, rlmQueryMany, and rlmCode
• worker usage included in root-session token totals
• immutable shared flake.nix and flake.lock fixture
• Nix package and development closures prebuilt before timed runs
• byte-for-byte grading that agents did not modify the supplied flake fixture

Controlled result

Run on office-builder on August 24, 2026 with gpt-5.6-terra, medium effort, and one trial per runner. Every runner preserved the supplied flake, self-verified, and passed the independent grader.

runner	passed	seconds	input	uncached	cached	output
agent-cli	yes	250.51	268,201	27,305	240,896	6,385
agent-cli RLM	yes	420.11	695,037	122,237	572,800	17,084
Codex	yes	190.98	659,186	40,690	618,496	6,759

Direct agent-cli used 59.3% fewer total input tokens than Codex, but took 31.2% longer. RLM took 67.7% longer than direct agent-cli and used 159.1% more total input, 347.7% more uncached input, 137.8% more cached input, and 167.6% more output.

Interpretation

The controlled run did not show an advantage for RLM on this task. The RLM worker produced code with faulty imports; the root then reread and substantially rewrote the complete file and performed several repair turns. Delegation duplicated implementation and review context instead of reducing root work.

This is a small, tightly specified, one-file task. Worker startup and handoff overhead are large relative to the implementation. RLM may still be useful for larger tasks with separable research or implementation units, but that requires a different eval.

Why the prebuilt fixture matters

Earlier runs let each agent construct its own flake. Those timings were dominated by different first-use Nix closures and produced misleading comparisons. The final controlled fixture gives every runner identical pinned dependencies and prebuilds them before timing.

Caveats

• The final controlled comparison has only one trial per runner; it is a smoke result, not a stable median.
• Product prompts and tool contracts still differ between agent-cli and Codex.
• The RLM prototype needs better worker task scoping, validation, and handoff before another performance evaluation.
• Earlier cold/warm runs are retained in the documentation as historical data, with their comparability limitations explained.

Validation

• agent-cli test suite: 807 examples, 0 failures
• evaluator and agent-cli executables build with GHC 9.10.3
• shared flake prebuild smoke-tested
• all three controlled generated applications passed the external grader
• full methodology and transcript analysis in docs/evals/agent-vs-codex-todo.md

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 919d641e75

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Bound readiness by wall-clock time

When the generated server accepts connections but does not complete an HTTP response, each of these 600 attempts can consume the two-second curl --max-time plus the 500 ms delay. (curl --help all describes --max-time as the maximum time allowed for a transfer.) The purported five-minute readiness check can therefore take about 25 minutes per run, allowing one broken solution to stall a multi-trial evaluation for hours; enforce a five-minute deadline around the whole readiness loop instead.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Isolate agent-cli from machine-wide MCP configuration

On hosts with ~/.haskell-agent/config.json, the agent runner still unconditionally loads that file through loadHarnessConfig in Agent.CLI, including enabled MCP servers, while this invocation explicitly disables Codex user configuration. Those servers can alter agent-cli's available tools and timing or even make startup fail, so identical eval commands can produce host-dependent and asymmetric results; run agent-cli with an isolated harness configuration as well while preserving only the required credentials.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Equalize the runner turn budget

When a solution needs more than 50 model/tool rounds but remains within the configured wall-clock timeout, agent-cli is forcibly stopped while the Codex invocation receives no equivalent evaluator-imposed limit. Agent.CLI.Options documents this flag as stopping after N model turns and gives agent-cli a default of 500, so lowering only one runner to 50 introduces an extra failure budget that the advertised shared timeout does not capture; remove this override or impose the same budget on both runners.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Require successful status on every GET

If GET /tasks returns the expected two-element JSON body with an error status after tasks are created, this check passes because taskCount examines only the body; the first-persistence and final-deletion checks likewise omit the response status. The evaluator can therefore report that CRUD behavior passed even though every nonempty GET returns HTTP 500, so each GET assertion should also require status 200 and a JSON content type.

Useful? React with 👍 / 👎.

[mpscholten]

Revised self-verifying eval results

I reran the suite on office-builder after correcting the fairness problems found in the first transcripts:

• agent-cli used --no-subagents --no-ghci --bash;
• delegation tools were removed from its tool surface;
• both runners had to personally verify GHC 9.10, build with Nix, start with nix run, and exercise GET/POST/DELETE with curl;
• a run only counted when it recorded self-verification, did not delegate, exited successfully, and passed the independent grader;
• uncached input is now reported separately.

runner	passed	median successful seconds	median input	median uncached	median output	median cached
agent-cli	2/3	232.20	383,787	48,491	7,136	335,296
Codex	3/3	213.94	657,318	51,364	6,835	610,816

Among successful runs, agent-cli used 41.6% fewer total input tokens and 45.1% fewer cached-input tokens. Its uncached-input advantage was only 5.6%, and it used 4.4% more output tokens. Codex had the 8.5% faster median and completed all three trials.

Every completed run self-verified without delegation and passed the external grader. agent-cli trial 3 stalled immediately after creating its plan, produced no project files, and hit the 900-second timeout.

The corrected conclusion is therefore narrower: agent-cli still shows materially lower cached-context amplification, but it was not faster or more reliable in this sample.

The full individual-run table and the original-run caveat are now committed in docs/evals/agent-vs-codex-todo.md.

[mpscholten]

In-process RLM update and gpt-5.6-terra smoke results

Implemented the Prime Intellect-style recursive orchestration as in-process subagents, not child agent-cli processes:

• opt-in --rlm root mode with only the GHCi tool exposed
• rlmQuery, rlmQueryMany, and rlmCode mailbox helpers
• read-only workers can run concurrently; coding workers are serialized
• worker turn/time/call budgets and read-only tool filtering
• worker token usage is accumulated into the root session totals
• evaluator runner: --runner agent-cli-rlm

All corrected runs below self-verified and passed the independent GHC 9.10, MVar, and HTTP CRUD grader.

runner	seconds	input	uncached	cached	output
agent-cli direct	206.61	371,798	28,886	342,912	6,545
agent-cli RLM, warm Nix cache	248.30	155,518	49,790	105,728	9,342
Codex	713.20	1,886,187	111,851	1,774,336	9,637

Compared with direct agent-cli, the warm RLM run used 58.2% fewer total input tokens and 69.2% fewer cached tokens, but 72.4% more uncached input, 42.7% more output, and 20.2% more time. Compared with Codex, RLM used 91.8% fewer total input, 94.0% fewer cached input, 55.5% fewer uncached input, and 65.2% less time.

Why the token use dropped

The RLM root delegated workspace inspection, initial implementation, and a focused API correction to short-lived in-process workers. It then inspected the files and ran Nix/HTTP verification itself. Large implementation histories therefore stayed in worker contexts instead of being replayed through every root turn. That sharply reduced cached-context amplification. The extra worker prompts/responses explain the increase in uncached input and output versus direct agent-cli.

Timing caveat and startup fixes

The first corrected RLM run passed but took 835.26 seconds (333,778 input; 245,504 cached; 9,133 output). Its generated flake selected an unwarmed Nix package set; the first GHC check exhausted the command timeout while dependencies built, then succeeded on retry. After warming that exact package set, RLM took 248.30 seconds. Codex's 713.20-second run was also dominated by a first-use Nix closure, so this one-trial timing comparison is not controlled enough for a broad speed claim.

The first two RLM attempts also exposed real integration defects, now fixed in this PR: the benchmark process initially lacked ghci on PATH, and the generated rlmStart helper had an invalid local binding. The packaged agent now adds GHC to PATH, and the helper passes a direct GHCi startup smoke test.

Full methodology, transcripts analysis, cold/warm caveat, and measurements are committed in docs/evals/agent-vs-codex-todo.md.

[mpscholten]

Controlled prebuilt-flake eval

Implemented the suggested fixture change:

• every workspace receives the same pinned flake.nix and flake.lock;
• the evaluator builds the package and development closures before starting the timer;
• the provided flake runs app/Main.hs and exposes GHC 9.10, Aeson, WAI, Warp, and http-types;
• prompts explicitly forbid changing either flake file;
• the independent grader requires both files to remain byte-for-byte identical.

This removes runner-selected Nix dependencies and first-use closure compilation from timed agent work.

First controlled gpt-5.6-terra smoke run

All three runners preserved the fixture, self-verified, and passed the grader.

runner	seconds	input	uncached	cached	output
agent-cli	250.51	268,201	27,305	240,896	6,385
agent-cli RLM	420.11	695,037	122,237	572,800	17,084
Codex	190.98	659,186	40,690	618,496	6,759

The controlled result is substantially different from the earlier run. Direct agent-cli used 59.3% fewer input tokens than Codex but took 31.2% longer. RLM took 67.7% longer than direct and used 159.1% more total input.

Transcript inspection shows that RLM delegated implementation, received faulty imports, then the root re-read and substantially rewrote the complete file and performed several repair turns. For this small one-file task, delegation duplicated context and work rather than isolating it. This is currently one trial, so it should be treated as a smoke result rather than a stable median.

Implementation and detailed analysis are committed in fb60f64c and ef5a7b97.