Add Tools V2 (Orchestrate): a composable tool engine alongside V1

.claude/orchestrate-spec.md

@@ -0,0 +1,88 @@
+# Orchestrate: the specification, as tests
+
+Three scopes, each with its own seam. Nothing below a scope's seam is real inside it.
+
+| Scope | Under test | Everything else |
+|---|---|---|
+| Engine | what a run does with stages | fake tools that answer for themselves |
+| Program | turning an executor's answer into a stage's | `FakeExecutor` |
+| Executor | closing a pipe becoming a real kill | real processes |
+
+## Engine, with fake tools
+
+A stage's outcome. One per stage, and each test says what the stage produced, what the report
+says, and what the stages after it did.
+
+1. A stage runs to the end and its tool calls it done.
+2. A stage runs to the end and its tool calls it a failure.
+3. A stage is ended by a signal its tool reports.
+4. A stage is refused before it runs.
+5. A stage never starts, because the one before it failed.
+6. A stage never starts, because the one before it was refused.
+7. A stage is stopped for producing more than can be held.
+8. A stage throws.
+9. The call is cancelled while a stage is running.
+
+Bytes between stages.
+
+10. What one stage produces is what the next receives, byte for byte.
+11. Nothing between stages interprets those bytes: no separator, encoding or size is assumed.
+12. A producer runs ahead of a stalled reader by one buffer and no more.
+13. Adding a stage adds a fixed amount to that, not a multiple.
+14. A producer whose reader has gone is told to stop, and so is one two stages back.
+15. A producer that never ends still terminates the call.
+
+Joining stages.
+
+16. `&&` runs the next stage only when the previous one succeeded.
+17. `||` runs it only when the previous one failed.
+18. `;` runs it whatever happened.
+19. `|` gives the next stage the previous one's bytes.
+20. A refusal counts as a failure for 16 and 17.
+
+Xargs.
+
+21. The engine puts an Xargs stage's output into the next tool's declared field.
+22. What that field already held is kept, and the new values follow.
+23. A sequence where Xargs feeds a tool with no such field is refused before anything runs.
+24. An Xargs output larger than can be held stops the producer, and the stage it fed does not run.
+
+What is held.
+
+25. A batch shown for approval is bounded; reaching the bound refuses rather than showing part of it.
+26. The run's own result is bounded; reaching it stops the producer and the report says so.
+
+Judging.
+
+27. Every stage is put to the decision, including one that touches nothing.
+28. A stage is judged on what it will really do, after its variables are resolved.
+29. What is published for approval is what the caller wrote.
+
+## Xargs, as a tool on its own
+
+30. It splits what it reads into one argument per line.
+31. A trailing separator does not produce an empty argument.
+32. Bytes with no separator in them are one argument.
+
+## Program, with a fake executor
+
+33. An executor reporting exit code zero is a stage that succeeded.
+34. An executor reporting a non-zero exit code is a stage that failed.
+35. An executor reporting a signal is a stage ended by that signal.
+36. The process's bytes are the stage's bytes, unchanged.
+37. What was piped in reaches the process's input, unchanged.
+38. Closing the stage's output asks the executor to stop the process.
+39. A stage is not answered for until the executor says the process is finished with.
+
+## Executor, with real processes
+
+40. Closing the read end of a running process's output kills it with SIGPIPE.
+41. A process that ignores that is killed anyway.
+42. A process that ends on its own reports its own exit code.
+43. Nothing is left running once a run is over.
+
+## Captures, engine with a fake executor
+
+44. A capture holds the stage's whole output.
+45. It reaches a later command through the environment that command runs under.
+46. It never appears in what is published for approval.

.claude/plans/orchestrate.md

@@ -0,0 +1,201 @@
+# Orchestrate — Plan
+
+Execution checklist. Reasoning and decisions live in `.claude/orchestrate-design.md` —
+this file is only the phases and their status, so a new session can resume without
+re-deriving the plan from chat history.
+
+Tools V2: Orchestrate is a genuinely separate registration/approval system from the
+existing `packages/claude-sdk` `ToolRegistry`/`ApprovalCoordinator`, not a tool bolted
+onto it. See the design doc's "This is Tools V2" section for why. Every tool eventually
+becomes a ToolV2 — V2 replaces V1 entirely, catalogue-wide. A tool not yet ported (Memory,
+History, TypeScript, AzCli, GitHub, AzureDevOps, and everything else besides the handful
+built so far) is simply not done yet, not excluded from this plan — priority, not scope.
+
+## Phase 1 — `packages/orchestrate-core` — DONE
+
+The engine: `Leaf<TIn,TOut>`, `FsOperation` (`fs.list`/`fs.read`/`fs.write`/`fs.delete`/
+`fs.exec`), `Op` (`|`/`&&`/`||`, absent = `;`), `plan()` (buffer-vs-stream per stage from
+the live approval grant), `execute()` (gating, operator semantics, capture/reference,
+Xargs bridging, centralized stderr policy), `XargsStage`. Real vitest specs, builds/
+lints/type-checks clean.
+
+## Phase 2 — Rewrite `Pipe`'s six stages as genuinely lazy leaves — DONE
+
+`packages/claude-sdk-tools/src/Orchestrate/leaves/`: Find, Match, Head, Tail, Range,
+Read, plus Program (the ExecV3/ExecV2 successor, backed by `@shellicar/exec-core`'s
+real Executor). Real tests under `packages/claude-sdk-tools/test/Orchestrate/`, using
+`MemoryFileSystem`/`FakeExecutor`, never real fs/processes. Full existing suite green
+throughout.
+
+## Phase 3 — Build the actual `Orchestrate` tool — IN PROGRESS
+
+Wraps `orchestrate-core`, registered as its own thing per the Tools V2 decision — not
+touching the existing `ToolRegistry`.
+
+**Naming correction (SC caught this):** "Leaf" was never the settled name and implied a
+tree structure the design explicitly rejects. Renamed throughout `orchestrate-core` and
+`claude-sdk-tools`: `Leaf` → `ToolV2`, `LeafResult` → `ToolV2Result`, `LeafStage` →
+`ToolStage` (`kind: 'leaf'` → `kind: 'tool'`), `createXLeaf` → `createXToolV2`, the
+`leaves/` directory → `tools/`. These are tools — the same concept as a V1 tool, built to
+a streaming contract. Orchestrate is not a tool that encapsulates a fixed set of them; it's
+a tool that can run *any* registered one.
+
+**Architecture correction (SC caught this too):** the first pass hand-wrote a second copy
+of every tool's shape into a separate `wireSchema.ts`, kept "in lockstep by hand" with the
+registry — two sources of truth for the same thing. Collapsed into one real
+`ToolsV2Registry` (`registry.ts`): each tool is `defineToolV2`-shaped and self-describing
+(carries its own zod `model`, like a V1 `ToolDefinition` carries its own schema). The
+registry derives, from that one list:
+- `wireTools: BetaTool[]` — every registered tool gets its own wire entry, same as V1's
+  `Find`/`Paths` sources are both a pipe step and standalone-callable. This is the "it
+  needs all the tools" point — a V2 tool is genuinely callable on its own, not only
+  reachable through Orchestrate's stage array.
+- `stageSchema` — the `Orchestrate` wire tool's `stages` array, a discriminated union
+  built at construction time from every registered tool's own `model`. Generated, not
+  hand-authored — no second schema to drift out of lockstep.
+- `toStage(wire)` — resolves one wire stage into a real `orchestrate-core` `Stage`,
+  validating that stage's `input` against its own tool's `model`.
+
+Done so far, real code + tests, in `packages/claude-sdk-tools/src/Orchestrate/`:
+- `defineToolV2.ts` — the V2 tool contract (name, description, operation, model, run),
+  mirroring V1's `defineTool`/`ToolDefinition`.
+- `tools/` — Find, Match, Head, Tail, Range, Read, Program, each `defineToolV2`-shaped.
+- `registry.ts` — `ToolsV2Registry` / `createToolsV2Registry(deps)`, as above.
+- `runOrchestrateCall.ts` — the one function a V2 dispatch path needs to call: raw
+  `tool_use.input` in, `{ ok, content } | { ok, error }` out, matching V1's handler-result
+  shape so the consumer doesn't need a second result taxonomy.
+- Proven end-to-end (`test/Orchestrate/runOrchestrateCall.spec.ts`) against real tools:
+  parse → resolve → `execute()` runs, and `execute()`'s existing `approve(stageName, batch)`
+  hook already fires once per gated stage with no new engine work needed — confirmed via a
+  scratch POC (`.claude/poc/orchestrate-tool-v2-dispatch.ts`) before writing the real files.
+
+Three of the four touch points are now DONE, real code + tests:
+
+1. **Wire tools list** — DONE. Real merge point turned out to be `DurableConfig.toolsV2?:
+   BetaTool[]` (new field, `packages/claude-sdk/src/public/types.ts`), threaded through
+   `RequestBuilder`/`TurnRunner` alongside `serverTools`, populated by
+   `apps/claude-sdk-cli/src/setup/ToolsV2Service.ts` (`toolsV2WireTools(registry)`) and
+   consumed in `DurableConfigFactory.#build()`. NOT `IToolRegistry.wireTools` (unused by the
+   real request path) and NOT folded into V1's `AnyToolDefinition[]`/`ToolRegistry` —
+   genuinely separate arrays merged only at the wire-params level.
+2. **Dispatch** — DONE. `packages/claude-sdk/src/public/interfaces.ts` gained
+   `IOrchestrateEngine` (`owns(name)`, `run(name, input, requestApproval?)`), injected into
+   `QueryRunner` and consulted before the V1 registry in `#runTools` — a V2 name never
+   reaches `IToolRegistry.resolve`. Concrete impl: `OrchestrateEngine` in
+   `claude-sdk-tools/src/Orchestrate/OrchestrateEngine.ts`, backed by `ToolsV2Registry` +
+   `runToolV2Call` (handles both `Orchestrate` composed calls and a direct single-tool call,
+   e.g. calling `Find` on its own — both reduce to the same `execute()` call). Registered in
+   `apps/claude-sdk-cli/src/setup/container.ts`.
+4. **Approval/permissions** — DONE (settled earlier, now wired). V2 never touches V1's
+   permission matrix (`apps/claude-sdk-cli/src/permissions.ts`). `QueryRunner`'s
+   `#runOrchestrateTool` builds a `requestApproval` callback that reuses
+   `ApprovalCoordinator`'s existing keyed request/response plumbing and the
+   `tool_approval_request`/`response` wire messages — reused mechanism, not reused policy:
+   fires once per gated STAGE (`${toolUseId}:${stageIndex}`), showing that stage's own
+   resolved input, honouring only `requireToolApproval` (off → auto-approve everything).
+
+3. **Tool rendering** — NOT DONE. The TUI has no shape yet for a multi-stage V2 result
+   (`ExecuteResult`/`StageReport[]`) distinct from a V1 single result. Right now a V2 call's
+   `tool_result` is just the flattened text `runToolV2Call.summarise()` produces — functional,
+   not yet rendered richly. **Real priority (SC), needs design thought before starting** —
+   not a quick follow-on to anything already built.
+5. **Approval rendering** — DONE (this session, after the plan text above was written).
+   `QueryRunner`'s wire message now sends the gated stage's own resolved `input` (e.g.
+   `Program`'s real `program`/`args`), not just the piped batch (which was `[]` for any
+   producer stage — that was the actual bug behind "I don't see any input"). Confirmed live:
+   a real approval prompt now shows the real command about to run.
+
+**Fixed: ESC-cancel.** An `AbortSignal` now flows `QueryRunner` → `IOrchestrateEngine.run` →
+`runToolV2Call` → `execute()` → every `ToolV2.run` unconditionally (optional param, most tools
+ignore it). `execute()`'s only job is to stop advancing to further stages once the signal is
+aborted; each tool decides for itself whether/how to react (`Program` ties it into the real
+process kill it already had for its own timeout/caps). `QueryRunner` registers the same shared
+`toolController` around the V2 dispatch that V1's phase already used, so ESC routes to it as a
+tool-cancel. Proven with a full-stack integration test (real `QueryRunner`/`ApprovalCoordinator`/
+`OrchestrateEngine`/registry/`execute()`, only the OS process faked) plus unit tests in
+orchestrate-core and Program.spec.ts.
+
+## Policy — the unified V1+V2 approval ACL, built and live (separate from the four
+## touch points above, but part of this same thread)
+
+Replaces `permissions`/`tools.rules`/`tools.blockedCommands` with one ordered rule list
+(`packages/claude-sdk-tools/src/Policy/`) — ACL-shaped (tower/mvp's `bridge::permissions`
+is the model), not the old fixed inside/outside grid. `disabledTools` stays separate on
+purpose — it's a registration-time decision (does the model see this tool at all), not an
+approval-time one, so it was never in scope for this merge.
+
+Done: `matchTool`/`matchInput`/`matchValue`/`matchPath`/`resolve`/`resolveSet`, each
+concern tested in isolation plus one composed-policy integration test proving genuine
+parity with every real `Exec/ruleConfig.ts` `defaultRules` entry. `validatePolicy` (three
+cases: wrong shape → invalid; a rule scoped to a currently-loaded tool referencing a field
+it doesn't have → invalid; a rule scoped to a tool that isn't loaded yet → warning only,
+not invalid) and `PolicyStore` (never updates to an invalid policy, never has no policy at
+all — falls back to a safe ask-everything default). Wired into real V2 approval
+(`createPolicyGatedApproval`, consulted by `OrchestrateEngine` before any human-ask).
+`policy` is a real, live, top-level (not `tools.policy`) config field with its own
+independent watch/notice (`ConfigPolicyProvider`, mirroring `tools.rules`'s own
+independent-watch pattern exactly) — `⚠️ policy is invalid` / `✅ policy valid again` /
+`🛡️ policy updated`, spliced into the primary view, confirmed live (editing `policy` while
+the CLI is running takes effect immediately, no restart).
+
+V2 tool schemas now carry `isPath` markers (`Find.path`, `Paths.paths`, `Program.cwd`,
+`Delete.files`) so `collectPaths` can extract real paths for path-scoped rules
+(`$PWD`/`*`) to match against — without this, every path-scoped rule was silently
+unreachable (`paths` was always `[]`). Two real resolver bugs found and fixed live during
+testing, both now covered by tests: (1) `path: '*'` was being skipped when `paths` was
+empty, defeating the one rule meant to catch everything, since a wildcard imposes no real
+constraint and should match regardless — same principle `tool: '*'` already had right.
+(2) A rule that matched but was silent on the specific operation being asked about (no
+`operations` entry for it, no `default`) was resolving to `ask` right there instead of
+falling through to the next matching rule — meaning an earlier, narrower rule (e.g. a path
+zone that only ever talks about read/write) could silently block a later, more general
+rule from ever being consulted for an operation the earlier rule never mentioned.
+
+Built the unified V2 `Delete` tool (files and directories in one, no `kind` branch — same
+principle as `Match` losing its own) specifically to have something with a real
+`fs.delete`-tier `isPath`-marked field to test the above against.
+
+**Not a concern in itself (SC), but the real fix is porting more tools to V2, not a Policy
+change:** V1 tools do not go through Policy at all yet (confirmed live — `ReadMemory`
+bypasses it entirely). The fix isn't special-casing V1 inside Policy — it's moving more of
+the catalogue onto ToolV2, same as `Find`/`Program`/`Delete` already are.
+
+**Full gap analysis (checked against `createAppTools.ts` vs `registry.ts`), current at
+time of writing:**
+- V2 built, V1 not yet retired (both live): `Find`/`Paths`/`Match`/`Head`/`Tail`/`Range`/
+  `Read`/`ReadBinaryFile` (V1's `Pipe`/`ReadFile` already retired, no collision), `Program`
+  (V1's `Exec`/`ExecV2`/`ExecV3` still separate), `Delete` (V1's `DeleteFile`/
+  `DeleteDirectory` still separate).
+- No V2 equivalent at all: `TsDiagnostics`/`TsHover`/`TsReferences`/`TsDefinition`, the
+  Memory tools, `Skill`, the History tools, the GitHub PR tools, the AzureDevOps PR tools,
+  `AzCli`/`EscalatedAzCli`.
+
+**Urgent (SC) — DONE: the file tools + `Ref` + the text/binary split.** `EditFile`,
+`CreateFile`, `AppendFile`, `Ref`, and the `ReadFile` → `Read`/`ReadBinaryFile` split (Phase
+6, folded into this push) all built as V2 tools this thread. V1's `ReadFile` fully retired.
+`ReadBinaryFile` needed a new general mechanism, since its output (a native attachment) is
+unlike every other V2 tool's `Stream<string>`: `ToolV2Definition.excludeFromStages` (keeps a
+tool individually callable via `wireTools` while excluding it from `Orchestrate`'s own
+`stages` discriminated union — absent/false is the ordinary, composable case, no other tool
+needs the flag) and a `ToolV2Result.attachments?: () => unknown[]` channel (opaque to
+orchestrate-core, threaded through `execute()` → `runToolV2Call` → `OrchestrateEngine` into
+the existing `ToolOutcome.blocks`). `Skill` is the other tool this same exemption will apply
+to once it's ported — flagged by the SC, not yet built.
+
+## Phase 5 — Retire `Pipe`/`ExecV3` from the catalogue — PARTIALLY DONE
+
+`Paths` (the one Pipe tool Orchestrate didn't yet have — the other source alongside `Find`)
+built as `createPathsToolV2` (`fs.list` tier, same fatal-on-first-missing-path behaviour as
+V1). With that, Orchestrate now covers all seven Pipe tools (Find, Paths, Read, Match, Head,
+Tail, Range). `Pipe` retired from `createAppTools.ts` — no longer registered, so V1's
+standalone `Find`/`Paths` (which collided by name with V2's) are gone too.
+
+`ExecV3` NOT yet retired — still registered alongside V2's `Program`. That's a separate call
+(different tool, not blocked by anything above).
+
+## Explicitly out of scope for this plan
+
+- Env scrubbing at spawn time, and `context`-as-escalation-mechanism — separate
+  concerns, not sequenced here, not a prerequisite for anything above.
+- The GitHub/AzureDevOps `number`-required-argument fix — a different, unrelated task
+  (the SC ruled this out explicitly when it came up mid-design).

CLAUDE.md

@@ -42,6 +42,7 @@ Full detail: `.claude/five-banana-pillars.md`
 | `packages/mcp-typescript/` | MCP server wrapping TsDiagnostics, TsHover, TsReferences, TsDefinition, backed by a real `tsserver` |
 | `packages/mcp-internals/` | Private, never-published source-of-truth for MCP helpers (e.g. `getDataDir`), designed to be inlined by any MCP server that uses it rather than shipped as a runtime dependency. `private: true` |
 | `packages/exec-core/` | Process-spawning core: stream-based single-process spawn behind a shared interface |
+| `packages/orchestrate-core/` | Tool orchestration runtime: plan/execute a pipeline of stages joined by `\|`, `&&`, `;`, with streaming, Xargs fan-out and named captures |
 | `packages/keychain-native/` | Minimal N-API binding to macOS Keychain generic-password reads, for holding credentials the CLI's own exec surface never sees |
 | `platforms/claude-sdk-cli-darwin-arm64/` | Published prebuilt SEA binary (macOS arm64) for the CLI, selected via the CLI's optional dependency. Bumped in lockstep whenever `claude-sdk-cli` is released. |
 
@@ -155,7 +156,7 @@ All releases are pre-releases until 1.0.0. The current version series is `1.0.0-
 
 5. Single PR with all version bumps, changelog updates, and lock file changes. Body is one line: "Bumps N packages to <version>." No feature bullets, no changelog recap — the diff is only version lines, so the body says only that.
 
-6. After merge, create a GitHub release for each bumped **buildable** package (`claude-core`, `claude-sdk`, `claude-sdk-tools`, `mcp-exec`, `mcp-history`, `mcp-memory`, `mcp-typescript`, `claude-sdk-cli`):
+6. After merge, create a GitHub release for each bumped **buildable** package (`claude-core`, `claude-sdk`, `claude-sdk-tools`, `orchestrate-core`, `mcp-exec`, `mcp-history`, `mcp-memory`, `mcp-typescript`, `claude-sdk-cli`):
    ```bash
    gh release create "<package>@<version>" --title "<package>@<version>" --target <main-sha> --notes "<unreleased section>" --prerelease
    ```