test: add a repository-wide CLI consistency gate

[philcunliffe]

What this adds

test/core/cli-consistency-gate.test.js: a structural sweep over the whole
registered CLI surface, picked up automatically by npm test (the runner
walks test/**/*.test.js).

The focused CLI suites each prove one mechanism against one hand-written
example. Nothing enforced agreement across every registration at once, which
is how the audit found drift while 57 focused tests were green. This gate
asserts, for every visible core command together:

Registration inventory (registry only, no dispatch)

• one-line summary, one-line usage, a run();
• usage invokes the command it documents (hyp <name> ...), so a rename
  cannot leave an unrunnable usage line behind;
• balanced <required> / [optional] groups;
• every option named in a usage line is spelled the way the parsers accept
  (catches --dry_run, --dryRun, -dry);
• every subcommand hangs off a registered bare command or a registered group,
  so no leaf lands under a namespace nothing describes;
• every alias resolves to its owner and shadows nothing;
• every core verb projects the usage usageForVerb generates from its schema
  (LLP 0034: one declaration, no CLI/MCP drift).

Help surface (through real dispatch)

• --help never reaches a command body (proved with a fixture whose run
  throws), pinning LLP 0009's central interception;
• hyp --help lists exactly the visible top-level tokens, sorted, with
  subcommands one level down and hidden commands absent but still
  dispatchable;
• -h renders the same page as --help;
• rendering help writes nothing under HYP_HOME;
• every visible group renders its registry-backed subcommand table including
  its own usage line and every visible child;
• every visible leaf renders hyp <name> - <summary> plus usage: <usage>
  with an empty stderr;
• every help-only group rejects an unknown subcommand with exit 2 and names
  the real ones; an unknown top-level command exits 2 and points at help;
• a public alias dispatches to its owner, help included.

Active-plugin fixture

Group, leaf, and verb-projected help all render through the same dispatch; a
namespace with no bare command earns one synthesized top-level row; an
internal (hidden) command stays out of both. A manifestRuntimeDrift helper
compares manifest-declared commands against runtime registrations and is
pinned on all three findings it must produce (manifest-only, runtime-only
public, summary drift), so the comparison is not a tautology.

Safety-critical prose

Deliberately few exact assertions, on the two destructive commands only:
hyp purge and hyp report delete keep their verbatim warnings.

Isolation

Every dispatch runs with a fresh HYP_HOME, an empty workspaceDir (so no
bundled manifest is discovered), and an injected kernel, so nothing boots, no
listener binds, and no real user state is read or written. Only --help and
unknown-subcommand argv are dispatched, so no command body runs and no
service manager is reachable (LLP 0181). The sweeps carry an explicit
timeout, so a regression in the help interception fails the gate instead of
hanging on a command body that waits for input.

Evidence it catches drift

Each mutation was applied to the tree, the gate run, and the tree restored:

Injected inconsistency	Gate result
rename query maintain to query compact, leaving usage stale	every usage line invokes the command it documents fails
--dry-run misspelled --dry_run in a usage line	every option named in a usage line is spelled as an option the parsers accept fails
add collect list with no collect bare command or group	every subcommand hangs off a registered bare command or a registered group fails
drop "Prompts on a TTY" from hyp purge help	destructive commands keep their exact warnings fails
renderGroupHelp stops printing its usage line; listGroupChildren stops filtering hidden	2 sweeps fail
renderHelp stops filtering hidden commands	3 tests fail

Clean tree: 22/22 pass. Full suite: HYP_HOME=$(mktemp -d) npm test is
4279 pass / 0 fail / 1 skipped. npm run typecheck is clean.

Acceptance criteria deliberately not implemented here

Four of the issue's criteria cannot be satisfied on master today because
the drift they exist to catch is still present and is the subject of the
sibling issues in this batch. Landing them now would put a red gate on
master, and weakening them to pass would defeat the point, so they are
left to the issues that own the corresponding fix:

• "Verify every visible core command rejects a representative unknown
  option with exit 2." Unknown-option handling is currently ad hoc: hyp backfill zzz exits 1, version / daemon stop / daemon restart ignore
  arguments entirely. That is Standardize core command argument validation and usage generation #836, whose own acceptance criteria already
  call for exactly this parameterized test once the parsers are unified.
• "Compare declared usage/options with parser-accepted positionals and
  flags." The parseCommandArgv schemas are inline in the command bodies
  and not exported, so there is nothing repo-wide to compare against yet.
  Standardize core command argument validation and usage generation #836 covers generating usage from the schema or adding the agreement
  check. The verb-projected half of this, where the schema is reachable,
  is included above.
• "Validate literal repair commands against the command registry where
  possible." src/core/commands/clients.js and the ai-gateway session
  command still print (hyp start), which the registry does not answer to.
  That is CLI repair paths recommend nonexistent hyp start #834.
• "Add an active-plugin fixture checking manifest ... help" over the real
  bundled plugins, and "Cover selected-but-activation-failed plugin
  behavior once Enforce plugin manifest and runtime command agreement #837 settles it." @hypaware/context-graph-enrich has live
  manifest/runtime summary drift, and Enforce plugin manifest and runtime command agreement #837 explicitly owns the generic
  bundled-plugin contract test. The fixture-level comparison here proves the
  mechanism without asserting over the drifted tree.

The inactive-plugin fixture criterion is already met by the existing
test/core/dispatch-inactive-plugin.test.js, so it is not duplicated.

Once #834, #836, and #837 land, extending this file with those assertions is
additive: the inventory helpers and the isolated harness are already here.

No LLP was minted. The gate tests invariants LLP 0009, 0034, and 0181 already
settled; it does not decide anything new.

Fixes #841

[philcunliffe]

Neutral review of #848 at 611ab67e - findings, fixed on the branch

Verdict: approve after the follow-up commit 9a656231. The gate is real, fast (0.25s standalone, full suite 16.0s), deterministic, and order-independent. Six findings, all low/medium, all in the new file, all fixed here. Nothing remains open.

(a) Non-vacuity: independently verified, all six

I applied each mutation to my own worktree, ran only the gate, and reverted. Every one was caught, by the named test:

Mutation	Result
core_commands.js:144 query maintain -> query compact, usage left stale	not ok 2 - every usage line invokes the command it documents
core_commands.js:146 --dry-run -> --dry_run	not ok 4 - every option named in a usage line is spelled as an option the parsers accept
collect list leaf inserted with no collect command or group	not ok 5 - every subcommand hangs off a registered bare command or a registered group
core_commands.js:458 "Prompts on a TTY" deleted from purge help	not ok 22 - destructive commands keep their exact warnings
group_help.js:94 renderGroupHelp stops printing its usage line	not ok 14 - every visible group renders its registry-backed subcommand table
group_help.js:40 listGroupChildren stops filtering hidden	not ok 19 - an active plugin renders group, leaf, and verb-projected help ...
dispatch.js:749 renderHelp stops filtering hidden	not ok 9, not ok 12, not ok 19 (3 failures)

The evidence table in the body is accurate. One wording nit: the listGroupChildren half of row 5 is caught by the fixture test, not by a sweep, so "2 sweeps fail" is really "1 sweep + 1 fixture".

I then probed the isolation claim directly. Ambient HYP_DEV_TELEMETRY=1, DEV_RUN_ID, and a HYP_CONFIG pointing at a plugin-enabling config each left the gate 22/22, so it is not env-sensitive in practice. Not vacuous.

Findings (all fixed in 9a656231)

1. medium - test/core/cli-consistency-gate.test.js:98: the injected kernel was bound to the developer's real ~/.hyp, not to the temp HYP_HOME.
createKernelRuntime({ commandRegistry }) passes no cacheRoot, so defaultCacheRoot() (src/core/runtime/activation.js:106) falls back to process.env.HYP_HOME || ~/.hyp - and the harness only ever put HYP_HOME into the env object handed to dispatch, never into process.env. Confirmed empirically: the kernel resolved to /home/<user>/.hyp/hypaware/cache, and the LLP 0070 usage-policy resolver to the sibling real state dir. This contradicts the header claim on line 14 ("no real user state is read or written"), and it makes the readdir(hypHome) guard on line 300 watch a directory nothing was ever pointed at. Harmless today because only --help argv is dispatched, but the guard cannot fail. Verified by mutation: making defaultCacheRoot() throw produced 13 failures at 611ab67e and 0 after the fix. Fixed by passing cacheRoot: path.join(hypHome, 'hypaware', 'cache').

2. low - :109: HYP_CONFIG was not neutralized.
resolveConfigPath (src/core/runtime/boot.js:466) honours env.HYP_CONFIG ahead of hypHome, and the harness spread process.env while overriding only HYP_HOME. The sibling help tests do this correctly - test/core/command-dispatch.test.js:183,225,240,348,390,1065 all pass HYP_CONFIG: '' alongside HYP_HOME. Latent only because workspaceDir is empty so no manifests exist to select against. Fixed by matching the sibling convention.

3. low - :293: the @ref overclaimed what the test holds.
@ref LLP 0009#top-level-help-lists-plugin-commands-without-booting [tests]: help touches no state reads as pinning the pre-boot property, but harness() always injects opts.kernel and dispatch skips bootKernel outright when it does. Verified: with bootKernel mutated to throw unconditionally, the gate is 22/22 green while test/core/command-dispatch.test.js takes 4 failures. So the pre-boot property is pinned there, not here. Per CLAUDE.md ("keep refs honest"; a ref must tell you something the code does not), the gloss now states what this test does hold - that no registration's help page seeds the install - and the comment points at the suite that owns the boot property.

4. low - :287: "stay dispatchable" was asserted against the registry, not through dispatch.
The test called registry.match(...) directly, a registry unit check that cannot see a guard added inside dispatch. Verified: injecting a visibility guard at src/core/cli/dispatch.js:325 (matched.command.hidden ? undefined : matched) left the gate 22/22 green at 611ab67e and now fails not ok 12. This mattered - hyp smoke <flow> is a hidden command, so the regression would have shipped. Fixed by dispatching each hidden command's --help through the harness.

5. low - :161: the option-spelling check rejected a spelling the parsers accept.
The split class [\s[\]<>|]+ omits =, and LONG_OPTION forbids it, so a usage line documenting [--format=<fmt>] would fail the gate as an invalid spelling even though src/core/cli/verb_codec.js:63,154 parses --flag=value. Latent (no current usage line uses the form). Fixed by splitting on = so the check applies to the option name. Re-verified the --dry_run mutation is still caught after the relaxation.

6. low - :15 and :305/:323: two accuracy/robustness nits.
(i) The header said "no command body executes", but hyp <group> zzz-not-a-subcommand does run makeGroupCommand's run - deliberately, and its body is only a registry read plus an error, so the behaviour is right and the prose was wrong. Reworded. (ii) The two --help sweeps skip on children.length === 0, so both go silently empty if listGroupChildren ever regresses to returning nothing - precisely the failure they exist to catch. Added assert.ok(checked > 0, ...) to both, matching the guard the help-only-group sweep already had at :351.

(b) Scope: Fixes #841 is correct as written - no change made

I checked each deferred criterion against the sibling issues' own acceptance criteria, and every one has a verbatim home on an open issue, so auto-closing #841 loses nothing:

Deferred #841 criterion	Tracked on
"every visible core command rejects a representative unknown option with exit 2"	#836: "Add a parameterized test proving every visible core command rejects a representative unknown flag."
"Compare declared usage/options with parser-accepted positionals and flags"	#836: "Generate registered usage from the schema ... or add an automated usage/schema agreement check."
"Validate literal repair commands against the command registry"	#834: "Add or extend a test that executes/resolves the recommended command spelling against the core command registry."
active-plugin fixture over the real bundled plugins	#837: "Add a generic bundled-plugin contract test that compares every manifest-declared command name and summary with runtime registration ... Cover verb-projected commands and plugin group metadata."
selected-but-activation-failed behavior	#837: "Make dispatch misses distinguish at least inactive, selected-but-unavailable/activation-failed, and genuinely unknown commands."

The last row is the thinnest link - #837 owns the behavior, and the gate-side coverage is implied rather than spelled out - but not enough to justify leaving #841 open indefinitely behind three in-flight PRs (#844, #849, #850). I did not edit the PR body and did not open a follow-up issue: a new issue here would duplicate criteria that already exist on #834/#836/#837, and duplicated backlog is its own defect. The body's own "Acceptance criteria deliberately not implemented here" section is an accurate record for whoever reads the closed issue later.

Other checks

• No LLP needed - agreed. The gate asserts invariants LLP 0009/0034/0181 already settled and decides nothing new; the three file-level @refs are [tests]/[constrained-by] and now honest.
• Tier placement correct per CLAUDE.md: deterministic registry/render logic, no daemon, no network, no service manager, so a traditional test rather than a smoke.
• Not slow, flaky, or order-dependent. Standalone 0.25s; full suite unchanged at ~16.0s; every dispatch gets its own mkdtemp HYP_HOME, and each of the 22 tests passes in isolation. The three long sweeps carry SWEEP_TIMEOUT_MS; I added it to the hidden-command test too, since it now dispatches in a loop.
• Style clean: no semicolons, no em dashes, JSDoc types only.
• npm test 4279 pass / 0 fail / 1 skipped, npm run typecheck clean, both before and after the fix commit.

Nit, not fixed (non-blocking)

harness() creates two mkdtemp directories per call and removes neither, leaking ~24 empty temp dirs per run. Consistent with the rest of the suite, so left alone.

Post-push CI note: the one red check is not this PR's

Pushing 9a656231 turned the pull_request test job red on both Node 22 and 24, with exactly one failure: not ok 1960 - no tool transcript is tracked in the repo (test/core/repo-scratch-hygiene.test.js:80). It is not caused by this PR. The push job at the same SHA, which builds the branch alone, is green; only the merge-with-master job fails, because origin/master tracks x/npm-test.log and x/typecheck.log. I confirmed by checking out ec3361bb on its own: master fails that test with no PR merged in at all.

The timeline explains why 611ab67e looked green: those files landed on master in adb448ab (#785) at 19:24 UTC and the lint that forbids them landed one commit later in bf9e4773 (#796); this PR's earlier pull_request run was at 19:05 UTC, before either.

Already tracked - issue #852, with PR #853 ("Delete the two committed tool transcripts that hold master red") open to fix it. Nothing to do on #848; it just cannot show green until #853 lands.

I also merged current origin/master into the branch locally and ran the full suite: 4504 pass / 1 fail, the single failure being that same master-owned test, and the gate itself 22/22. So this branch is compatible with master as it stands today. The merge was local only and was not pushed; origin/fix/issue-841 is exactly 9a656231.

Summary

Fixed and pushed as 9a656231: findings 1-6, all in test/core/cli-consistency-gate.test.js. Open: nothing. Findings 1, 3, and 4 were each confirmed by a mutation that the gate at 611ab67e could not see and now can.

[philcunliffe]

Review: PR #848 (fix/issue-841) at e79f0098

Verdict: findings (2 actionable, both fixed and pushed as 5dddbf47; 2 more raised and deliberately left open).

The PR adds one file and no production code: test/core/cli-consistency-gate.test.js. I read dispatch.js, group_help.js, registry/commands.js, verb_command.js, core_verbs.js, observability/env.js, observability/index.js, and scripts/run-tests.js to check that each assertion pins what its comment claims. The assertions themselves are sound: no false-positive sweep, no tautology that matters, and the isolation reasoning about cacheRoot and HYP_CONFIG is correct as far as it goes. The two findings are about the harness, not the assertions.

Fixed

1. medium - test/core/cli-consistency-gate.test.js:118 (reviewed head): the harness is not inert, and rendering help writes nothing under HYP_HOME is falsely green when it is not.

The harness injects env into dispatch, but dispatch calls installObservability() with no argument (src/core/cli/dispatch.js:209), and that reads readObservabilityEnv() off the real process.env (src/core/observability/env.js:18-23), never the injected object. So with HYP_DEV_TELEMETRY=1 exported (the value CLAUDE.md's hermetic smokes set, and one a developer plausibly has in their shell) the JSONL exporters in tracer.js/logger.js/meter.js write under the real HYP_HOME for every one of the ~60 help renders, while fs.readdir(hypHome) on the temp dir at line 322 still returns []. OTEL_EXPORTER_OTLP_ENDPOINT escapes the same way.

Reproduced on the reviewed head:

$ PROBE=$(mktemp -d)
$ HYP_HOME=$PROBE HYP_DEV_TELEMETRY=1 node --test test/core/cli-consistency-gate.test.js
# pass 22
# fail 0
$ find $PROBE -type f
/tmp/tmp.xnq3b0lBTR/hypaware/dev-telemetry/metrics-1066980.jsonl
/tmp/tmp.xnq3b0lBTR/hypaware/dev-telemetry/traces-1066980.jsonl
/tmp/tmp.xnq3b0lBTR/hypaware/dev-telemetry/logs-1066980.jsonl

That contradicts the file's own docstring ("no real user state is read or written") and the name of the test at line 316. The mechanism is already documented in the repo: test/core/daemon-maintenance-tick-status.test.js:122-126 calls out that installObservability() reads real process.env and arms the exporter on purpose there. The header comment at lines 99-101 reasons about exactly this hazard for cacheRoot and at 116-117 for HYP_CONFIG, but the observability path was missed.

Fixed by clearing HYP_DEV_TELEMETRY and OTEL_EXPORTER_OTLP_ENDPOINT on process.env at module scope (the only thing readObservabilityEnv() actually reads) and blanking them in the injected env too. node --test runs each file in its own process, so the mutation cannot reach a sibling suite. Same command now leaves the probe directory empty, still 22/22.

2. low - test/core/cli-consistency-gate.test.js:97-98 (reviewed head): harness() leaks two temp directories per call.

Two mkdtemp calls, no removal, and harness() runs once per test. Measured on the reviewed head: 26 directories left behind in os.tmpdir() after a single run of the file. Sibling suites (ai-gateway-dataset.test.js, action-attach.test.js, attach-all-notes.test.js) pair their mkdtemp with fs.rm. Fixed with a file-level after hook that removes every directory the harness minted; the count is now 0.

Raised, left open

3. low/medium - manifestRuntimeDrift (line 414) never runs against a real plugin. The helper is documented as "the shape a plugin author owes" and is the file's only manifest-vs-runtime check, but its only two call sites (lines 520, 529) feed it the synthetic FIXTURE_MANIFEST_COMMANDS/activateFixturePlugin pair. No manifest under hypaware-core/plugins-workspace/ is compared with its own contributes.commands, so the drift class it exists to catch still ships unnoticed for every real plugin. Left open deliberately: the PR body already states this, names the live drift on @hypaware/context-graph-enrich, and defers to #837, which owns the generic bundled-plugin contract test. Landing it now would put a red gate on master. The three-direction pinning at line 529 does make the helper non-tautological, which is the right thing to have proved here.

4. low - two latent brittleness points in the hidden-command sweep (line 297). command.name is interpolated into a RegExp unescaped; harmless today because smoke is the only hidden command and contains no metacharacters. More interestingly, if a hidden bare command ever gained a visible child, renderHelp deliberately synthesizes a top-level Subcommands: ... row for it (dispatch.js:767-769, and this very file asserts that behavior for the fixture at line 516), and the assertion would fail with "leaked into help" against correct rendering. Left open: speculative, no current registration triggers it, and keying the assertion on the bare command's own summary instead is a design choice for whoever adds such a command.

Two things I checked and am not reporting as findings:

• The option-spelling sweep (line 172) splits on [\s[\]<>|=]+, so a usage line wrapping options in parentheses ((--a|--b)) would yield --b) and fail spuriously. No core usage line contains parentheses today, so this is hypothetical.
• every core verb projects the usage its schema generates (line 214) is close to tautological, since verbToCommand sets usage: usageForVerb(verb.name, verb.inputSchema) from the same two inputs (src/core/cli/verb_command.js:27). It still earns its place as a guard against a hand-written usage override, just not for the reason the comment gives.

Also noted: hyp --help lists exactly the visible top-level tokens (line 246) builds expected from registry.list() only. The registry exposes no listGroups, so a future core namespace registered via registerGroup with no bare command would fail this test even though renderHelp synthesizing a row for it is correct. Core registers no such group today, and there is no accessor to fix it with, so this is a note rather than a finding.

Checks

Run in a clean worktree off origin/fix/issue-841 with the fixes applied:

• HYP_HOME=$(mktemp -d) npm test: 4505 pass / 0 fail / 1 skipped
• npm run typecheck: clean (exit 0)
• node --test test/core/cli-consistency-gate.test.js: 22/22

Nothing was already failing on the unmodified head. The fix commit touches only the test file, adding 33 lines and removing 2; no assertion was weakened or removed.

[philcunliffe]

Triage at head 5dddbf47434838bf0336e7d3c4f605a6fa06a4e2: both findings left open by the final review round (findings 3 and 4 of the e79f0098 review) are classified non-blocking. The PR adds one test file and no production code, so neither the fixture-only manifestRuntimeDrift coverage (whose real-plugin extension is owned by #837) nor the latent brittleness in the hidden-command sweep can cause a production defect. Deferred to follow-up issue #889. The PR can merge.