diff --git a/llp/0217-compaction-effectiveness-verdict.decision.md b/llp/0217-compaction-effectiveness-verdict.decision.md index a1209e9c..b1b50a22 100644 --- a/llp/0217-compaction-effectiveness-verdict.decision.md +++ b/llp/0217-compaction-effectiveness-verdict.decision.md @@ -6,7 +6,7 @@ **Author:** Kenny / Claude **Date:** 2026-08-13 **Related:** LLP 0199 (the baseline gate this extends), LLP 0209 (the writer change that made a stale verdict reachable), LLP 0027 (the re-settle baseline the gate was built from) -**Extended-by:** [LLP 0218](./0218-compaction-failed-attempt-reported.decision.md) (the stamp a failed retry writes carries no effectiveness claim, so the skip report below cannot speak for it: the stamp records when the attempt failed and the skip is reported on that instead) +**Extended-by:** [LLP 0218](./0218-compaction-failed-attempt-reported.decision.md) (the stamp a failed retry writes carries no effectiveness claim, so the skip report below cannot speak for it: the stamp records when the attempt failed and the skip is reported on that instead); [LLP 0228](./0228-maintenance-skips-are-a-standing-surface.decision.md) (the verdict's skip report stops being visible only to whoever runs `hyp query maintain` or queries the trace: the daemon tick summarizes it into `status.json` and `hyp status` renders it) > The partition cursor records the data-file count a rewrite started from > beside the count it produced, so "the live count sits on its baseline" can be diff --git a/llp/0218-compaction-failed-attempt-reported.decision.md b/llp/0218-compaction-failed-attempt-reported.decision.md index dfc67e02..b4a88f43 100644 --- a/llp/0218-compaction-failed-attempt-reported.decision.md +++ b/llp/0218-compaction-failed-attempt-reported.decision.md @@ -6,7 +6,7 @@ **Author:** Kenny / Claude **Date:** 2026-08-13 **Related:** LLP 0217 (the effectiveness verdict and the retry stamp this extends), LLP 0199 (the baseline gate both sit on), LLP 0207 (the recognition path that writes the same stamp) -**Extended-by:** [LLP 0220](./0220-maintenance-walk-survives-a-partition.decision.md) (the tick in which the attempt fails is reported too: `failed` is this tick's error, beside the `compactionAttemptFailed` skip an earlier one recorded) +**Extended-by:** [LLP 0220](./0220-maintenance-walk-survives-a-partition.decision.md) (the tick in which the attempt fails is reported too: `failed` is this tick's error, beside the `compactionAttemptFailed` skip an earlier one recorded); [LLP 0228](./0228-maintenance-skips-are-a-standing-surface.decision.md) (the standing record this document describes stops being a `hyp query maintain` line and a span attribute: the daemon tick summarizes both skip reasons into `status.json`, where `hyp status` finds them) > The stamp a failed compaction retry writes records the moment the attempt > failed, so the ticks that skip the partition afterwards can say why. A diff --git a/llp/0228-maintenance-skips-are-a-standing-surface.decision.md b/llp/0228-maintenance-skips-are-a-standing-surface.decision.md new file mode 100644 index 00000000..575efd5b --- /dev/null +++ b/llp/0228-maintenance-skips-are-a-standing-surface.decision.md @@ -0,0 +1,172 @@ +# LLP 0228: A partition maintenance leaves fragmented is named on a standing daemon surface + +**Type:** Decision +**Status:** Accepted +**Systems:** Cache, Daemon, CLI +**Author:** Kenny / Claude +**Date:** 2026-08-13 +**Related:** LLP 0217 (the effectiveness verdict whose skip reason this carries), LLP 0218 (the failed-attempt skip reason this carries), LLP 0199 (the baseline gate both sit on, and the walk order this borrows), LLP 0164 (the precedent: an operator-facing fact the daemon alone knows travels through `status.json`), LLP 0021 (the trace surface this stops being the only one) + +> The daemon's maintenance tick stops discarding the report `maintainCache` +> hands back. Every completed tick overwrites one bounded snapshot in +> `status.json` naming the partitions it deliberately left fragmented and why; +> `hyp status` renders that snapshot and raises one warning diagnostic from it, +> and the tick logs one line whenever the count is nonzero. Retention is +> exactly the last tick, because a skip reason is a standing state the tick +> re-derives from the cursor, not an event that happened once. + +## Context {#context} + +LLP 0217 and LLP 0218 made a partition maintenance deliberately leaves +fragmented a *stated* outcome rather than an absence in a "0 partitions +compacted" summary: `MaintenancePartitionReport` carries +`compactionIneffective` and `compactionAttemptFailed`, `hyp query maintain` +prints both, and the `maintenance.partition` span carries +`compaction_ineffective` / `compaction_attempt_failed`. + +Both statements reach an operator only if the operator goes looking. The +daemon runs the same walk hourly and `src/core/daemon/runtime.js` awaited +`maintainCache` and dropped its return value on the floor, so on an ordinary +install the only daemon-side evidence of a frozen partition was a span +attribute. A span attribute is a real observability channel (the repo's +Log-Driven Development rule says so), and it is the wrong one for this fact: +it is only there if tracing was on at the moment the tick ran, and finding it +means knowing to query the `traces` dataset for an attribute whose name you +already know. The state it describes, meanwhile, is not a moment. It persists +until something rewrites the cursor. + +Reported as #742, deferred out of PR #741 on the grounds that widening the +surface LLP 0217/0218 settled is a new request rather than a defect in them. + +## Decision {#decision} + +**The daemon's status file carries it, +and `hyp status` reads it.** The maintenance tick summarizes its report into +`DaemonStatus.maintenance` and persists it with the snapshot the daemon +already writes on every sink tick. `hyp status` lifts it from `status.json`, +renders a `maintenance:` block, carries it under `--json`, and raises one +`maintenance_partitions_skipped` warning diagnostic naming the count and the +repair (`hyp query maintain --dry-run` to enumerate, `--force` to retry). + +This is LLP 0164's route, for LLP 0164's reason. The daemon is the only +process that runs the hourly walk, `hyp status` activates no plugins and reads +no cache, and a CLI that re-ran `maintainCache` to answer the question would +be a *second* walk (loading metadata, stat-ing data files) fired by a status +command. The file is the only place the answer can come from without doing the +work twice. + +The alternatives were weighed and are not enough on their own. A `fileLog` +line alone (#742's option 1) is a record of a tick, not a description of a +state: it scrolls away, and it is the shape of surface that already existed +for the failing tick in LLP 0218's context section. A new field on the +maintenance report alone changes nothing, because the caller that discards +the report is the whole defect. So the log line is kept as the record (the +tick logs `daemon.maintenance_skipped` when the count is nonzero) and the +status file is the discovery, which is #742's option 3. + +Nothing about *when* a partition is compacted moves. This decision reads the +report the walk already produces, and it must never be a reason to stat a +data file: proving a skipped partition is also still fragmented is exactly the +per-tick cost the LLP 0199 gate exists to avoid, and LLP 0217 and LLP 0218 +both already declined to pay it for their own reports. + +**Retention is the last completed tick, and only +that.** Every completed tick overwrites the snapshot whole, including a tick +that skipped nothing (which writes zeros). + +A skip reason is per-partition and per-tick, so a standing surface needs a +retention rule, and the reason itself decides which one. Both reasons are read +off the partition cursor on every tick: they describe a state that is still +true, not an event that occurred. So the newest tick is the only tick whose +answer is current, and a partition that thaws (a `--force` rewrite, new data +flushing in, the next writer generation) drops off the surface on the tick +after it thaws, with no expiry rule and nothing to invalidate. That is the same +self-clearing property LLP 0218 built into the report itself. A last-N history +would go stale against the cursor, would need an eviction rule of its own, and +would duplicate the `fileLog`, which is already the append-only record with +timestamps. + +The snapshot is bounded three ways, because `status.json` is read back and +printed to a terminal. `reasons` is a fixed key set, one integer per reason. +`partitions` is capped at `MAX_SKIPPED_PARTITIONS_REPORTED` entries, taken in +walk order, which is LLP 0199#neediest-first: descending live data-file count, +so the named partitions are the worst ones and the cap costs no sort of its +own. `skippedTotal` is the true count, so the cap is never a lie, and the +render says how many were not named and which command lists them all. The cap +and the sanitizing are re-applied on read as well as on write, for the reason +LLP 0164 states: core reads a *file*, and must not assume the daemon that +wrote it was this build. + +Each named partition carries its dataset, its partition tuple, its reason, and +the one number or timestamp that reason is about (the recorded rewrite's +data-file count, or when the spent attempt failed). Those are dataset and +partition identifiers and kernel-side counters. Nothing from a row, a prompt, +a credential, or a config value is anywhere near this path. + +**The reason ids are the span +attribute names, verbatim.** `compaction_ineffective` (LLP +0217#record-effectiveness) and `compaction_attempt_failed` (LLP +0218#report-the-spent-attempt) are the ids on every operator-facing surface: +the status file, `hyp status` text, `hyp status --json`, and the daemon log +line. They are already the `maintenance.partition` span's attribute names, and +those are in turn named after the `MaintenancePartitionReport` fields, so an +operator moving between a trace query, `hyp status`, and the daemon log reads +one spelling. Minting a third name for the same fact is exactly the parallel +vocabulary this repo's skip reporting has so far avoided. + +Two things a maintenance tick also does are deliberately *not* reasons here. A +partition sitting on its baseline with a rewrite recorded as effective is +converged (LLP 0199#baseline-gate): that is the healthy majority of every +cache, it is why the gate exists, and naming it would put most partitions on +the surface and none of the interesting ones. A rebaseline (LLP 0207) is work +the tick performed, not a partition it left alone. The surface names only the +partitions the kernel knows are still fragmented and has stopped rewriting. + +A partition whose rewrite throws on every tick is knowingly deferred, not +named here. On the ordinary growth path a throwing rewrite writes no LLP +0217 retry stamp, so the tick re-attempts and re-throws forever without +`compactionAttemptFailed` ever being set, and this surface says nothing +about it. That is the correct behavior for the two reasons this document +settled (an operator sees nothing rather than a false "0 skipped"), not a +defect in either; a third id for a tick's own error is the additive +extension this decision's Consequences section already anticipates, left +for whoever picks it up next. + +## Consequences {#consequences} + +- `hyp status` on an install with no frozen partition is unchanged: the text + block renders only when the count is nonzero, the way every other + conditional section in the status render does. `--json` always carries the + key (null before any tick has run), per that surface's "missing values + surface as null rather than being omitted" contract. +- The diagnostic is a warning and never flips `overall` to `degraded`. A + frozen partition is a thing to know about, not an outage: the daemon is + running, capture is working, and queries answer. This puts it beside + `recent_errors` and a failed client action rather than beside a missing + config. +- The snapshot can only say what the last tick saw. A tick cut short by + `max_tick_ms` (LLP 0199#neediest-first) reports the partitions it reached and + no others, so `partitionsVisited` is on the snapshot beside the counts: "3 + skipped of 12 visited" is honest in a way "3 skipped" is not when the cache + holds 400 partitions. +- A daemon that has never run a maintenance tick, and an install whose + maintenance is disabled, both leave the field absent. Absent means "no tick + has reported", which is not "nothing is frozen", and the render says nothing + rather than claiming a clean cache. +- `status.json` grows by a bounded constant. It is rewritten in full on every + persist, so there is no accumulation across ticks and no file growth to + bound over a daemon's lifetime. +- The surface is additive to the report, so a later per-partition outcome + worth standing (this tick's own error, say, which is a different fact from + either reason here) is a new id and a new count key, not a new shape. + +## Extends {#extends} + +LLP 0217 settled that a partition skipped because its rewrite achieved nothing +is skipped explicitly, and LLP 0218 settled the same for a partition whose one +retry was spent by an attempt that threw. Both stand exactly as written: this +document changes neither the verdicts, nor what is recorded on the cursor, nor +when a partition is retried. What it adds is that the daemon, which runs the +walk that reaches those verdicts, stops throwing away the report that states +them, so a frozen partition is discoverable by an operator who never runs +`hyp query maintain` and never queries the `traces` dataset. diff --git a/src/core/commands/status.js b/src/core/commands/status.js index e31fd647..3979759a 100644 --- a/src/core/commands/status.js +++ b/src/core/commands/status.js @@ -1,7 +1,7 @@ // @ts-check import { Attr, withSpan } from '../observability/index.js' -import { collectHypAwareStatus } from '../daemon/status.js' +import { collectHypAwareStatus, describeMaintenanceSkipReasons } from '../daemon/status.js' import { sanitizeLabel } from '../util/json_util.js' import { ENV_VAR_NAME } from '../daemon/launchd_env.js' import { formatFirstSyncDeadline } from '../usage-policy/first_sync_hold.js' @@ -215,6 +215,24 @@ export function renderStatusJson({ report, clientNames, datasets, cacheRoot }) { last_seen: e.lastSeen, rows: e.rows, })), + // What the daemon's last maintenance tick deliberately left fragmented + // (LLP 0228). Null until a daemon has reported a tick: absent is "no tick + // has said", which is not "nothing is frozen". + maintenance: report.maintenance + ? { + tick_at: report.maintenance.tickAt, + partitions_visited: report.maintenance.partitionsVisited, + skipped_total: report.maintenance.skippedTotal, + reasons: report.maintenance.reasons, + skipped: report.maintenance.partitions.map((p) => ({ + dataset: p.dataset, + partition: p.partition, + reason: p.reason, + ...(p.dataFiles !== undefined ? { data_files: p.dataFiles } : {}), + ...(p.failedAt ? { failed_at: p.failedAt } : {}), + })), + } + : null, // Capture health per otel-attached client (LLP 0257 S17). Always an // array so a consumer can pin the key; empty means no configured client // is otel-attached, which keeps the pre-otel payload shape unchanged. @@ -589,6 +607,36 @@ export function renderStatusText({ report, clientNames, datasets, cacheRoot, std ) } + // Partitions the daemon's last maintenance tick deliberately left + // fragmented (LLP 0228). Rendered only when there are some, like every + // other conditional block here, so an ordinary install's text surface is + // unchanged. `formatEntrypointAge` is the file's coarse-age formatter (it + // is named for its first caller): the question is "is this tick's answer + // hours or weeks old?", not the exact instant. + // @ref LLP 0228#status-file-is-the-surface [implements]: hyp status is where an operator who never runs `hyp query maintain` finds a frozen partition + if (report.maintenance && report.maintenance.skippedTotal > 0) { + const m = report.maintenance + const breakdown = describeMaintenanceSkipReasons(m.reasons) + stdout.write(' maintenance:\n') + stdout.write( + ` ${m.skippedTotal} of ${m.partitionsVisited} partitions left fragmented, as of the tick ${formatEntrypointAge(m.tickAt)} (${breakdown})\n` + ) + for (const p of m.partitions) { + // The count is the one the recorded rewrite ran over, not the live one: + // the sentence is about that rewrite, and `hyp query maintain` draws the + // same distinction. + const detail = + p.reason === 'compaction_attempt_failed' + ? (p.failedAt ? ` the retry failed at ${p.failedAt}` : '') + : (p.dataFiles !== undefined ? ` the last rewrite of ${p.dataFiles} files reduced nothing` : '') + stdout.write(` - ${p.dataset}/${p.partition} [${p.reason}]${detail}\n`) + } + const unnamed = m.skippedTotal - m.partitions.length + if (unnamed > 0) { + stdout.write(` ... and ${unnamed} more (hyp query maintain --dry-run lists them all)\n`) + } + } + // Local entries the central layer overrides (LLP 0031): dropped at // merge, listed here with their reason. Loud, but not an outage signal. // The gateway runs fine on the central config. diff --git a/src/core/daemon/runtime.js b/src/core/daemon/runtime.js index b53eb3f9..17e57a29 100644 --- a/src/core/daemon/runtime.js +++ b/src/core/daemon/runtime.js @@ -31,7 +31,7 @@ import { writePidFile, } from './pid.js' import { openDaemonLog } from './logs.js' -import { statusFilePath, writeStatusFile } from './status.js' +import { statusFilePath, summarizeMaintenanceSkips, writeStatusFile } from './status.js' /** * @import { AiGatewayCapability, JsonObject } from '../../../hypaware-plugin-kernel-types.js' @@ -703,6 +703,30 @@ export async function runDaemon(opts = {}) { storage: boot.runtime.storage, getSettleHook: (dataset) => boot.runtime.query.getDataset(dataset)?.resettleBatch, }) + // @ref LLP 0228#status-file-is-the-surface [implements]: the tick + // stops discarding the report. A partition this walk deliberately + // left fragmented was, until now, a span attribute and nothing + // else, so an operator who did not have tracing on when the tick + // ran had no way to find it at all. + const skips = summarizeMaintenanceSkips(report) + persist({ maintenance: skips }) + span.setAttribute('partitions_visited', skips.partitionsVisited) + span.setAttribute('partitions_skipped', skips.skippedTotal) + if (skips.skippedTotal > 0) { + // The log line is the record and the status file is the + // discovery (LLP 0228#status-file-is-the-surface). One line per + // tick, not one per partition: the counts are the fact, and the + // status file already names the worst of them. + fileLog.warn('daemon.maintenance_skipped', { + partitions_visited: skips.partitionsVisited, + partitions_skipped: skips.skippedTotal, + compaction_ineffective: skips.reasons.compaction_ineffective, + compaction_attempt_failed: skips.reasons.compaction_attempt_failed, + worst: skips.partitions[0] + ? `${skips.partitions[0].dataset}/${skips.partitions[0].partition}` + : null, + }) + } // @ref LLP 0220#tick-reports-degraded [implements]: the walk now // survives a partition that throws, so the rejected promise has // stopped being how the daemon hears about one. Read the failures diff --git a/src/core/daemon/status.js b/src/core/daemon/status.js index c4834f83..c5c392b0 100644 --- a/src/core/daemon/status.js +++ b/src/core/daemon/status.js @@ -51,7 +51,8 @@ import { /** * @import { HypAwareV2Config, PluginConfigInstance } from '../../../hypaware-plugin-kernel-types.js' * @import { ClientActionStatus, ConfigControlStatus, ConfigValidationError } from '../../../src/core/config/types.js' - * @import { CaptureHealthReport, ClientActionReport, ClientActionsReport, ClientAttachReport, CollectStatusOptions, DaemonStatus, DroppedUpstreamAttribution, HypAwareStatusReport, ProxyTrustReport, RecentEntrypoint, ServiceState, SinkSnapshot, SourceSnapshot, StatusDiagnostic } from '../../../src/core/daemon/types.js' + * @import { CaptureHealthReport, ClientActionReport, ClientActionsReport, ClientAttachReport, CollectStatusOptions, DaemonStatus, DroppedUpstreamAttribution, HypAwareStatusReport, MaintenanceSkippedPartition, MaintenanceSkipReason, MaintenanceSkipSnapshot, ProxyTrustReport, RecentEntrypoint, ServiceState, SinkSnapshot, SourceSnapshot, StatusDiagnostic } from '../../../src/core/daemon/types.js' + * @import { MaintenancePartitionReport, MaintenanceReport } from '../../../src/core/cache/types.js' * @import { Dirent } from 'node:fs' * @import { ClientDescriptor, LoadedManifest, PluginCatalog } from '../../../src/core/types.js' * @import { FolderAskMode } from '../../../src/core/usage-policy/types.js' @@ -500,6 +501,229 @@ export function recentEntrypointsFromSources(sources) { return out.slice(0, MAX_RECENT_ENTRYPOINTS) } +/* ---------- maintenance skips (LLP 0228) ---------- */ + +/** + * How many skipped partitions the standing surface names. The counts beside + * the list are exact, so this bounds the terminal block and the status file + * without hiding the size of the problem; `hyp query maintain` is where an + * operator enumerates every one. Eight is a screenful, and a cache with more + * than eight frozen partitions has a story the count already tells. + */ +export const MAX_SKIPPED_PARTITIONS_REPORTED = 8 + +/** Every reason id, in the order the render lists them. */ +const MAINTENANCE_SKIP_REASONS = Object.freeze( + /** @type {MaintenanceSkipReason[]} */ (['compaction_ineffective', 'compaction_attempt_failed']) +) + +/** + * The reason breakdown as one phrase, e.g. `2 compaction_ineffective, 1 + * compaction_attempt_failed`. Reasons no partition was skipped for are left + * out rather than printed as zeros, and the ids are printed verbatim: they + * are the span attribute names, so this phrase is also the trace query + * (LLP 0228#reason-ids-are-span-attribute-names). + * + * Both call sites interpolate this unconditionally into a sentence that + * already committed to a parenthetical, so an empty phrase would render as a + * bare `()`. That is unreachable from a snapshot this build wrote (every + * skip has one of the two known reasons by construction), but not from a + * `status.json` a later build wrote: LLP 0228#consequences names a third + * reason id as exactly the kind of extension this shape absorbs, and a + * snapshot whose only nonzero reasons are ones this build does not + * recognize is precisely `skippedTotal > 0` with every known count at zero. + * The fallback names that case instead of leaving the parenthetical empty. + * + * @param {Record} reasons + * @returns {string} + */ +export function describeMaintenanceSkipReasons(reasons) { + const phrase = MAINTENANCE_SKIP_REASONS + .filter((reason) => (reasons[reason] ?? 0) > 0) + .map((reason) => `${reasons[reason]} ${reason}`) + .join(', ') + return phrase === '' ? 'reasons this build does not recognize' : phrase +} + +/** + * The partition tuple as one label: exactly the shape `hyp query maintain` + * prints after the dataset name, so the same partition reads identically on + * both surfaces. + * + * @param {Record | undefined} partition + * @returns {string} + */ +function partitionLabel(partition) { + if (!isPlainObject(partition)) return 'all' + const parts = Object.entries(partition) + .filter(([, v]) => typeof v === 'string') + .map(([k, v]) => `${k}=${v}`) + return parts.length > 0 ? parts.join('/') : 'all' +} + +/** + * Why this tick left the partition fragmented, or undefined when it did not. + * + * A partition the tick *rewrote* is not on this surface even when the rewrite + * achieved nothing: that is a run that did work, and the verdict it recorded + * puts the partition on the next tick's snapshot as a skip. What this names is + * the standing state, the partition the kernel has stopped rewriting. + * + * @param {MaintenancePartitionReport} p + * @returns {MaintenanceSkipReason | undefined} + * @ref LLP 0218#verdict-outranks-error [constrained-by]: maintenance already makes the two mutually exclusive, so this order only has to agree about which one a reader is owed if that ever stops holding + */ +function skipReasonOf(p) { + if (p.compacted || p.rebaselined) return undefined + if (p.compactionIneffective) return 'compaction_ineffective' + if (p.compactionAttemptFailed) return 'compaction_attempt_failed' + return undefined +} + +/** + * Summarize a maintenance tick's report into the snapshot the daemon persists + * (`DaemonStatus.maintenance`). Pure, and deliberately cheap: it reads the + * report the walk already produced and stats nothing, because proving a + * skipped partition is also still fragmented is the per-tick cost the LLP 0199 + * baseline gate exists to avoid. + * + * A tick that skipped nothing still produces a snapshot (all-zero counts, an + * empty list). The snapshot is the *current* answer, so a partition that + * thawed has to be able to leave it. + * + * @param {MaintenanceReport} report + * @param {{ at?: string }} [opts] + * @returns {MaintenanceSkipSnapshot} + * @ref LLP 0228#last-tick-only [implements]: one bounded snapshot per tick, named partitions capped and taken in the walk's own neediest-first order + */ +export function summarizeMaintenanceSkips(report, opts = {}) { + const visited = Array.isArray(report?.partitions) ? report.partitions : [] + /** @type {Record} */ + const reasons = { compaction_ineffective: 0, compaction_attempt_failed: 0 } + /** @type {MaintenanceSkippedPartition[]} */ + const partitions = [] + let skippedTotal = 0 + for (const p of visited) { + const reason = skipReasonOf(p) + if (reason === undefined) continue + reasons[reason] += 1 + skippedTotal += 1 + // No sort: the report is already in walk order, which is descending live + // data-file count (LLP 0199#neediest-first), so the first entries past the + // cap are the most fragmented ones by construction. + if (partitions.length >= MAX_SKIPPED_PARTITIONS_REPORTED) continue + partitions.push({ + // Sanitized here too, not only on read: LLP 0228#last-tick-only says the + // cap and the sanitizing are both re-applied on read, which only holds + // if the write side already produced a clean label. `dataset` and + // `partition` are kernel-side identifiers in the ordinary case, but + // `partition`'s values come off a captured row's `client_name` by way + // of `resolveSourceSegments` -> `sanitizePathSegment`, which strips only + // path-hostile bytes and applies no length clamp or bidi/zero-width + // filtering. Unsanitized here, the daemon log line at + // `runtime.js`'s `worst` field (which reads `partitions[0]` straight) + // would be the one surface on this path with nothing downstream to + // clean it. + // @ref LLP 0228#last-tick-only [implements]: the write side sanitizes and clamps, not only the read side + dataset: sanitizeLabel(p.dataset) ?? 'unknown', + partition: sanitizeLabel(partitionLabel(p.partition)) ?? 'all', + reason, + // The count the recorded rewrite ran over, not the live one: the same + // distinction `hyp query maintain` draws, for the same reason. + ...(reason === 'compaction_ineffective' && typeof p.compactionIneffectiveFiles === 'number' + ? { dataFiles: p.compactionIneffectiveFiles } + : {}), + ...(reason === 'compaction_attempt_failed' && typeof p.compactionAttemptFailedAt === 'string' + ? { failedAt: p.compactionAttemptFailedAt } + : {}), + }) + } + return { + tickAt: opts.at ?? new Date().toISOString(), + partitionsVisited: visited.length, + skippedTotal, + reasons, + partitions, + } +} + +/** + * Lift the maintenance snapshot out of a status file, or null when no daemon + * has reported a tick for this state root. + * + * Validated, sanitized and re-capped on read as well as on write, for the + * reason `recentEntrypointsFromSources` states: `status.json` is a file, this + * build did not necessarily write it, and everything here is about to be + * printed to a terminal. Dataset and partition labels are the only free-form + * strings on the path and both are kernel-side identifiers, but they are + * cleaned anyway rather than trusted. + * + * Not liveness-gated, for LLP 0164's reason: "these partitions were frozen as + * of the tick at T" stays true after the daemon exits, and the rendered age + * carries the staleness. + * + * @param {DaemonStatus | null} status + * @returns {MaintenanceSkipSnapshot | null} + * @ref LLP 0228#status-file-is-the-surface [implements]: hyp status answers from status.json rather than running a second maintenance walk + */ +export function maintenanceSkipsFromStatus(status) { + const raw = status?.maintenance + if (!isPlainObject(raw)) return null + const tickAt = raw.tickAt + // No timestamp, no snapshot: every render of this block is relative to when + // the tick ran, and "frozen, at some unknown time" is not worth printing. + if (typeof tickAt !== 'string' || Number.isNaN(Date.parse(tickAt))) return null + + const rawReasons = isPlainObject(raw.reasons) ? raw.reasons : {} + /** @type {Record} */ + const reasons = { compaction_ineffective: 0, compaction_attempt_failed: 0 } + for (const reason of MAINTENANCE_SKIP_REASONS) { + reasons[reason] = nonNegativeInt(rawReasons[reason]) ?? 0 + } + + /** @type {MaintenanceSkippedPartition[]} */ + const partitions = [] + const rawPartitions = Array.isArray(raw.partitions) ? raw.partitions : [] + for (const item of rawPartitions) { + if (partitions.length >= MAX_SKIPPED_PARTITIONS_REPORTED) break + if (!isPlainObject(item)) continue + const reason = item.reason + // An unknown reason id is dropped rather than printed: a name this build + // cannot explain is worse than a shorter list, and the counts above still + // account for it. + if (typeof reason !== 'string' || !MAINTENANCE_SKIP_REASONS.includes(/** @type {MaintenanceSkipReason} */ (reason))) continue + const dataset = sanitizeLabel(item.dataset) + const partition = sanitizeLabel(item.partition) + if (dataset === undefined || partition === undefined) continue + const dataFiles = nonNegativeInt(item.dataFiles) + const failedAt = sanitizeLabel(item.failedAt) + partitions.push({ + dataset, + partition, + reason: /** @type {MaintenanceSkipReason} */ (reason), + ...(dataFiles !== undefined ? { dataFiles } : {}), + ...(failedAt !== undefined ? { failedAt } : {}), + }) + } + + const recordedTotal = nonNegativeInt(raw.skippedTotal) + ?? MAINTENANCE_SKIP_REASONS.reduce((sum, reason) => sum + reasons[reason], 0) + return { + tickAt, + // Floored at the skipped total (and the named list, which the total + // itself is already floored at below): "visited" can never be smaller + // than "skipped", or the render says "5 of 0 partitions" for a snapshot + // no tick could have produced. A file this build did not write can claim + // whatever it wants here, so the floor is enforced rather than trusted. + partitionsVisited: Math.max(nonNegativeInt(raw.partitionsVisited) ?? 0, recordedTotal, partitions.length), + // The list is capped, so the count leads; but a count smaller than the + // list would render "2 partitions" above three lines of them. + skippedTotal: Math.max(recordedTotal, partitions.length), + reasons, + partitions, + } +} + /** * Resolve the AI gateway's live bound base URL from the on-disk daemon status * snapshot, **guarded by a daemon-liveness check** so a stale snapshot from a @@ -951,6 +1175,29 @@ export async function collectHypAwareStatus(opts = {}) { // @ref LLP 0164#not-liveness-gated [implements]: a last-seen timestamp survives its daemon; the rendered age carries the staleness const recentEntrypoints = recentEntrypointsFromSources(daemonStatusFile?.sources) + // ----- partitions maintenance left fragmented (LLP 0228) ----- + // Same route and the same reason as the block above: the daemon runs the + // hourly walk, and `hyp status` reads no cache, so answering this any other + // way would mean firing a second maintenance walk from a status command. + const maintenance = maintenanceSkipsFromStatus(daemonStatusFile) + if (maintenance && maintenance.skippedTotal > 0) { + const one = maintenance.skippedTotal === 1 + const breakdown = describeMaintenanceSkipReasons(maintenance.reasons) + // Warning, never an error: the daemon is running, capture works, and + // queries answer. A frozen partition costs disk and query time, so it is + // a thing to know about rather than an outage, which is why it sits with + // `recent_errors` outside the set that degrades `overall` below. + diagnostics.push({ + severity: 'warning', + kind: 'maintenance_partitions_skipped', + message: `cache maintenance is leaving ${maintenance.skippedTotal} partition${one ? '' : 's'} fragmented (${breakdown}), as of its tick at ${maintenance.tickAt}`, + repair: [ + 'hyp query maintain --dry-run', + 'hyp query maintain --force', + ], + }) + } + // Sinks are derived from the loaded config (so the count reflects // "how many sinks does the user have configured?", the same number // a fresh kernel boot or a running daemon would surface). When @@ -1518,6 +1765,7 @@ export async function collectHypAwareStatus(opts = {}) { usagePolicy, firstSyncHoldDeadline, recentEntrypoints, + maintenance, captureHealth, proxyTrust, } diff --git a/src/core/daemon/types.d.ts b/src/core/daemon/types.d.ts index c5ea98f7..e7f0455e 100644 --- a/src/core/daemon/types.d.ts +++ b/src/core/daemon/types.d.ts @@ -66,6 +66,70 @@ export interface SinkSnapshot { nextScheduledAt?: string } +/** + * Why a maintenance tick deliberately left a partition fragmented instead + * of rewriting it (LLP 0228#reason-ids-are-span-attribute-names). The ids + * are the `maintenance.partition` span's attribute names verbatim, which + * are themselves named after the `MaintenancePartitionReport` fields, so + * one spelling covers the trace, the status file, `hyp status`, and the + * daemon log. + * + * - `compaction_ineffective`: this writer already rewrote the partition and + * reproduced the same file count (LLP 0217#record-effectiveness). + * - `compaction_attempt_failed`: the one retry the writer generation owed it + * was spent by a rewrite that threw (LLP 0218#report-the-spent-attempt). + * + * Convergence (LLP 0199#baseline-gate) is not on this list: it is the + * healthy majority of a cache, not a partition left fragmented. + */ +export type MaintenanceSkipReason = 'compaction_ineffective' | 'compaction_attempt_failed' + +/** One partition the last maintenance tick left fragmented, and why. */ +export interface MaintenanceSkippedPartition { + dataset: string + /** Partition tuple as `k=v/k=v`, or `all` for an unpartitioned dataset. */ + partition: string + reason: MaintenanceSkipReason + /** + * The data-file count the recorded rewrite ran over, not the live one + * (LLP 0217). Set for `compaction_ineffective` when the cursor records it. + */ + dataFiles?: number + /** + * ISO time the spent attempt failed, as the cursor records it. Set for + * `compaction_attempt_failed`. + */ + failedAt?: string +} + +/** + * What the daemon's last completed maintenance tick left alone, summarized + * for a standing surface (LLP 0228). Overwritten whole by every tick, + * including one that skipped nothing: a skip reason is a state the tick + * re-derives from the partition cursor, so only the newest tick's answer is + * current, and a partition that thaws drops off by itself + * (LLP 0228#last-tick-only). + * + * Absent means no tick has reported (a daemon that has not reached one, or + * maintenance disabled), which is not the same as "nothing is frozen". + */ +export interface MaintenanceSkipSnapshot { + /** ISO time of the tick this snapshot describes. */ + tickAt: string + /** Partitions the tick actually visited (a budget can cut the walk short). */ + partitionsVisited: number + /** Partitions skipped for a stated reason, all reasons summed. */ + skippedTotal: number + /** How many partitions each reason accounts for. Zero keys are kept. */ + reasons: Record + /** + * The worst of them by name, in walk order (LLP 0199#neediest-first, so + * descending live data-file count), capped at + * `MAX_SKIPPED_PARTITIONS_REPORTED`. `skippedTotal` is the true count. + */ + partitions: MaintenanceSkippedPartition[] +} + export interface DaemonStatus { state: DaemonState pid: number @@ -85,6 +149,11 @@ export interface DaemonStatus { configPath?: string sources: SourceSnapshot[] sinks: SinkSnapshot[] + /** + * What the last completed cache-maintenance tick left fragmented, and why + * (LLP 0228#status-file-is-the-surface). Absent until a tick has run. + */ + maintenance?: MaintenanceSkipSnapshot warnings?: string[] } @@ -107,6 +176,7 @@ export type StatusDiagnosticKind = | 'remote_config_rolled_back' | 'local_only_list_unreadable' | 'client_sync_list_unreadable' + | 'maintenance_partitions_skipped' | 'capture_gap' /** @@ -404,6 +474,15 @@ export interface HypAwareStatusReport { * and reads no cache, so this is the only place the answer can come from. */ recentEntrypoints: RecentEntrypoint[] + /** + * What the daemon's last cache-maintenance tick deliberately left + * fragmented (LLP 0228). Read from `status.json`, like + * `recentEntrypoints`: the daemon runs the hourly walk, and `hyp status` + * activates no plugins and reads no cache, so re-deriving this would mean + * running a second walk from a status command. Null when no daemon has + * reported a tick for this state root. + */ + maintenance: MaintenanceSkipSnapshot | null /** * Capture health for every otel-attached client (LLP 0257#status-and-health, * the RFC 0262 open-question-1 duty): last event seen on the telemetry path diff --git a/test/core/status-maintenance-skips.test.js b/test/core/status-maintenance-skips.test.js new file mode 100644 index 00000000..fbd68246 --- /dev/null +++ b/test/core/status-maintenance-skips.test.js @@ -0,0 +1,653 @@ +// @ts-check + +// A partition maintenance deliberately leaves fragmented (LLP 0217, LLP 0218) +// was stated on exactly two surfaces an operator has to go looking for: a +// `hyp query maintain` line and a span attribute. The daemon, which runs the +// walk hourly, threw the report away. These tests pin the standing surface +// that replaces that silence: the snapshot the tick persists, its bound, the +// reason vocabulary it uses, and what `hyp status` does with it. +// @ref LLP 0228#status-file-is-the-surface [tests]: + +import test from 'node:test' +import assert from 'node:assert/strict' +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' + +import { + collectHypAwareStatus, + MAX_SKIPPED_PARTITIONS_REPORTED, + maintenanceSkipsFromStatus, + summarizeMaintenanceSkips, + writeStatusFile, +} from '../../src/core/daemon/status.js' +import { runDaemon } from '../../src/core/daemon/runtime.js' +import { renderStatusJson, renderStatusText } from '../../src/core/commands/status.js' +import { maintainCache } from '../../src/core/cache/maintenance.js' +import { appendRowsToSourceTable, readCursorSync, writeCursor } from '../../src/core/cache/partition.js' +import { defaultConfigPath } from '../../src/core/config/schema.js' + +/** + * @import { ColumnSpec } from '../../hypaware-plugin-kernel-types.js' + * @import { MaintenancePartitionReport, MaintenanceReport, PartitionCursor } from '../../src/core/cache/types.js' + * @import { CollectStatusOptions, MaintenanceSkipSnapshot } from '../../src/core/daemon/types.js' + */ + +/* ---------- report fixtures ---------- */ + +/** + * One entry of a `maintainCache` report. Defaults are a healthy converged + * partition: the overwhelming majority of every cache, and the thing the + * surface must stay quiet about. + * + * @param {Partial} patch + * @returns {MaintenancePartitionReport} + */ +function partitionReport(patch = {}) { + return { + dataset: 'ai_gateway_messages', + partition: { source: 'claude' }, + path: '/cache/datasets/ai_gateway_messages/source=claude', + snapshotsExpired: 0, + compacted: false, + rowCount: 10, + dataFilesBefore: 4, + dataFilesAfter: 4, + ...patch, + } +} + +/** + * @param {MaintenancePartitionReport[]} partitions + * @returns {MaintenanceReport} + */ +function maintenanceReport(partitions) { + return { + partitions, + totalSnapshotsExpired: 0, + totalCompacted: partitions.filter((p) => p.compacted).length, + totalRebaselined: partitions.filter((p) => p.rebaselined).length, + // Derived like every other total, so a fixture that seeds a partition + // this tick failed on (LLP 0220) stays a report the walk could produce. + // A failure is not a skip: it reports no reason here and stays off this + // surface, which is what `a partition this tick failed on is not a skip` + // pins below. + totalFailed: partitions.filter((p) => p.failed).length, + dryRun: false, + elapsedMs: 12, + } +} + +/* ---------- the summary the daemon persists ---------- */ + +// The ids are the `maintenance.partition` span's attribute names, which are +// themselves the report's field names: one spelling across the trace, the +// status file, `hyp status`, and the daemon log. +// @ref LLP 0228#reason-ids-are-span-attribute-names [tests]: +test('summarizeMaintenanceSkips names both skip reasons in the vocabulary the report already uses', () => { + const snapshot = summarizeMaintenanceSkips(maintenanceReport([ + partitionReport({ + dataset: 'ai_gateway_messages', + partition: { source: 'claude' }, + compactionIneffective: true, + compactionIneffectiveFiles: 1521, + }), + partitionReport({ + dataset: 'traces', + partition: { source: 'codex' }, + compactionAttemptFailed: true, + compactionAttemptFailedAt: '2026-08-12T21:55:35.168Z', + }), + partitionReport({ dataset: 'logs', partition: { source: 'claude' } }), + ]), { at: '2026-08-13T09:00:00.000Z' }) + + assert.deepEqual(snapshot, { + tickAt: '2026-08-13T09:00:00.000Z', + partitionsVisited: 3, + skippedTotal: 2, + reasons: { compaction_ineffective: 1, compaction_attempt_failed: 1 }, + partitions: [ + { + dataset: 'ai_gateway_messages', + partition: 'source=claude', + reason: 'compaction_ineffective', + dataFiles: 1521, + }, + { + dataset: 'traces', + partition: 'source=codex', + reason: 'compaction_attempt_failed', + failedAt: '2026-08-12T21:55:35.168Z', + }, + ], + }) +}) + +// The cap on write bounds entry *count*; this pins that the write side also +// sanitizes and clamps the bytes *inside* each entry, not only the count. +// LLP 0228#last-tick-only says the cap and the sanitizing are "re-applied on +// read as well as on write", which presumes the write side already produced +// something clean. Partition labels are row-derived +// (`resolveSourceSegments` -> `sanitizePathSegment`, which strips only +// path-hostile bytes, not bidi/zero-width/DEL, and applies no length +// clamp), so a hostile `client_name` reaches `summarizeMaintenanceSkips` +// unsanitized. The daemon log's `worst` field reads `partitions[0]` straight +// off this snapshot with no read-side cleanup of its own, so this is also +// the only thing standing between a row-derived label and `tail -f`. +// @ref LLP 0228#last-tick-only [tests]: the write side sanitizes and clamps too, not only the read side +test('summarizeMaintenanceSkips sanitizes and clamps hostile dataset and partition labels on write, into status.json itself', async () => { + const BIDI = String.fromCharCode(0x202e) // right-to-left override + const ZW = String.fromCharCode(0x200b) // zero-width space + const SHY = String.fromCharCode(0x00ad) // soft hyphen + const DEL = String.fromCharCode(0x7f) + const hostile = `claude${BIDI}${ZW}${SHY}${DEL}` + 'x'.repeat(200) + + const snapshot = summarizeMaintenanceSkips(maintenanceReport([ + partitionReport({ + dataset: hostile, + partition: { source: hostile }, + compactionIneffective: true, + compactionIneffectiveFiles: 3, + }), + ])) + + assert.equal(snapshot.partitions.length, 1) + const [p] = snapshot.partitions + for (const bad of [BIDI, ZW, SHY, DEL]) { + assert.equal(p.dataset.includes(bad), false, 'dataset must not carry hostile bytes') + assert.equal(p.partition.includes(bad), false, 'partition must not carry hostile bytes') + } + assert.ok(p.dataset.length <= 120, `dataset must be clamped, got ${p.dataset.length} chars`) + assert.ok(p.partition.length <= 120, `partition must be clamped, got ${p.partition.length} chars`) + + // Not only the in-memory snapshot: the bytes that actually land in + // status.json, which is what the daemon writes and what a `tail -f` on + // the daemon log's `worst` field is downstream of. + const hypHome = await fs.mkdtemp(path.join(os.tmpdir(), 'hyp-status-maintenance-sanitize-')) + try { + const stateRoot = path.join(hypHome, 'hypaware') + await fs.mkdir(path.join(stateRoot, 'run'), { recursive: true }) + writeStatusFile(stateRoot, /** @type {any} */ ({ state: 'healthy', sources: [], sinks: [], maintenance: snapshot })) + const raw = await fs.readFile(path.join(stateRoot, 'run', 'status.json'), 'utf8') + for (const bad of [BIDI, ZW, SHY, DEL]) { + assert.equal(raw.includes(bad), false, `status.json bytes must not carry ${JSON.stringify(bad)}`) + } + assert.ok(raw.length < 2000, `a single hostile label must not balloon status.json, got ${raw.length} bytes`) + } finally { + await fs.rm(hypHome, { recursive: true, force: true }) + } +}) + +// Convergence (LLP 0199#baseline-gate) is the healthy case, a rebaseline (LLP +// 0207) is work the tick did, and a rewrite that achieved nothing is a run +// that ran. None of the three is a partition the kernel has stopped +// rewriting, which is what this surface is for. +test('a converged, rebaselined, or freshly rewritten partition is not on the surface', () => { + const snapshot = summarizeMaintenanceSkips(maintenanceReport([ + partitionReport({ dataset: 'logs' }), + partitionReport({ dataset: 'traces', rebaselined: true }), + partitionReport({ dataset: 'metrics', compacted: true, compactionIneffective: true, compactionIneffectiveFiles: 12 }), + ])) + + assert.equal(snapshot.skippedTotal, 0) + assert.deepEqual(snapshot.partitions, []) + assert.deepEqual(snapshot.reasons, { compaction_ineffective: 0, compaction_attempt_failed: 0 }) + assert.equal(snapshot.partitionsVisited, 3) +}) + +// LLP 0220 gave the walk a per-partition `failed`, which is a different fact +// from either reason here: `failed` means this tick attempted work and it +// threw, while both ids on this surface mean nothing was attempted, for a +// stated reason. A tick that attempted a rewrite is not a tick that skipped +// one, so a failed partition stays off the surface and is reported by its own +// per-partition `daemon.maintenance_failed` line instead. Without this, the +// two would double-report the same partition under contradictory +// descriptions. +// @ref LLP 0220#this-tick-versus-a-recorded-one [tests]: this tick's failure is not a skip, so it does not reach the skip surface +test('a partition this tick failed on is not a skip', () => { + const snapshot = summarizeMaintenanceSkips(maintenanceReport([ + partitionReport({ + dataset: 'ai_gateway_messages', + failed: true, + errorKind: 'maintenance_partition_failed', + errorMessage: 'torn data file', + }), + partitionReport({ + dataset: 'traces', + compactionAttemptFailed: true, + compactionAttemptFailedAt: '2026-08-12T21:55:35.168Z', + }), + ])) + + // Visited counts both: the walk did reach them. + assert.equal(snapshot.partitionsVisited, 2) + // Only the recorded skip is named, and it is the one an earlier tick froze. + assert.equal(snapshot.skippedTotal, 1) + assert.deepEqual(snapshot.partitions.map((p) => p.dataset), ['traces']) + assert.deepEqual(snapshot.reasons, { compaction_ineffective: 0, compaction_attempt_failed: 1 }) +}) + +// The status file is read back and printed to a terminal, so the named list +// is bounded. The counts beside it are not, so the bound never hides the size +// of the problem. +// @ref LLP 0228#last-tick-only [tests]: +test('the named list is capped in the walk order the tick already used, and the count stays exact', () => { + const snapshot = summarizeMaintenanceSkips(maintenanceReport( + // Walk order is descending live data-file count (LLP 0199#neediest-first), + // so position in the report is the ranking: index 0 is the worst. + Array.from({ length: 20 }, (_, i) => partitionReport({ + dataset: `dataset_${String(i).padStart(2, '0')}`, + compactionIneffective: true, + compactionIneffectiveFiles: 500 - i, + })) + )) + + assert.equal(snapshot.skippedTotal, 20) + assert.equal(snapshot.reasons.compaction_ineffective, 20) + assert.equal(snapshot.partitions.length, MAX_SKIPPED_PARTITIONS_REPORTED) + assert.equal(MAX_SKIPPED_PARTITIONS_REPORTED, 8) + assert.equal(snapshot.partitions[0].dataset, 'dataset_00') + assert.equal(snapshot.partitions[7].dataset, 'dataset_07') +}) + +// The retention rule is "the last tick, whole": a skip reason is a state the +// tick re-derives from the cursor, so a partition that thawed has to be able +// to leave the surface without an expiry rule. +test('a tick that skipped nothing still produces a snapshot, so the surface clears itself', () => { + const snapshot = summarizeMaintenanceSkips(maintenanceReport([partitionReport({ compacted: true })])) + assert.equal(snapshot.skippedTotal, 0) + assert.equal(snapshot.partitionsVisited, 1) + assert.ok(!Number.isNaN(Date.parse(snapshot.tickAt)), 'a snapshot always timestamps its tick') +}) + +/* ---------- reading a file this build did not necessarily write ---------- */ + +test('maintenanceSkipsFromStatus is null for a daemon that has never reported a tick', () => { + assert.equal(maintenanceSkipsFromStatus(null), null) + assert.equal(maintenanceSkipsFromStatus(/** @type {any} */ ({ state: 'healthy' })), null) + // No usable tick timestamp: every render of the block is relative to it. + assert.equal( + maintenanceSkipsFromStatus(/** @type {any} */ ({ maintenance: { skippedTotal: 3, partitions: [] } })), + null + ) + assert.equal( + maintenanceSkipsFromStatus(/** @type {any} */ ({ maintenance: { tickAt: 'not a date' } })), + null + ) +}) + +test('a foreign status file is capped, cleaned, and stripped of reasons this build cannot explain', () => { + const ESC = String.fromCharCode(27) + const LF = String.fromCharCode(10) + const snapshot = maintenanceSkipsFromStatus(/** @type {any} */ ({ + maintenance: { + tickAt: '2026-08-13T09:00:00.000Z', + partitionsVisited: -4, + skippedTotal: 900, + reasons: { compaction_ineffective: 900, compaction_attempt_failed: 'lots', compaction_from_the_future: 5 }, + partitions: [ + { dataset: `ai_gateway_messages${ESC}[2K${LF} daemon: FORGED`, partition: 'source=claude', reason: 'compaction_ineffective', dataFiles: 1521 }, + { dataset: 'traces', partition: 'source=codex', reason: 'compaction_from_the_future' }, + { dataset: 'logs', reason: 'compaction_ineffective' }, + null, + ...Array.from({ length: 40 }, (_, i) => ({ + dataset: `dataset_${i}`, + partition: 'source=claude', + reason: 'compaction_attempt_failed', + failedAt: '2026-08-12T21:55:35.168Z', + })), + ], + }, + })) + + assert.ok(snapshot) + assert.equal(snapshot.partitions.length, MAX_SKIPPED_PARTITIONS_REPORTED) + assert.equal(snapshot.partitions[0].dataset.includes(ESC), false, 'no escape byte survives') + assert.equal(snapshot.partitions[0].dataset.includes(LF), false, 'no newline survives') + // An unknown reason id and an entry with no partition tuple are dropped + // rather than printed: a line this build cannot explain is worse than a + // shorter list. + assert.deepEqual( + snapshot.partitions.map((p) => p.reason), + ['compaction_ineffective', ...Array.from({ length: 7 }, () => 'compaction_attempt_failed')] + ) + // Unknown reason keys never reach the counts, and a non-numeric one reads + // as zero rather than as text on a terminal. + assert.deepEqual(snapshot.reasons, { compaction_ineffective: 900, compaction_attempt_failed: 0 }) + // A negative count is not a count, and 900 partitions were recorded + // skipped, so visited is floored at the skipped total rather than at the + // capped list length: "8 of 900" is exactly the impossible-looking + // sentence a missing floor would render. + assert.equal(snapshot.partitionsVisited, 900, 'partitionsVisited must never read smaller than skippedTotal') + assert.equal(snapshot.skippedTotal, 900) +}) + +test('a total smaller than the list it labels is corrected upward, never rendered as a lie', () => { + const snapshot = maintenanceSkipsFromStatus(/** @type {any} */ ({ + maintenance: { + tickAt: '2026-08-13T09:00:00.000Z', + skippedTotal: 1, + reasons: {}, + partitions: [ + { dataset: 'a', partition: 'source=claude', reason: 'compaction_ineffective' }, + { dataset: 'b', partition: 'source=claude', reason: 'compaction_ineffective' }, + { dataset: 'c', partition: 'source=claude', reason: 'compaction_ineffective' }, + ], + }, + })) + assert.ok(snapshot) + assert.equal(snapshot.skippedTotal, 3) +}) + +/* ---------- hyp status ---------- */ + +async function makeHome() { + const hypHome = await fs.mkdtemp(path.join(os.tmpdir(), 'hyp-status-maintenance-')) + const stateRoot = path.join(hypHome, 'hypaware') + await fs.mkdir(path.join(stateRoot, 'run'), { recursive: true }) + await fs.writeFile(defaultConfigPath(hypHome), JSON.stringify({ version: 2, plugins: [] }) + '\n') + return { hypHome, stateRoot } +} + +/** + * @param {string} hypHome + * @returns {CollectStatusOptions} + */ +function collectOpts(hypHome) { + return { + env: { ...process.env, HYP_HOME: hypHome, HYP_CONFIG: '' }, + platform: 'darwin', + isLaunchAgentInstalled: () => false, + } +} + +/** @returns {{ write(chunk: string): void, text(): string }} */ +function buffer() { + /** @type {string[]} */ + const chunks = [] + return { write: (chunk) => { chunks.push(chunk) }, text: () => chunks.join('') } +} + +/** @param {string} stateRoot @param {MaintenanceSkipSnapshot | undefined} maintenance */ +function writeDaemonStatus(stateRoot, maintenance) { + writeStatusFile(stateRoot, /** @type {any} */ ({ + state: 'healthy', + sources: [], + sinks: [], + ...(maintenance ? { maintenance } : {}), + })) +} + +test('hyp status names the frozen partitions, and says so without calling the install broken', async () => { + const { hypHome, stateRoot } = await makeHome() + try { + writeDaemonStatus(stateRoot, { + tickAt: new Date(Date.now() - 5 * 60_000).toISOString(), + partitionsVisited: 12, + skippedTotal: 3, + reasons: { compaction_ineffective: 2, compaction_attempt_failed: 1 }, + partitions: [ + { dataset: 'ai_gateway_messages', partition: 'source=claude', reason: 'compaction_ineffective', dataFiles: 1521 }, + { dataset: 'traces', partition: 'source=codex', reason: 'compaction_attempt_failed', failedAt: '2026-08-12T21:55:35.168Z' }, + ], + }) + + const report = await collectHypAwareStatus(collectOpts(hypHome)) + assert.equal(report.maintenance?.skippedTotal, 3) + + const diagnostic = report.diagnostics.find((d) => d.kind === 'maintenance_partitions_skipped') + assert.ok(diagnostic, 'a frozen partition raises a diagnostic an operator scanning status will see') + assert.equal(diagnostic.severity, 'warning') + assert.match(diagnostic.message, /3 partitions fragmented|leaving 3 partitions fragmented/) + assert.ok(diagnostic.repair.includes('hyp query maintain --force')) + // The daemon is running and capture works: this is a thing to know + // about, not an outage. + assert.equal(report.overall, 'healthy') + + const stdout = buffer() + renderStatusText({ report, clientNames: [], datasets: [], cacheRoot: path.join(stateRoot, 'cache'), stdout }) + const text = stdout.text() + assert.match(text, /maintenance:/) + assert.match(text, /3 of 12 partitions left fragmented, as of the tick 5m ago \(2 compaction_ineffective, 1 compaction_attempt_failed\)/) + assert.match(text, /- ai_gateway_messages\/source=claude {2}\[compaction_ineffective\] {2}the last rewrite of 1521 files reduced nothing/) + assert.match(text, /- traces\/source=codex {2}\[compaction_attempt_failed\] {2}the retry failed at 2026-08-12T21:55:35\.168Z/) + // Two of three are named, so the render says where the third is. + assert.match(text, /\.\.\. and 1 more \(hyp query maintain --dry-run lists them all\)/) + + const json = renderStatusJson({ report, clientNames: [], datasets: [], cacheRoot: path.join(stateRoot, 'cache') }) + assert.equal(json.maintenance?.skipped_total, 3) + assert.equal(json.maintenance?.partitions_visited, 12) + assert.deepEqual(json.maintenance?.reasons, { compaction_ineffective: 2, compaction_attempt_failed: 1 }) + assert.deepEqual(json.maintenance?.skipped[0], { + dataset: 'ai_gateway_messages', + partition: 'source=claude', + reason: 'compaction_ineffective', + data_files: 1521, + }) + } finally { + await fs.rm(hypHome, { recursive: true, force: true }) + } +}) + +test('an install with nothing frozen keeps the V1 text surface, and a daemon that never ran reports null', async () => { + const { hypHome, stateRoot } = await makeHome() + try { + // A tick ran and skipped nothing. + writeDaemonStatus(stateRoot, { + tickAt: new Date().toISOString(), + partitionsVisited: 12, + skippedTotal: 0, + reasons: { compaction_ineffective: 0, compaction_attempt_failed: 0 }, + partitions: [], + }) + const clean = await collectHypAwareStatus(collectOpts(hypHome)) + assert.equal(clean.maintenance?.skippedTotal, 0) + assert.equal(clean.diagnostics.some((d) => d.kind === 'maintenance_partitions_skipped'), false) + const stdout = buffer() + renderStatusText({ report: clean, clientNames: [], datasets: [], cacheRoot: path.join(stateRoot, 'cache'), stdout }) + assert.doesNotMatch(stdout.text(), /maintenance:/) + + // No tick has ever reported. Absent is not "nothing is frozen", so + // nothing is claimed either way. + writeDaemonStatus(stateRoot, undefined) + const silent = await collectHypAwareStatus(collectOpts(hypHome)) + assert.equal(silent.maintenance, null) + const json = renderStatusJson({ report: silent, clientNames: [], datasets: [], cacheRoot: path.join(stateRoot, 'cache') }) + assert.equal(json.maintenance, null) + } finally { + await fs.rm(hypHome, { recursive: true, force: true }) + } +}) + +// `describeMaintenanceSkipReasons` filters to reasons in the known +// vocabulary with a nonzero count and joins; when a status.json's only +// nonzero counts are for reasons this build does not recognize, the join is +// '', and both `hyp status`'s text render and its +// `maintenance_partitions_skipped` diagnostic interpolate that +// unconditionally, so the rendered line ends "fragmented ()". This is +// unreachable from a file this build wrote (every skip it records has one of +// the two known reasons by construction), but the read path's whole +// justification is that this build did not necessarily write the file, and +// LLP 0228#consequences names exactly this extension ("a new id and a new +// count key, not a new shape") as the forward-compatible path. The existing +// foreign-file test above keeps `compaction_ineffective: 900` alongside its +// unknown reason id, so that case never hits the all-unknown branch. +// +// Separately, `partitionsVisited` fell back to `partitions.length` with no +// floor at `skippedTotal`, while `skippedTotal` is floored at +// `partitions.length`. A status.json omitting `partitionsVisited` (as an +// older or different build might) alongside a positive `skippedTotal` and an +// empty `partitions` list rendered "5 of 0 partitions" - a sentence no tick +// can produce, since visited can never be smaller than skipped. +// @ref LLP 0228#last-tick-only [tests]: a foreign status file's reason breakdown and visited count stay sentences a tick could actually produce +test('a status.json carrying only reasons this build does not recognize renders no empty parenthetical and no impossible visited count', async () => { + const { hypHome, stateRoot } = await makeHome() + try { + writeStatusFile(stateRoot, /** @type {any} */ ({ + state: 'healthy', + sources: [], + sinks: [], + maintenance: { + tickAt: new Date(Date.now() - 5 * 60_000).toISOString(), + // No partitionsVisited at all: a later or different build might not + // have written one either. + skippedTotal: 5, + reasons: { compaction_from_the_future: 5 }, + partitions: [], + }, + })) + + const report = await collectHypAwareStatus(collectOpts(hypHome)) + assert.equal(report.maintenance?.skippedTotal, 5) + // The floor: visited can never render smaller than skipped. + assert.ok( + (report.maintenance?.partitionsVisited ?? 0) >= 5, + `partitionsVisited must be floored at skippedTotal, got ${report.maintenance?.partitionsVisited}` + ) + + const diagnostic = report.diagnostics.find((d) => d.kind === 'maintenance_partitions_skipped') + assert.ok(diagnostic, 'a positive skippedTotal still raises the diagnostic') + assert.doesNotMatch(diagnostic.message, /\(\)/, 'no bare parenthetical in the diagnostic message') + assert.match(diagnostic.message, /reasons this build does not recognize/) + + const stdout = buffer() + renderStatusText({ report, clientNames: [], datasets: [], cacheRoot: path.join(stateRoot, 'cache'), stdout }) + const text = stdout.text() + assert.doesNotMatch(text, /\(\)/, 'no bare parenthetical in the rendered text') + assert.doesNotMatch(text, /5 of 0 partitions/, 'visited must never render smaller than skipped') + } finally { + await fs.rm(hypHome, { recursive: true, force: true }) + } +}) + +/* ---------- against a real frozen partition ---------- */ + +/** @type {ColumnSpec[]} */ +const SESSION_COLUMNS = [ + { name: 'id', type: 'INT32', nullable: false }, + { name: 'session_id', type: 'STRING', nullable: false }, + { name: 'attributes', type: 'STRING', nullable: true }, +] + +/** + * The production shape behind #723: one identity-partitioned tuple per + * session, so the partition already sits on its file-count floor and every + * rewrite reproduces it. + */ +const SESSION_DECLARATION = { + source: { columns: ['source'] }, + iceberg: { fields: [{ column: 'session_id', transform: 'identity' }] }, +} + +/** + * Seed a partition and plant the cursor a rewrite that achieved nothing + * leaves behind: the baseline sits on the live count (so the LLP 0199 gate + * skips it), the record says the rewrite started from that same count (so the + * verdict is "reduced nothing"), and the stamp names the writer running now + * (so no retry is owed). + * + * @param {string} cacheRoot + * @param {number} sessions + * @returns {Promise} the partition directory + */ +async function seedFrozenPartition(cacheRoot, sessions) { + const rows = Array.from({ length: sessions }, (_, i) => ({ + id: i, + session_id: `s-${i}`, + attributes: `{"gateway":{"session":"s-${i}"}}`, + })) + await appendRowsToSourceTable( + cacheRoot, 'ai_gateway_messages', ['source=claude'], SESSION_COLUMNS, rows, + { declaration: SESSION_DECLARATION } + ) + const dir = path.join(cacheRoot, 'datasets', 'ai_gateway_messages', 'source=claude') + const cursor = readCursorSync(dir) + /** @type {PartitionCursor} */ + const next = { + ...cursor, + compaction: { + previousTableDir: 'table', + compactedAt: '2026-08-12T21:55:35.168Z', + resettleBaselineFiles: sessions, + dataFilesBefore: sessions, + writerGeneration: 2, + }, + } + await writeCursor(dir, next) + return dir +} + +// The vocabulary is only worth anything if it is the vocabulary a real walk +// produces. This runs maintenance over a partition frozen exactly the way +// #723's was and summarizes what comes back. +test('a real maintenance walk over a frozen partition summarizes into the reason it reported', async () => { + const cacheRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'hyp-maintenance-frozen-')) + try { + await seedFrozenPartition(cacheRoot, 6) + const report = await maintainCache({ cacheRoot, compactOnly: true }) + assert.equal(report.totalCompacted, 0, 'fixture invariant: the partition must be skipped, not rewritten') + + const snapshot = summarizeMaintenanceSkips(report) + assert.equal(snapshot.skippedTotal, 1) + assert.deepEqual(snapshot.reasons, { compaction_ineffective: 1, compaction_attempt_failed: 0 }) + assert.deepEqual(snapshot.partitions, [{ + dataset: 'ai_gateway_messages', + partition: 'source=claude', + reason: 'compaction_ineffective', + dataFiles: 6, + }]) + } finally { + await fs.rm(cacheRoot, { recursive: true, force: true }) + } +}) + +// The defect itself: `runMaintenance` awaited `maintainCache` and dropped the +// result, so nothing the walk decided reached any surface the daemon keeps. +// @ref LLP 0228#status-file-is-the-surface [tests]: the tick persists what it left fragmented +test('the daemon maintenance tick persists what it left fragmented into status.json', async () => { + const hypHome = await fs.mkdtemp(path.join(os.tmpdir(), 'hyp-daemon-maintenance-')) + const stateRoot = path.join(hypHome, 'hypaware') + let handle + try { + await seedFrozenPartition(path.join(stateRoot, 'cache'), 6) + const configPath = defaultConfigPath(hypHome) + await fs.mkdir(path.dirname(configPath), { recursive: true }) + await fs.writeFile(configPath, JSON.stringify({ + version: 2, + plugins: [], + // 0.01 minutes: the maintenance timer is not clamped the way the sink + // tick is, so a tick lands inside the wait below. + query: { cache: { maintenance: { interval_minutes: 0.01 } } }, + })) + + handle = await runDaemon({ + hypHome, + configPath, + env: { ...process.env, HYP_HOME: hypHome }, + runId: 'maintenance-skip-tick', + tickIntervalMs: 1, + installSignalHandlers: false, + }) + + const deadline = Date.now() + 30_000 + /** @type {MaintenanceSkipSnapshot | null} */ + let maintenance = null + while (Date.now() < deadline) { + await new Promise((resolve) => setTimeout(resolve, 100)) + const raw = await fs.readFile(path.join(stateRoot, 'run', 'status.json'), 'utf8').catch(() => null) + maintenance = raw ? (JSON.parse(raw).maintenance ?? null) : null + if (maintenance && maintenance.skippedTotal > 0) break + } + assert.ok(maintenance, 'no maintenance snapshot ever reached status.json') + assert.equal(maintenance.skippedTotal, 1) + assert.deepEqual(maintenance.reasons, { compaction_ineffective: 1, compaction_attempt_failed: 0 }) + assert.equal(maintenance.partitions[0].dataset, 'ai_gateway_messages') + assert.equal(maintenance.partitions[0].reason, 'compaction_ineffective') + } finally { + if (handle) { + await handle.stop() + await handle.done + } + await fs.rm(hypHome, { recursive: true, force: true }) + } +})