From 1129a1276709c7007a44c55ef16a38f1fc7e52be Mon Sep 17 00:00:00 2001 From: test Date: Thu, 13 Aug 2026 23:56:37 +0000 Subject: [PATCH 1/3] Daemon maintenance skip reasons reach a standing surface (#742) A partition maintenance deliberately leaves fragmented (LLP 0217, LLP 0218) was stated only on surfaces an operator has to go looking for: a `hyp query maintain` line and a `maintenance.partition` span attribute. The daemon's hourly tick awaited `maintainCache` and discarded the report. The tick now summarizes that report into `DaemonStatus.maintenance` and persists it with the snapshot it already writes; `hyp status` renders the block, carries it under `--json`, and raises one warning diagnostic, and the tick logs one line when the count is nonzero. Retention is the last completed tick, whole: both reasons are re-derived from the partition cursor every tick, so the newest tick is the only current answer and a thawed partition leaves the surface by itself. Bounded by a fixed reason-key set and an 8-entry named list taken in the walk's own neediest-first order, with the exact total beside it. The reason ids are the span attribute names verbatim. Design record: LLP 0224, with Extended-by forward-refs on LLP 0217 and 0218. Co-Authored-By: Claude --- ...mpaction-effectiveness-verdict.decision.md | 2 +- ...action-failed-attempt-reported.decision.md | 1 + ...e-skips-are-a-standing-surface.decision.md | 162 ++++++ src/core/commands/status.js | 50 +- src/core/daemon/runtime.js | 30 +- src/core/daemon/status.js | 222 +++++++- src/core/daemon/types.d.ts | 79 +++ test/core/status-maintenance-skips.test.js | 495 ++++++++++++++++++ 8 files changed, 1035 insertions(+), 6 deletions(-) create mode 100644 llp/0224-maintenance-skips-are-a-standing-surface.decision.md create mode 100644 test/core/status-maintenance-skips.test.js diff --git a/llp/0217-compaction-effectiveness-verdict.decision.md b/llp/0217-compaction-effectiveness-verdict.decision.md index a1209e9c..7295952c 100644 --- a/llp/0217-compaction-effectiveness-verdict.decision.md +++ b/llp/0217-compaction-effectiveness-verdict.decision.md @@ -6,7 +6,7 @@ **Author:** Kenny / Claude **Date:** 2026-08-13 **Related:** LLP 0199 (the baseline gate this extends), LLP 0209 (the writer change that made a stale verdict reachable), LLP 0027 (the re-settle baseline the gate was built from) -**Extended-by:** [LLP 0218](./0218-compaction-failed-attempt-reported.decision.md) (the stamp a failed retry writes carries no effectiveness claim, so the skip report below cannot speak for it: the stamp records when the attempt failed and the skip is reported on that instead) +**Extended-by:** [LLP 0218](./0218-compaction-failed-attempt-reported.decision.md) (the stamp a failed retry writes carries no effectiveness claim, so the skip report below cannot speak for it: the stamp records when the attempt failed and the skip is reported on that instead); [LLP 0224](./0224-maintenance-skips-are-a-standing-surface.decision.md) (the verdict's skip report stops being visible only to whoever runs `hyp query maintain` or queries the trace: the daemon tick summarizes it into `status.json` and `hyp status` renders it) > The partition cursor records the data-file count a rewrite started from > beside the count it produced, so "the live count sits on its baseline" can be diff --git a/llp/0218-compaction-failed-attempt-reported.decision.md b/llp/0218-compaction-failed-attempt-reported.decision.md index 2ada3317..5afef5f8 100644 --- a/llp/0218-compaction-failed-attempt-reported.decision.md +++ b/llp/0218-compaction-failed-attempt-reported.decision.md @@ -6,6 +6,7 @@ **Author:** Kenny / Claude **Date:** 2026-08-13 **Related:** LLP 0217 (the effectiveness verdict and the retry stamp this extends), LLP 0199 (the baseline gate both sit on), LLP 0207 (the recognition path that writes the same stamp) +**Extended-by:** [LLP 0224](./0224-maintenance-skips-are-a-standing-surface.decision.md) (the standing record this document describes stops being a `hyp query maintain` line and a span attribute: the daemon tick summarizes both skip reasons into `status.json`, where `hyp status` finds them) > The stamp a failed compaction retry writes records the moment the attempt > failed, so the ticks that skip the partition afterwards can say why. A diff --git a/llp/0224-maintenance-skips-are-a-standing-surface.decision.md b/llp/0224-maintenance-skips-are-a-standing-surface.decision.md new file mode 100644 index 00000000..8d24cf8f --- /dev/null +++ b/llp/0224-maintenance-skips-are-a-standing-surface.decision.md @@ -0,0 +1,162 @@ +# LLP 0224: A partition maintenance leaves fragmented is named on a standing daemon surface + +**Type:** Decision +**Status:** Accepted +**Systems:** Cache, Daemon, CLI +**Author:** Kenny / Claude +**Date:** 2026-08-13 +**Related:** LLP 0217 (the effectiveness verdict whose skip reason this carries), LLP 0218 (the failed-attempt skip reason this carries), LLP 0199 (the baseline gate both sit on, and the walk order this borrows), LLP 0164 (the precedent: an operator-facing fact the daemon alone knows travels through `status.json`), LLP 0021 (the trace surface this stops being the only one) + +> The daemon's maintenance tick stops discarding the report `maintainCache` +> hands back. Every completed tick overwrites one bounded snapshot in +> `status.json` naming the partitions it deliberately left fragmented and why; +> `hyp status` renders that snapshot and raises one warning diagnostic from it, +> and the tick logs one line whenever the count is nonzero. Retention is +> exactly the last tick, because a skip reason is a standing state the tick +> re-derives from the cursor, not an event that happened once. + +## Context {#context} + +LLP 0217 and LLP 0218 made a partition maintenance deliberately leaves +fragmented a *stated* outcome rather than an absence in a "0 partitions +compacted" summary: `MaintenancePartitionReport` carries +`compactionIneffective` and `compactionAttemptFailed`, `hyp query maintain` +prints both, and the `maintenance.partition` span carries +`compaction_ineffective` / `compaction_attempt_failed`. + +Both statements reach an operator only if the operator goes looking. The +daemon runs the same walk hourly and `src/core/daemon/runtime.js` awaited +`maintainCache` and dropped its return value on the floor, so on an ordinary +install the only daemon-side evidence of a frozen partition was a span +attribute. A span attribute is a real observability channel (the repo's +Log-Driven Development rule says so), and it is the wrong one for this fact: +it is only there if tracing was on at the moment the tick ran, and finding it +means knowing to query the `traces` dataset for an attribute whose name you +already know. The state it describes, meanwhile, is not a moment. It persists +until something rewrites the cursor. + +Reported as #742, deferred out of PR #741 on the grounds that widening the +surface LLP 0217/0218 settled is a new request rather than a defect in them. + +## Decision {#decision} + +**The daemon's status file carries it, +and `hyp status` reads it.** The maintenance tick summarizes its report into +`DaemonStatus.maintenance` and persists it with the snapshot the daemon +already writes on every sink tick. `hyp status` lifts it from `status.json`, +renders a `maintenance:` block, carries it under `--json`, and raises one +`maintenance_partitions_skipped` warning diagnostic naming the count and the +repair (`hyp query maintain --dry-run` to enumerate, `--force` to retry). + +This is LLP 0164's route, for LLP 0164's reason. The daemon is the only +process that runs the hourly walk, `hyp status` activates no plugins and reads +no cache, and a CLI that re-ran `maintainCache` to answer the question would +be a *second* walk (loading metadata, stat-ing data files) fired by a status +command. The file is the only place the answer can come from without doing the +work twice. + +The alternatives were weighed and are not enough on their own. A `fileLog` +line alone (#742's option 1) is a record of a tick, not a description of a +state: it scrolls away, and it is the shape of surface that already existed +for the failing tick in LLP 0218's context section. A new field on the +maintenance report alone changes nothing, because the caller that discards +the report is the whole defect. So the log line is kept as the record (the +tick logs `daemon.maintenance_skipped` when the count is nonzero) and the +status file is the discovery, which is #742's option 3. + +Nothing about *when* a partition is compacted moves. This decision reads the +report the walk already produces, and it must never be a reason to stat a +data file: proving a skipped partition is also still fragmented is exactly the +per-tick cost the LLP 0199 gate exists to avoid, and LLP 0217 and LLP 0218 +both already declined to pay it for their own reports. + +**Retention is the last completed tick, and only +that.** Every completed tick overwrites the snapshot whole, including a tick +that skipped nothing (which writes zeros). + +A skip reason is per-partition and per-tick, so a standing surface needs a +retention rule, and the reason itself decides which one. Both reasons are read +off the partition cursor on every tick: they describe a state that is still +true, not an event that occurred. So the newest tick is the only tick whose +answer is current, and a partition that thaws (a `--force` rewrite, new data +flushing in, the next writer generation) drops off the surface on the tick +after it thaws, with no expiry rule and nothing to invalidate. That is the same +self-clearing property LLP 0218 built into the report itself. A last-N history +would go stale against the cursor, would need an eviction rule of its own, and +would duplicate the `fileLog`, which is already the append-only record with +timestamps. + +The snapshot is bounded three ways, because `status.json` is read back and +printed to a terminal. `reasons` is a fixed key set, one integer per reason. +`partitions` is capped at `MAX_SKIPPED_PARTITIONS_REPORTED` entries, taken in +walk order, which is LLP 0199#neediest-first: descending live data-file count, +so the named partitions are the worst ones and the cap costs no sort of its +own. `skippedTotal` is the true count, so the cap is never a lie, and the +render says how many were not named and which command lists them all. The cap +and the sanitizing are re-applied on read as well as on write, for the reason +LLP 0164 states: core reads a *file*, and must not assume the daemon that +wrote it was this build. + +Each named partition carries its dataset, its partition tuple, its reason, and +the one number or timestamp that reason is about (the recorded rewrite's +data-file count, or when the spent attempt failed). Those are dataset and +partition identifiers and kernel-side counters. Nothing from a row, a prompt, +a credential, or a config value is anywhere near this path. + +**The reason ids are the span +attribute names, verbatim.** `compaction_ineffective` (LLP +0217#record-effectiveness) and `compaction_attempt_failed` (LLP +0218#report-the-spent-attempt) are the ids on every operator-facing surface: +the status file, `hyp status` text, `hyp status --json`, and the daemon log +line. They are already the `maintenance.partition` span's attribute names, and +those are in turn named after the `MaintenancePartitionReport` fields, so an +operator moving between a trace query, `hyp status`, and the daemon log reads +one spelling. Minting a third name for the same fact is exactly the parallel +vocabulary this repo's skip reporting has so far avoided. + +Two things a maintenance tick also does are deliberately *not* reasons here. A +partition sitting on its baseline with a rewrite recorded as effective is +converged (LLP 0199#baseline-gate): that is the healthy majority of every +cache, it is why the gate exists, and naming it would put most partitions on +the surface and none of the interesting ones. A rebaseline (LLP 0207) is work +the tick performed, not a partition it left alone. The surface names only the +partitions the kernel knows are still fragmented and has stopped rewriting. + +## Consequences {#consequences} + +- `hyp status` on an install with no frozen partition is unchanged: the text + block renders only when the count is nonzero, the way every other + conditional section in the status render does. `--json` always carries the + key (null before any tick has run), per that surface's "missing values + surface as null rather than being omitted" contract. +- The diagnostic is a warning and never flips `overall` to `degraded`. A + frozen partition is a thing to know about, not an outage: the daemon is + running, capture is working, and queries answer. This puts it beside + `recent_errors` and a failed client action rather than beside a missing + config. +- The snapshot can only say what the last tick saw. A tick cut short by + `max_tick_ms` (LLP 0199#neediest-first) reports the partitions it reached and + no others, so `partitionsVisited` is on the snapshot beside the counts: "3 + skipped of 12 visited" is honest in a way "3 skipped" is not when the cache + holds 400 partitions. +- A daemon that has never run a maintenance tick, and an install whose + maintenance is disabled, both leave the field absent. Absent means "no tick + has reported", which is not "nothing is frozen", and the render says nothing + rather than claiming a clean cache. +- `status.json` grows by a bounded constant. It is rewritten in full on every + persist, so there is no accumulation across ticks and no file growth to + bound over a daemon's lifetime. +- The surface is additive to the report, so a later per-partition outcome + worth standing (this tick's own error, say, which is a different fact from + either reason here) is a new id and a new count key, not a new shape. + +## Extends {#extends} + +LLP 0217 settled that a partition skipped because its rewrite achieved nothing +is skipped explicitly, and LLP 0218 settled the same for a partition whose one +retry was spent by an attempt that threw. Both stand exactly as written: this +document changes neither the verdicts, nor what is recorded on the cursor, nor +when a partition is retried. What it adds is that the daemon, which runs the +walk that reaches those verdicts, stops throwing away the report that states +them, so a frozen partition is discoverable by an operator who never runs +`hyp query maintain` and never queries the `traces` dataset. diff --git a/src/core/commands/status.js b/src/core/commands/status.js index cd5648fc..c7617246 100644 --- a/src/core/commands/status.js +++ b/src/core/commands/status.js @@ -1,7 +1,7 @@ // @ts-check import { Attr, withSpan } from '../observability/index.js' -import { collectHypAwareStatus } from '../daemon/status.js' +import { collectHypAwareStatus, describeMaintenanceSkipReasons } from '../daemon/status.js' import { formatFirstSyncDeadline } from '../usage-policy/first_sync_hold.js' /** @@ -205,6 +205,24 @@ export function renderStatusJson({ report, clientNames, datasets, cacheRoot }) { last_seen: e.lastSeen, rows: e.rows, })), + // What the daemon's last maintenance tick deliberately left fragmented + // (LLP 0224). Null until a daemon has reported a tick: absent is "no tick + // has said", which is not "nothing is frozen". + maintenance: report.maintenance + ? { + tick_at: report.maintenance.tickAt, + partitions_visited: report.maintenance.partitionsVisited, + skipped_total: report.maintenance.skippedTotal, + reasons: report.maintenance.reasons, + skipped: report.maintenance.partitions.map((p) => ({ + dataset: p.dataset, + partition: p.partition, + reason: p.reason, + ...(p.dataFiles !== undefined ? { data_files: p.dataFiles } : {}), + ...(p.failedAt ? { failed_at: p.failedAt } : {}), + })), + } + : null, datasets: datasets.map((d) => ({ name: d.name, plugin: d.plugin })), cache: { dir: cacheRoot, @@ -440,6 +458,36 @@ export function renderStatusText({ report, clientNames, datasets, cacheRoot, std ) } + // Partitions the daemon's last maintenance tick deliberately left + // fragmented (LLP 0224). Rendered only when there are some, like every + // other conditional block here, so an ordinary install's text surface is + // unchanged. `formatEntrypointAge` is the file's coarse-age formatter (it + // is named for its first caller): the question is "is this tick's answer + // hours or weeks old?", not the exact instant. + // @ref LLP 0224#status-file-is-the-surface [implements]: hyp status is where an operator who never runs `hyp query maintain` finds a frozen partition + if (report.maintenance && report.maintenance.skippedTotal > 0) { + const m = report.maintenance + const breakdown = describeMaintenanceSkipReasons(m.reasons) + stdout.write(' maintenance:\n') + stdout.write( + ` ${m.skippedTotal} of ${m.partitionsVisited} partitions left fragmented, as of the tick ${formatEntrypointAge(m.tickAt)} (${breakdown})\n` + ) + for (const p of m.partitions) { + // The count is the one the recorded rewrite ran over, not the live one: + // the sentence is about that rewrite, and `hyp query maintain` draws the + // same distinction. + const detail = + p.reason === 'compaction_attempt_failed' + ? (p.failedAt ? ` the retry failed at ${p.failedAt}` : '') + : (p.dataFiles !== undefined ? ` the last rewrite of ${p.dataFiles} files reduced nothing` : '') + stdout.write(` - ${p.dataset}/${p.partition} [${p.reason}]${detail}\n`) + } + const unnamed = m.skippedTotal - m.partitions.length + if (unnamed > 0) { + stdout.write(` ... and ${unnamed} more (hyp query maintain --dry-run lists them all)\n`) + } + } + // Local entries the central layer overrides (LLP 0031): dropped at // merge, listed here with their reason. Loud, but not an outage signal. // The gateway runs fine on the central config. diff --git a/src/core/daemon/runtime.js b/src/core/daemon/runtime.js index ab7ee341..55e8c5b5 100644 --- a/src/core/daemon/runtime.js +++ b/src/core/daemon/runtime.js @@ -28,7 +28,7 @@ import { writePidFile, } from './pid.js' import { openDaemonLog } from './logs.js' -import { statusFilePath, writeStatusFile } from './status.js' +import { statusFilePath, summarizeMaintenanceSkips, writeStatusFile } from './status.js' /** * @import { AiGatewayCapability, JsonObject } from '../../../hypaware-plugin-kernel-types.js' @@ -677,8 +677,8 @@ export async function runDaemon(opts = {}) { daemon_mode: mode, status: 'ok', }, - async () => { - await maintainCache({ + async (span) => { + const report = await maintainCache({ cacheRoot: boot.runtime.storage.cacheRoot, budgetMs: mCfg.max_tick_ms, config: mCfg, @@ -689,6 +689,30 @@ export async function runDaemon(opts = {}) { storage: boot.runtime.storage, getSettleHook: (dataset) => boot.runtime.query.getDataset(dataset)?.resettleBatch, }) + // @ref LLP 0224#status-file-is-the-surface [implements]: the tick + // stops discarding the report. A partition this walk deliberately + // left fragmented was, until now, a span attribute and nothing + // else, so an operator who did not have tracing on when the tick + // ran had no way to find it at all. + const skips = summarizeMaintenanceSkips(report) + persist({ maintenance: skips }) + span.setAttribute('partitions_visited', skips.partitionsVisited) + span.setAttribute('partitions_skipped', skips.skippedTotal) + if (skips.skippedTotal > 0) { + // The log line is the record and the status file is the + // discovery (LLP 0224#status-file-is-the-surface). One line per + // tick, not one per partition: the counts are the fact, and the + // status file already names the worst of them. + fileLog.warn('daemon.maintenance_skipped', { + partitions_visited: skips.partitionsVisited, + partitions_skipped: skips.skippedTotal, + compaction_ineffective: skips.reasons.compaction_ineffective, + compaction_attempt_failed: skips.reasons.compaction_attempt_failed, + worst: skips.partitions[0] + ? `${skips.partitions[0].dataset}/${skips.partitions[0].partition}` + : null, + }) + } }, { component: 'daemon' } ).catch((err) => { diff --git a/src/core/daemon/status.js b/src/core/daemon/status.js index ec137dd7..489c1b42 100644 --- a/src/core/daemon/status.js +++ b/src/core/daemon/status.js @@ -48,7 +48,8 @@ import { /** * @import { HypAwareV2Config, PluginConfigInstance } from '../../../hypaware-plugin-kernel-types.js' * @import { ClientActionStatus, ConfigControlStatus, ConfigValidationError } from '../../../src/core/config/types.js' - * @import { ClientActionReport, ClientActionsReport, ClientAttachReport, CollectStatusOptions, DaemonStatus, DroppedUpstreamAttribution, HypAwareStatusReport, RecentEntrypoint, ServiceState, SinkSnapshot, SourceSnapshot, StatusDiagnostic } from '../../../src/core/daemon/types.js' + * @import { ClientActionReport, ClientActionsReport, ClientAttachReport, CollectStatusOptions, DaemonStatus, DroppedUpstreamAttribution, HypAwareStatusReport, MaintenanceSkippedPartition, MaintenanceSkipReason, MaintenanceSkipSnapshot, RecentEntrypoint, ServiceState, SinkSnapshot, SourceSnapshot, StatusDiagnostic } from '../../../src/core/daemon/types.js' + * @import { MaintenancePartitionReport, MaintenanceReport } from '../../../src/core/cache/types.js' * @import { Dirent } from 'node:fs' * @import { ClientDescriptor, LoadedManifest, PluginCatalog } from '../../../src/core/types.js' * @import { FolderAskMode } from '../../../src/core/usage-policy/types.js' @@ -426,6 +427,201 @@ export function recentEntrypointsFromSources(sources) { return out.slice(0, MAX_RECENT_ENTRYPOINTS) } +/* ---------- maintenance skips (LLP 0224) ---------- */ + +/** + * How many skipped partitions the standing surface names. The counts beside + * the list are exact, so this bounds the terminal block and the status file + * without hiding the size of the problem; `hyp query maintain` is where an + * operator enumerates every one. Eight is a screenful, and a cache with more + * than eight frozen partitions has a story the count already tells. + */ +export const MAX_SKIPPED_PARTITIONS_REPORTED = 8 + +/** Every reason id, in the order the render lists them. */ +const MAINTENANCE_SKIP_REASONS = Object.freeze( + /** @type {MaintenanceSkipReason[]} */ (['compaction_ineffective', 'compaction_attempt_failed']) +) + +/** + * The reason breakdown as one phrase, e.g. `2 compaction_ineffective, 1 + * compaction_attempt_failed`. Reasons no partition was skipped for are left + * out rather than printed as zeros, and the ids are printed verbatim: they + * are the span attribute names, so this phrase is also the trace query + * (LLP 0224#reason-ids-are-span-attribute-names). + * + * @param {Record} reasons + * @returns {string} + */ +export function describeMaintenanceSkipReasons(reasons) { + return MAINTENANCE_SKIP_REASONS + .filter((reason) => (reasons[reason] ?? 0) > 0) + .map((reason) => `${reasons[reason]} ${reason}`) + .join(', ') +} + +/** + * The partition tuple as one label: exactly the shape `hyp query maintain` + * prints after the dataset name, so the same partition reads identically on + * both surfaces. + * + * @param {Record | undefined} partition + * @returns {string} + */ +function partitionLabel(partition) { + if (!isPlainObject(partition)) return 'all' + const parts = Object.entries(partition) + .filter(([, v]) => typeof v === 'string') + .map(([k, v]) => `${k}=${v}`) + return parts.length > 0 ? parts.join('/') : 'all' +} + +/** + * Why this tick left the partition fragmented, or undefined when it did not. + * + * A partition the tick *rewrote* is not on this surface even when the rewrite + * achieved nothing: that is a run that did work, and the verdict it recorded + * puts the partition on the next tick's snapshot as a skip. What this names is + * the standing state, the partition the kernel has stopped rewriting. + * + * @param {MaintenancePartitionReport} p + * @returns {MaintenanceSkipReason | undefined} + * @ref LLP 0218#verdict-outranks-error [constrained-by]: maintenance already makes the two mutually exclusive, so this order only has to agree about which one a reader is owed if that ever stops holding + */ +function skipReasonOf(p) { + if (p.compacted || p.rebaselined) return undefined + if (p.compactionIneffective) return 'compaction_ineffective' + if (p.compactionAttemptFailed) return 'compaction_attempt_failed' + return undefined +} + +/** + * Summarize a maintenance tick's report into the snapshot the daemon persists + * (`DaemonStatus.maintenance`). Pure, and deliberately cheap: it reads the + * report the walk already produced and stats nothing, because proving a + * skipped partition is also still fragmented is the per-tick cost the LLP 0199 + * baseline gate exists to avoid. + * + * A tick that skipped nothing still produces a snapshot (all-zero counts, an + * empty list). The snapshot is the *current* answer, so a partition that + * thawed has to be able to leave it. + * + * @param {MaintenanceReport} report + * @param {{ at?: string }} [opts] + * @returns {MaintenanceSkipSnapshot} + * @ref LLP 0224#last-tick-only [implements]: one bounded snapshot per tick, named partitions capped and taken in the walk's own neediest-first order + */ +export function summarizeMaintenanceSkips(report, opts = {}) { + const visited = Array.isArray(report?.partitions) ? report.partitions : [] + /** @type {Record} */ + const reasons = { compaction_ineffective: 0, compaction_attempt_failed: 0 } + /** @type {MaintenanceSkippedPartition[]} */ + const partitions = [] + let skippedTotal = 0 + for (const p of visited) { + const reason = skipReasonOf(p) + if (reason === undefined) continue + reasons[reason] += 1 + skippedTotal += 1 + // No sort: the report is already in walk order, which is descending live + // data-file count (LLP 0199#neediest-first), so the first entries past the + // cap are the most fragmented ones by construction. + if (partitions.length >= MAX_SKIPPED_PARTITIONS_REPORTED) continue + partitions.push({ + dataset: p.dataset, + partition: partitionLabel(p.partition), + reason, + // The count the recorded rewrite ran over, not the live one: the same + // distinction `hyp query maintain` draws, for the same reason. + ...(reason === 'compaction_ineffective' && typeof p.compactionIneffectiveFiles === 'number' + ? { dataFiles: p.compactionIneffectiveFiles } + : {}), + ...(reason === 'compaction_attempt_failed' && typeof p.compactionAttemptFailedAt === 'string' + ? { failedAt: p.compactionAttemptFailedAt } + : {}), + }) + } + return { + tickAt: opts.at ?? new Date().toISOString(), + partitionsVisited: visited.length, + skippedTotal, + reasons, + partitions, + } +} + +/** + * Lift the maintenance snapshot out of a status file, or null when no daemon + * has reported a tick for this state root. + * + * Validated, sanitized and re-capped on read as well as on write, for the + * reason `recentEntrypointsFromSources` states: `status.json` is a file, this + * build did not necessarily write it, and everything here is about to be + * printed to a terminal. Dataset and partition labels are the only free-form + * strings on the path and both are kernel-side identifiers, but they are + * cleaned anyway rather than trusted. + * + * Not liveness-gated, for LLP 0164's reason: "these partitions were frozen as + * of the tick at T" stays true after the daemon exits, and the rendered age + * carries the staleness. + * + * @param {DaemonStatus | null} status + * @returns {MaintenanceSkipSnapshot | null} + * @ref LLP 0224#status-file-is-the-surface [implements]: hyp status answers from status.json rather than running a second maintenance walk + */ +export function maintenanceSkipsFromStatus(status) { + const raw = status?.maintenance + if (!isPlainObject(raw)) return null + const tickAt = raw.tickAt + // No timestamp, no snapshot: every render of this block is relative to when + // the tick ran, and "frozen, at some unknown time" is not worth printing. + if (typeof tickAt !== 'string' || Number.isNaN(Date.parse(tickAt))) return null + + const rawReasons = isPlainObject(raw.reasons) ? raw.reasons : {} + /** @type {Record} */ + const reasons = { compaction_ineffective: 0, compaction_attempt_failed: 0 } + for (const reason of MAINTENANCE_SKIP_REASONS) { + reasons[reason] = nonNegativeInt(rawReasons[reason]) ?? 0 + } + + /** @type {MaintenanceSkippedPartition[]} */ + const partitions = [] + const rawPartitions = Array.isArray(raw.partitions) ? raw.partitions : [] + for (const item of rawPartitions) { + if (partitions.length >= MAX_SKIPPED_PARTITIONS_REPORTED) break + if (!isPlainObject(item)) continue + const reason = item.reason + // An unknown reason id is dropped rather than printed: a name this build + // cannot explain is worse than a shorter list, and the counts above still + // account for it. + if (typeof reason !== 'string' || !MAINTENANCE_SKIP_REASONS.includes(/** @type {MaintenanceSkipReason} */ (reason))) continue + const dataset = sanitizeLabel(item.dataset) + const partition = sanitizeLabel(item.partition) + if (dataset === undefined || partition === undefined) continue + const dataFiles = nonNegativeInt(item.dataFiles) + const failedAt = sanitizeLabel(item.failedAt) + partitions.push({ + dataset, + partition, + reason: /** @type {MaintenanceSkipReason} */ (reason), + ...(dataFiles !== undefined ? { dataFiles } : {}), + ...(failedAt !== undefined ? { failedAt } : {}), + }) + } + + const recordedTotal = nonNegativeInt(raw.skippedTotal) + ?? MAINTENANCE_SKIP_REASONS.reduce((sum, reason) => sum + reasons[reason], 0) + return { + tickAt, + partitionsVisited: nonNegativeInt(raw.partitionsVisited) ?? partitions.length, + // The list is capped, so the count leads; but a count smaller than the + // list would render "2 partitions" above three lines of them. + skippedTotal: Math.max(recordedTotal, partitions.length), + reasons, + partitions, + } +} + /** * Resolve the AI gateway's live bound base URL from the on-disk daemon status * snapshot, **guarded by a daemon-liveness check** so a stale snapshot from a @@ -772,6 +968,29 @@ export async function collectHypAwareStatus(opts = {}) { // @ref LLP 0164#not-liveness-gated [implements]: a last-seen timestamp survives its daemon; the rendered age carries the staleness const recentEntrypoints = recentEntrypointsFromSources(daemonStatusFile?.sources) + // ----- partitions maintenance left fragmented (LLP 0224) ----- + // Same route and the same reason as the block above: the daemon runs the + // hourly walk, and `hyp status` reads no cache, so answering this any other + // way would mean firing a second maintenance walk from a status command. + const maintenance = maintenanceSkipsFromStatus(daemonStatusFile) + if (maintenance && maintenance.skippedTotal > 0) { + const one = maintenance.skippedTotal === 1 + const breakdown = describeMaintenanceSkipReasons(maintenance.reasons) + // Warning, never an error: the daemon is running, capture works, and + // queries answer. A frozen partition costs disk and query time, so it is + // a thing to know about rather than an outage, which is why it sits with + // `recent_errors` outside the set that degrades `overall` below. + diagnostics.push({ + severity: 'warning', + kind: 'maintenance_partitions_skipped', + message: `cache maintenance is leaving ${maintenance.skippedTotal} partition${one ? '' : 's'} fragmented (${breakdown}), as of its tick at ${maintenance.tickAt}`, + repair: [ + 'hyp query maintain --dry-run', + 'hyp query maintain --force', + ], + }) + } + // Sinks are derived from the loaded config (so the count reflects // "how many sinks does the user have configured?", the same number // a fresh kernel boot or a running daemon would surface). When @@ -1152,6 +1371,7 @@ export async function collectHypAwareStatus(opts = {}) { usagePolicy, firstSyncHoldDeadline, recentEntrypoints, + maintenance, } } diff --git a/src/core/daemon/types.d.ts b/src/core/daemon/types.d.ts index 16551df2..29fc3d07 100644 --- a/src/core/daemon/types.d.ts +++ b/src/core/daemon/types.d.ts @@ -66,6 +66,70 @@ export interface SinkSnapshot { nextScheduledAt?: string } +/** + * Why a maintenance tick deliberately left a partition fragmented instead + * of rewriting it (LLP 0224#reason-ids-are-span-attribute-names). The ids + * are the `maintenance.partition` span's attribute names verbatim, which + * are themselves named after the `MaintenancePartitionReport` fields, so + * one spelling covers the trace, the status file, `hyp status`, and the + * daemon log. + * + * - `compaction_ineffective`: this writer already rewrote the partition and + * reproduced the same file count (LLP 0217#record-effectiveness). + * - `compaction_attempt_failed`: the one retry the writer generation owed it + * was spent by a rewrite that threw (LLP 0218#report-the-spent-attempt). + * + * Convergence (LLP 0199#baseline-gate) is not on this list: it is the + * healthy majority of a cache, not a partition left fragmented. + */ +export type MaintenanceSkipReason = 'compaction_ineffective' | 'compaction_attempt_failed' + +/** One partition the last maintenance tick left fragmented, and why. */ +export interface MaintenanceSkippedPartition { + dataset: string + /** Partition tuple as `k=v/k=v`, or `all` for an unpartitioned dataset. */ + partition: string + reason: MaintenanceSkipReason + /** + * The data-file count the recorded rewrite ran over, not the live one + * (LLP 0217). Set for `compaction_ineffective` when the cursor records it. + */ + dataFiles?: number + /** + * ISO time the spent attempt failed, as the cursor records it. Set for + * `compaction_attempt_failed`. + */ + failedAt?: string +} + +/** + * What the daemon's last completed maintenance tick left alone, summarized + * for a standing surface (LLP 0224). Overwritten whole by every tick, + * including one that skipped nothing: a skip reason is a state the tick + * re-derives from the partition cursor, so only the newest tick's answer is + * current, and a partition that thaws drops off by itself + * (LLP 0224#last-tick-only). + * + * Absent means no tick has reported (a daemon that has not reached one, or + * maintenance disabled), which is not the same as "nothing is frozen". + */ +export interface MaintenanceSkipSnapshot { + /** ISO time of the tick this snapshot describes. */ + tickAt: string + /** Partitions the tick actually visited (a budget can cut the walk short). */ + partitionsVisited: number + /** Partitions skipped for a stated reason, all reasons summed. */ + skippedTotal: number + /** How many partitions each reason accounts for. Zero keys are kept. */ + reasons: Record + /** + * The worst of them by name, in walk order (LLP 0199#neediest-first, so + * descending live data-file count), capped at + * `MAX_SKIPPED_PARTITIONS_REPORTED`. `skippedTotal` is the true count. + */ + partitions: MaintenanceSkippedPartition[] +} + export interface DaemonStatus { state: DaemonState pid: number @@ -85,6 +149,11 @@ export interface DaemonStatus { configPath?: string sources: SourceSnapshot[] sinks: SinkSnapshot[] + /** + * What the last completed cache-maintenance tick left fragmented, and why + * (LLP 0224#status-file-is-the-surface). Absent until a tick has run. + */ + maintenance?: MaintenanceSkipSnapshot warnings?: string[] } @@ -106,6 +175,7 @@ export type StatusDiagnosticKind = | 'remote_config_rolled_back' | 'local_only_list_unreadable' | 'client_sync_list_unreadable' + | 'maintenance_partitions_skipped' /** * Diagnostic surfaced by `hyp status`. Carries a severity, the @@ -326,6 +396,15 @@ export interface HypAwareStatusReport { * and reads no cache, so this is the only place the answer can come from. */ recentEntrypoints: RecentEntrypoint[] + /** + * What the daemon's last cache-maintenance tick deliberately left + * fragmented (LLP 0224). Read from `status.json`, like + * `recentEntrypoints`: the daemon runs the hourly walk, and `hyp status` + * activates no plugins and reads no cache, so re-deriving this would mean + * running a second walk from a status command. Null when no daemon has + * reported a tick for this state root. + */ + maintenance: MaintenanceSkipSnapshot | null } export interface CollectStatusOptions { diff --git a/test/core/status-maintenance-skips.test.js b/test/core/status-maintenance-skips.test.js new file mode 100644 index 00000000..d4fbbab5 --- /dev/null +++ b/test/core/status-maintenance-skips.test.js @@ -0,0 +1,495 @@ +// @ts-check + +// A partition maintenance deliberately leaves fragmented (LLP 0217, LLP 0218) +// was stated on exactly two surfaces an operator has to go looking for: a +// `hyp query maintain` line and a span attribute. The daemon, which runs the +// walk hourly, threw the report away. These tests pin the standing surface +// that replaces that silence: the snapshot the tick persists, its bound, the +// reason vocabulary it uses, and what `hyp status` does with it. +// @ref LLP 0224#status-file-is-the-surface [tests]: + +import test from 'node:test' +import assert from 'node:assert/strict' +import fs from 'node:fs/promises' +import os from 'node:os' +import path from 'node:path' + +import { + collectHypAwareStatus, + MAX_SKIPPED_PARTITIONS_REPORTED, + maintenanceSkipsFromStatus, + summarizeMaintenanceSkips, + writeStatusFile, +} from '../../src/core/daemon/status.js' +import { runDaemon } from '../../src/core/daemon/runtime.js' +import { renderStatusJson, renderStatusText } from '../../src/core/commands/status.js' +import { maintainCache } from '../../src/core/cache/maintenance.js' +import { appendRowsToSourceTable, readCursorSync, writeCursor } from '../../src/core/cache/partition.js' +import { defaultConfigPath } from '../../src/core/config/schema.js' + +/** + * @import { ColumnSpec } from '../../hypaware-plugin-kernel-types.js' + * @import { MaintenancePartitionReport, MaintenanceReport, PartitionCursor } from '../../src/core/cache/types.js' + * @import { CollectStatusOptions, MaintenanceSkipSnapshot } from '../../src/core/daemon/types.js' + */ + +/* ---------- report fixtures ---------- */ + +/** + * One entry of a `maintainCache` report. Defaults are a healthy converged + * partition: the overwhelming majority of every cache, and the thing the + * surface must stay quiet about. + * + * @param {Partial} patch + * @returns {MaintenancePartitionReport} + */ +function partitionReport(patch = {}) { + return { + dataset: 'ai_gateway_messages', + partition: { source: 'claude' }, + path: '/cache/datasets/ai_gateway_messages/source=claude', + snapshotsExpired: 0, + compacted: false, + rowCount: 10, + dataFilesBefore: 4, + dataFilesAfter: 4, + ...patch, + } +} + +/** + * @param {MaintenancePartitionReport[]} partitions + * @returns {MaintenanceReport} + */ +function maintenanceReport(partitions) { + return { + partitions, + totalSnapshotsExpired: 0, + totalCompacted: partitions.filter((p) => p.compacted).length, + totalRebaselined: partitions.filter((p) => p.rebaselined).length, + dryRun: false, + elapsedMs: 12, + } +} + +/* ---------- the summary the daemon persists ---------- */ + +// The ids are the `maintenance.partition` span's attribute names, which are +// themselves the report's field names: one spelling across the trace, the +// status file, `hyp status`, and the daemon log. +// @ref LLP 0224#reason-ids-are-span-attribute-names [tests]: +test('summarizeMaintenanceSkips names both skip reasons in the vocabulary the report already uses', () => { + const snapshot = summarizeMaintenanceSkips(maintenanceReport([ + partitionReport({ + dataset: 'ai_gateway_messages', + partition: { source: 'claude' }, + compactionIneffective: true, + compactionIneffectiveFiles: 1521, + }), + partitionReport({ + dataset: 'traces', + partition: { source: 'codex' }, + compactionAttemptFailed: true, + compactionAttemptFailedAt: '2026-08-12T21:55:35.168Z', + }), + partitionReport({ dataset: 'logs', partition: { source: 'claude' } }), + ]), { at: '2026-08-13T09:00:00.000Z' }) + + assert.deepEqual(snapshot, { + tickAt: '2026-08-13T09:00:00.000Z', + partitionsVisited: 3, + skippedTotal: 2, + reasons: { compaction_ineffective: 1, compaction_attempt_failed: 1 }, + partitions: [ + { + dataset: 'ai_gateway_messages', + partition: 'source=claude', + reason: 'compaction_ineffective', + dataFiles: 1521, + }, + { + dataset: 'traces', + partition: 'source=codex', + reason: 'compaction_attempt_failed', + failedAt: '2026-08-12T21:55:35.168Z', + }, + ], + }) +}) + +// Convergence (LLP 0199#baseline-gate) is the healthy case, a rebaseline (LLP +// 0207) is work the tick did, and a rewrite that achieved nothing is a run +// that ran. None of the three is a partition the kernel has stopped +// rewriting, which is what this surface is for. +test('a converged, rebaselined, or freshly rewritten partition is not on the surface', () => { + const snapshot = summarizeMaintenanceSkips(maintenanceReport([ + partitionReport({ dataset: 'logs' }), + partitionReport({ dataset: 'traces', rebaselined: true }), + partitionReport({ dataset: 'metrics', compacted: true, compactionIneffective: true, compactionIneffectiveFiles: 12 }), + ])) + + assert.equal(snapshot.skippedTotal, 0) + assert.deepEqual(snapshot.partitions, []) + assert.deepEqual(snapshot.reasons, { compaction_ineffective: 0, compaction_attempt_failed: 0 }) + assert.equal(snapshot.partitionsVisited, 3) +}) + +// The status file is read back and printed to a terminal, so the named list +// is bounded. The counts beside it are not, so the bound never hides the size +// of the problem. +// @ref LLP 0224#last-tick-only [tests]: +test('the named list is capped in the walk order the tick already used, and the count stays exact', () => { + const snapshot = summarizeMaintenanceSkips(maintenanceReport( + // Walk order is descending live data-file count (LLP 0199#neediest-first), + // so position in the report is the ranking: index 0 is the worst. + Array.from({ length: 20 }, (_, i) => partitionReport({ + dataset: `dataset_${String(i).padStart(2, '0')}`, + compactionIneffective: true, + compactionIneffectiveFiles: 500 - i, + })) + )) + + assert.equal(snapshot.skippedTotal, 20) + assert.equal(snapshot.reasons.compaction_ineffective, 20) + assert.equal(snapshot.partitions.length, MAX_SKIPPED_PARTITIONS_REPORTED) + assert.equal(MAX_SKIPPED_PARTITIONS_REPORTED, 8) + assert.equal(snapshot.partitions[0].dataset, 'dataset_00') + assert.equal(snapshot.partitions[7].dataset, 'dataset_07') +}) + +// The retention rule is "the last tick, whole": a skip reason is a state the +// tick re-derives from the cursor, so a partition that thawed has to be able +// to leave the surface without an expiry rule. +test('a tick that skipped nothing still produces a snapshot, so the surface clears itself', () => { + const snapshot = summarizeMaintenanceSkips(maintenanceReport([partitionReport({ compacted: true })])) + assert.equal(snapshot.skippedTotal, 0) + assert.equal(snapshot.partitionsVisited, 1) + assert.ok(!Number.isNaN(Date.parse(snapshot.tickAt)), 'a snapshot always timestamps its tick') +}) + +/* ---------- reading a file this build did not necessarily write ---------- */ + +test('maintenanceSkipsFromStatus is null for a daemon that has never reported a tick', () => { + assert.equal(maintenanceSkipsFromStatus(null), null) + assert.equal(maintenanceSkipsFromStatus(/** @type {any} */ ({ state: 'healthy' })), null) + // No usable tick timestamp: every render of the block is relative to it. + assert.equal( + maintenanceSkipsFromStatus(/** @type {any} */ ({ maintenance: { skippedTotal: 3, partitions: [] } })), + null + ) + assert.equal( + maintenanceSkipsFromStatus(/** @type {any} */ ({ maintenance: { tickAt: 'not a date' } })), + null + ) +}) + +test('a foreign status file is capped, cleaned, and stripped of reasons this build cannot explain', () => { + const ESC = String.fromCharCode(27) + const LF = String.fromCharCode(10) + const snapshot = maintenanceSkipsFromStatus(/** @type {any} */ ({ + maintenance: { + tickAt: '2026-08-13T09:00:00.000Z', + partitionsVisited: -4, + skippedTotal: 900, + reasons: { compaction_ineffective: 900, compaction_attempt_failed: 'lots', compaction_from_the_future: 5 }, + partitions: [ + { dataset: `ai_gateway_messages${ESC}[2K${LF} daemon: FORGED`, partition: 'source=claude', reason: 'compaction_ineffective', dataFiles: 1521 }, + { dataset: 'traces', partition: 'source=codex', reason: 'compaction_from_the_future' }, + { dataset: 'logs', reason: 'compaction_ineffective' }, + null, + ...Array.from({ length: 40 }, (_, i) => ({ + dataset: `dataset_${i}`, + partition: 'source=claude', + reason: 'compaction_attempt_failed', + failedAt: '2026-08-12T21:55:35.168Z', + })), + ], + }, + })) + + assert.ok(snapshot) + assert.equal(snapshot.partitions.length, MAX_SKIPPED_PARTITIONS_REPORTED) + assert.equal(snapshot.partitions[0].dataset.includes(ESC), false, 'no escape byte survives') + assert.equal(snapshot.partitions[0].dataset.includes(LF), false, 'no newline survives') + // An unknown reason id and an entry with no partition tuple are dropped + // rather than printed: a line this build cannot explain is worse than a + // shorter list. + assert.deepEqual( + snapshot.partitions.map((p) => p.reason), + ['compaction_ineffective', ...Array.from({ length: 7 }, () => 'compaction_attempt_failed')] + ) + // Unknown reason keys never reach the counts, and a non-numeric one reads + // as zero rather than as text on a terminal. + assert.deepEqual(snapshot.reasons, { compaction_ineffective: 900, compaction_attempt_failed: 0 }) + assert.equal(snapshot.partitionsVisited, MAX_SKIPPED_PARTITIONS_REPORTED, 'a negative count is not a count') + assert.equal(snapshot.skippedTotal, 900) +}) + +test('a total smaller than the list it labels is corrected upward, never rendered as a lie', () => { + const snapshot = maintenanceSkipsFromStatus(/** @type {any} */ ({ + maintenance: { + tickAt: '2026-08-13T09:00:00.000Z', + skippedTotal: 1, + reasons: {}, + partitions: [ + { dataset: 'a', partition: 'source=claude', reason: 'compaction_ineffective' }, + { dataset: 'b', partition: 'source=claude', reason: 'compaction_ineffective' }, + { dataset: 'c', partition: 'source=claude', reason: 'compaction_ineffective' }, + ], + }, + })) + assert.ok(snapshot) + assert.equal(snapshot.skippedTotal, 3) +}) + +/* ---------- hyp status ---------- */ + +async function makeHome() { + const hypHome = await fs.mkdtemp(path.join(os.tmpdir(), 'hyp-status-maintenance-')) + const stateRoot = path.join(hypHome, 'hypaware') + await fs.mkdir(path.join(stateRoot, 'run'), { recursive: true }) + await fs.writeFile(defaultConfigPath(hypHome), JSON.stringify({ version: 2, plugins: [] }) + '\n') + return { hypHome, stateRoot } +} + +/** + * @param {string} hypHome + * @returns {CollectStatusOptions} + */ +function collectOpts(hypHome) { + return { + env: { ...process.env, HYP_HOME: hypHome, HYP_CONFIG: '' }, + platform: 'darwin', + isLaunchAgentInstalled: () => false, + } +} + +/** @returns {{ write(chunk: string): void, text(): string }} */ +function buffer() { + /** @type {string[]} */ + const chunks = [] + return { write: (chunk) => { chunks.push(chunk) }, text: () => chunks.join('') } +} + +/** @param {string} stateRoot @param {MaintenanceSkipSnapshot | undefined} maintenance */ +function writeDaemonStatus(stateRoot, maintenance) { + writeStatusFile(stateRoot, /** @type {any} */ ({ + state: 'healthy', + sources: [], + sinks: [], + ...(maintenance ? { maintenance } : {}), + })) +} + +test('hyp status names the frozen partitions, and says so without calling the install broken', async () => { + const { hypHome, stateRoot } = await makeHome() + try { + writeDaemonStatus(stateRoot, { + tickAt: new Date(Date.now() - 5 * 60_000).toISOString(), + partitionsVisited: 12, + skippedTotal: 3, + reasons: { compaction_ineffective: 2, compaction_attempt_failed: 1 }, + partitions: [ + { dataset: 'ai_gateway_messages', partition: 'source=claude', reason: 'compaction_ineffective', dataFiles: 1521 }, + { dataset: 'traces', partition: 'source=codex', reason: 'compaction_attempt_failed', failedAt: '2026-08-12T21:55:35.168Z' }, + ], + }) + + const report = await collectHypAwareStatus(collectOpts(hypHome)) + assert.equal(report.maintenance?.skippedTotal, 3) + + const diagnostic = report.diagnostics.find((d) => d.kind === 'maintenance_partitions_skipped') + assert.ok(diagnostic, 'a frozen partition raises a diagnostic an operator scanning status will see') + assert.equal(diagnostic.severity, 'warning') + assert.match(diagnostic.message, /3 partitions fragmented|leaving 3 partitions fragmented/) + assert.ok(diagnostic.repair.includes('hyp query maintain --force')) + // The daemon is running and capture works: this is a thing to know + // about, not an outage. + assert.equal(report.overall, 'healthy') + + const stdout = buffer() + renderStatusText({ report, clientNames: [], datasets: [], cacheRoot: path.join(stateRoot, 'cache'), stdout }) + const text = stdout.text() + assert.match(text, /maintenance:/) + assert.match(text, /3 of 12 partitions left fragmented, as of the tick 5m ago \(2 compaction_ineffective, 1 compaction_attempt_failed\)/) + assert.match(text, /- ai_gateway_messages\/source=claude {2}\[compaction_ineffective\] {2}the last rewrite of 1521 files reduced nothing/) + assert.match(text, /- traces\/source=codex {2}\[compaction_attempt_failed\] {2}the retry failed at 2026-08-12T21:55:35\.168Z/) + // Two of three are named, so the render says where the third is. + assert.match(text, /\.\.\. and 1 more \(hyp query maintain --dry-run lists them all\)/) + + const json = renderStatusJson({ report, clientNames: [], datasets: [], cacheRoot: path.join(stateRoot, 'cache') }) + assert.equal(json.maintenance?.skipped_total, 3) + assert.equal(json.maintenance?.partitions_visited, 12) + assert.deepEqual(json.maintenance?.reasons, { compaction_ineffective: 2, compaction_attempt_failed: 1 }) + assert.deepEqual(json.maintenance?.skipped[0], { + dataset: 'ai_gateway_messages', + partition: 'source=claude', + reason: 'compaction_ineffective', + data_files: 1521, + }) + } finally { + await fs.rm(hypHome, { recursive: true, force: true }) + } +}) + +test('an install with nothing frozen keeps the V1 text surface, and a daemon that never ran reports null', async () => { + const { hypHome, stateRoot } = await makeHome() + try { + // A tick ran and skipped nothing. + writeDaemonStatus(stateRoot, { + tickAt: new Date().toISOString(), + partitionsVisited: 12, + skippedTotal: 0, + reasons: { compaction_ineffective: 0, compaction_attempt_failed: 0 }, + partitions: [], + }) + const clean = await collectHypAwareStatus(collectOpts(hypHome)) + assert.equal(clean.maintenance?.skippedTotal, 0) + assert.equal(clean.diagnostics.some((d) => d.kind === 'maintenance_partitions_skipped'), false) + const stdout = buffer() + renderStatusText({ report: clean, clientNames: [], datasets: [], cacheRoot: path.join(stateRoot, 'cache'), stdout }) + assert.doesNotMatch(stdout.text(), /maintenance:/) + + // No tick has ever reported. Absent is not "nothing is frozen", so + // nothing is claimed either way. + writeDaemonStatus(stateRoot, undefined) + const silent = await collectHypAwareStatus(collectOpts(hypHome)) + assert.equal(silent.maintenance, null) + const json = renderStatusJson({ report: silent, clientNames: [], datasets: [], cacheRoot: path.join(stateRoot, 'cache') }) + assert.equal(json.maintenance, null) + } finally { + await fs.rm(hypHome, { recursive: true, force: true }) + } +}) + +/* ---------- against a real frozen partition ---------- */ + +/** @type {ColumnSpec[]} */ +const SESSION_COLUMNS = [ + { name: 'id', type: 'INT32', nullable: false }, + { name: 'session_id', type: 'STRING', nullable: false }, + { name: 'attributes', type: 'STRING', nullable: true }, +] + +/** + * The production shape behind #723: one identity-partitioned tuple per + * session, so the partition already sits on its file-count floor and every + * rewrite reproduces it. + */ +const SESSION_DECLARATION = { + source: { columns: ['source'] }, + iceberg: { fields: [{ column: 'session_id', transform: 'identity' }] }, +} + +/** + * Seed a partition and plant the cursor a rewrite that achieved nothing + * leaves behind: the baseline sits on the live count (so the LLP 0199 gate + * skips it), the record says the rewrite started from that same count (so the + * verdict is "reduced nothing"), and the stamp names the writer running now + * (so no retry is owed). + * + * @param {string} cacheRoot + * @param {number} sessions + * @returns {Promise} the partition directory + */ +async function seedFrozenPartition(cacheRoot, sessions) { + const rows = Array.from({ length: sessions }, (_, i) => ({ + id: i, + session_id: `s-${i}`, + attributes: `{"gateway":{"session":"s-${i}"}}`, + })) + await appendRowsToSourceTable( + cacheRoot, 'ai_gateway_messages', ['source=claude'], SESSION_COLUMNS, rows, + { declaration: SESSION_DECLARATION } + ) + const dir = path.join(cacheRoot, 'datasets', 'ai_gateway_messages', 'source=claude') + const cursor = readCursorSync(dir) + /** @type {PartitionCursor} */ + const next = { + ...cursor, + compaction: { + previousTableDir: 'table', + compactedAt: '2026-08-12T21:55:35.168Z', + resettleBaselineFiles: sessions, + dataFilesBefore: sessions, + writerGeneration: 2, + }, + } + await writeCursor(dir, next) + return dir +} + +// The vocabulary is only worth anything if it is the vocabulary a real walk +// produces. This runs maintenance over a partition frozen exactly the way +// #723's was and summarizes what comes back. +test('a real maintenance walk over a frozen partition summarizes into the reason it reported', async () => { + const cacheRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'hyp-maintenance-frozen-')) + try { + await seedFrozenPartition(cacheRoot, 6) + const report = await maintainCache({ cacheRoot, compactOnly: true }) + assert.equal(report.totalCompacted, 0, 'fixture invariant: the partition must be skipped, not rewritten') + + const snapshot = summarizeMaintenanceSkips(report) + assert.equal(snapshot.skippedTotal, 1) + assert.deepEqual(snapshot.reasons, { compaction_ineffective: 1, compaction_attempt_failed: 0 }) + assert.deepEqual(snapshot.partitions, [{ + dataset: 'ai_gateway_messages', + partition: 'source=claude', + reason: 'compaction_ineffective', + dataFiles: 6, + }]) + } finally { + await fs.rm(cacheRoot, { recursive: true, force: true }) + } +}) + +// The defect itself: `runMaintenance` awaited `maintainCache` and dropped the +// result, so nothing the walk decided reached any surface the daemon keeps. +// @ref LLP 0224#status-file-is-the-surface [tests]: the tick persists what it left fragmented +test('the daemon maintenance tick persists what it left fragmented into status.json', async () => { + const hypHome = await fs.mkdtemp(path.join(os.tmpdir(), 'hyp-daemon-maintenance-')) + const stateRoot = path.join(hypHome, 'hypaware') + let handle + try { + await seedFrozenPartition(path.join(stateRoot, 'cache'), 6) + const configPath = defaultConfigPath(hypHome) + await fs.mkdir(path.dirname(configPath), { recursive: true }) + await fs.writeFile(configPath, JSON.stringify({ + version: 2, + plugins: [], + // 0.01 minutes: the maintenance timer is not clamped the way the sink + // tick is, so a tick lands inside the wait below. + query: { cache: { maintenance: { interval_minutes: 0.01 } } }, + })) + + handle = await runDaemon({ + hypHome, + configPath, + env: { ...process.env, HYP_HOME: hypHome }, + runId: 'maintenance-skip-tick', + tickIntervalMs: 1, + installSignalHandlers: false, + }) + + const deadline = Date.now() + 30_000 + /** @type {MaintenanceSkipSnapshot | null} */ + let maintenance = null + while (Date.now() < deadline) { + await new Promise((resolve) => setTimeout(resolve, 100)) + const raw = await fs.readFile(path.join(stateRoot, 'run', 'status.json'), 'utf8').catch(() => null) + maintenance = raw ? (JSON.parse(raw).maintenance ?? null) : null + if (maintenance && maintenance.skippedTotal > 0) break + } + assert.ok(maintenance, 'no maintenance snapshot ever reached status.json') + assert.equal(maintenance.skippedTotal, 1) + assert.deepEqual(maintenance.reasons, { compaction_ineffective: 1, compaction_attempt_failed: 0 }) + assert.equal(maintenance.partitions[0].dataset, 'ai_gateway_messages') + assert.equal(maintenance.partitions[0].reason, 'compaction_ineffective') + } finally { + if (handle) { + await handle.stop() + await handle.done + } + await fs.rm(hypHome, { recursive: true, force: true }) + } +}) From 69a05160977ac89022878459abbae21662b3fc3b Mon Sep 17 00:00:00 2001 From: test Date: Fri, 14 Aug 2026 00:33:03 +0000 Subject: [PATCH 2/3] Sanitize maintenance skip labels on write, not only on read (#742) --- ...e-skips-are-a-standing-surface.decision.md | 10 ++ src/core/daemon/status.js | 36 +++++- test/core/status-maintenance-skips.test.js | 122 +++++++++++++++++- 3 files changed, 163 insertions(+), 5 deletions(-) diff --git a/llp/0224-maintenance-skips-are-a-standing-surface.decision.md b/llp/0224-maintenance-skips-are-a-standing-surface.decision.md index 8d24cf8f..0a7eeb6f 100644 --- a/llp/0224-maintenance-skips-are-a-standing-surface.decision.md +++ b/llp/0224-maintenance-skips-are-a-standing-surface.decision.md @@ -122,6 +122,16 @@ the surface and none of the interesting ones. A rebaseline (LLP 0207) is work the tick performed, not a partition it left alone. The surface names only the partitions the kernel knows are still fragmented and has stopped rewriting. +A partition whose rewrite throws on every tick is knowingly deferred, not +named here. On the ordinary growth path a throwing rewrite writes no LLP +0217 retry stamp, so the tick re-attempts and re-throws forever without +`compactionAttemptFailed` ever being set, and this surface says nothing +about it. That is the correct behavior for the two reasons this document +settled (an operator sees nothing rather than a false "0 skipped"), not a +defect in either; a third id for a tick's own error is the additive +extension this decision's Consequences section already anticipates, left +for whoever picks it up next. + ## Consequences {#consequences} - `hyp status` on an install with no frozen partition is unchanged: the text diff --git a/src/core/daemon/status.js b/src/core/daemon/status.js index 489c1b42..e09de468 100644 --- a/src/core/daemon/status.js +++ b/src/core/daemon/status.js @@ -450,14 +450,25 @@ const MAINTENANCE_SKIP_REASONS = Object.freeze( * are the span attribute names, so this phrase is also the trace query * (LLP 0224#reason-ids-are-span-attribute-names). * + * Both call sites interpolate this unconditionally into a sentence that + * already committed to a parenthetical, so an empty phrase would render as a + * bare `()`. That is unreachable from a snapshot this build wrote (every + * skip has one of the two known reasons by construction), but not from a + * `status.json` a later build wrote: LLP 0224#consequences names a third + * reason id as exactly the kind of extension this shape absorbs, and a + * snapshot whose only nonzero reasons are ones this build does not + * recognize is precisely `skippedTotal > 0` with every known count at zero. + * The fallback names that case instead of leaving the parenthetical empty. + * * @param {Record} reasons * @returns {string} */ export function describeMaintenanceSkipReasons(reasons) { - return MAINTENANCE_SKIP_REASONS + const phrase = MAINTENANCE_SKIP_REASONS .filter((reason) => (reasons[reason] ?? 0) > 0) .map((reason) => `${reasons[reason]} ${reason}`) .join(', ') + return phrase === '' ? 'reasons this build does not recognize' : phrase } /** @@ -528,8 +539,20 @@ export function summarizeMaintenanceSkips(report, opts = {}) { // cap are the most fragmented ones by construction. if (partitions.length >= MAX_SKIPPED_PARTITIONS_REPORTED) continue partitions.push({ - dataset: p.dataset, - partition: partitionLabel(p.partition), + // Sanitized here too, not only on read: LLP 0224#last-tick-only says the + // cap and the sanitizing are both re-applied on read, which only holds + // if the write side already produced a clean label. `dataset` and + // `partition` are kernel-side identifiers in the ordinary case, but + // `partition`'s values come off a captured row's `client_name` by way + // of `resolveSourceSegments` -> `sanitizePathSegment`, which strips only + // path-hostile bytes and applies no length clamp or bidi/zero-width + // filtering. Unsanitized here, the daemon log line at + // `runtime.js`'s `worst` field (which reads `partitions[0]` straight) + // would be the one surface on this path with nothing downstream to + // clean it. + // @ref LLP 0224#last-tick-only [implements]: the write side sanitizes and clamps, not only the read side + dataset: sanitizeLabel(p.dataset) ?? 'unknown', + partition: sanitizeLabel(partitionLabel(p.partition)) ?? 'all', reason, // The count the recorded rewrite ran over, not the live one: the same // distinction `hyp query maintain` draws, for the same reason. @@ -613,7 +636,12 @@ export function maintenanceSkipsFromStatus(status) { ?? MAINTENANCE_SKIP_REASONS.reduce((sum, reason) => sum + reasons[reason], 0) return { tickAt, - partitionsVisited: nonNegativeInt(raw.partitionsVisited) ?? partitions.length, + // Floored at the skipped total (and the named list, which the total + // itself is already floored at below): "visited" can never be smaller + // than "skipped", or the render says "5 of 0 partitions" for a snapshot + // no tick could have produced. A file this build did not write can claim + // whatever it wants here, so the floor is enforced rather than trusted. + partitionsVisited: Math.max(nonNegativeInt(raw.partitionsVisited) ?? 0, recordedTotal, partitions.length), // The list is capped, so the count leads; but a count smaller than the // list would render "2 partitions" above three lines of them. skippedTotal: Math.max(recordedTotal, partitions.length), diff --git a/test/core/status-maintenance-skips.test.js b/test/core/status-maintenance-skips.test.js index d4fbbab5..4c29bc69 100644 --- a/test/core/status-maintenance-skips.test.js +++ b/test/core/status-maintenance-skips.test.js @@ -117,6 +117,61 @@ test('summarizeMaintenanceSkips names both skip reasons in the vocabulary the re }) }) +// The cap on write bounds entry *count*; this pins that the write side also +// sanitizes and clamps the bytes *inside* each entry, not only the count. +// LLP 0224#last-tick-only says the cap and the sanitizing are "re-applied on +// read as well as on write", which presumes the write side already produced +// something clean. Partition labels are row-derived +// (`resolveSourceSegments` -> `sanitizePathSegment`, which strips only +// path-hostile bytes, not bidi/zero-width/DEL, and applies no length +// clamp), so a hostile `client_name` reaches `summarizeMaintenanceSkips` +// unsanitized. The daemon log's `worst` field reads `partitions[0]` straight +// off this snapshot with no read-side cleanup of its own, so this is also +// the only thing standing between a row-derived label and `tail -f`. +// @ref LLP 0224#last-tick-only [tests]: the write side sanitizes and clamps too, not only the read side +test('summarizeMaintenanceSkips sanitizes and clamps hostile dataset and partition labels on write, into status.json itself', async () => { + const BIDI = String.fromCharCode(0x202e) // right-to-left override + const ZW = String.fromCharCode(0x200b) // zero-width space + const SHY = String.fromCharCode(0x00ad) // soft hyphen + const DEL = String.fromCharCode(0x7f) + const hostile = `claude${BIDI}${ZW}${SHY}${DEL}` + 'x'.repeat(200) + + const snapshot = summarizeMaintenanceSkips(maintenanceReport([ + partitionReport({ + dataset: hostile, + partition: { source: hostile }, + compactionIneffective: true, + compactionIneffectiveFiles: 3, + }), + ])) + + assert.equal(snapshot.partitions.length, 1) + const [p] = snapshot.partitions + for (const bad of [BIDI, ZW, SHY, DEL]) { + assert.equal(p.dataset.includes(bad), false, 'dataset must not carry hostile bytes') + assert.equal(p.partition.includes(bad), false, 'partition must not carry hostile bytes') + } + assert.ok(p.dataset.length <= 120, `dataset must be clamped, got ${p.dataset.length} chars`) + assert.ok(p.partition.length <= 120, `partition must be clamped, got ${p.partition.length} chars`) + + // Not only the in-memory snapshot: the bytes that actually land in + // status.json, which is what the daemon writes and what a `tail -f` on + // the daemon log's `worst` field is downstream of. + const hypHome = await fs.mkdtemp(path.join(os.tmpdir(), 'hyp-status-maintenance-sanitize-')) + try { + const stateRoot = path.join(hypHome, 'hypaware') + await fs.mkdir(path.join(stateRoot, 'run'), { recursive: true }) + writeStatusFile(stateRoot, /** @type {any} */ ({ state: 'healthy', sources: [], sinks: [], maintenance: snapshot })) + const raw = await fs.readFile(path.join(stateRoot, 'run', 'status.json'), 'utf8') + for (const bad of [BIDI, ZW, SHY, DEL]) { + assert.equal(raw.includes(bad), false, `status.json bytes must not carry ${JSON.stringify(bad)}`) + } + assert.ok(raw.length < 2000, `a single hostile label must not balloon status.json, got ${raw.length} bytes`) + } finally { + await fs.rm(hypHome, { recursive: true, force: true }) + } +}) + // Convergence (LLP 0199#baseline-gate) is the healthy case, a rebaseline (LLP // 0207) is work the tick did, and a rewrite that achieved nothing is a run // that ran. None of the three is a partition the kernel has stopped @@ -221,7 +276,11 @@ test('a foreign status file is capped, cleaned, and stripped of reasons this bui // Unknown reason keys never reach the counts, and a non-numeric one reads // as zero rather than as text on a terminal. assert.deepEqual(snapshot.reasons, { compaction_ineffective: 900, compaction_attempt_failed: 0 }) - assert.equal(snapshot.partitionsVisited, MAX_SKIPPED_PARTITIONS_REPORTED, 'a negative count is not a count') + // A negative count is not a count, and 900 partitions were recorded + // skipped, so visited is floored at the skipped total rather than at the + // capped list length: "8 of 900" is exactly the impossible-looking + // sentence a missing floor would render. + assert.equal(snapshot.partitionsVisited, 900, 'partitionsVisited must never read smaller than skippedTotal') assert.equal(snapshot.skippedTotal, 900) }) @@ -362,6 +421,67 @@ test('an install with nothing frozen keeps the V1 text surface, and a daemon tha } }) +// `describeMaintenanceSkipReasons` filters to reasons in the known +// vocabulary with a nonzero count and joins; when a status.json's only +// nonzero counts are for reasons this build does not recognize, the join is +// '', and both `hyp status`'s text render and its +// `maintenance_partitions_skipped` diagnostic interpolate that +// unconditionally, so the rendered line ends "fragmented ()". This is +// unreachable from a file this build wrote (every skip it records has one of +// the two known reasons by construction), but the read path's whole +// justification is that this build did not necessarily write the file, and +// LLP 0224#consequences names exactly this extension ("a new id and a new +// count key, not a new shape") as the forward-compatible path. The existing +// foreign-file test above keeps `compaction_ineffective: 900` alongside its +// unknown reason id, so that case never hits the all-unknown branch. +// +// Separately, `partitionsVisited` fell back to `partitions.length` with no +// floor at `skippedTotal`, while `skippedTotal` is floored at +// `partitions.length`. A status.json omitting `partitionsVisited` (as an +// older or different build might) alongside a positive `skippedTotal` and an +// empty `partitions` list rendered "5 of 0 partitions" - a sentence no tick +// can produce, since visited can never be smaller than skipped. +// @ref LLP 0224#last-tick-only [tests]: a foreign status file's reason breakdown and visited count stay sentences a tick could actually produce +test('a status.json carrying only reasons this build does not recognize renders no empty parenthetical and no impossible visited count', async () => { + const { hypHome, stateRoot } = await makeHome() + try { + writeStatusFile(stateRoot, /** @type {any} */ ({ + state: 'healthy', + sources: [], + sinks: [], + maintenance: { + tickAt: new Date(Date.now() - 5 * 60_000).toISOString(), + // No partitionsVisited at all: a later or different build might not + // have written one either. + skippedTotal: 5, + reasons: { compaction_from_the_future: 5 }, + partitions: [], + }, + })) + + const report = await collectHypAwareStatus(collectOpts(hypHome)) + assert.equal(report.maintenance?.skippedTotal, 5) + // The floor: visited can never render smaller than skipped. + assert.ok( + (report.maintenance?.partitionsVisited ?? 0) >= 5, + `partitionsVisited must be floored at skippedTotal, got ${report.maintenance?.partitionsVisited}` + ) + + const diagnostic = report.diagnostics.find((d) => d.kind === 'maintenance_partitions_skipped') + assert.ok(diagnostic, 'a positive skippedTotal still raises the diagnostic') + assert.doesNotMatch(diagnostic.message, /\(\)/, 'no bare parenthetical in the diagnostic message') + assert.match(diagnostic.message, /reasons this build does not recognize/) + + const stdout = buffer() + renderStatusText({ report, clientNames: [], datasets: [], cacheRoot: path.join(stateRoot, 'cache'), stdout }) + const text = stdout.text() + assert.doesNotMatch(text, /\(\)/, 'no bare parenthetical in the rendered text') + assert.doesNotMatch(text, /5 of 0 partitions/, 'visited must never render smaller than skipped') + } finally { + await fs.rm(hypHome, { recursive: true, force: true }) + } +}) + /* ---------- against a real frozen partition ---------- */ /** @type {ColumnSpec[]} */ From ca4510c8f9c554b104d21fc7ae172f24723e8735 Mon Sep 17 00:00:00 2001 From: test Date: Sat, 15 Aug 2026 01:05:57 +0000 Subject: [PATCH 3/3] Point LLP 0218's Extended-by at 0228, not the renumbered 0226 The forward-ref was written when this document was LLP 0224, and the 0224 to 0226 to 0228 renumbering left the link one step behind. 0226 on master is the prune decision, so the link resolved to the wrong document. --- llp/0218-compaction-failed-attempt-reported.decision.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/llp/0218-compaction-failed-attempt-reported.decision.md b/llp/0218-compaction-failed-attempt-reported.decision.md index fb8f38df..b4a88f43 100644 --- a/llp/0218-compaction-failed-attempt-reported.decision.md +++ b/llp/0218-compaction-failed-attempt-reported.decision.md @@ -6,7 +6,7 @@ **Author:** Kenny / Claude **Date:** 2026-08-13 **Related:** LLP 0217 (the effectiveness verdict and the retry stamp this extends), LLP 0199 (the baseline gate both sit on), LLP 0207 (the recognition path that writes the same stamp) -**Extended-by:** [LLP 0220](./0220-maintenance-walk-survives-a-partition.decision.md) (the tick in which the attempt fails is reported too: `failed` is this tick's error, beside the `compactionAttemptFailed` skip an earlier one recorded); [LLP 0226](./0226-maintenance-skips-are-a-standing-surface.decision.md) (the standing record this document describes stops being a `hyp query maintain` line and a span attribute: the daemon tick summarizes both skip reasons into `status.json`, where `hyp status` finds them) +**Extended-by:** [LLP 0220](./0220-maintenance-walk-survives-a-partition.decision.md) (the tick in which the attempt fails is reported too: `failed` is this tick's error, beside the `compactionAttemptFailed` skip an earlier one recorded); [LLP 0228](./0228-maintenance-skips-are-a-standing-surface.decision.md) (the standing record this document describes stops being a `hyp query maintain` line and a span attribute: the daemon tick summarizes both skip reasons into `status.json`, where `hyp status` finds them) > The stamp a failed compaction retry writes records the moment the attempt > failed, so the ticks that skip the partition afterwards can say why. A