[WIP] CAS draft (adopting to CI/CD, not for review / merge)#2073
Draft
filimonov wants to merge 2670 commits into
Draft
[WIP] CAS draft (adopting to CI/CD, not for review / merge)#2073filimonov wants to merge 2670 commits into
filimonov wants to merge 2670 commits into
Conversation
filimonov
added a commit
that referenced
this pull request
Jul 17, 2026
…ent_addressed stateless lanes) Config Workflow check failed with 'Workflows are outdated' for master.yml, pull_request.yml, pull_request_community.yml, release_builds.yml. Regenerated via 'python3 -m praktika yaml'. The regeneration adds the two CAS stateless jobs to the generated workflows: 'Stateless tests (arm_binary, content_addressed storage, parallel)' and 'Stateless tests (arm_binary, content_addressed s3 storage, parallel)' (the rustfs-backed lane). CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=927ea142c9cb14759623861eb004261d0b4b1c8f&name_0=PR&name_1=Config+Workflow PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 17, 2026
…teless lane The lane's start_rustfs expected a pre-extracted binary at ci/tmp/rustfs and failed on CI runners where nothing provisions it (the workflow wipes ci/tmp on every run). Download the static musl build for the runner architecture from the RustFS GitHub release (1.0.0-beta.9) when the binary is absent, mirroring how setup_minio.sh downloads minio/mc. Validated locally: the beta.9 binary passes the conditional-operation semantics the CA pool requires (second 'If-None-Match: *' PUT -> 412, wrong-etag conditional DELETE -> 412, right-etag DELETE succeeds), and download_rustfs provisions an executable binary end-to-end. PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 17, 2026
Fast test fails at cmake generation: 'Target "dbms" links to ch_contrib::crc32c but the target was not found' — the fast-test job initializes a limited submodule list that does not include contrib/crc32c, so the unconditional add_contrib is skipped while the dbms link line still references the target. The dependency is dead: it was wired in for per-block CRC32C in the early CAS run-file format (5f1272c), which was later replaced by the text record-stream codecs; no source file includes the library today. Restore the pre-CAS state: crc32c is built only for google-cloud-cpp, and dbms does not link it. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=835251f81cb5af73ad9eaa3a835f50f0c8b678db&name_0=PR&name_1=Fast+test PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 17, 2026
Fast test builds without SSL and failed on the unconditional 'openssl/evp.h' include in CasBlobHashingWriteBuffer.cpp. Wrap the OpenSSL-backed Sha256 hashing write buffer and the one-shot digest in '#if USE_SSL'; on non-SSL builds selecting blob_hash = 'sha256' now fails closed with SUPPORT_IS_DISABLED. CityHash128 and XXH3-128 blob hashes are unaffected. PR: #2073 Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 18, 2026
… test regression) CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=aeb13b24394023fa8cd9d310d4cbcbc308380af1&name_0=PR&name_1=Fast+test PR: #2073 A CAS parser commit grouped `RELOAD_DICTIONARY`/`RELOAD_MODEL`/ `RELOAD_FUNCTION` with `CONTENT_ADDRESSED_GARBAGE_COLLECTION` into a format case that prints only the optional disk, dropping the reload targets: `SYSTEM RELOAD MODEL my_model` formatted as `SYSTEM RELOAD MODEL` (failed 04117_parser_system_query_variants and 04124_parser_system_query_extra in Fast test). Fold all four types back into the generic target-printing case (table / target_model / target_function / disk else-if chain) — for the CA GC command the disk branch produces the identical output. Both stateless tests verified locally via clickhouse-local against their references. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 18, 2026
…als (arm_tidy, T13 batch 1) CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=aeb13b24394023fa8cd9d310d4cbcbc308380af1&name_0=PR&name_1=Build+(arm_tidy) PR: #2073 Removes default arguments from all virtual/override methods flagged by `google-default-arguments` (147 sites: `CasBackend.h` interface, `IObjectStorage.h`/`S3ObjectStorage.h`, all backend implementers, test helpers/fixtures) and adds non-virtual convenience overloads on the base classes that forward the previous default values. Derived classes gain `using` declarations to unhide the base overloads. Qualified parent-implementation calls in test fault backends switched to the explicit 3-arg form — the 2-arg form would now route through the base forwarder and re-enter the derived override virtually (double fault injection; caught by the battery). Bulk edits produced by codex (gpt-5.6-luna) per the T13 brief; overload visibility and qualified-call fixes plus verification by Claude. Battery 919/919 green. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
added a commit
that referenced
this pull request
Jul 18, 2026
CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=aeb13b24394023fa8cd9d310d4cbcbc308380af1&name_0=PR&name_1=Build+(arm_tidy) PR: #2073 Semantics-preserving conformance for the remaining flagged classes: readability-container-contains, readability-isolate-declaration, google-runtime-int (AWS SDK retry-API overrides keep `long` with targeted NOLINT — the override contract owns the type), readability-duplicate-include, cppcoreguidelines-init-variables, cert-msc, modernize-raw-string-literal, modernize-use-starts-ends-with, bugprone-empty-catch (comments only — no new behavior), googletest naming, bugprone-argument-comment, bugprone-optional-value-conversion, bugprone-misplaced-widening-cast (CasTypes.h site audited: not a real precision bug — the value is range-validated to 0-5; cast made explicit without value change). CasRefCowMap's own `contains` keeps its `find` with NOLINT (self-recursion). Bulk edits by codex (gpt-5.6-luna) per the T13 brief (.superpowers/sdd/task-13-batch2-report.md); one over-removed include (PartFolderAccess.h) restored and verification by Claude. Battery 919/919. Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…cs, ca-soak (F1) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…S GC concept) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ot_id/process_id/renewal_sequence/min_active_build_sequence) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nknown-key rejection (F4a) Introduces `ContentAddressedSettings` (pimpl/traits shape mirroring `FileCacheSettings`) holding the 22 `content_addressed` disk keys that currently live as inline `config.getX` calls in `MetadataStorageFactory.cpp`'s `registerContentAddressedMetadataStorage` lambda. `loadFromConfig` rejects unknown non-object-storage keys (fail closed), anchors/defaults `scratch_path`, expands macros in `server_root_id`, and `validate` fails closed on `gc_interval_sec`/ `gc_shards` == 0, an invalid `server_root_id`, and unparseable `blob_hash`/`staging_backend`/`part_folder_validate` values. Refactors `ContentAddressedMetadataStorage::parseStagingBackend` and `parsePartFolderValidate` into `const String &`-taking overloads, with the existing config-taking overloads now thin wrappers so existing callers (the factory, `gtest_cas_s3_staging.cpp`, `gtest_cas_part_folder_access.cpp`) keep compiling unchanged. The factory is not yet wired onto this struct — that is a follow-up task; this change only introduces the struct and its tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… args and duplicated defaults removed (F4b) Also fixes a real bug found while wiring this up: ContentAddressedSettings::validate() now distinguishes an ABSENT `server_root_id` (typed NO_ELEMENTS_IN_CONFIG, matching the pre-F4b factory's behavior) from a PRESENT-but-invalid one (Cas::validateServerRootId's BAD_ARGUMENTS) — loadFromConfig's unconditional macro-expand reassignment was marking the field `.changed` even when the key was absent from config, defeating that distinction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…follow-up) loadFromConfig anchored a relative scratch_path override to default_scratch_path (the per-disk default, e.g. .../disks/<name>/cas_scratch/) instead of the server data path, silently nesting the override two levels deeper than the pre-F4b factory resolved it — a regression hit by shipped configs (e.g. tests/config/config.d/content_addressed_storage_policy_for_merge_tree_by_default.xml's <scratch_path>content_addressed_scratch/</scratch_path>). Splits the single default_scratch_path parameter into scratch_path_anchor_if_relative (server data path, used only to anchor a relative override) and default_scratch_path (used only when the key is absent), mirroring FileCacheSettings::loadFromConfig's cache_path_prefix_if_relative/default_cache_path split. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed by S3ObjectStorage (F5a) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lone build (F5a review) single_attempt_client_base was a raw pointer compared for identity against the current disk client; after a rotation freed the old client, a later rotation could reallocate a new client at the same address and false-match, serving a stale single-attempt clone (e.g. built from retired credentials) indefinitely. Hold it as a shared_ptr instead, pinning at most one retired client version until the next rebuild, which makes the identity comparison sound. Also restructure writeObject's client selection into an if/else chain so getSingleAttemptClient() (which locks and may clone) is only invoked when the SingleAttempt profile is actually selected and no s3_client_override is set, instead of being built eagerly before the override's priority is resolved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… the S3 client pointer (F5b); SDK tripwire -> S3SingleAttemptRetryConsultations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…apshot-publish/anomaly-diag workers (F6) Registers six new ThreadName enum values (CasGcSched, CasGcHeartbeat, CasRemount, CasLeaseKeeper, CasRefSnapPub, CasAnomalyDiag) and names the corresponding background threads at their entry points, so `ps`, gdb, and /proc show a meaningful name instead of the pool's generic worker name. Includes an audit of every ThreadFromGlobalPool spawn site under ContentAddressed/: the four loops (GC scheduler, heartbeat, remount, lease-keeper renewal) and two one-shot maintenance tasks (ref-ledger snapshot publish, pool anomaly diagnostics GET) all now get a name. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… style (F7) Rewritten by codex gpt-5.6-luna (edit-only), verified: 127 events, names and ValueTypes byte-identical, zero 'Counts' leftovers in the Cas block. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…TTL becomes a commented example (F8) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…after_upload_override (F9: same polarity as the setting it overrides) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…family, no drift with isRetryableError) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…closes the F1-F11 consistency pass Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… in ContentAddressedSettings unknown-key gate — stateless-lane startup fix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ess tests and every CAS integration-test config/inline-disk() use No new generic keys surfaced (both sources resolve to path/name/use_fake_transaction, already skip-listed); the enumeration comment now documents all four scan sources as fully covered. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…st rename, backlog entries Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ated, not just /ping A post-fault recovery checkpoint raced a restarting node's async table load and SYSTEM SYNC REPLICA transiently failed with "is not replicated" (observed on the f1f11 5h soak: the high-churn stage made ch1's CA table load slow enough to lose the race). Pre-existing soak-driver hole, not a product regression — the table loads correctly; the driver just asked too early. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n't abort the server (STID 0883) CI PR#2073 (content_addressed storage lanes) crashed with "Too large size (9223372036854775870) passed to allocator" (LOGICAL_ERROR, server abort) running the regression test 04070_no_crash_extreme_compress_block _size. Root cause: an extreme max_compress_block_size (2^63-1) flows into ContentAddressedTransaction::writeFile's buf_size and, unclamped, reaches the CaContentWriteBuffer base-class allocation (Memory::alloc), where checkSize (>= 0x8000000000000000) fires. The ordinary MergeTree writers clamp compress -block sizes to 256 MiB (MergeTreeWriterSettings::MAX_COMPRESS_BLOCK_SIZE) for exactly this reason; the CAS write path received the value unclamped. Fix: clamp buf_size and adaptive_write_buffer_initial_size to 256 MiB at the CAS allocation site (both CaContentWriteBuffer ctors), mirroring the ordinary clamp. New gtest CasContentWriteBuffer.ExtremeBufferSizeIsClampedNotPassedToAllocator reproduces the exact crash number without the clamp (verified RED) and passes with it. CA gtest gate 1057/1057. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2073&sha=latest&name_0=PR Related: #2073
…, cap-rationale truthfulness (minors) Finding 2 (sealed_from completeness): after an unclean recovery publishes a new seal, the `RecoveryResult` kept the pre-seal base's `sealed_from` while `newest_snapshot_id` moved to the seal -- describing the new seal with the predecessor's observed-region bound. Set `result.sealed_from = seal.sealed_from` at that fold. Add a `sealed_from` field to `RefTableRuntime`, copy it in `installRecoveryResult` (so the "copies EVERY field" inventory contract holds), and a `sealedFromForTest` accessor. New `RefWriterRecoverySeal.RuntimeInventoryCarriesSealSealedFrom` asserts the unclean-seal value; `RecoveryResultInventoryComplete` now also pins the clean-recovery `nullopt`. Finding 3 (legacy "pl" rejection): the removed payload field `"pl"` was silently `skipUnknown`'d by the tolerant snapshot committed-row and generic ref-op readers. It is a KNOWN-removed field, not a genuinely-unknown one -- reject it with `CORRUPTED_DATA` naming the field. Negative decode pins added for both. Finding 4 (cap-rationale truthfulness): 64 MiB is the chosen default per-task memory budget, not a cap on blob bodies (only `RefLog`/`RefSnapshot` objects are capped at 64 MiB; an overweight condemned body is admitted exclusively). Corrected the setting description and the pool header/impl comments, and replaced the stale "mutable per-part files ride inside the ref payload" comment with the ref-to-manifest model. Review: tmp/codex_stage1_full_review_result.md (findings 2-4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HKgdqVjZwkpWPxLyHzduPb
…ifecycle (verify residuals) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HKgdqVjZwkpWPxLyHzduPb
…g summary Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HKgdqVjZwkpWPxLyHzduPb
`use_reader_executor` never engages under content-addressed storage: `DiskObjectStorage::readFile` always adds a `file_view` stage for a CA-blob-backed part (a byte window inside a shared blob), and `ReadPipeline::tryBuildReaderExecutor` falls back whenever `file_view` is set — the same reason these tests already skip distributed-cache and encrypted-storage configs. Root-caused during PR#2073 CI triage; not a product bug, just a missing tag (finding A). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…bles scrape
`dump_system_tables()` stops the server and re-reads its data with
`clickhouse local --only-system-tables`, which opens the same disks with
its own (zero) server identity. For a content-addressed disk this hits a
normal (writable) open, which claims server-root ownership
(`Pool::mountWritable` -> `claimOwnerOrThrow`) and fails closed with
"CAS server-root ... is owned by a different server" against the real
server's persisted owner uuid ("Scraping system tables" CI failure,
finding D from PR#2073 triage).
A read-only open (`Pool::open`: `if (!config.read_only) mountWritable(...)`)
skips that claim entirely, which is all a read-only dump needs. Patch the
already-stopped-server's config.xml to add `<readonly>true</readonly>`
right after the CAS marker tag (`<metadata_type>content_addressed
</metadata_type>`), keyed on the tag rather than the disk name so it
covers every content-addressed disk regardless of naming, mirroring the
existing `<log>`/`<errorlog>` sed overrides in this same function.
Verified locally end-to-end: a real server claims ownership of a
local-object-storage-backed CAS root and is stopped; `clickhouse local`
against the same data dir reproduces "owned by a different server
(... ours=00000000000000000000000000000000)" verbatim without the patch,
and completes cleanly with it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ustFS
The CAS mount capability probe (`CasProbe.cpp` step 6) requires the
backend to enforce conditional-DELETE (If-Match) semantics: a delete with
a mismatching token must come back `TokenMismatch`, never succeed. Stock
MinIO honors the mismatched-token delete instead of rejecting it, so the
probe fails closed with `Code 48 NOT_IMPLEMENTED` ~2.4s into server
startup; the server exits, and the harness reports it as a 311s
container-start timeout ("Container status: running") because the
`tail -f /dev/null` entrypoint keeps the container up after clickhouse
dies (PR#2073 CI triage finding).
Swap `with_minio=True` for the already-established `with_rustfs=True`
fixture (same pattern as `test_cas_insert_fault_recovery` /
`test_content_addressed_drop_pool_member`), point the disk at
`rustfs1:11121`, and use `cluster.rustfs_client`/`rustfs_bucket` instead
of the MinIO equivalents.
Verified locally: `python -m ci.praktika run integration --test
test_content_addressed_gc_s3` -- 1 passed in 11.46s.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…blob id `CasBlobInDegree.cpp`/`CasGc.cpp` logged a `Warning`-level "structurally impossible under the ack floor ... investigate" for a delete_pending retired entry that regains in-degree. Root-caused during PR#2073 CI triage (56 occurrences/run, no data loss, all correctly spared): a graduated (delete_pending) blob carries NO surviving prior edges by construction, so any in-degree recovery is necessarily a fresh this-generation edge -- a writer's `observeAndAdmit` point-read of the per-hash meta raced GC's `Condemned` write and adopted the token instead of resurrecting from source (write-once PUT -> 412 -> dedup-adopt). This is the expected shape of that TOCTOU race, not an ack-floor violation; the "investigate" wording was misleading and fired a Warning on every occurrence of an ordinary, already-safely-handled race. Downgrade both log sites to Debug, fix the wording to name the actual cause, and add a dedicated `CasGcRetiredSparedByReref` ProfileEvent (subset of the existing `CasGcRetiredSpared`) so the specific delete_pending-recovered-in-degree case stays observable as a metric without paging on it. Enrich the message with the blob id (the `CasBlobInDegree.cpp` site had none at all) and both the condemn round and the observing round, so a genuinely anomalous case -- one with no plausible dedup-adopt explanation -- can still be told apart by correlating against `system.content_addressed_log`. Scope note: a full attribution fix (join against the actual `BlobReuseAdopt` event, moving the real invariant check to the writer's edge-commit) is a larger, writer-side change and is tracked separately in BACKLOG.md (RECOVERED-INDEGREE-ATTRIBUTION) rather than done here. Verified: full rebuild clean; `unit_tests_dbms --gtest_filter= "*CasBlobInDegree*:*CasGc*:*ProfileEvents*"` -- 172 passed, 0 failed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…superseded v1 (single new classifier branch) was refuted by the first codex gpt-5.6-sol adversarial review: "same uuid + same epoch + unfenced" does not prove our own prior write (the allocateWriterEpoch empty-root hole can hand a second process the same pair), the superseded branch is also a normal fencing outcome that aborts ASan builds, and the codebase already owns the principled cure (putOverwriteControlled's resolve-by-GET contract). v2 design, approved interactively: - Layer 1: resolve-on-ambiguity inline in SingleWriterSlot::renewOnce — on any renewal-PUT exception, ONE GET three-way resolve (bytes==ours -> committed; token==expected -> provably not applied, rethrow transient; else -> confirmed mismatch). Kills the self-race at the source. - Layer 2: exhaustive non-aborting classification in MountLeaseKeeper::onRenewMismatch — new honest same_epoch_state_uncertain branch (ABORTED, forensic seq/pid/hostname), superseded downgraded from LOGICAL_ERROR to ABORTED (TLA already models it as localLost), foreign_writer deliberately stays loud. Heartbeat death-test repartitioned. - Layer 3: allocateWriterEpoch refuses to re-mint epoch 1 while a mount object exists (closes the same-pair two-process hole). Gates before implementation: second adversarial codex review of this spec, TLA+ CaCasMountCore extension (ambiguous-landed-write + resolve + uncertain fence; 3 negative controls), TDD gtests incl. a new apply-then-throw fault backend (the existing transient mock throws before applying and cannot reproduce the bug). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The second codex gpt-5.6-sol adversarial review returned UNSAFE on rev.1 (verdict kept at tmp/codex_review_mountlease_v2_verdict.md). rev.2 resolves every finding, with a traceability map in the spec: 1. (Critical) delayed resolve re-arming a deposed writer -> prepare-time deadline anchoring (D1) everywhere + resolve-commit refuses to re-arm on an elapsed anchored deadline; the safety argument no longer depends on any latency constant. 2. byte-equality not attempt-unique -> per-beat random nonce in the lease body (D2; pre-release format change). 3. native get's incoherent (bytes_newer, token_older) pair + content-ETag ABA -> HEAD-GET-HEAD coherence sandwich (D3); "token == expected" downgraded to "no commit evidence", never proof of non-application. 4. get-absence is non-authoritative -> layer 3 gates on probeSentinelRaw's ProbeOutcome; only authoritative KeyAbsent mints, everything else fails closed. 5. TLA model cannot express the scenarios -> gate 1 scope expanded (process identity separate from uuid, epoch-presence state, request/response split with stale delivery, anchored deadline, fairness stated; negative control (a) is the exact finding-1 trace). 6. layer 3 broke openForDecommission's supported recovery -> explicit policy parameter (NormalMount enforces, DecommissionRecovery bypasses), pinned by a test. 7. untestable fence assertions + missing race test -> gate 2 rewritten: real backgroundLoop (or seam), barrier-controlled stale-resolve race test, mixed HEAD/GET-token test, frozen-clock test, decommission test. Layer 2 (non-aborting classification) was endorsed by round 2 unchanged. Next: targeted third review of the rev.2 deltas (gate 0). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pproved) rev.2's complexity all served one implicit requirement: RESCUE an ambiguous-but-landed renewal in place (resolve -> adopt -> continue), which demanded three proofs (body nonce for attempt identity, HEAD-GET-HEAD read coherence, prepare-time-anchored commit gates) plus a large TLA+ rework. rev.3 restates the problem with plain lease semantics: a lease holder that cannot confirm its renewal is not live -- fence, then the existing battle-tested self-remount. The "did my write land?" question is made irrelevant instead of answered. This is the conservative option the round-2 review itself endorsed. rev.2's full rescue design stays in git history (dc68a7e) in case metrics ever justify it. What remains: - Phase A (the crash fix, round-2-endorsed verbatim): exhaustive non-aborting classification -- new same_epoch_state_uncertain branch (ABORTED), superseded downgraded to ABORTED, foreign_writer deliberately stays loud. - Phase B (~5 lines): prepare-time deadline anchoring, closing the pre-existing response-latency skew by construction (anchor captured before the payload stamp). - Phase C: probe-gated epoch re-mint guard (authoritative KeyAbsent only) with a DecommissionRecovery policy bypass -- round-2 findings 4 and 6 as prescribed. Round-2 findings 1-3 and the race-test half of 7 are MOOT (they attacked the rescue); the TLA gate shrinks to aligning the uncertain branch with the model's existing localLost transition + one guard negative-control. Accepted cost, stated in the spec: an ambiguous-but-landed renewal now costs one self-remount cycle (~36.5 s fenced writes with defaults) instead of an in-place rescue -- observed ~once per 2h only under a deliberately flapping backend in CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The third codex round reviewed rev.2 (pre-simplification); six of its ten findings attacked the dropped rescue machinery and are MOOT under rev.3's fence-not-rescue reframing. The four that survive are now incorporated: - №1 (Critical): the DecommissionRecovery policy is no longer a blind bypass — it requires the surviving mount to be TERMINAL (a live mount refuses, preserving decommission's live-member refusal) and mints an epoch distinct by construction (surviving.writer_epoch + 1), making the same-pair state unrepresentable on that path. New tests + a TLA negative control pin it. - №2 (Critical): Phase B anchoring extended to the startup-arm site — a materialization_grace_ms wait that consumes the TTL forces one fresh conditional claim (fails closed on a successor's token) before arming; zero cost for sane configs. - №4: clock-domain wording made precise — one pre-I/O stamp per domain (wall for the keeper deadline, boottime for the runtime fence), safety stated as the standard per-domain lease argument + the protocol's existing 5% rate allowance; Phase B removes the request-latency term only. - №8/№9 + closing notes: probeSentinelRaw confirmed implemented for both backend modes; fence-latch testability via startBackground + loss callback; all three mountWritable allocation sites take the policy parameter, self-remount stays NormalMount. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…c rev.4) Seven tasks: TLA gate (epoch-wipe twin + re-mint guard + decommission branch, two sabotage configs), Phase A classifier (TDD, repartitioned death tests), Phase A end-to-end (apply-then-throw fault backend through the real backgroundLoop), Phase B keeper anchoring (per-domain pre-I/O stamps), Phase B startup-arm redo after a TTL-consuming materialization grace, Phase C probe-gated epoch guard with the hardened DecommissionRecovery policy, final gate + docs. All code inlined against verified signatures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-mint guard Phase A is model-alignment (confirmed mismatch already maps to localLost). New: WipeEpoch environmental action + guarded RemintEpoch; SabEpochGuardOff sabotage config RED, honest config GREEN, full existing battery unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ollow-up) CaCasMountCore_RESULTS.md conflated the causes of the two dropped honest-config invariants under one "WipeEpoch alone" narrative. Corrected per independent review re-run: WriterEpochMonotoneUnique IS tripped by WipeEpoch alone (depth 6, no re-mint); FenceCostsEpoch is NOT — it requires RemintEpoch's honest branch to also fire, whose literal-1 mint has no distinctness protection against fencedEpochs (unlike RemintEpochDecom's mount.epoch + 1). Added a follow-up note (not implemented) that minting epochCeiling + 1 there would plausibly close the FenceCostsEpoch hole and narrow the residual gap. Comment-only changes across CaCasMountCore_RESULTS.md, CaCasMountCore_stage1.cfg, CaCasMountCore_rev6_observe.cfg, CaCasMountCore_sab_epochwipelive.cfg — no CONSTANTS/INVARIANT lines touched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r aborts (STID 3982-3b48 part 2, Phase A) New exhaustive classification in MountLeaseKeeper::onRenewMismatch: a same-uuid/same-epoch/unfenced body is state uncertainty (ambiguous landed renewal, or a same-pair twin after epoch-state loss) -> ABORTED, fence, self-remount; superseded (same uuid, newer epoch) downgraded from LOGICAL_ERROR to ABORTED (a normal fencing outcome, the TLA model's localLost); foreign uuid deliberately stays LOGICAL_ERROR-loud (protocol- unreachable past the owner anchor). The base-class fallthrough is now unreachable for this keeper and its call is removed. Root cause (CI, Altinity PR#2073 asan CAS-s3): a renewal PUT timed out client-side but landed server-side; the next beat's confirmed mismatch re-read our own live-looking body, matched no branch, and the base LOGICAL_ERROR aborted the server at exception construction. Also repartitions BackgroundLoopFencesImmediatelyOnConfirmedMismatch: its "foreign" fixture was actually same-uuid/same-epoch (the exact fallthrough shape above), so under the new classification it no longer dies; it now observes the fence via on_lost instead of EXPECT_DEATH. Spec: docs/superpowers/specs/2026-07-24-cas-mount-lease-self-race-fix-v2-design.md (rev.4) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…hase A e2e) New ApplyThenThrowPutOverwriteFaultBackend (applies, then throws — the landed-but-unacked case the existing transient mock cannot model) + a real backgroundLoop test reproducing the CI crash shape: beat 1 lands ambiguous, beat 2's confirmed mismatch takes the same_epoch_state_uncertain branch, on_lost latches the fence, no process death. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… (Phase B, keeper) Both deadline sites (the keeper's wall-domain confirmed_deadline_ms and the runtime's boot-domain mayMutate fence) previously refreshed from RESPONSE time, exceeding the durable authorization by the request latency — bounded only by the S3 request timeout (30 s default), not by any protocol constant. prepareRenew now stashes one pre-I/O anchor per clock domain; the success hooks and on_renew_ok(anchor) consume them. A slow ack can no longer extend a local fence past what the durable body it acknowledges authorizes. Also fixes a pre-existing test-isolation leak in Task 3's BackgroundLoopSurvivesAmbiguousLandedRenewal: it flipped DB::abort_on_logical_error to true directly in the test process (unlike the EXPECT_DEATH-wrapped uses elsewhere in this file, which only affect a forked child) and never restored it, so any LOGICAL_ERROR raised by a later test in the same gtest binary aborted instead of throwing. Restored to false before return. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ming grace forces a lease re-write (Phase B, startup) mountWritable armed bootMsNow()+TTL AFTER the (unbounded, operator-configured) materialization grace — a grace longer than the stale-token observation threshold let a successor legally reclaim during the wait while the predecessor then armed an already-superseded claim without revalidation. The arm now anchors at the claim attempt's pre-I/O instant; if the grace consumed the TTL, one fresh conditional lease write (keeperRenewOnce, fails closed via the Phase A classification if the slot changed hands) re-anchors before arming. Zero cost for sane configs (grace << TTL). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…x (Phase B addendum) The self-remount path's write-fence arm still computed its deadline at mount_runtime.bootMsNow() taken at arm time, mirroring the response-time bug mountWritable had before this task. Anchor it at the claim attempt's pre-I/O instant instead, captured at the same site as mountWritable's claim_anchor_boot_ms (right after installKeeper, right before keeperStart()). Unlike mountWritable, this path's materialization-grace wait runs BEFORE that anchor point, not after, so no wait can land between the anchor and the arm — no TTL-consumed redo branch is needed here; the anchor alone suffices (comment states this explicitly). Also fixes a Minor from the task-5 review: the new StartupArmRedoesLeaseWriteWhenGraceConsumesTtl test seeded prior.writer_epoch = 1, colliding with the pool's own first-allocated epoch and silently taking one extra FencedSelf fence-recovery iteration before actually exercising the different-epoch Fenced path the comment claimed. Reseeded at epoch 7 (matching FencedPriorPaysOnlyTmat) and corrected the comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ase B addendum 2) Turns the addendum-1 review's probe into a permanent regression test: CasMountRuntime::bootMsNow re-invokes PoolConfig::boot_ms_fn on every call with zero memoization, so a sequenced fake clock deterministically distinguishes the remount arm's early (anchor) reading from a later (response-time) one, with no real sleep and no threads. Verified TDD against both branches: temporarily swapping in the pre-addendum-1 CasPool.cpp (commit e0ee7af) makes the new test fail with mayMutate() == true (armed from the inflated later reading); the fixed code passes with mayMutate() == false (armed from the anchor, expired exactly at anchor + ttl). Also rewords the remount-arm comment to lead with "no UNBOUNDED wait can land between this anchor and the arm" — quiesceRefTablesForRemount IS a wait, just one bounded by cas_request_budget < ttl, and the old phrasing read as a direct contradiction of that fact two sentences later. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…se C) allocateWriterEpoch's absent-epoch branch minted epoch 1 whenever the data subtrees were empty — ignoring the mount/epoch CONTROL objects, so an epoch object lost under a live mount handed a second same-uuid process the same (uuid, epoch) pair (codex round-2 finding 1). The branch now gates on probeSentinelRaw's AUTHORITATIVE outcomes (get-absence flattens transport faults into not-found and must not gate a lifecycle decision): KeyAbsent mints, Present refuses (CORRUPTED_DATA with recovery guidance), everything else fails closed naming the probe outcome. Decommission (EpochMintPolicy::DecommissionRecovery, derived from MountClaimPolicy::NoWait) is NOT a blind bypass (codex round-3 finding 1): it requires the surviving mount to be TERMINAL (a live member refuses, preserving CasDecommission.RefusesLiveMember semantics) and mints surviving.writer_epoch + 1 — distinct by construction. Also fixes three gtest_cas_pool.cpp tests (UncleanOpenWaitsMaterializationGrace, FencedPriorPaysOnlyTmat, StartupArmRedoesLeaseWriteWhenGraceConsumesTtl) that planted a predecessor's mount lease directly via claimMount/putIfAbsent without ever durably minting the matching epoch object — a test-harness shortcut that is unrepresentable in production (allocateWriterEpoch always runs before the mount claim) and now correctly trips the new guard; seeded the missing epoch object in each to restore a realistic precondition. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hase C follow-up) allocateWriterEpoch's DecommissionRecovery liveness test uses a bare wall-clock comparison, unlike claimMount's reclaim gate in the same file (gc_fenced / clean-farewell / proven-dead-token only, never bare wall-clock, because clock skew can misjudge). This is deliberate and safe, not a regression: the mint is distinct-by-construction (no same-pair state is ever representable), and claimMount's own strong liveness gate runs right after and refuses a genuinely live member regardless — so a clock-skewed misread here can only burn one epoch number on a doomed decommission attempt, never admit a claim. Document that divergence at the call site; the check itself is unchanged. Also fixes a double space after an arrow in CasServerRoot.h's doc comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ine; docs/test precision 1. (behavioral, fail-closed) `SingleWriterSlot::renewOnce` now calls a new protected virtual `onRenewCommitted()` hook right after recording a successful write — reached by EVERY caller of `renewOnce`, background-driven or direct. `MountLeaseKeeper` moves its `refreshConfirmedDeadline` call from `onRenewSucceeded` (background-loop-only) into this new override, so the startup-arm redo in `CasPool.cpp`'s `mountWritable` (which calls `keeperRenewOnce()` directly, never through `onRenewSucceeded`) now also refreshes `confirmed_deadline_ms` from the redo attempt's own wall anchor instead of leaving it stale at the pre-grace anchor. `onRenewSucceeded` now fires only the boot-domain `on_renew_ok` callback. Fixed the now-stale doc comments on `refreshConfirmedDeadline`, the base hooks, and `CasPool.cpp`'s `backgroundLoop` accordingly. Added a failing-first test, `CasHeartbeat.DirectRenewOnceRefreshesConfirmedDeadlineWithoutOnRenewSucceeded`, driving a direct `renewOnce` with no `onRenewSucceeded` call and asserting `shouldFenceOnTransientRenewFailure` reflects the refreshed deadline. 2. (spec text) Softened the Phase A forensics sentence in the mount-lease self-race-fix-v2 design doc to match reality: the exception `message` carries the observed body's identity (`describeMountHolder`) plus OUR local `seq`; the `MountConflict` event carries only the observed body's identity, with `expires_at_ms` (not `started_at_ms`). Also softened Phase B's "precede the payload's own wall-clock stamp" wording to "precede-or-equal (the wall anchor IS the payload stamp)". 3. (doc comments) `CasServerRoot.cpp`'s `onRenewFailed` comment and `CasServerRoot.h`'s `onRenewMismatch` doc now name the full five-way classified-branch split (`fenced_by_gc`/`same_epoch_state_uncertain`/`superseded`/`foreign_writer`/`vanished`) and its non-aborting semantics. 4. (test precision) `gtest_cas_heartbeat.cpp`'s `SameEpochUnfencedTouchIsUncertainNotFatal` now also asserts the local-seq message fragment (`"vs our seq="`) is present, so dropping it from the exception message fails the test. 5. (stale comment) `gtest_cas_pool.cpp`'s `StartupArmRedoesLeaseWriteWhenGraceConsumesTtl` comment corrected: after the zero-write-bootstrap epoch seeding in this test, the pool's own first-allocated `writer_epoch` is 8, not colliding with the seeded epoch-7 prior by construction — no silent `FencedSelf` fence-recovery detour to account for. 6. (TLA results note) `CaCasMountCore_RESULTS.md`'s epochCeiling+1 follow-up note now states that the model's honest `RemintEpoch` omits the code's `serverRootSubtreeEmpty` precondition (`rootEmpty` never returns TRUE in the model after the first write), so the honest-config `FenceCostsEpoch` counterexample over-approximates the code's real exposure — conservative in direction, to revisit together with the epochCeiling+1 refinement. Build: `ninja -C build` green. Tests: `unit_tests_dbms --gtest_filter='CasHeartbeat.*:CasPool.*:CasPoolRemount.*'` — 52/52 pass. Full gate `--gtest_filter='Cas*:CA*'` — 1186/1186 pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKmSZa7T87WbRGKkNkSXky
…ref-lane landing
The parallel stage1 round rewrote the ref lane this spec leans on (chunked flush
via commitRefChunk, counts-only admission caps, streaming recovery, SetPublishedAt).
Re-checked every load-bearing claim against HEAD. Design unchanged; three
substantive updates:
(a) The post-durable-PUT window NARROWED. commitRefChunk now swallows a
post-apply overlay-fold failure as an optimization-only step (CasRefLedger.cpp
:1718-1728, coherent-on-throw), so materializeCommitted is no longer part of
the window. What remains is applyRefLogTxn itself (:1694) throwing on
ALLOCATION failure — COW erase tombstones a base-only row
(CasRefCowMap.cpp:162) and the tracked allocator can throw
MEMORY_LIMIT_EXCEEDED. The outer catch (:1730-1755) fails survivors and
rethrows LOGICAL_ERROR; completeOwnedItemsAndReleaseLeadership (:1100-1126)
clears leader_active — no wedge, no fence: the "durable, unapplied, unwedged,
idle, warm" state is still reachable (release-only; LOGICAL_ERROR aborts under
sanitizers). Phase 7's scope shrinks to the apply step only.
(b) The poison marker must be STICKY — cleared only by a fresh recovery, never by
the next successful flush: the cache stays divergent from durable truth and
later flushes would apply on top of the stale base. Effect: relink from that
sender/table degrades to bytes until recovery (fail-closed, self-limiting).
(c) Chunked flush adds a mid-tenure partially-durable state (chunk 1 applied,
chunks 2..N unwritten). Already covered by the leader_active predicate — not a
hole — but it widens the unknown window under write load. Verified chunk
boundaries fall on ITEM boundaries only (:1448-1454, oversized item rejected
at :1426), so precommitAdd durability-on-return stays atomic.
New finding folded into phase 7: the two post-durable apply sites are asymmetric —
the wedge-resolution arm (:1205+) has no inner swallow, so a materializeCommitted
throw there leaves the txn applied but the wedge unreset => next resolution
re-applies it and double-bumps the tail counters. Same class, same fix.
Re-verified unchanged: precommitAdd still validates only the namespace
(CasPartWriteTxn.cpp:926-929) so the mint-tightening is still required;
~PartWriteTxn still only retires build_seq (:119-124); gate 0's grabOldParts
Deleting transition (MergeTreeData.cpp:3538); publishEntries (:338); the relink
advertise gate (DataPartsExchange.cpp:545) and adoption catch
(ContentAddressedMetadataStorage.cpp:2002). All ledger citations re-anchored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012UJqPB1gNiKk5AKi91zZyY
…OCATIONS_IN_SCOPE (user suggestion)
Phase 7's guarantee was carried by a static_assert(noexcept) plus prose. Adding
the existing upstream mechanism makes it mechanically enforced:
DENY_ALLOCATIONS_IN_SCOPE (Common/MemoryTracker.h:35) sets a thread-local flag
that makes MemoryTracker throw LOGICAL_ERROR on ANY allocation attempt in the
scope (no-op in Release). Direct upstream precedent with the identical intent:
AsyncLoader.cpp:363-364 ("We do not want any exception to be thrown after this
point, because the following code is not exception-safe"), also ThreadPool.cpp:795,
ProcessList, StorageBuffer.
Why it is the right tool here: the defect class is invisible in ordinary testing
because an allocation inside the post-durable window normally SUCCEEDS — nothing
fails until real memory pressure in production. The deny-scope makes the presence
of an allocation the failure signal, so every debug/CI run that commits a ref op
enforces the invariant instead of restating it in a comment.
Spec updates:
- Phase 7: wrap EXACTLY the post-durability install region (no-throw move + atomic
tail-counter bumps) at BOTH call sites (commitRefChunk and wedge resolution).
Scope discipline spelled out: the overlay fold (materializeCommitted, legitimately
allocating and optimization-only), survivor completion under ref_queue_mutex, and
event/ProfileEvents emission stay OUTSIDE (or in an explicit
ALLOW_ALLOCATIONS_IN_SCOPE). Limits stated: debug-only (CI-grade invariant, not a
production guarantee — production safety comes from the restructure) and only
tracker-routed allocations are caught.
- Poison marker (item 1): arm/clear go inside the deny scope too — arming the marker
must never itself be the thing that throws.
- Testing: no dedicated "does it allocate?" test — the whole CAS gtest battery and
the CA stateless lane become the enforcement surface; add only a negative control
proving the guard is armed and the region is entered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012UJqPB1gNiKk5AKi91zZyY
…f the CAS write path) User request: a soak-shaped card with memory_tracker_fault_probability (and cannot_allocate_thread_fault_injection_probability) enabled — everything must stay consistent despite allocation errors. Gap it closes: the suite faults nodes (kill/restart) and the object store (s3faultproxy) but never the ALLOCATOR, so the whole exception-safety class the ref-lane hardening addresses — a throw between a durable PUT and its in-memory apply, leaving the cache diverged from the journal — is invisible today, and in ordinary testing allocations simply succeed. Verified mechanisms: memory_tracker_fault_probability is a per-query Float Setting (Core/Settings.cpp:2312) and is NOT debug-gated (MemoryTracker.cpp:340-342), so it runs on the ordinary RelWithDebInfo soak image and faults allocations inside the CAS commit path; cannot_allocate_thread_fault_injection_probability is a ServerSetting (ServerSettings.cpp:233) applied via SYSTEM RELOAD CONFIG (InterpreterSystemQuery.cpp:1158), reaching the background pools the query-level knob cannot. Three legs (A query-thread faults, B thread-allocation faults, C disarm + quiesce + GC-to-fixpoint + fsck + RESTART with pre/post view equality — the diverged-cache oracle nothing else in the suite catches). Oracle allows query failures but not invariant violations: no LOGICAL_ERROR/abort, acked-vs-lost = 0, replicas agree, fsck clean, GC recovers, no permanently wedged lane, no hung query. Mandatory S39-style soundness guard (nonzero injected failures or inconclusive, never a vacuous pass). Optional debug-image mode additionally enforces every DENY_ALLOCATIONS_IN_SCOPE region under real allocation pressure. Next free id is S42 (s41_wide_insert_baseline.py is taken). Design detail in the publish-confirm spec §testing; this registers it suite-side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012UJqPB1gNiKk5AKi91zZyY
…xes test_cancel_backup) ContentAddressedMetadataStorage::tryFromDisk detected non-CA disks by calling getMetadataStorage and catching its NOT_IMPLEMENTED throw — but constructing a DB::Exception increments system.errors even when caught. The function runs on every asynchronous-metrics tick for every configured disk (the CAS GC-health gauges), so every pure-local server accumulated a steady +N/s stream of invisible NOT_IMPLEMENTED errors — caught by strict-error tests as a stray system.errors entry: test_backup_restore_on_cluster/test_cancel_backup.py's NoTrashChecker failed all 7 subtests across the asan/msan/tsan integration lanes of Altinity PR#2073 (run 30019911967). Fix: check the throw-free `isContentAddressed` predicate first; getMetadataStorage is reached only for genuine CA disks. Also forward `isContentAddressed` through ReadOnlyDiskWrapper (it forwarded getMetadataStorage but not the predicate, so a wrapped CA disk would have silently dropped out of the CAS introspection paths under the new gate). TDD: new gtest CasWiring.TryFromDiskOnLocalDiskIsExceptionFreeAndCountsNoError reads the ErrorCodes counter around the call — failing-first confirmed on the old code. Gate Cas*:CA* 1187/1187; test_cancel_backup::test_cancel_backup re-run locally end-to-end: 1 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…disks_app typo Three fixes from the PR#2073 full CI reconciliation, all verified locally: 1. test_content_addressed_s3 + test_content_addressed_shared_pool: swap with_minio -> with_rustfs (same class and recipe as test_content_addressed_gc_s3's fix): the CA mount capability probe requires enforced conditional-DELETE semantics (CasProbe step 6), MinIO OSS silently honors a mismatched-token DELETE, so the fail-closed probe aborted server startup — reported by the harness as a 311s container start-timeout on all three sanitizer lanes. 2. test_pool_survives_node_crash additionally needed start_clickhouse(start_wait_sec=150): a post-SIGKILL restart legitimately pays the unclean-reclaim protocol cost before serving (stale-token observation ~36.5s + materialization grace 30s + lease re-write; ~71s observed) — over the harness's 60s default. Previously masked by the MinIO startup abort. 3. test_disks_app_func_rm_shared_recursive: the "d/a" write is a dropped-prefix typo for "a/d/a" (present since 54aadd8, 2024-07) that has ALWAYS failed internally; clickhouse-disks --query used to swallow command failures and exit 0, and this branch's deliberate ee80535 (fsck exit-code contract) made the latent failure visible. Fix the typo and the four ls --recursive expectations it changes. No product code touched: DiskObjectStorageTransaction was verified unchanged in behavior for plain disks (dispatch() degenerates to the old push_back; the TXN-ONE-PIPELINE hook is a no-op for non-CA metadata). Verified: test_content_addressed_s3 2/2, shared_pool 2/2 (pool_survives 1 passed in 98s after the timeout fix), disks_app_func_rm_shared_recursive passed — local praktika integration runs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… actual invariant The test (authored 2026-07-16) predated the R3 acked-data-loss fix (7748419, 2026-07-17) by one day and asserted the OLD commit ordering: disk commit after the Keeper multi, so the failpoint left a phantom ZK part and ordinary lost-part recovery had to run (ReplicatedDataLoss bump + empty covering part). R3 deliberately reversed the order (renameParts closes the part's disk transaction BEFORE the Keeper multi — a part must be durable before its block_id is registered), so the failpoint now aborts the INSERT before anything reaches ZK: no phantom part, nothing to recover, and the old predicate waited forever (600s timeouts on all three sanitizer CI lanes of PR#2073 run 30019911967 and on a local release build). Rewritten to assert the new, strictly stronger invariant: (1) the failed INSERT leaves no trace across a restart (no ZK entry, no queue debris, ReplicatedDataLoss unchanged); (2) the R3 guard itself — retrying the SAME insert (same block_id) genuinely lands instead of being silently deduped by a phantom block_id (the acked-data-loss class the reordering prevents); (3) the no-wedge guards stay (no LOGICAL_ERROR, fresh INSERT replicates). Verified locally: 1 passed in 35s (was: 600s timeout). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
content addressable storage - draft PR
Documentation entry for user-facing changes
TBD.
Exclude tests:
Regression jobs to run: