docs(storage): settings-driven section prefixes, per-schema hash paths, GC semantics#196
docs(storage): settings-driven section prefixes, per-schema hash paths, GC semantics#196dimitri-yatsenko wants to merge 6 commits into
Conversation
…s, GC semantics Reflects the storage corrections in datajoint/datajoint-python#1479, which aligned the code to these docs' long-documented model: the per-store hash_prefix/schema_prefix/filepath_prefix settings control the layout and are consumed by writers, garbage collection, and <filepath@> validation alike. Emphasis per review: hash-addressed paths embed the schema name ({hash_prefix}/{schema}/{hash}), which scopes deduplication per schema and is what garbage collection relies on to attribute stored objects to schemas — and, because storage-side listing covers the whole section, collect() must be given every schema that uses the store. The GC how-to's old 'you only need to scan schemas that share the same database' note said the opposite and is replaced with a warning stating the real obligation. - how-to/garbage-collection.md: corrected multi-schema warning; layout blocks parameterized by the prefix settings; token-per-write versioning, folder objects + manifest sidecars and coverage-based orphan matching; legacy root-level layout note (2.3.0 and earlier); filepath_prefix never touched. - reference/specs/object-store-configuration.md: normative Consumption paragraph (all components read the same spec values; defaults applied to every spec incl. plugin protocols; no component-level fallbacks); hash-paths-embed-schema note; changing-prefixes-on-existing-store caveat. - reference/specs/type-system.md: per-project dedup claim corrected to per-schema; storage tree and codec table paths sectioned and parameterized ({hash_prefix}/{schema}/{hash}; {schema_prefix}/{schema}/{table}/...). - explanation/type-system.md, how-to/use-object-storage.md, how-to/manage-pipeline-project.md, reference/specs/staged-insert.md, reference/specs/npy-codec.md: root-level schema-addressed paths corrected to the sectioned layout with write tokens. - reference/configuration.md, how-to/configure-storage.md, how-to/choose-storage-type.md, object-store-configuration.md: token filename format corrected to {field}_{token}{ext} (docs showed dot- separated); configure-storage gains the changing-prefixes warning and the single-source consumption note.
…helper format_stats was removed from datajoint-python (it presumed one display format and the library never called it). Examples now read the plain dict scan()/ collect() return, and note that collect(dry_run=True) reports counts only — use scan() for per-item paths and reclaimable bytes.
|
Added a commit removing |
…ombined totals Reflects the GC refactor in datajoint/datajoint-python#1479: garbage collection is a store-bound GarbageCollector class, not module-level functions. - All examples use dj.gc.GarbageCollector(store=...).scan(...) / .collect(...). - The "Multiple Schemas" section is inverted: per-schema scoping makes any SUBSET of a store's schemas safe to collect — the old "always pass every schema or lose data" warning was the opposite of the shipped behavior and is replaced with a per-schema-attribution note. - Named-store examples bind the store at construction (no store_name= param). - Combined totals (orphaned/orphaned_bytes/referenced/stored) removed from the statistics reference and examples; readers sum the per-section fields.
|
Added a commit updating the GC docs to the store-bound |
…n) replaces scan Follows datajoint/datajoint-python#1479: the collector takes its schemas at construction (dj.gc.GarbageCollector(schema1, schema2, store=...)), and the separate scan() is gone — collect(dry_run=True) is the default read-only report (full orphan paths + reclaimable bytes), collect(dry_run=False) deletes.
…h_paths) Mirror the schema-addressed naming on the hash side of the collect() report (datajoint/datajoint-python#1479): hash_referenced->hash_paths_referenced, hash_stored->hash_paths_stored, hash_orphaned->hash_paths_orphaned, hash_orphaned_bytes->hash_paths_orphaned_bytes, orphaned_hashes->orphaned_hash_paths.
- type-system.md <hash@>: fix stale intro path to {hash_prefix}/{schema}/{hash}
matching the tree diagram and codec table
- type-system.md <hash@>: correct the two bullets that still claimed per-project
scope / cross-schema sharing to per-schema, consistent with the Deduplication bullet
- npy-codec.md: update the example and JSON metadata paths to the
{schema_prefix}/.../{attribute}_{token}.npy form used elsewhere
Documentation counterpart to datajoint/datajoint-python#1479 (in review), which aligned the code to the model these docs already described: the per-store
hash_prefix/schema_prefix/filepath_prefixsettings control the storage layout and are consumed by writers, garbage collection, and<filepath@>validation from the same spec values (no component-level fallbacks).The load-bearing point (per maintainer review)
Hash-addressed paths embed the schema name —
{hash_prefix}/{schema}/{hash}— which:collect()must be given every schema that uses the store — the GC how-to's old note ("you only need to scan schemas that share the same database") asserted the opposite and is replaced with an explicit warning. Per-schema attribution is also what makes dropped-schema debris reclaimable.Other corrections
{schema}/…layout (that's the ≤2.3.0 legacy layout, noted as such; GC lists both).{field}_{token}{ext}— several pages showed a dot-separated{field}.{token}.{ext}form that never matched the code..manifest.jsonsidecars with coverage-based orphan matching, and thefilepath_prefixnever-touched guarantee.Sequencing
Merge after datajoint/datajoint-python#1479 — these pages describe the post-fix behavior. No file overlap with the open #195.