Restructured Save/Upload output as a BIDS-Study tree - #41
Merged
Conversation
…ta/derivatives, BEP028 dataset_description.json provenance) - sourcedata/ and derivatives/clip-extractor/ mirror the source video's own subject/session, with files named per BEP047's sub-/ses-/recording-/desc- entities and _video/_image suffixes - the extracted clip or frame's own sidecar JSON now carries the delivery's description and this app's full provenance record (nested), replacing the separate provenance file - an upload folds a BEP028-style GeneratedBy entry into the dandiset's dataset_description.json and into derivatives/clip-extractor/dataset_description.json, without disturbing anything already there Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
|
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #41 +/- ##
==========================================
+ Coverage 81.19% 84.90% +3.70%
==========================================
Files 36 40 +4
Lines 1617 1802 +185
Branches 394 459 +65
==========================================
+ Hits 1313 1530 +217
+ Misses 304 272 -32
🚀 New features to boost your workflow:
|
CodyCBakerPhD
marked this pull request as ready for review
August 20, 2026 02:21
…echnical sidecar, and added mock_sub/mock_ses test injections - a loaded .slp now lands under derivatives/clip-extractor/ alongside the extract, since it is a pose estimation pipeline's output rather than raw recording content - the source video's own sourcedata sidecar now carries its real duration/frame rate/dimensions, already read off it when it was loaded, rather than nothing at all - ?test&mock_video&mock_sub=<label>[&mock_ses=<label>] previews the Save/Upload tree for a known archive subject and session, alongside the existing sub-unknown fallback Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
…isambiguator in sourcedata A known subject/session (opened out of the archive rather than dropped locally) means sourcedata's one copy of the source video is meant to be reused/overwritten across deliveries, not duplicated, so it now carries no disambiguating entity at all; derivatives, still needing one per run, spells it as separate date-/time- entities rather than the compact recording- stamp, which now applies only to the sub-unknown fallback in both trees. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
… own dataset_description.json sourcedata/ itself is not validated against BIDS naming at all, but sourcedata/rawbids/ is now built as a complete, independently valid DatasetType: raw BIDS dataset (its own dataset_description.json, entity-named files throughout), so a validator run directly against that subtree passes on its own, not just as part of the dandiset as a whole. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
CodyCBakerPhD
commented
Aug 20, 2026
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
… the source video in provenance
- the root dataset_description.json is now DatasetType: "study" (BIDS's own type for a root that
organizes source, raw and derived data together, which is exactly what this app's tree is) and,
when freshly created, named after the delivery that made it ("<Snippet|Frame> extracted using the
Clip Extractor on <ISO datetime>") rather than just the dandiset id
- every dataset_description.json's GeneratedBy entry now carries as much as is known about the
source video (filename, checksum, url, fps, dimensions, frame count), and two different videos at
the same tool version get their own entries instead of being collapsed into one
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
… its description Sources is now omitted entirely rather than written as [] when a companion file has no upstream to name; the source video's own sidecar description reads 'The source video this selection was clipped from.' Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
… names to the study, named the source in SourceDatasets - BEP047 sidecars now carry VideoCodec when it is actually known: h264 for anything this app encoded with ffmpeg.wasm, the source's own codec for a streamed clip copied through untouched, and omitted rather than guessed for a re-encode mediabunny performed itself - derivatives/clip-extractor/ and sourcedata/rawbids/'s own dataset_description.json now share the study root's Name verbatim, suffixed "(Extracted)"/"(Original)" - the derivatives description's SourceDatasets now names the actual source video (URL when streamed, Filename/Checksum otherwise) instead of a placeholder "." URL Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
…tion ?test&mock_video alone lands on the gated 'describe it first' state, which needed a frame picked and a description typed by hand before Save/Upload output could actually be previewed. mock_ready does both automatically, so the link alone lands on a saveable state.
…asetLinks - dataset_description.json files now credit the signed-in archive account as a minimal Authors entry. - mock_sub now also gives the mock video a fake asset URL, so it previews SourceDatasets the way a real EMBER-sourced clip would. - The study root and the derivatives file cross-link each other via a DatasetLinks alias (clip / raw). - GeneratedBy entries order their keys Name, Description, then the version-y fields.
…c/pixel-format detail - GeneratedBy no longer carries a SourceVideo (not BEP028 vocabulary); SourceDatasets now accumulates one entry per distinct video across repeat deliveries instead. - The study root's own dataset_description.json now also names sourcedata/rawbids/ directly via a plain 'source' key. - Video/image sidecars now carry a Checksum field (MD5 and dandi-etag, SPDX-shaped), and, where a StreamingVideoBackend can name them, VideoCodecRFC6381, ImagePixelFormat and ImageBitDepth; BEP047's own technical keys are now grouped together, last. - A ?test&mock_video preview reports codec/pixel-format detail as h264's own typical values rather than leaking the mock's own VP8 encoding.
…ed instead - ?test&mock_video&mock_sub=/&mock_ses= are gone. mock_video is always the 'dropped locally' case now (sub-unknown fallback); fakeArchiveBrowse's fake video paths are zero-padded, proper BIDS entities (sub-01/ses-01/..., not sub-1/session-1) instead, previewing the dandiset-sourced shape. - The study root's plain 'source' key is folded into its existing DatasetLinks object instead of sitting beside it. - Minor: GeneratedBy's own Description text trimmed.
…ive cases, dropped ad hoc provenance block sourcedata/rawbids's copy of the source video is now named sub-<label>[_ses-<label>]_video.<ext> instead of keeping its original filename, and a repeat delivery of the same subject/session overwrites it instead of piling up duplicates. ?test&mock_video&mock_ready gained a =snippet variant and can be crossed with &from_archive, previewing a video opened out of the archive (a known sub-01/ses-02, with a real source URL) instead of dropped locally. The mock video itself is now named/typed .mp4 to match what a real archive-sourced recording would actually be, and from_archive's fixed mock path uses sub-01/ses-02 to keep subject and session visibly distinct. Dropped the ad hoc clip-extractor provenance block nested in the sidecar JSON; a later PR will add a proper W3C PROV record in its place.
…erlay The extracted video/frame previously carried no desc entity at all, unlike its pose-overlay sibling's desc-overlay — BIDS labels may concatenate multiple applicable ones with a plus sign, so the plain extract now reads desc-extracted+clip: the thing this app extracted, and the clip it is named for.
…ocal file GeneratedBy is already recorded once in each dataset_description.json, so repeating it in every sidecar was redundant; sidecars now carry only Description/Checksum(/technical keys). SourceDatasets previously named a locally dropped video by filename and checksum alone when it had no real URL, but that pair only repeated what the file's own sidecar already says and isn't what BIDS means by a source dataset. A local delivery now leaves SourceDatasets out entirely; only a video opened from a real address (streamed from the archive) gets an entry. Also extended the mock_ready/from_archive 2x2 Playwright coverage to assert on that contrast directly, and fixed a couple of stale sub-01/ses-01 doc comments left over from the from_archive sub-01/ses-02 change.
…irectory Every derivatives filename previously repeated date-/time- (or recording-, for a locally dropped sub-unknown video) to tell one delivery apart from another. That entity now names a directory instead: derivatives/clip-extractor/sub-.../beh/<date-..._time-...|recording-.../>, holding every file the delivery wrote. Filenames underneath keep only sub-/ses-/desc-, so a listing of beh/ reads as one subdirectory per delivery rather than one repeated stamp per file. The Save bundle's own filename carries neither disambiguator: it is the outer container, not a file in the tree it holds, so saving the same source again names the same bundle rather than minting an ever-changing name around the delivery's own instant.
The four ?test&mock_video&mock_ready links now name both halves of what they preview rather than leaving each implicit: ?test&mock_video&mock_ready&from_local&frame ?test&mock_video&mock_ready&from_local&snippet ?test&mock_video&mock_ready&from_ember&frame ?test&mock_video&mock_ready&from_ember&snippet &from_ember replaces &from_archive and &from_local spells out the default it leaves alone; &frame/&snippet replace mock_ready=snippet, with &frame likewise the explicit spelling of the default. mock_ready also marks a selection somebody could plausibly have made instead of one that only proves the gate opened: frame 12, or frames 6-21, of mock_video's own 30, held to the loaded video's own bounds so a shorter mock_video=<n> still lands somewhere real. &frame=<n> and &snippet=<lo>-<hi> name others. The frame case now moves the playhead rather than only the marks, since frame mode extracts whatever state.cur points at.
The four links below it spell the same grid out, and &from_local&frame is that bare link exactly. Listing both invited pasting the one that leans on defaults rather than the one that says what it previews. The defaults themselves stay: &from_local and &frame remain the explicit spelling of what mock_ready does on its own.
…dataset names The saved bundle carried desc-frame/desc-snippet; it now carries the same desc-extracted+clip as the extract inside it. The outer container says nothing about which kind of selection it holds, so the same source saved again names the same bundle either way. The three dataset_description.json files no longer stamp their Name with the delivery's instant. These files are dataset-level, not per-delivery: the first delivery to find one missing creates it and every later one folds into what is already there, so a name built around whichever moment happened to be first would sit there permanently describing an instant that means nothing to anything added after it. When a delivery happened is still recorded where it belongs, in the date-/time- directory its own derivatives sit in. Also corrected generatedBy.ts's header, which still claimed the same GeneratedBy entry is written into every sidecar.
…ity back
The plain extract is a delivery's primary output, so it needs no desc- to
set it apart from anything; only the pose overlay keeps one. Its files do
carry the delivery's own disambiguating entity again, alongside the
directory already named for it — the directory is what makes a listing of
beh/ read as one group per delivery, and the filename repeating it is what
keeps each file self-describing once it is copied or downloaded somewhere
that directory does not follow.
derivatives/clip-extractor/sub-unknown/beh/recording-<label>/
sub-unknown_recording-<label>_image.png
sub-unknown_recording-<label>_image.json
sub-unknown_recording-<label>_desc-overlay_image.png
The bundle keeps desc-extracted+clip as the one thing naming what the
container holds, and loses the _bundle suffix: it is a download, not a
file in a BIDS tree, so behFilename's sibling bundleFilename builds it
without a suffix or a disambiguator.
sub-unknown_desc-extracted+clip.tar.gz
Also consolidated this PR's CHANGELOG entries, several of which described
steps later ones reversed rather than the net change.
Eleven entries had accumulated, one per follow-up request, several of them narrating steps a later commit reversed rather than where the change landed. They are now a single entry describing what someone using the app would notice: Save and Upload write a BIDS-Study tree. Added the corresponding guideline to CLAUDE.md, since the sprawl came from adding an entry per commit rather than revising the PR's own.
…tirely The saved bundle carried sub-/ses- and a desc-. None of that belonged on it: the bundle is a download, not a file in the tree it holds, and it stops existing the moment it is unpacked. A subject in its name also claimed the container holds only that subject, which is not something it promises. It is now named for the dandiset it is destined for, or clip-extractor when no dataset is picked: 9900001.tar.gz clip-extractor.tar.gz bundleFileName takes the dandiset id rather than BehEntities, so bidsPath's bundle-specific builder and its optional suffix are gone again.
…to 214000
SourceDatasets named the source *file* — its URL, filename and checksum —
which both overstated what BIDS means by the field and repeated what the
extract's own sidecar already carries. It now names the dataset the video
came out of, with the asset path within it as a detail:
"SourceDatasets": [
{
"URL": "https://api-dandi.emberarchive.org/api/dandisets/214000",
"Path": "sub-01/ses-02/test-injection-mock-video.mp4"
}
]
The dandiset is carried on AppState as sourceDandisetId, set from the
browsed asset (ArchiveVideo already knew it) and by &from_ember for the
mock. A locally dropped file belongs to no dataset, so it still gets no
entry at all.
The fake dandiset id moves from 9900001 to 214000: six digits like a real
one, so it exercises the same parsing, and far enough up EMBER's
sequential range that nothing can collide with it.
ProvenanceInput collapses to the description alone, since the source
identity it used to carry now lives in SourceDatasets and in each file's
own Checksum.
DANDI stores an asset's bytes at a bucket URL ending in the blob's own id
(.../blobs/<x>/<y>/<uuid>), so the id the archive content-addresses the
source by is derivable from the listing this app already reads. It is held
to a UUID, so a URL shaped some other way yields nothing rather than
whatever its last segment happened to be, and an embargoed asset — whose
bucket URL is signed on demand rather than listed — simply gets no BlobID.
The three loose source fields threaded through loadVideo collapse into one
ArchiveSource ({dandisetId, path, blobId}), which is also what a delivery
records; a locally dropped file has none of it and still gets no entry.
The mock's own source deliberately carries no BlobID: that video is
synthesized in the page and was never stored as a blob, and inventing an
id would make it the one part of the preview that is not a truthful stand-
in for the real thing.
Each file's Checksum now leads with SHA-256, the digest most tooling outside the archive reaches for, ahead of the MD5 and the dandi-etag it already carried. This needed hash-wasm: Web Crypto's crypto.subtle.digest has no incremental API, so it would have to buffer a whole file to hash it, and the MD5 here deliberately streams in 16MB reads precisely so a multi-GB source video never lands in memory whole. hash-wasm's hashers take the bytes a chunk at a time, so SHA-256 runs as a third pass in the same shape, and checksum progress is now reported in thirds rather than halves. The new digest is verified against node's own crypto in etag.test.ts, including across the chunk boundary where the streaming loop actually runs, and for the empty blob it never enters for.
bundleFileName read currentConfig().dandisetId — the dataset picked to upload TO — so a save while signed in was named after wherever the next upload would go (e.g. 000475), which says nothing about what is inside. It now reads state.sourceArchive: the dandiset the video came OUT of, or local-dataset for a video that came from no dataset at all. 200123.tar.gz (from_ember: the mock source dandiset) local-dataset.tar.gz (from_local: a locally dropped file) The from_ember mock source is its own 200123 rather than one of the 214000+ fake destination datasets, so a preview keeps its source and its destination visibly distinct, the way a real extraction's usually are. SourceDatasets follows to .../dandisets/200123 in the mock.
…xel format off the PNG itself The URL-state reflector was stripping a mock link's own flags (frame, mode collide with session params) and respelling every bare &flag as &flag=; under a test injection the harness owns the address, so nothing rewrites it now, and carried-through bare flags keep their spelling elsewhere too. Frame-mode sidecars (the extract's and the overlay's) now carry BEP047's ImagePixelFormat/ImageBitDepth, read off the IHDR header of the PNG the browser actually encoded rather than assumed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
for more information, see https://pre-commit.ci
…rites Mirrors the expected_output fixture style of data-ingest-task-force's labs/kemere/tests: each &frame live-test link is driven to a real save, the bundle unpacked, its file listing held against a committed manifest and every JSON file held in full against a committed copy. Run-varying values (delivery timestamps, checksums of MediaRecorder-made bytes, the app version) are normalized to spelled-out placeholders; regenerate with UPDATE_EXPECTED=1. listTar moved to helpers.ts for the new spec to share. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
Both formats describe themselves from the inside — skeleton, frame indices and provenance all live in the file — so a companion .json beside them only restated what opening the file says. The pose file still travels with the delivery and is still checksummed; BEP047 media assets keep their sidecars, where the technical keys are defined to live. CompanionSidecarInput's optional technical and nullable checksum existed only for the .slp, so both are now required. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
… encode The snippet cases were left out because a snippet extraction goes through ffmpeg.wasm, whose ~32MB core the app fetches from a CDN. Serving that core from a new @ffmpeg/core devDependency instead keeps the encode the app's own while needing no network, so all four live-test links are now fixtured. The extract's own VideoFrameCount is held exactly (16 frames, 6 through 21); only the synthesized source's recorder-decided values stay normalized. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NVfLrtMvcNB1qau7yTAva4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Save and Upload wrote a single ad hoc, timestamped folder (
sourcedata/raw/clip-extractor/date-…_time-…_type-…/). This PR replaces that with a real BIDS-Study tree:sourcedata/— the source video and any.slp, mirroring the dandiset's ownsub-<label>/[ses-<label>/]structure (parsed off the source asset's own path when it was opened from the archive; falls back tosub-unknownfor a locally dropped file).derivatives/clip-extractor/— the extracted clip/frame and its pose overlay, named per BEP047'ssub-/ses-/recording-/desc-entities and_video/_imagesuffixes (recording-<label>is this delivery's own timestamp — BEP047 has no entity for "which delivery is this";desc-overlayis the standard BIDS derivatives entity for a second output of the same recording)._video.json/_image.jsoncarries the description typed for the delivery, BEP047's technical keys, a BEP028-styleGeneratedByentry, and this app's full existing provenance record nested under its ownclip-extractorkey, so nothing from the old format is lost. The overlay gets its own lighter sidecar pointing back at the primary one.dataset_description.json— an upload now folds aGeneratedByentry (BEP028) into the dandiset's own rootdataset_description.jsonand intoderivatives/clip-extractor/dataset_description.json, fetching and merging into whichever already exists rather than overwriting it (creating either fresh only when nothing is registered yet).A saved bundle holds the exact same tree an upload would have written,
dataset_description.jsonfiles included, built fresh since there's no archive to merge against.New modules
lib/bidsPath.ts— BEP047 entity parsing and path/filename builderslib/generatedBy.ts— BEP028GeneratedByentries anddataset_description.jsonmerginglib/datasetDescription.ts— reads whateverdataset_description.jsonfiles already exist at the destinationTesting
lib/provenance.ts.npm run typecheck,npm run lint,npm test,npm run build, and the full Playwright suite all pass locally;pre-commit run --all-filesis clean.Original prompt
OK now last thing to polish before v1.1.0 deployment is to polish the output data structure to be completely BIDS-Study compliant (sourcedata BEP047 and dataset_description.json using provenance records from BEP028), description text and extracted snippet is then derivative, which should roughly follow same BEP046 structure but can have that text annotation in sidecar json instead of its own file
BEP bids-standard/bids-specification#2231
examples bids-standard/bids-examples#523
Generated by Claude Code