[ENH] BEP047 - Add audio/video recordings to behavioral experiments by bendichter · Pull Request #2231 · bids-standard/bids-specification

bendichter · 2025-10-25T16:42:26Z

Add comprehensive support for audio and video recordings in behavioral experiments: - Add audio file extensions (mp3, wav) and video file extensions (mp4, mkv, avi) with corresponding _audio and _video suffixes - Document usage of audio/video recordings in beh directory for capturing vocalizations, speech, facial expressions, and body movements - Add metadata schema for audio/video device information and stream properties - Include privacy warnings about personally identifiable information in human subject recordings - Update behavioral experiments title to remove "with no neural recordings" restriction, clarifying data can be stored with or without neural recordings - Add examples for file organization including multi-angle recordings and split files - Define optional entities: task, acquisition, run, recording, split

for more information, see https://pre-commit.ci

@yarikoptic

…ee macros - Change section title from 'Behavioral experiments' to 'Behavioral recordings' - Convert file tree examples to use MACROS___make_filetree_example for consistent rendering - Address review comments from @yarikoptic in PR #2231

…asets

…umentation

effigies

Overall this makes sense to me. It would be good to get some feedback from contributors to related BEPs, such as eye-tracking (20), motion (29), stimuli (44) and physio (45). Even if this PR doesn't propose adding this as an associated file to those data types, the potential is there and it's worth getting opinions and identifying potential conflicts.

cc @bids-standard/bep029 @bids-standard/bep044
cc @mszinte @julia-pfarr @oesteban (BEP020)
cc @m-miedema @smoia @SouravKulkarni (?) (BEP045)

codecov · 2025-10-28T13:29:50Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.09%. Comparing base (39978da) to head (a8745d6).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2231   +/-   ##
=======================================
  Coverage   83.09%   83.09%           
=======================================
  Files          22       22           
  Lines        1698     1698           
=======================================
  Hits         1411     1411           
  Misses        287      287

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Co-authored-by: Chris Markiewicz <markiewicz@stanford.edu>

yarikoptic · 2026-06-05T15:59:02Z

A great usecase came from @xiaonansun is video recording of epileptic patients along with EEG or iEEG or other (single units) recording.

yarikoptic · 2026-06-05T16:06:29Z

and @mvdm 's lab records rodent faces for eye tracking + potential whiskers analysis etc!

aimeme-ag · 2026-06-12T18:49:05Z

I wanted to ask whether is considering support for subtitle or caption files, such as .srt or .vtt, associated with the recordings. In our MoBI arts datasets, especially a Balinese performance project, subtitles could be useful for storing time-aligned transcriptions in the original language, as well as English translations.

Would subtitle files be better represented as external files linked to the video, as _events.tsv annotations, or should subtitle/caption formats be considered as part of the audio/video extension?

Per the July 1 BEP047 discussion, generalize the camera-position field so it also covers audio recording devices (for example, microphone arrays): - Rename the `CameraPosition` metadata object to `DevicePosition` and broaden its description to any recording device. - Move it into the `AudioVideoImageDevice` sidecar group so it applies to the `audio`, `video`, `audiovideo`, and `image` suffixes, and drop the separate `CameraProperties` group. - Update the behavioral-experiments tables and example sidecar accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

# Conflicts: # src/schema/objects/extensions.yaml

The BEP047 metadata field was generalized from `CameraPosition` to `DevicePosition` (bids-standard/bids-specification#2231) so it applies to any recording device. Update the example sidecars and README accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

bendichter · 2026-07-01T16:06:59Z

Consolidation update

This PR now carries the complete BEP047 proposal in one place. It combines what were previously three separate PRs:

Add common media file definitions for BEP044/BEP047 #2367 (Yarik's common media file definitions for BEP044/BEP047) — now closed
Add BEP047 audio/video behavioral recordings on top of common media definitions yarikoptic/bids-specification#2 (BEP047 behavioral layer on top of those definitions) — now closed
the original BEP047 work on this branch

Both upstream PRs have been closed with pointers here, so #2231 is the single canonical PR to review. The branch has been merged onto current master.

What's included (12 files)

src/appendices/media-files.md — shared appendix defining media file naming, accepted formats/codecs, and common stream metadata (duration, audio/video/image properties). Applies to both BEP044 (stimuli) and BEP047 (recordings).
src/modality-specific-files/behavioral-experiments.md — the BEP047 behavioral layer: _audio, _video, _audiovideo, and _image recordings in beh/, with the recording/run/split entities and behavioral-specific device metadata.
Schema objects/rules (extensions, suffixes, metadata, rules/files/raw/beh, rules/sidecars/{beh,media}) plus the mkdocs/macro plumbing for the metadata tables.

bendichter · 2026-07-01T16:07:26Z

bids-standard/bids-examples#523 is up do date with this branch

yarikoptic

I think this is in a good shape and would allow also BEP042 to start making use of the metadata fields and macros

yarikoptic · 2026-07-07T18:39:59Z

Since this would relate to the software which would use such recordings, I would love to invite @talmo @MMathisLab @niksirbi @roaldarbol as relating to

Motion trajectories & pose extracted data for animal beh research #2057

Please have a look at this PR changes and/or rendered version at https://bids-specification--2231.org.readthedocs.build/en/2231/modality-specific-files/behavioral-experiments.html . It would be great to hear your opinion about the suggested BIDS file naming conventions for audiovideo recordings and metadata to come along in the sidecars.

edit: note -- you are welcome to submit an "official" review of the PR

niksirbi · 2026-07-16T15:45:11Z

Thanks for tagging me @yarikoptic! I'm quite busy at the moment, but would definitely want to review this. Is there a time horizon / deadline for the review period?

roaldarbol · 2026-07-19T12:25:38Z

Currently on holidays, I'll have a look in early August. :-)

bendichter · 2026-08-17T21:30:54Z

@roaldarbol checking in about a review

Each variant now lands as its own session under the beh datatype directory (sub-01/ses-<label>/beh/sub-01_ses-<label>_video.<ext>), following bids-standard/bids-specification#2231 as clip-extractor writes it. A JSON sidecar next to each video carries BEP047's technical keys as ffprobe extracts them plus the demo detail under an encoding-helper key, and a dataset_description.json at the root replaces manifest.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn

* Add Inspect-tab demo set generation and EMBER upload scripts scripts/generate-inspect-demos.sh derives 28 short videos from the committed source recording (scripts/data/Video_S1.m4v), each varying one aspect the Inspect tab surfaces: atom layout (faststart, fragmentation), container (MOV/MKV/WebM/AVI), codec and profile (H.264 Baseline/High 10/High 4:4:4, HEVC, AV1, VP8/9, MJPEG, FFV1), GOP and B-frame structure, bitrate behaviour (CBR, starved cap), track properties (rotation, VFR, resolution, no audio) and metadata tags. Each file carries its own explanation in its title/comment tags, and a manifest.json describes the set for future in-app integration. scripts/upload-inspect-demos.sh pushes a generated set to a dandiset on the EMBER archive with the dandi CLI, and the Inspect demos workflow (workflow_dispatch) runs both against EMBER dandiset 000527, keeping the set as a run artifact and supporting a dry run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Limit the demos workflow's GITHUB_TOKEN to contents: read The job only checks out the repo; the EMBER upload uses its own key. Flagged by CodeQL on #27. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Rename the demo set away from the Inspect tab The demos exercise the whole app, not just one tab: the workflow is now "Upload demos to EMBER" (upload-demos.yml), the scripts are generate-demos.sh and upload-demos.sh, and the default asset prefix is demos/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Fix the upload target and lay demos out BEP047-style The dandiset (000527), instance (ember-dandi) and layout are no longer workflow inputs: the workflow now dispatches with no parameters and the demo files land as sub-01/video/sub-01_desc-<label>_video.<ext>, BIDS entity naming with a distinct desc- per variant, the way clip-extractor names its output. The manifest sits beside the videos. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Write the dandiset.yaml instead of downloading it The dandi download of the dandiset record silently produced nothing on the runner, and with the target directory then missing, cp collapsed the sub-01/ level into it. dandi upload only reads the identifier out of dandiset.yaml, so the script now writes that one line itself and copies the demo tree into a directory that is guaranteed to exist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Pass the API key as EMBER_DANDI_API_KEY The dandi CLI reads the key for a given instance from {INSTANCE_NAME}_API_KEY, so for ember-dandi it ignored DANDI_API_KEY and fell back to an interactive prompt, which EOFs in CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Include the webm demos and the manifest in the upload dandi upload only considers extensions it recognizes as assets, which silently left the two .webm files and manifest.json behind; DANDI_DEVEL with --allow-any-path uploads the whole tree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Made the demo output a BEP047 dataset of one session per variant Each variant now lands as its own session under the beh datatype directory (sub-01/ses-<label>/beh/sub-01_ses-<label>_video.<ext>), following bids-standard/bids-specification#2231 as clip-extractor writes it. A JSON sidecar next to each video carries BEP047's technical keys as ffprobe extracts them plus the demo detail under an encoding-helper key, and a dataset_description.json at the root replaces manifest.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Enriched the sidecars and moved the demo index into dataset_description Sidecars gain ImageBitDepth and, where one can be determined (via ffmpeg's DASH muxer, or built from the level for HEVC), VideoCodecRFC6381; the per-session demo detail moves out of them into dataset_description.json under the encoding-helper key, so one fetch indexes the set. The dataset also now records its CC-BY-4.0 license, the PLOS ONE article the source video is supplementary material of as SourceDatasets, and the generation date in GeneratedBy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Named the source publication in SourceDatasets The entry now carries the article title, authors, year, license and a note that the source recording is its Supporting Information Video S1; the dataset Description says "the original video" instead of naming the reference session. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Dropped Version from the source entry and led with Name, Description The publication year moved into the Description; the year is not a dataset version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Published the source recording under sourcedata/rawbids The recording the demos are derived from now travels with them, in the place BIDS keeps a raw original (with its own dataset_description.json, as clip-extractor's rawbids copy has), and the generator fetches it from there when there is no local copy. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Removed the source recording from the repository The archive now holds it under sourcedata/rawbids/, so the generator fetches it from there and no video blob has to live in the git history. scripts/data/ is ignored, and the workflow passes the archive key to the generate step so the fetch works against a non-public dandiset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Made the source recording ses-original instead of sourcedata The dataset is raw at its outer level, so the unmodified recording needs no sourcedata/rawbids/ layer of its own: it is simply the first session, with the same sidecar every other session gets and an index entry that carries no ffmpeg_args, since no ffmpeg call produced it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Fall back to the old source path for one migration run The archive still holds the recording where it sat before it became ses-original, and a runner has no local copy to publish the new path from. The fallback lets the first run after the move find it, and comes out once that run has published ses-original. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn * Dropped the migration fallback now that ses-original is published Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ErEUUQ4hdFhkcGptkwGgAn --------- Signed-off-by: Cody Baker <51133164+CodyCBakerPhD@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

roaldarbol · 2026-08-25T11:57:24Z

@bendichter Finally found some time to look through it. Generally I think it looks very thorough and well thought out. I mentioned in a comment that if the idea with images (or one of the things that it should help with) is regarding reproducibility of pose estimation or other models, then the annotations on those images also need to be stored somewhere. Is that already possible, like is there place those could meaningfully belong? And if so, it may be worth adding that extra bit of context in the section that discusses images and explicitly mentions pose estimation.

Great work! I'll be looking more closely at your metadata sidecars for adoption in animovement.

CodyCBakerPhD · 2026-08-25T15:06:54Z

I mentioned in a comment that if the idea with images (or one of the things that it should help with) is regarding reproducibility of pose estimation or other models, then the annotations on those images also need to be stored somewhere.

Such annotations would count as 'derivative' data since they could be tagged or interpreted in multiple ways, possibly even automatically by software

BTW this little conveience app exports in BEP047-compliant format (and supports both image type as well as associated text descriptions): https://clip-extractor.brain-bbqs.org/

You can try it out with pose overlay on the SLEAP tutorial data: https://github.com/talmolab/sleap-tutorial-data/tree/main/new_data

roaldarbol · 2026-08-28T23:25:21Z

@CodyCBakerPhD You could also see https://roald-arboel.com/croppy/ which is also for extracting clips (and command lab video operations). If there's anything I could do to make it better fit BEP047, please let me know on the repo! :-)

bendichter added 2 commits October 25, 2025 12:31

fix(metadata): remove invalid enum reference for DatasetType

ea2e5d1

bendichter requested a review from erdalkaraca as a code owner October 25, 2025 16:42

bendichter and others added 3 commits October 25, 2025 12:59

Merge branch 'master' into audio-video-clean

e1d95e8

[pre-commit.ci] auto fixes from pre-commit.com hooks

ffccafc

for more information, see https://pre-commit.ci

Apply suggestion from @bendichter

4707a33

bendichter commented Oct 25, 2025

View reviewed changes

Comment thread src/schema/objects/metadata.yaml Outdated

Apply suggestion from @bendichter

38310aa

bendichter commented Oct 25, 2025

View reviewed changes

Comment thread src/schema/objects/metadata.yaml Outdated

Apply suggestion from @bendichter

fa98d09

bendichter commented Oct 25, 2025

View reviewed changes

Comment thread src/schema/objects/metadata.yaml Outdated

Apply suggestion from @bendichter

8884985

yarikoptic reviewed Oct 25, 2025

View reviewed changes

Comment thread src/modality-specific-files/behavioral-experiments.md Outdated

yarikoptic reviewed Oct 25, 2025

View reviewed changes

Comment thread src/modality-specific-files/behavioral-experiments.md Outdated

yarikoptic changed the title ~~SCHEMA: Add audio video~~ SCHEMA: Add audio video behavioral data support Oct 25, 2025

yarikoptic added the schema Issues related to the YAML schema representation of the specification. Patch version release. label Oct 25, 2025

bendichter added 4 commits October 26, 2025 12:55

fix(docs): update link to BIDS examples repository for behavioral dat…

7cacffd

…asets

refactor(beh): streamline description of behavioral recordings in doc…

9b262a3

…umentation

Merge branch 'master' into audio-video-clean

7bc990a

effigies reviewed Oct 27, 2025

View reviewed changes

Comment thread src/modality-specific-files/behavioral-experiments.md

effigies reviewed Oct 28, 2025

View reviewed changes

Comment thread src/modality-specific-files/behavioral-experiments.md Outdated

Comment thread src/modality-specific-files/behavioral-experiments.md Outdated

Comment thread src/schema/rules/sidecars/beh.yaml Outdated

effigies changed the title ~~SCHEMA: Add audio video behavioral data support~~ [ENH] Add audio/video recordings to behavioral experiments Oct 28, 2025

Merge branch 'master' into audio-video-clean

e8a8bf3

Update src/modality-specific-files/behavioral-experiments.md

3ee2c6c

Co-authored-by: Chris Markiewicz <markiewicz@stanford.edu>

bendichter commented Oct 28, 2025

View reviewed changes

Comment thread src/modality-specific-files/behavioral-experiments.md

Apply suggestion from @bendichter

826e25a

bendichter commented Oct 28, 2025

View reviewed changes

Comment thread src/schema/rules/files/raw/beh.yaml

Apply suggestion from @bendichter

00e3e26

Merge branch 'mediafiles-updated' into mediafiles-bep047

96a3386

bendichter and others added 2 commits July 1, 2026 10:48

Merge remote-tracking branch 'origin/master' into pr2-mediafiles-bep047

07ca171

# Conflicts: # src/schema/objects/extensions.yaml

bendichter force-pushed the audio-video-clean branch from 9418f93 to 07ca171 Compare July 1, 2026 14:58

bendichter mentioned this pull request Jul 1, 2026

[BLOG] BEP047 audio/video behavioral recordings community review bids-standard/bids-website#863

Draft

yarikoptic reviewed Jul 1, 2026

View reviewed changes

Comment thread tools/mkdocs_macros_bids/macros.py

yarikoptic approved these changes Jul 6, 2026

View reviewed changes

yarikoptic mentioned this pull request Jul 7, 2026

Potentially unify listing (table) of extensions #2461

Open

yarikoptic mentioned this pull request Jul 12, 2026

bids: needs knowledge about how to validate against BEP OpenScience-Collective/osa#346

Open

Merge branch 'master' into audio-video-clean

3d90540

CodyCBakerPhD mentioned this pull request Aug 20, 2026

Restructured Save/Upload output as a BIDS-Study tree brain-bbqs/clip-extractor#41

Merged

CodyCBakerPhD added 2 commits August 21, 2026 10:23

Merge branch 'master' into audio-video-clean

a8745d6

Merge branch 'master' into audio-video-clean

681b892

CodyCBakerPhD mentioned this pull request Aug 23, 2026

Added the demo set generator and the EMBER upload workflow brain-bbqs/encoding-helper#27

Merged

CodyCBakerPhD mentioned this pull request Aug 25, 2026

Named an audio-bearing source _audiovideo and described its sound brain-bbqs/clip-extractor#47

Merged

Uh oh!

Conversation

bendichter commented Oct 25, 2025 • edited by yarikoptic Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

effigies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

yarikoptic commented Jun 5, 2026

Uh oh!

yarikoptic commented Jun 5, 2026

Uh oh!

aimeme-ag commented Jun 12, 2026

Uh oh!

bendichter commented Jul 1, 2026

Consolidation update

What's included (12 files)

Uh oh!

bendichter commented Jul 1, 2026

Uh oh!

Uh oh!

yarikoptic left a comment

Choose a reason for hiding this comment

Uh oh!

yarikoptic commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

niksirbi commented Jul 16, 2026

Uh oh!

roaldarbol commented Jul 19, 2026

Uh oh!

bendichter commented Aug 17, 2026

Uh oh!

roaldarbol commented Aug 25, 2026

Uh oh!

CodyCBakerPhD commented Aug 25, 2026

Uh oh!

roaldarbol commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

bendichter commented Oct 25, 2025 •

edited by yarikoptic

Loading

codecov Bot commented Oct 28, 2025 •

edited

Loading

yarikoptic commented Jul 7, 2026 •

edited

Loading

roaldarbol commented Aug 28, 2026 •

edited

Loading