Sharded dataset generation - #73

Open

PatrickRMiles wants to merge 16 commits into

PatrickRMiles:miles30/sharded_sample_generation

Commits on Jun 5, 2026

fix fractalgen bug: computing acceptance criteria variance across wrong axis
Patrick Miles
committed
pass absolute fractal dir path to scaffold calls to ensure existing fractals found
Patrick Miles
committed
revert more complex changes, focus on simpler implementation: dataset sharding matches distconv spec
Patrick Miles
committed
sharded volgen WIP
Patrick Miles
committed
normalize shard layout in dataset config hashing so equivalent shard layouts hash the same way
Patrick Miles
committed
small tweaks to shrink diff
Patrick Miles
committed
ruff
Patrick Miles
committed
add docstrings
Patrick Miles
committed

Commits on Jun 10, 2026

update README
Patrick Miles
committed
volumegen: every rank ensures needed dirs exist
Patrick Miles
committed

Commits on Jun 11, 2026

rework sharded volgen
Patrick Miles
committed

Commits on Jun 25, 2026

ruff
Patrick Miles
committed

Commits on Jul 16, 2026

update readme
Patrick Miles
committed
remove v2 support
Patrick Miles
committed

Commits on Jul 29, 2026

WIP oom debug w/ diagnostics/memory tracking. will likely remove later
Patrick Miles
committed
Merge branch 'main' into miles30/sharded_sample_generation
Patrick Robert Miles
committed