Support zarr v3

[alejoe91]

Fixes #4014

Implement bulk changes from zarr v2 to zarr v3

• create_dataset -> create_array
• Serialization of non JSON serializable objects (recording/sorting/extensions) not supported anymore
• Compressors use zarr.codecs, which wrap numcodecs compressor (e.g., Blosc -> BloscCodec)
• Codec Pipeline: improved generation of codec pipeline to automatically chain filters (ArrayArrayCodec), serializers (standard unless an ArrayBytesCodec is used - e.g. Wavpack), and compressors (BytesBytesCodec)
• Clean up of tests

TODO:

• Add sharding parameter to save function
• Backcompatibility
• Test v2 reading in v3: make an action that generates v2 zarr for Recording/Sorting/Template/SortingAnalyzer using an old SI version and reads it in the new version

Depends on:

• ProbeInterface: Support zarr v3 probeinterface#387
• NWB (hdmf-zarr): Migrate hdmf-zarr from zarr-python v2 to v3 hdmf-dev/hdmf-zarr#325

[chrishalcrow]

I'm having issues loading principal_components, quality_metrics and template_metrics from a v2 analyzer. Looking into it...

[alejoe91]

I'm having issues loading principal_components, quality_metrics and template_metrics from a v2 analyzer. Looking into it...

We should add full analyzers in the cross-serialization checks!

[chrishalcrow]

This type of hack works for v2 backwards compat. Annoyingly, we don't necessarily know the names of the pc models apiori: they're called pca_model_by_channel_local_323 in my dataset - I guess 323 are the channel_indices...

Suggested change

	for ext_data_name in extension_group.keys():
	if self.extension_name =='template_metrics':
	extension_group_keys = ['metrics', 'main_channel_templates', 'peaks_data']
	elif self.extension_name =='quality_metrics':
	extension_group_keys = ['metrics']
	elif self.extension_name =='principal_components':
	extension_group_keys = ['pca_projection']
	else:
	extension_group_keys = extension_group.keys()
	for ext_data_name in extension_group_keys:

[alejoe91]

In hdmf-zarr we made a V2 / V3 compatible iter function..let me look it up! Otherwise, I think it's fine to drop loading the PCA models. They are saved with the pkl codec anyways so not sure.if they will be compatible!

[alejoe91]

We can use something like this. LEt me give it a try!

https://github.com/alejoe91/hdmf-zarr/blob/nwb-zarr-v2-io/src/hdmf_zarr/backend_zarrv2.py#L466

[alejoe91]

This is not limited to the member itself: Group.keys(), Group.members(),
key in group and zarr.open_consolidated() all fail, so a single legacy |O array
makes the whole group (or store) unreadable. In practice this hit every v2
SortingAnalyzer:

• data["runtime_s"] (per-metric run times, a dict) in quality_metrics,
  template_metrics, spiketrain_metrics → JSON codec
• data["pca_model_*"] (sklearn IncrementalPCA) in principal_components → Pickle codec
• recording / sorting_provenance at the analyzer root → JSON or Pickle codec

The fix (reading)

New helpers in core/zarr_tools.py, usable for both zarr v2 and v3 groups:

• iterate_zarr_group(group) — yields (name, member). Member names come from
  consolidated metadata when present, otherwise from store.list_dir; each member is then
  opened individually, so one unopenable member no longer breaks the iteration.
• read_legacy_zarr_object_array(store, path) / LegacyZarrObjectArray — decodes a
  legacy |O array directly with numcodecs (reads .zarray, undoes the compressor and
  the object codec) and exposes the small part of the zarr.Array API the loading code
  needs (attrs, [i], shape, dtype, len). Members that can be neither opened nor
  decoded are skipped with a warning instead of raising.
• get_zarr_group_keys(group) — member names without opening them (for in-style checks).
• get_zarr_attr_or_legacy_object(group, name) — reads from group attrs (>= 0.105
  layout) and falls back to a legacy |O array of the same name.

Call sites updated:

• AnalyzerExtension.load_data (zarr branch) — the loop that was failing; also handles the
  legacy dict attr (v2) next to the new dict_data attr (v3), and the dataframe column loop
• SortingAnalyzer.get_saved_extension_names
• SortingAnalyzer.load and get_sorting_provenance — v2 recording /
  sorting_provenance object arrays are read again, so the recording and provenance
  links of old analyzers are no longer silently lost
• ZarrRecordingExtractor / ZarrSortingExtractor property loops
• _guess_object_from_zarr in core/loading.py

sklearn models are now saved without pickle (writing)

Previously the zarr branch of _save_data warned
"Data type of pca_model_... not supported for zarr saving, skipping", so newly computed
principal_components silently lost their models. They are now saved as plain data:

• save_sklearn_model_to_zarr_group — writes every ndarray in vars(model) as a zarr
  array in a sub-group (honouring the analyzer's saving_options) and stores the remaining
  scalars/params in a sklearn_model attribute, together with the class path, the array key
  list and the writing sklearn.__version__.
• load_sklearn_model_from_zarr_group — imports the class, instantiates with
  cls.__new__(cls) (constructor bypassed on purpose, the state is restored verbatim) and
  sets the attributes and arrays back.
• Models whose state is neither array nor json-serializable (e.g. a Pipeline with nested
  estimators) warn and are skipped, as before — no half-written group is left behind.

No pickle is involved in either direction for new stores, and legacy pickled models are
still readable, so an old analyzer re-saved with save_as(format="zarr") is migrated to the
array representation automatically.

Cross-version caveat, for the record: the estimator state is restored verbatim, so a model
written by a much newer scikit-learn could still miss an attribute a much older one expects.
The writing version is stored in the attribute for diagnostics (no warning is emitted, since
by_channel_local would produce one per channel).

Validation

Extended the cross-serialization tests to creating a SortingAnalyzer with all available extensions and storing the data fields for every extension to check against. Also extended the cross-serialization tests to test against the current dev version for round-trip.

[alejoe91]

@chrishalcrow can you test everything again? I think it's all fixed now!