Support ome-zarr v05

CLAUDE.md

@@ -73,7 +73,7 @@ The library is layered. A typical `mobie.add_*` call flows top-to-bottom through
 
 ## Current task: migrate from cluster_tools to bioimage-py
 
-The active work on this branch (`migrate-bioimage-py`) is replacing the `cluster_tools` compute backend with **`bioimage-py`**. `environment.yaml` already swaps `cluster_tools`/`numba`/`python-elf<0.9` for `bioimage-py` + `python-elf>0.9`. **The migration is now complete: `cluster_tools` is no longer imported anywhere in the mobie source.**
+This work replaced the `cluster_tools` compute backend with **`bioimage-py`**. `environment.yaml` already swaps `cluster_tools`/`numba`/`python-elf<0.9` for `bioimage-py` + `python-elf>0.9`. **The migration is complete: `cluster_tools` is no longer imported anywhere in the mobie source, and both additional goals — dropping `vigra` and supporting ome.zarr on zarr v2 *and* v3 — are done too** (see the ome.zarr version-support bullet below).
 
 **Progress so far:** the **foundation shim** (`mobie/utils.py::get_run_config`) and the **core downscaling path** (`mobie/import_data/utils.py::downscale`, `compute_max_id`, plus the new `mobie/import_data/_format_metadata.py`) are migrated to bioimage-py and tested. The import path now reads **all** inputs through `bp.open_source` consistently — `downscale` plus `image_data.py::_get_default_contrast_limits` — so mobie can ingest every format bioimage-py supports (hdf5/n5/zarr/**tif**/**mrc**/**nifti**/msr/knossos); mrc and nifti import are now tested (`test/import_data/test_image.py`). The 2d→3d promotion for bdv outputs no longer writes an on-disk temp file: `ensure_volume` was **deleted** and `downscale` promotes 2d sources on the fly via the new bioimage-py wrapper primitive **`ExpandDimsSource`** (added to `bioimage_py.wrapper`). The **segmentation-table morphology workflow** is migrated too: `tables/default_table.py` now drives `bp.morphology.morphology` (+ optional `bp.morphology.regionprops` for anchor correction) for a unified 2d/3d path, and the **HTM table path** (`htm/data_import.py::_add_tables`) computes tables by parallelizing `compute_default_table` over images via `bp.get_runner().map` (the old `htm/table_impl.py` luigi task was deleted). **`vigra` is now fully gone from the source** (the table paths were its last importers). The **HTM copy path** (`htm/data_import.py::_copy_image_data`, used by `add_images`/`add_segmentations`) is migrated too: it now parallelizes over sources via `bp.get_runner().map` (one task per source) and routes each through the single-source import path (`import_image_data` / `import_segmentation`) — so `htm/data_import.py` is fully cluster_tools/luigi-free. The **registration** paths (affine/transformix/coordinate) were migrated onto `bp.wrapper.AffineSource`/`bp.transformation` + the `transformix` CLI dispatched via `bp.get_runner().map`. Finally, the **node-label import** (`import_data/from_node_labels.py`) was migrated onto **`bp.segmentation.relabel`** and the outdated **paintera project import** (`import_data/paintera.py`) was removed — which made `utils.py::write_global_config`/`BaseClusterTask` dead code, so they were deleted. **`cluster_tools`/`luigi` are no longer imported anywhere in `mobie/`** (only historical references survive in comments).
 
@@ -82,5 +82,5 @@ The migration means rewriting the layer-2 import code to drive `bioimage-py` ins
 - **bioimage-py provides *primitives*; mobie *composes* them.** Do NOT take "implement what is missing in bioimage-py" to mean adding off-the-shelf, mobie-specific high-level functions there. bioimage-py stays a clean, general primitive library (block-wise `copy`/`downsample`/`stats`/`morphology`, the runner, source wrappers). When something is missing, add the missing *primitive* (e.g. a `SourceWrapper`) — and keep MoBIE-specific orchestration and all spec/format-metadata writing here in mobie. Concretely: the multiscale-pyramid loop and the ome.zarr/bdv metadata writing (via pybdv / ome-zarr) belong in mobie, not bioimage-py.
 - **Execution model:** bioimage-py replaces the luigi `*Workflow` + `luigi.build` + on-disk `*.config`/`global.config` model with a runner — every op takes `job_type` (`local`/`subprocess`/`slurm`), `num_workers`, `block_shape`, and a `RunnerConfig`/`SlurmConfig`. This collapses `write_global_config`, the per-task `.config` files, and `BaseClusterTask` into per-call kwargs + one config object. Concentrate changes in `mobie/import_data/utils.py` (the shared engine) and `mobie/utils.py`; the format-specific wrappers and the metadata/validation layers should be largely unaffected.
 - Preserve the public-API surface (`target`, `max_jobs`, function signatures) so callers and the `add_*` flow described above keep working. Note: **LSF is dropped** (only `local`/`subprocess`/`slurm` are supported), and the legacy `int_to_uint` option has been **removed**.
-- **ome.zarr is currently written as zarr v2 via z5py** (not `elf.io.open_file`, which writes zarr v3 in this env — incompatible with MoBIE NGFF v0.4). **Additional goal — support BOTH zarr v2 and v3** as output formats (v3 not yet implemented; a writer/metadata toggle threaded through `downscale` + `_format_metadata.py`).
+- **ome.zarr version support (zarr v2 *and* v3): DONE.** The ome.zarr / NGFF version is selected via a `@` suffix on `file_format` (e.g. `file_format="ome.zarr@0.5"`; no suffix ⇒ default `"0.4"`). `mobie/utils.py::parse_file_format` splits the version off at each `add_*` entry — so the on-disk folder name and the `dataset.json` source key stay the canonical `ome.zarr` — and threads an internal `ome_zarr_version` through the import layer. `0.4`→zarr v2, `0.5`→zarr v3, **both written via z5py** (`z5py.File(..., zarr_format=3)`; z5py 3.0.2 supports v3 + sharding, so `elf.io.open_file` is still avoided for writing). v0.5 writes its NGFF metadata under `zarr.json`'s `attributes.ome` (version at the `ome` level); the v2-vs-v3 on-disk layout (`.zattrs`/`.zarray` vs `zarr.json`) is handled on read by helpers in `mobie/validation/utils.py` (`load_ngff_group_attrs`/`load_ngff_array_shape`/`ngff_multiscales`/`ngff_version`, shared by the local and s3 paths), and validation selects the matching NGFF 0.4 vs 0.5 schema (the 0.5 schema's external `$ref` is resolved via a `referencing.Registry`). A **`shards`** kwarg (on the `add_*`/import functions + a `--shards` CLI flag) enables zarr-v3 sharding and is only valid for `ome.zarr@0.5`. Full design + rationale: the 2026-07-03 increment in [[bioimage-py-migration-decisions]].
 - **Additional goal — drop the `vigra` dependency: DONE.** No `import vigra` remains in the source (the morphology-table 2d path + `htm/table_impl.py` were its last users). `vigra` was never an explicit env-file entry, so no env change was needed; `cluster_tools` may still pull it in transitively until that dependency is removed.

mobie/htm/data_import.py

@@ -14,7 +14,8 @@
 
 
 def _import_one_source(index, input_files, output_files, names, key, file_format,
-                       resolution, unit, scale_factors, chunks, is_seg):
+                       resolution, unit, scale_factors, chunks, is_seg,
+                       ome_zarr_version="0.4", shards=None):
     # Each source is converted in-memory (target "local"); parallelization happens over
     # sources in `_copy_image_data`, not via bioimage-py's within-source block runner.
     import_fn = import_segmentation if is_seg else import_image_data
@@ -23,14 +24,16 @@ def _import_one_source(index, input_files, output_files, names, key, file_format
         resolution, scale_factors, chunks,
         tmp_folder=None, target="local", max_jobs=1,
         unit=unit, source_name=names[index], file_format=file_format,
+        ome_zarr_version=ome_zarr_version, shards=shards,
     )
 
 
 def _copy_image_data(files, key, root,
                      dataset_name, source_names,
                      file_format,  resolution, unit,
                      scale_factors, chunks,
-                     tmp_folder, target, max_jobs, is_seg=False):
+                     tmp_folder, target, max_jobs, is_seg=False,
+                     ome_zarr_version="0.4", shards=None):
     assert len(files) == len(source_names)
     ds_folder = os.path.join(root, dataset_name)
     sources = list(metadata.read_dataset_metadata(ds_folder).get("sources", {}).keys())
@@ -53,7 +56,8 @@ def _copy_image_data(files, key, root,
         functools.partial(_import_one_source, input_files=input_files, output_files=output_files,
                           names=input_names, key=key, file_format=file_format,
                           resolution=resolution, unit=unit, scale_factors=scale_factors,
-                          chunks=chunks, is_seg=is_seg),
+                          chunks=chunks, is_seg=is_seg,
+                          ome_zarr_version=ome_zarr_version, shards=shards),
         len(input_files), num_workers=num_workers, has_return_val=False, name="htm-import",
     )
     return input_names, metadata_paths
@@ -128,6 +132,7 @@ def add_images(
     unit: str = "micrometer",
     is_default_dataset: bool = False,
     is2d: Optional[bool] = None,
+    shards: Optional[Sequence[int]] = None,
 ) -> None:
     """Add images from a high-content microscopy experiment to a MoBIE dataset.
 
@@ -149,9 +154,15 @@ def add_images(
         is_default_dataset: Whether this is the default dataset.
             Only relevant if the dataset will be created.
         is2d: Whether this is a 2D datasets.
+        shards: The shard shape for zarr v3 sharding. Only supported for the ome.zarr v0.5 format
+            (pass file_format='ome.zarr@0.5').
     """
     assert len(files) == len(image_names), f"{len(files)}, {len(image_names)}"
 
+    # the ome.zarr / NGFF version may be encoded as a suffix on the file format (e.g. 'ome.zarr@0.5').
+    file_format, ome_zarr_version = utils.parse_file_format(file_format)
+    utils.check_shards(shards, file_format, ome_zarr_version)
+
     # require the dataset
     if is2d is None:
         is2d = len(resolution) == 2
@@ -163,7 +174,8 @@ def add_images(
                                                     dataset_name, image_names,
                                                     file_format,  resolution, unit,
                                                     scale_factors, chunks,
-                                                    tmp_folder, target, max_jobs, is_seg=False)
+                                                    tmp_folder, target, max_jobs, is_seg=False,
+                                                    ome_zarr_version=ome_zarr_version, shards=shards)
 
     # add metadata for all the images
     if source_names:
@@ -187,6 +199,7 @@ def add_segmentations(
     unit: str = "micrometer",
     is_default_dataset: bool = False,
     is2d: Optional[bool] = None,
+    shards: Optional[Sequence[int]] = None,
 ) -> None:
     """Add segmentation data for a high-content microscopy experiment to a MoBIE dataset.
 
@@ -209,9 +222,15 @@ def add_segmentations(
         is_default_dataset: Whether this is the default dataset.
             Only relevant if the dataset will be created.
         is2d: Whether this is a 2D datasets.
+        shards: The shard shape for zarr v3 sharding. Only supported for the ome.zarr v0.5 format
+            (pass file_format='ome.zarr@0.5').
     """
     assert len(files) == len(segmentation_names)
 
+    # the ome.zarr / NGFF version may be encoded as a suffix on the file format (e.g. 'ome.zarr@0.5').
+    file_format, ome_zarr_version = utils.parse_file_format(file_format)
+    utils.check_shards(shards, file_format, ome_zarr_version)
+
     # require the dataset
     if is2d is None:
         is2d = len(resolution) == 2
@@ -224,7 +243,8 @@ def add_segmentations(
                                                     dataset_name, segmentation_names,
                                                     file_format,  resolution, unit,
                                                     scale_factors, chunks,
-                                                    tmp_folder, target, max_jobs, is_seg=True)
+                                                    tmp_folder, target, max_jobs, is_seg=True,
+                                                    ome_zarr_version=ome_zarr_version, shards=shards)
 
     if add_default_tables:
         table_folders = _add_tables(file_format, metadata_paths,

mobie/image_data.py

@@ -220,6 +220,7 @@ def add_image(
     channel: Optional[int] = None,
     skip_add_to_dataset: bool = False,
     use_memmap: bool = False,
+    shards: Optional[Sequence[int]] = None,
 ) -> None:
     """Add an image source to a MoBIE dataset.
 
@@ -257,7 +258,14 @@ def add_image(
         use_memmap: Whether to use memmap for loading the input data.
             This option is only supported for inputs in tif file format that can be loaded via `tifffile.memmap`.
             This does not work for images that are compressed or have an otherwise non-standard format.
+        shards: The shard shape for zarr v3 sharding. Only supported for the ome.zarr v0.5 format
+            (pass file_format='ome.zarr@0.5').
     """
+    # the ome.zarr / NGFF version may be encoded as a suffix on the file format (e.g. 'ome.zarr@0.5').
+    # split it off here so all downstream folder / metadata-key / dispatch logic sees the canonical name.
+    file_format, ome_zarr_version = utils.parse_file_format(file_format)
+    utils.check_shards(shards, file_format, ome_zarr_version)
+
     # TODO add 'setup_id' to the json schema for bdv formats to also support it there
     if channel is not None and file_format != "ome.zarr":
         raise NotImplementedError("Channel setting is currently only supported for ome.zarr")
@@ -296,7 +304,9 @@ def add_image(
                           max_jobs=max_jobs, unit=unit,
                           source_name=image_name,
                           file_format=file_format,
-                          channel=channel)
+                          channel=channel,
+                          ome_zarr_version=ome_zarr_version,
+                          shards=shards)
 
     if transformation is not None:
         utils.update_transformation_parameter(image_metadata_path, transformation, file_format)
@@ -323,4 +333,5 @@ def main():
               view=view, menu_name=args.menu_name,
               tmp_folder=args.tmp_folder, target=args.target, max_jobs=args.max_jobs,
               is_default_dataset=bool(args.is_default_dataset),
-              transformation=transformation, unit=args.unit)
+              transformation=transformation, unit=args.unit,
+              **utils.get_source_kwargs(args))

mobie/import_data/_format_metadata.py

@@ -21,12 +21,14 @@
 AXES_TYPES = {"t": "time", "c": "channel", "z": "space", "y": "space", "x": "space"}
 
 
-def _write_ome_zarr_metadata(path, metadata_dict, scale_factors):
+def _write_ome_zarr_metadata(path, metadata_dict, scale_factors, ome_zarr_version="0.4"):
     setup_name = metadata_dict.get("setup_name", None)
     setup_name = "data" if setup_name is None else setup_name
     unit = metadata_dict.get("unit", "pixel")
 
-    with z5py.File(path, mode="a", dimension_separator="/") as f:
+    # v0.4 -> zarr v2 (.zattrs, dimension_separator='/'); v0.5 -> zarr v3 (zarr.json).
+    open_kwargs = {"zarr_format": 3} if ome_zarr_version == "0.5" else {"dimension_separator": "/"}
+    with z5py.File(path, mode="a", **open_kwargs) as f:
         ndim = f["s0"].ndim
         axes_names = ["y", "x"] if ndim == 2 else ["z", "y", "x"]
         resolution = metadata_dict.get("resolution", [1.0] * ndim)
@@ -41,9 +43,13 @@ def _write_ome_zarr_metadata(path, metadata_dict, scale_factors):
             {"path": f"s{level}", "coordinateTransformations": [{"type": "scale", "scale": scale}]}
             for level, scale in enumerate(scales)
         ]
-        f.attrs["multiscales"] = [
-            {"axes": axes, "datasets": datasets, "name": setup_name, "version": "0.4"}
-        ]
+        multiscales = [{"axes": axes, "datasets": datasets, "name": setup_name}]
+        if ome_zarr_version == "0.5":
+            # NGFF v0.5: the metadata lives under an 'ome' key, with the version at the 'ome' level.
+            f.attrs["ome"] = {"version": "0.5", "multiscales": multiscales}
+        else:
+            multiscales[0]["version"] = ome_zarr_version
+            f.attrs["multiscales"] = multiscales
 
 
 def _write_bdv_metadata(metadata_format, path, metadata_dict, scale_factors):
@@ -66,20 +72,22 @@ def _write_bdv_metadata(metadata_format, path, metadata_dict, scale_factors):
         write_n5_metadata(path, scale_factors, resolution)
 
 
-def write_format_metadata(metadata_format, path, metadata_dict, scale_factors):
+def write_format_metadata(metadata_format, path, metadata_dict, scale_factors, ome_zarr_version="0.4"):
     """Write the multiscale metadata for the given storage format.
 
     Args:
         metadata_format: The storage format. One of 'ome.zarr', 'bdv.n5', 'bdv.hdf5'.
         path: The path to the (already written) multiscale data.
         metadata_dict: The metadata values, with keys 'resolution', 'unit' and 'setup_name'.
         scale_factors: The relative per-level downscaling factors (without the s0 identity).
+        ome_zarr_version: The ome.zarr / NGFF version to write. Only applies to the 'ome.zarr'
+            format ('0.4' -> zarr v2, '0.5' -> zarr v3).
 
     Raises:
         ValueError: If the storage format is not supported.
     """
     if metadata_format == "ome.zarr":
-        _write_ome_zarr_metadata(path, metadata_dict, scale_factors)
+        _write_ome_zarr_metadata(path, metadata_dict, scale_factors, ome_zarr_version)
     elif metadata_format in ("bdv", "bdv.n5", "bdv.hdf5"):
         _write_bdv_metadata(metadata_format, path, metadata_dict, scale_factors)
     else: