Skip to content

Add auto-shard configuration and manage anndata/dask pins#176

Merged
lorenzocerrone merged 6 commits into
mainfrom
chore/add-autoshard-config-anndata
May 18, 2026
Merged

Add auto-shard configuration and manage anndata/dask pins#176
lorenzocerrone merged 6 commits into
mainfrom
chore/add-autoshard-config-anndata

Conversation

@lorenzocerrone
Copy link
Copy Markdown
Collaborator

@lorenzocerrone lorenzocerrone commented May 18, 2026

Fix

  • Replace da.to_zarr with da.store(..., lock=False) in pyramid writes (_on_disk_dask_zoom, _on_disk_coarsen) and region slice writes (_ops_slices). Dask >=2025.11's to_zarr re-derives chunks via normalize_chunks(chunks="auto", ...) and emits a PerformanceWarning (treated as error by ngio's filterwarnings) when the result is not a multiple of the target's chunks; da.store writes blocks 1:1.
  • Copy object/string-dtype zarr arrays directly when consolidating groups: dask >=2025.11 raises NotImplementedError from auto-chunking for these dtypes, so they bypass dask and are copied via numpy.
  • Set auto_shard_zarr_v3 together with zarr_write_format on anndata's global settings via a new _update_anndata_global_settings helper, so reading/writing tables works correctly when mixing zarr v2 and v3 in the same session on anndata 0.12.

Chores

  • Pin anndata to >=0.12.0,<0.13.0.
  • Unpin dask (remove the <2025.11.0 upper bound introduced in v0.4.5).

Checklist before merging

  • I added an appropriate entry to CHANGELOG.md

@codecov
Copy link
Copy Markdown

codecov Bot commented May 18, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@lorenzocerrone lorenzocerrone merged commit 6025187 into main May 18, 2026
7 checks passed
@lorenzocerrone lorenzocerrone deleted the chore/add-autoshard-config-anndata branch May 18, 2026 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant