Request For Comment: Move external and large reference data out of Git and into a dynamic Pixi task / script

[kayhangultekin]

Problem

Our current repository layout includes several large reference datasets committed directly inside holodeck/data/. This introduces significant technical friction and data-compliance challenges.

1. Technical Friction & Git Bloat

Because Git tracks history permanently, storing binary formats (like .hdf5, .npz, and .tar.gz) breaks delta compression. Every modification forces Git to store a fresh copy of the asset.
You can reproduce and audit this bloat yourself using the following commands:

• Check the active workspace size for data and dotfiles:

du -sh * .[^.]* | sort -rh

• Verify the heaviest historical objects packing down the .git folder:

git verify-pack -v .git/objects/pack/pack-*.idx \
  | grep blob \
  | sort -k3 -nr \
  | head -n 10 \
  | while read -r hash type size size_pack offset; do \
      file=$(git rev-list --objects --all | grep "$hash" | awk '{print $2}'); \
      echo "$((size_pack / 1024 / 1024)) MB \t (uncompressed: $((size / 1024 / 1024)) MB) \t $file"; \
    done

Result: The local directory contains a little under 200 MB of active data (bad), but the historical .git/objects/pack binaries take up over 700 MB of redundant space (terrible) due to old snapshots of these files (e.g., multiple copies of 67 MB TNG HDF5 logs).

2. Licensing & Authorship Compliance

Almost all of these reference data are the product of external collaborations (such as IllustrisTNG). We don't own the copyright to these data and papers and can't distribute them in this git repo maybe at all, but definitely not under an MIT license. It also implies our authorship.

Proposed Strategy

We can decouple our functional code logic from external scientific data products

1. Purge History: Permanently scrub these heavy data paths out of the Git ledger using git-filter-repo --path holodeck/data/.
2. Ignore: Add holodeck/data/ straight to the .gitignore.
3. Automate: Add a native fetching helper (e.g., pixi task pixi run download-data in pyproject.toml or make a download-holodeck-auxdata.sh script) so users and contributors can download the active data directly to their local environments only when needed.

Items Up for Discussion

1. Storage Hosting: Where should we house these data? Do we use a NANOGrav Google Drive, institutional server, or GitLab Pages? Crucially, do we have explicit permission from the original creators to mirror and host copies of these specific datasets publicly?
2. Data Retations: Do we actively need all of these data from our core simulation runs or current unit testing suites, or can some of these legacy assets be retired entirely?

Please share your thoughts on access requirements and hosting setups below.

(This issue was brought up by a comment by @scottransom on slack. I investigated with the help of Google Gemini 3.5 Flash, confirmed my proposed solution with the LLM, and had the LLM write 99% of the above text.)

[astrolamb]

Storage hosting: use zenodo -- there is a 50GB limit per record, and it has version histories and DOIs

[kayhangultekin]

So some of the data files are actually used by holodeck for things like numerical interpolations. We can't simply delete them without breaking core operability, but we still shouldn't be tracking them in Git due to repository bloat.

Instead, we should host these large datasets externally (e.g., in a Zenodo repository) and supply a script or Pixi task to download them on demand.

To make this seamless and bulletproof for users, I propose we channel all holodeck/data file access through a centralized utility function (likely in utils.py). This utility will handle file loading and act as a defensive gatekeeper: if a required data file is missing locally, it will intercept the crash and print a clean, helpful error message, e.g., "You need to download xyz.dat. See data/README.md for details on how to fetch it." This keeps the repository lightweight while ensuring the onboarding experience doesn't completely break if someone runs code before downloading the assets.

[astrolamb]

PTArcade has a lovely utility function that downloads data when required. The function can be found via this link.

Downloading the same data set with each call to the function that requires it wouldn't be efficient for files more than a few MB. But perhaps the first call to the function could check if it's in holodeck/data, and then download it if it can't find it?

[kayhangultekin]

Thanks for that, @astrolamb! It looks like it uses astropy's download_file, which is a straightforward way to do this.

[kayhangultekin]

A downside of astropy's download_file is that it stores it in the users home directory, which may not be ideal for clusters.

[astrolamb]

There are alternatives that have more options, such as shutil

[kayhangultekin]

It's possible hack one's way to a solution with astropy (astropy/astropy#17507), but it could be problematic. Pooch seems like a great way to handle it.

[kayhangultekin]

Okay, like the monks of the Order of Dagon, I have rewritten history by purging large and/or binary files from the repository history using git-filter-repo. I have pushed the clean history over to the dev-offloadbindata branch. My commit message from there:

WARNING: This commit used git-filter-repo to rewrite history. Care will need to be taken when merging.

Summary

The primary objective of this commit is to dramatically reduce the size of the repository gitpack history to allow for a streamlined, lightweight, and high-performance local development footprint. Over years of active simulation tracking, the repository size inflated to ~1.5 GB due to the accumulation of heavy scientific datasets, lookup tables, and PDF reference files. By executing a targeted history purge using git-filter-repo, the permanent repository size has been successfully optimized down to an incredibly lean ~43 MB.

Long-Term Asset & Development Strategy

To prevent the codebase history from re-inflating and to ensure rapid cloning moving forward, we have transitioned to a decoupled data architecture:

1. Dynamic Data On-Demand (data_manager.py & pooch):
   Large raw binary datasets (such as galaxy merger trees, stellar mass functions, and simulation tables) have been offloaded externally to public cloud storage hosting. The codebase now resolves these assets dynamically at runtime using a cryptographic hash registry and automated caching via Pooch. See holodeck/data/README.md for more.

2. Strict Tracking Rules (.gitignore Modifications):
   The root .gitignore has been updated with a rigid barrier protecting holodeck/data/*. It actively fences off heavy structures while explicit lookahead whitelists ensure essential, human-readable layout keys, configuration texts, and text lookup manifests remain cleanly tracked.

3. Reference Policy (docs/ Restrictions):
   Academic and literature reference PDFs are explicitly banned from direct repository versioning. External literature citations should be captured textually in documentation, and core project assets can be packaged together externally rather than filling the code source history with binary blocks. See docs/references/README.md for more.

Complete Audit of Purged Historical Files

The following large binaries, dataset packages, reference assets, media files, and transient development notebooks were fully expunged from the historical tree: