We currently add a step to all of our CI builds to uninstall the NetCDF4 library, as it tends to run into problems when running on GitHub Workflows. Testing seems to work without issue when simply using h5netcdf/h5py.
My hypothesis is that when we launch tests using pytest-xdist, we are opening test NetCDF files and potentially performing operations that are not thread-safe while they are open. I can see a few options going forward to fix this:
One potential workaround is to copy files into separate pytest worker-related caches (this is what we did in xclim when we were using the NetCDF4 library) given that the C and Python libraries are not thread-safe.
Alternatively, we could spend more effort ensuring that NetCDF files loaded by each pytest worker are written to a test file and reopened as (similar to the solution above, but could be managed entirely by a pytest fixture/decorator).
Finally, we could drop the NetCDF4 library entirely in favour of h5netcdf/h5py. It's "good enough" for running the tests already. Whether users want to use NetCDF4 could be left entirely up to them.
We currently add a step to all of our CI builds to uninstall the
NetCDF4library, as it tends to run into problems when running on GitHub Workflows. Testing seems to work without issue when simply usingh5netcdf/h5py.My hypothesis is that when we launch tests using
pytest-xdist, we are opening test NetCDF files and potentially performing operations that are not thread-safe while they are open. I can see a few options going forward to fix this:One potential workaround is to copy files into separate
pytestworker-related caches (this is what we did inxclimwhen we were using theNetCDF4library) given that the C and Python libraries are not thread-safe.Alternatively, we could spend more effort ensuring that NetCDF files loaded by each
pytestworker are written to a test file and reopened as (similar to the solution above, but could be managed entirely by apytestfixture/decorator).Finally, we could drop the
NetCDF4library entirely in favour ofh5netcdf/h5py. It's "good enough" for running the tests already. Whether users want to useNetCDF4could be left entirely up to them.