added the reinflate api#499
Conversation
| zarr_ref_store: dict, | ||
| ) -> dict: | ||
| """ | ||
| Given a zarr_store hierarchy, pull out the variables present in the |
There was a problem hiding this comment.
There should be a one-line summary at the start of the docstring, since this is what help() and sphynx use as the title describing the method.
martindurant
left a comment
martindurant
left a comment
There was a problem hiding this comment.
The functions here are identical to the originals by @emfdavid ?
I am OK to include these now, but I think the next PR should move all of the tree/idx-related functions to another module (_grib_idx ?), and only import the public-facing functions into this module. As it is, the module is now extremely long compared to other formats, and it's not so obvious where a user should start.
|
Separate module and only exposing public functions sounds good to me. |
|
Is there a better name for this function? I am terrible at names... |
You mean "reinflate"? Maybe if you write a 2-sentence description, the name will present itself :) |
|
@martindurant what should be the module name? The module should be included in a sub-package like |
|
I suggested |
|
Comments on naming (not my strong suit)... the Camus team had suggested "outbasing" as the name for this general approach of ingesting an index of chunks to a database and constructing the dataset/tree on the fly. The "reinflate" method in particular builds the refspec dictionary for a zarr/xarray tree representing a set of grib data. The inputs are the static metadata for the refspec (everything but the chunks for variables with a time dimension), a set of chunk indexes for specific variables (usually stored in a DB), the aggregation type, and the set of axes that define how to arrange those chunks. I wanted to be general, supporting all the FMRC slices which are represented in the AggregationType enum. The api allows passing a list of axes such that you can express a set of runtimes, a set of valid times, a set of horizons, or the 'best available' as of a certain time. I hope there is some simpler expression of these concepts that might use this method as a backend, but I suspect you would have to make assumptions about the backend database/schema to wrap it up nicely. |
|
|
Thank you for getting the process started. I will plan to make progress this month, but it will be a nights and weekends effort. If that is not progressing by the end of October we can regroup. |
|
Sure, I was simply book-keeping :) |
|
Check out #523 - I think that's intended to finish off the topic. |
|
@martindurant ooh, yes indeed, that looks like it! Thanks! |
|
Can we close this now that #523 is merged? |
4 participants
Added the
reinflatefunction to reinflate the index.