Using the _OPTIONS section with lists

[robdarvell]

I've been having a play with using lists in order to specify several levels for a particular variable rather than writing out each level as a separate section with netcdf files:
eg:
BOTH_VAR1_NAME = Temp
BOTH_VAR1_LEVELS = (@850,*,*)
BOTH_VAR2_NAME = Temp
BOTH_VAR2_LEVELS = (@500,*,*)

to (one way of listing it)
UPPER_LEVELS= "(@850,*,*)", "(@500,*,*)"
BOTH_VAR1_NAME = Temp
BOTH_VAR1_LEVELS = {UPPER_LEVELS}

As it stands this would put the (@850,*,*) into the level part of the results/output file.

At present I use a set_attr_level option within the _OPTIONS section in order to write the level out in a different way eg: to P850 (as that looks better than the original)

Is there any way in which this _OPTIONS setting can be written in order to cover what I have attempted to do in the method using the UPPER_LEVELS variable. I have tried creating a new variable and link that to the set_attr command, but that only appears to like a single entry from what I can see. I can also see other places where this could also shorten lists from what people (me!) are doing eg: if you are regridding before running gridstat and in the naming of the regridded variables.

But this is more a general question to begin with (and I think I have it in the correct category).

Many thanks
Rob

[j-opatz]

Hi Rob,

And thank you for your question. From my reading of your discussion, it seems like you'd like a way to list multiple levels for a given variable field. Given situations where the forecast and observation fields would have the same access information (variable name, pressure levels, etc.), this methodology would be a more streamlined experience compared to using a separate VAR entry for each level of interest.

Would that be an accurate summary of your question?

Unfortunately, there are no current methods to support that method of verification. I think the concern would be surrounding if users wanted to use that method, but also wanted to specify settings for one of those levels: using your example, if a user wanted the 850mb temps to have thresholds of 5 Celsius and 10 Celsius, but the 850mb temps to have thresholds of -15 Celsius and -20 Celsius. I'm not sure what the settings would look like.

But this seems like a potential future improvement I'd like to capture. Would you be willing to write out the configuration settings you would expect to do this type of logic? This can (ideally) include current settings that we could enhance, or completely new settings (less than ideal but still feasible).

I'll see about getting your thoughts synthesized and into an issue for METplus enhancement.

[robdarvell]

Thanks for the prompt reply. I think you have summarised what I was asked about in the response above, but will try and expand it a little bit below. I have not considered this approach for all possible MET tasks where it maybe thought about, so it may not fit all.
There are probably lots of different approaches which could be taken, but I was trying to work out which one may work out best. (but have probably not considered all possible options)
Summarising what I am thinking about. I’m using netcdf files here rather than GRIB.
If I have 15 different levels and 3 different fields each verifying on the 15 levels. I’m trying to come up with the best approach to reduce the size of the list in my METplus wrapper (to save me having 45 entries). I realise you could feed in one field at a time, but was trying to generalise things.

Some alternative options:

• Use the method as shown above. Set the UPPER_LEVELS as a variable and then use the BOTH_VAR1 (assuming everything is the same) which would give 3 VAR sections. This then doesn’t allow you to correct things in the output files eg: I was trying to avoid having the (@850,*,*) appearing in the results file so correct it using the _OPTIONS.
  You can I could guess, without this correction, manually run a Linux command to do a find and replace (@850,*,*) with P850 after you have run the task
• Using the CUSTOM_LOOP_LIST option. Set that to say to the list of the 15 levels and then again you could have the 3 field sections. This time though you could have the _LEVLES set to “(@{custom},*,*)” and then you could also use the {custom} in an _OPTIONS section. So that would work. You do though (please correct me if I have missed a variable setting), have to update your OUTPUT_PREFIX string to cover each option, otherwise you only get the final level in the results files. Down side of that option would be the number of output files you may end up with (especially if you want to run each forecast range separately to improve run times)
• Another way in which someone is running a task (but a slightly different one to this one but I could see it being adapted for this) has been they have invoked Jinja2 syntax into creating the wrapper script and are using a Jinja2 syntax to generate the loop before running the task. I not sure I really want to suggest that to people to work from.

So, from what we have at present, I was potentially thinking along the lines of something like this (which would answer my question, but may not be as flexible as it could be, noting part of your reply on thresholds usage, which I don’t have here). It sort of follows the use-case example where you had a different list of level info in a Pointstat job between the forecast and observations (but they needed to have the same number of entries):

UPPER_LEVELS = “(@1000,*,*)”, “(@850,*,*)”, “(@500,*,*)”
UPPER_OUTPUT = “P1000”, “P850”, “P500”
BOTH_VAR1_NAME= air_temperature
BOTH_VAR1_LEVELS = {UPPER_LEVELS}
BOTH_VAR1_OPTIONS = set_attr_level={UPPER_OUTPUT};
BOTH_VAR2_NAME= humidity
BOTH_VAR2_LEVELS = {UPPER_LEVELS}
BOTH_VAR2_OPTIONS = set_attr_level={UPPER_OUTPUT};

Whether a threshold list could also be envoked using a similar list structure (with some sort of missing data indicator if a level isn’t required) is another thought.
(don't know whether this syntax would even work)
UPPER_THRESH_1 = "<=100, <=200", "<=400, <=500", "<=600, <=750"
UPPER_THRESH_2 = "<=1, <=20", "<=4, <=50", "<=6, <=75"
BOTH_VAR1_THRESH = [{UPPER_THRESH_1}]
BOTH_VAR2_THRESH = [{UPPER_THRESH_2}]

Using the second option I had been playing with, it would look something like:

CUSTOM_LOOP_LIST = 1000, 850, 500
BOTH_VAR1_NAME= air_temperature
BOTH_VAR1_LEVELS = “(@{custom},*,*)”
BOTH_VAR1_OPTIONS = set_attr_level=”P{custom}”;
BOTH_VAR2_NAME= humidity
BOTH_VAR2_LEVELS = “(@{custom},*,*)”
BOTH_VAR2_OPTIONS = set_attr_level=”P{custom}”;

This has had less thought about the use of thresholds which a user may or may not wish to use (but whether a user would invoke this method should threshold be involved, you could do something else perhaps), but may just require some sort of additional variable to get all the values in the list to appear in the same output file rather than it overwriting each time.
(note I have not even considered how feasible that may or may not be from how the code base sits and how things loop)

[robdarvell]

Thanks for the prompt reply. I think you have summarised what I was asked about in the response above, but will try and expand it a little bit below. I have not considered this approach for all possible MET tasks where it maybe thought about, so it may not fit all.
There are probably lots of different approaches which could be taken, but I was trying to work out which one may work out best. (but have probably not considered all possible options)
Summarising what I am thinking about. I’m using netcdf files here rather than GRIB.
If I have 15 different levels and 3 different fields each verifying on the 15 levels. I’m trying to come up with the best approach to reduce the size of the list in my METplus wrapper (to save me having 45 entries). I realise you could feed in one field at a time, but was trying to generalise things.

Some alternative options:

Use the method as shown above. Set the UPPER_LEVELS as a variable and then use the BOTH_VAR1 (assuming everything is the same) which would give 3 VAR sections. This then doesn’t allow you to correct things in the output files eg: I was trying to avoid having the (@850,*,*) appearing in the results file so correct it using the _OPTIONS.
You can I could guess, without this correction, manually run a Linux command to do a find and replace (@850,*,*) with P850 after you have run the task
Using the CUSTOM_LOOP_LIST option. Set that to say to the list of the 15 levels and then again you could have the 3 field sections. This time though you could have the _LEVLES set to “(@{custom},*,*)” and then you could also use the {custom} in an _OPTIONS section. So that would work. You do though (please correct me if I have missed a variable setting), have to update your OUTPUT_PREFIX string to cover each option, otherwise you only get the final level in the results files. Down side of that option would be the number of output files you may end up with (especially if you want to run each forecast range separately to improve run times)
Another way in which someone is running a task (but a slightly different one to this one but I could see it being adapted for this) has been they have invoked Jinja2 syntax into creating the wrapper script and are using a Jinja2 syntax to generate the loop before running the task. I not sure I really want to suggest that to people to work from.
So, from what we have at present, I was potentially thinking along the lines of something like this (which would answer my question, but may not be as flexible as it could be, noting part of your reply on thresholds usage, which I don’t have here). It sort of follows the use-case example where you had a different list of level info in a Pointstat job between the forecast and observations (but they needed to have the same number of entries):

UPPER_LEVELS =“(@1000,*,*)”, “(@850,*,*)”, “(@500,*,*)”
UPPER_OUTPUT = “P1000”, “P850”, “P500”
BOTH_VAR1_NAME= air_temperature
BOTH_VAR1_LEVELS = {UPPER_LEVELS}
BOTH_VAR1_OPTIONS = set_attr_level={UPPER_OUTPUT};
BOTH_VAR2_NAME= humidity
BOTH_VAR2_LEVELS = {UPPER_LEVELS}
BOTH_VAR2_OPTIONS = set_attr_level={UPPER_OUTPUT};

Whether a threshold list could also be envoked using a similar list structure (with some sort of missing data indicator if a level isn’t required) is another thought.
(don't know whether this syntax would even work)
UPPER_THRESH_1 = "<=100, <=200", "<=400, <=500", "<=600, <=750"
UPPER_THRESH_2 = "<=1, <=20", "<=4, <=50", "<=6, <=75"
BOTH_VAR1_THRESH = [{UPPER_THRESH_1}]
BOTH_VAR2_THRESH = [{UPPER_THRESH_2}]

Using the second option I had been playing with, it would look something like:

CUSTOM_LOOP_LIST = 1000, 850, 500
BOTH_VAR1_NAME= air_temperature
BOTH_VAR1_LEVELS = “(@{custom},*,*)”
BOTH_VAR1_OPTIONS = set_attr_level=”P{custom}”;
BOTH_VAR2_NAME= humidity
BOTH_VAR2_LEVELS = “(@{custom},*,*)”
BOTH_VAR2_OPTIONS = set_attr_level=”P{custom}”;

This has had less thought about the use of thresholds which a user may or may not wish to use (but whether a user would invoke this method should threshold be involved, you could do something else perhaps), but may just require some sort of additional variable to get all the values in the list to appear in the same output file rather than it overwriting each time.
(note I have not even considered how feasible that may or may not be from how the code base sits and how things loop)

[georgemccabe]

Hi Rob @robdarvell,

There is an open METplus issue #2619 to add functionality to expand a list of strings to create a list of settings for each string in the list. I think this would allow you to do what you are trying to do in a single call. The custom loop list functionality is similar, but as you mentioned, it requires separate calls to the tools and therefore many additional output files.

With the changes in #2619 and the enhancement of VAR<n>_OPTIONS to support a list of values (that must match the number of VAR<n>_LEVELS), you should be able to do:

PRESSURE_LIST = 1000, 850, 500

BOTH_VAR1_NAME= air_temperature
BOTH_VAR1_LEVELS = “(@expand_string({PRESSURE_LIST}),*,*)”
BOTH_VAR1_OPTIONS = set_attr_level=”Pexpand_string({PRESSURE_LIST})”;

BOTH_VAR2_NAME= humidity
BOTH_VAR2_LEVELS = {BOTH_VAR1_LEVELS}
BOTH_VAR2_OPTIONS = {BOTH_VAR1_OPTIONS}

[robdarvell]

Hi George @georgemccabe ,
Yes that does appear as though it is going to solve what I have been requesting about above. Look forward to it appearing once funds, schedules etc can be arranged for it.