[2037]

[enssow]

Description

Fixes issue with selecting channels that don't use all available pressure levels or, a single level e.g. u_500 being treated as a surface variable.

The changes in the code will create a pressure index with all available levels but fill a speicifc variable at a given pressure level with Nans if not requested in hte code

e.g. only q_500 was requested to the ds["q"].values returns:

array([[[       nan],
        [       nan],
        [       nan],
        ...,
        [       nan],
        [       nan],
        [       nan]],

       [[0.00027466],
        [0.00028229],
        [0.00030136],
        ...,
        [0.00387573],
        [0.00354004],
        [0.00379944]]], shape=(2, 40320, 1), dtype=float32)

Closes two issues as refactored the find_pl to be better suited for the second issue as well

Issue Number

Closes #2037
Closes #2030

Is this PR a draft? Mark it as draft.

Checklist before asking for review

• I have performed a self-review of my code
• My changes comply with basic sanity checks:
  • I have fixed formatting issues with ./scripts/actions.sh lint
  • I have run unit tests with ./scripts/actions.sh unit-test
  • I have documented my code and I have updated the docstrings.
  • I have added unit tests, if relevant
• I have tried my changes with data and code:
  • I have run the integration tests with ./scripts/actions.sh integration-test
  • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
  • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
• I have informed and aligned with people impacted by my change:
  • for config changes: the MatterMost channels and/or a design doc
  • for changes of dependencies: the MatterMost software development channel

[s6sebusc]

looks like pl is returned but no longer used? could we just not return it?

[enssow]

oh yes that's true - I'm just going to double check no other parsers are using the function too

[s6sebusc]

Exporting single or multiple pressure levels works as expected now. Unfortunately exporting IMERG is broken again for me: part of the problem is that the channel tp_imerg_0 is recognized as pressure level zero. The reshape method therefore tries to add a pressure level to the data, which would work, except that the ipoint and channel dimensions are in the wrong order for IMERG for some reason. It then complains about

ValueError: conflicting sizes for dimension 'pressure_level': length 40320 on the data but length 1 on coordinate 'pressure_level'

I could fix that by saying data.sel(channel=old_vars).transpose("ipoint",...).values in l138 to ensure that ipoint is always first. Not sure if that's a good idea in general (shouldn't hurt though). The real problem is that IMERG shouldn't have a pressure level in the first place. Perhaps we add a check that changes pressurelevel 0 to None in find_pl since 0hPa isn't a real pressure level anyways?

[enssow]

Exporting single or multiple pressure levels works as expected now. Unfortunately exporting IMERG is broken again for me: part of the problem is that the channel tp_imerg_0 is recognized as pressure level zero. The reshape method therefore tries to add a pressure level to the data, which would work, except that the ipoint and channel dimensions are in the wrong order for IMERG for some reason. It then complains about

ValueError: conflicting sizes for dimension 'pressure_level': length 40320 on the data but length 1 on coordinate 'pressure_level'

I could fix that by saying data.sel(channel=old_vars).transpose("ipoint",...).values in l138 to ensure that ipoint is always first. Not sure if that's a good idea in general (shouldn't hurt though). The real problem is that IMERG shouldn't have a pressure level in the first place. Perhaps we add a check that changes pressurelevel 0 to None in find_pl since 0hPa isn't a real pressure level anyways?

I think having a check in find_pl is a good idea to avoid this - i'll update the PR now :)

[enssow]

Note to self: PR #1723 will have to be double checked for using find_pl too (I doubt it will be a big change as verif is using the surface variables so find_pl is not really actively used)