Redesign planning step
Currently the planning (or coordination) step of hiopy
isn't actually in hiopy. Furthermore, it's a monolitic block of code, which worked for the nextGEMS cycle3 runs, but is hard to maintain and hard to adapt to new features. My guess would be, at least #2, #4, #9, #11, #13 would be affected by this. The planning step should be rethought and become more modular and more generic.
current process
expand_plan reads in a configuration and the list of ICON variables. Based on those two files, it create a huge new yaml file in a single function, which in turn controls everything in hiopy
using the hiopy configure
tool. See also the overview diagram.
ideas for the future
- the list of icon variables probably stays a bit, it would be great to extract it from ICON, but it's currently not forseeable when / how this will happen
- tools (e.g. healpix-output, meteogram-output, pressure interpolation etc...) could probably declare generically what they need and provide, which would then be used by a new planning tool, such that the planning tool will autmatically adjust itself to new features
- weird assumptions should go away, currently these are (maybe incomplete):
- coarser time resolutions always produce averaged results
- some magic numbers of the healpix grid are hardcoded
- it's assumed that there is always a hierarchy in time and another one in space
- it's assumed that only one (hierarchy of) dataset is produced
- the count of levels in the model shouldn't be part of the output configuration
- the coupling configuration should probably be more generic or even go away. For YAC3 there's some hope, that we don't need it (#12 (closed))
- maybe we should think of the possibility of having other transports eventually (especially for communication between hiopy workers)