General export overhaul meta-issue
While the new export stuff is an improvement over its predecessor, there are still some major issues with the current implementation. I propose that this time, we actually lay out a plan on how the export API should look instead of just fiddling with details for a few weeks and then calling the emerging mess of functions and structs an API.
Features that I would like to see in an improved API are
-
a way to add more or less automated in-memory buffering to prevent Dorian's problem Context for new people: Our Alumnus Dorian managed to overload the OPA interconnects (THE OPA INTERCONNECTS!) by writing gazillions of tiny snapshots from hundreds of different jobs in a very short amount of time -
a consistent and expandable interface for the particle containers, so particle containers with custom dynamic objects don't have to overwrite core functions (e.g. readsnap
) to make io work. -
an intermediate representation of a snapshot (see #17) to prevent having to deserialise stuff three times in a row for three different analysis functions. Edit: we don't even need to add a custom type for this, I propose just using the tuple of stuff returned by readsnap
and adding asetstate!(sim, tuple_of_stuff...)
function
Feel free to contribute your own ideas here!
Edited by Lukas Hupe