Restart of model and writing output are inconsistent
restart = 0.15 days
(/work/mh0010/m300408/anaconda3) m300408@mlogin108% more logs/LOG.exp.DOM01large.run.29441112.o | grep "Time step:" | tail
Time step: 2151, model time: 2020-02-01 13:35:06.000 forecast time 00D 03H35M06S
Time step: 2152, model time: 2020-02-01 13:35:12.000 forecast time 00D 03H35M12S
Time step: 2153, model time: 2020-02-01 13:35:18.000 forecast time 00D 03H35M18S
Time step: 2154, model time: 2020-02-01 13:35:24.000 forecast time 00D 03H35M24S
Time step: 2155, model time: 2020-02-01 13:35:30.000 forecast time 00D 03H35M30S
Time step: 2156, model time: 2020-02-01 13:35:36.000 forecast time 00D 03H35M36S
Time step: 2157, model time: 2020-02-01 13:35:42.000 forecast time 00D 03H35M42S
Time step: 2158, model time: 2020-02-01 13:35:48.000 forecast time 00D 03H35M48S
Time step: 2159, model time: 2020-02-01 13:35:54.000 forecast time 00D 03H35M54S
Time step: 2160, model time: 2020-02-01 13:36:00.000 forecast time 00D 03H36M00S
restart has been changed to 0.5 days
(/work/mh0010/m300408/anaconda3) m300408@mlogin108% more logs/LOG.exp.DOM01large.run.29509395.o | grep "Time step:" | head
Time step: 2161, model time: 2020-02-01 13:36:06.000 forecast time 00D 03H36M06S
Time step: 2162, model time: 2020-02-01 13:36:12.000 forecast time 00D 03H36M12S
Time step: 2163, model time: 2020-02-01 13:36:18.000 forecast time 00D 03H36M18S
Time step: 2164, model time: 2020-02-01 13:36:24.000 forecast time 00D 03H36M24S
Time step: 2165, model time: 2020-02-01 13:36:30.000 forecast time 00D 03H36M30S
Time step: 2166, model time: 2020-02-01 13:36:36.000 forecast time 00D 03H36M36S
Time step: 2167, model time: 2020-02-01 13:36:42.000 forecast time 00D 03H36M42S
Time step: 2168, model time: 2020-02-01 13:36:48.000 forecast time 00D 03H36M48S
Time step: 2169, model time: 2020-02-01 13:36:54.000 forecast time 00D 03H36M54S
Time step: 2170, model time: 2020-02-01 13:37:00.000 forecast time 00D 03H37M00S
(/work/mh0010/m300408/anaconda3) m300408@mlogin108% more logs/LOG.exp.DOM01large.run.29509395.o | grep "Time step:" | tail
Time step: 9351, model time: 2020-02-02 01:35:06.000 forecast time 00D 15H35M06S
Time step: 9352, model time: 2020-02-02 01:35:12.000 forecast time 00D 15H35M12S
Time step: 9353, model time: 2020-02-02 01:35:18.000 forecast time 00D 15H35M18S
Time step: 9354, model time: 2020-02-02 01:35:24.000 forecast time 00D 15H35M24S
Time step: 9355, model time: 2020-02-02 01:35:30.000 forecast time 00D 15H35M30S
Time step: 9356, model time: 2020-02-02 01:35:36.000 forecast time 00D 15H35M36S
Time step: 9357, model time: 2020-02-02 01:35:42.000 forecast time 00D 15H35M42S
Time step: 9358, model time: 2020-02-02 01:35:48.000 forecast time 00D 15H35M48S
Time step: 9359, model time: 2020-02-02 01:35:54.000 forecast time 00D 15H35M54S
Time step: 9360, model time: 2020-02-02 01:36:00.000 forecast time 00D 15H36M00S
Job 29509395 run for 12 model hours, but the restart file has been written after 12 model hours since the original start time e.g. 2020-02-01 10:00:00 and so 3.5 hours are calculated again with the following job.
(/work/mh0010/m300408/anaconda3) m300408@mlogin108% more logs/LOG.exp.DOM01large.run.29509910.o | grep "Time step:" | head
Time step: 7201, model time: 2020-02-01 22:00:06.000 forecast time 00D 12H00M06S
Time step: 7202, model time: 2020-02-01 22:00:12.000 forecast time 00D 12H00M12S
Time step: 7203, model time: 2020-02-01 22:00:18.000 forecast time 00D 12H00M18S
Time step: 7204, model time: 2020-02-01 22:00:24.000 forecast time 00D 12H00M24S
Time step: 7205, model time: 2020-02-01 22:00:30.000 forecast time 00D 12H00M30S
Time step: 7206, model time: 2020-02-01 22:00:36.000 forecast time 00D 12H00M36S
Time step: 7207, model time: 2020-02-01 22:00:42.000 forecast time 00D 12H00M42S
Time step: 7208, model time: 2020-02-01 22:00:48.000 forecast time 00D 12H00M48S
Time step: 7209, model time: 2020-02-01 22:00:54.000 forecast time 00D 12H00M54S
Time step: 7210, model time: 2020-02-01 22:01:00.000 forecast time 00D 12H01M00S
(/work/mh0010/m300408/anaconda3) m300408@mlogin108% more logs/LOG.exp.DOM01large.run.29509910.o | grep "Time step:" | tail
Time step: 14391, model time: 2020-02-02 09:59:06.000 forecast time 00D 23H59M06S
Time step: 14392, model time: 2020-02-02 09:59:12.000 forecast time 00D 23H59M12S
Time step: 14393, model time: 2020-02-02 09:59:18.000 forecast time 00D 23H59M18S
Time step: 14394, model time: 2020-02-02 09:59:24.000 forecast time 00D 23H59M24S
Time step: 14395, model time: 2020-02-02 09:59:30.000 forecast time 00D 23H59M30S
Time step: 14396, model time: 2020-02-02 09:59:36.000 forecast time 00D 23H59M36S
Time step: 14397, model time: 2020-02-02 09:59:42.000 forecast time 00D 23H59M42S
Time step: 14398, model time: 2020-02-02 09:59:48.000 forecast time 00D 23H59M48S
Time step: 14399, model time: 2020-02-02 09:59:54.000 forecast time 00D 23H59M54S
Time step: 14400, model time: 2020-02-02 10:00:00.000 forecast time 01D 00H00M00S
This issue probably only occurs in case dt_restart
is changed in between. I expected dt_restart
to be referenced to each start time of a specific run and that the computation and the restart file writing process are aligned.