mo_name_list_output_mp_io_proc_write_name_list_() segmentation fault
- log: LOG.exp.DOM01+DOM02large.run.30867274.o
Time step: 87701, model time: 2020-01-15 12:10:06.000 forecast time 06D 02H10M06S
mo_name_list_output::write_name_list_output: closed EUREC4A_DOM01_radiation_20200115T120000Z.nc_part_2+
MAXABS VN, W in domain 1: 0.4970837748E+02 at level 39, 0.1395154706E+02 at level 79,
mo_name_list_output::open_output_file: opened EUREC4A_DOM02_radiation_20200115T120000Z.nc
mo_name_list_output::open_output_file: opened EUREC4A_DOM01_radiation_20200115T120000Z.nc
mo_name_list_output::write_name_list_output: Output to EUREC4A_DOM02_radiation_20200115T120000Z.nc_part_2+ at simulation time 2020-01-15T12:10:00.000 by PE 1917
mo_name_list_output::write_name_list_output: Output to EUREC4A_DOM01_radiation_20200115T120000Z.nc_part_2+ at simulation time 2020-01-15T12:10:00.000 by PE 1918
#################### I/O PE 1917 starting I/O at 222023.885
#################### I/O PE 1918 starting I/O at 222023.885
MAXABS VN, W in domain 1: 0.4970843914E+02 at level 39, 0.1397482876E+02 at level 79,
mo_name_list_output::open_output_file: opened EUREC4A_SYNSAT_RTTOV_FORWARD_MODEL_DOM01_ML_0145.nc
MAXABS VN, W in domain 1: 0.4970850127E+02 at level 39, 0.1401251604E+02 at level 79,
mo_name_list_output::write_name_list_output: Output to EUREC4A_SYNSAT_RTTOV_FORWARD_MODEL_DOM01_ML_0145.nc_part_2+ at simulation time 2020-01-15T12:10:00.000 by PE 1915
#################### I/O PE 1915 starting I/O at 222024.890
MAXABS VN, W in domain 1: 0.4970856158E+02 at level 39, 0.1404152798E+02 at level 79,
MAXABS VN, W in domain 1: 0.4970862040E+02 at level 39, 0.1402518413E+02 at level 79,
[m20989:16103:0] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))
mo_name_list_output::open_output_file: opened EUREC4A_SYNSAT_RTTOV_FORWARD_MODEL_DOM02_ML_0136.nc
mo_name_list_output::write_name_list_output: Output to EUREC4A_SYNSAT_RTTOV_FORWARD_MODEL_DOM02_ML_0136.nc_part_2+ at simulation time 2020-01-15T12:10:00.000 by PE 1916
==== backtrace ====
0 0x0000000002a2fba0 mo_name_list_output_mp_io_proc_write_name_list_() ???:0
1 0x0000000002a158b5 mo_name_list_output_mp_write_name_list_output_() ???:0
2 0x0000000002a2cd44 mo_name_list_output_mp_name_list_io_main_proc_() ???:0
3 0x000000000065052f mo_icon_output_tools_mp_init_io_processes_() ???:0
4 0x000000000041d0c8 mo_atmo_model_mp_construct_atmo_model_() ???:0
5 0x000000000041c095 mo_atmo_model_mp_atmo_model_() ???:0
6 0x000000000041627c MAIN__() ???:0
7 0x000000000041558e main() ???:0
8 0x000000000001ed20 __libc_start_main() ???:0
9 0x0000000000415469 _start() ???:0
===================
#################### I/O PE 1916 starting I/O at 222027.082
srun: error: m20989: task 1917: Segmentation fault
srun: Terminating job step 30867274.0
slurmstepd: error: *** STEP 30867274.0 ON m20072 CANCELLED AT 2021-06-27T22:20:27 ***
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
icon 00000000036F060F for__signal_handl Unknown Unknown
libpthread-2.12.s 00007FB881B967E0 Unknown Unknown Unknown
libmxm.so.2.0.32 00007FB82E0818F3 mxm_ud_verbs_ep_p Unknown Unknown
libmxm.so.2.0.32 00007FB82E05F68A mxm_progress Unknown Unknown
mca_mtl_mxm.so 00007FB82E364114 ompi_mtl_mxm_prog Unknown Unknown
libopen-pal.so.20 00007FB87C568D11 opal_progress Unknown Unknown
mca_osc_pt2pt.so 00007FB7E9CFDAC8 Unknown Unknown Unknown
mca_osc_pt2pt.so 00007FB7E9CFD585 ompi_osc_pt2pt_rg Unknown Unknown
libmpi.so.20.0.2 00007FB8808C815B PMPI_Rget Unknown Unknown
libmpi_mpifh.so.2 00007FB8821D0F46 PMPI_Rget_f08 Unknown Unknown
icon 0000000002A2FE6B Unknown Unknown Unknown
icon 0000000002A158B5 Unknown Unknown Unknown
icon 0000000002A2CD44 Unknown Unknown Unknown
icon 000000000065052F Unknown Unknown Unknown
icon 000000000041D0C8 Unknown Unknown Unknown
icon 000000000041C095 Unknown Unknown Unknown
icon 000000000041627C Unknown Unknown Unknown
icon 000000000041558E Unknown Unknown Unknown
libc-2.12.so 00007FB881811D20 __libc_start_main Unknown Unknown
icon 0000000000415469 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Edited by Hauke Schulz