Error messages not logged in log file

Create an issue to avoid putting too much detail into Merge request #1 (closed). More issues to follow (perhaps) in the same spirit.

It appears that the error message is not included in the log:

fangohr@mpsd-hpc-login2:/scratch/fangohr/mpsd-software-environments$ python3 install-mpsd-software-environment.py install dev-23a -- toolchains global
[+] /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/libgcrypt-1.10.1-maxwt3zdpktwlfpzkwdhyhj7s5ysyu42
[+] /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/munge-0.5.15-wx74yh6c35d6xaihwhwpnfbfcpe2fkwi
idge/gcc-11.3.0/octopus-12.1-p3jieav4smmioup2wpefbcplqiapybx2' '--with-blas=-L/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/openblas-0.3.20-yre2piwvwhekexfcoxzxmxlgvutgrsxu/lib -lopenblas' '--with-lapack=-L/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/openblas-0.3.20-yre2piwvwhekexfcoxzxmxlgvutgrsxu/lib -lopenblas' '--with-gsl-prefix=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/gsl-2.7.1-zp2gnohs5lxnujobwrbr3gicz2vdc5jj' '--with-libxc-prefix=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/libxc-5.2.3-qrnbphqzz4z5u4wlbjwiqjbncyrxtie7' '--enable-openmp' '--enable-mpi' 'CC=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/openmpi-4.1.4-xhcl4vfcvuexkqjxddxfntjpqnwn77af/bin/mpicc' 'FC=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/openmpi-4.1.4-xhcl4vfcvuexkqjxddxfntjpqnwn77af/bin/mpif90' '--with-fftw-prefix=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/fftw-3.3.10-xjdxnqmwsjp4bvpu6thrjsu5bzt7dof2' '--with-parmetis-prefix=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/parmetis-4.0.3-bvfeat4bq3oofns536ucnomoqla5ugy4' '--with-netcdf-prefix=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/netcdf-fortran-4.6.0-rzab7o6o3zqqouupx4oqgknxcjn3trdz' '--with-netcdf-include=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/netcdf-fortran-4.6.0-rzab7o6o3zqqouupx4oqgknxcjn3trdz/include' '--with-arpack=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/arpack-ng-3.8.0-pkcx62dghqslvftmvbddazhf3dblqquq/lib/libparpack.so /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/arpack-ng-3.8.0-pkcx62dghqslvftmvbddazhf3dblqquq/lib/libarpack.so' '--with-parpack=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/arpack-ng-3.8.0-pkcx62dghqslvftmvbddazhf3dblqquq/lib/libparpack.so /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/arpack-ng-3.8.0-pkcx62dghqslvftmvbddazhf3dblqquq/lib/libarpack.so' '--with-cgal-prefix=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/cgal-5.0.3-6kjht3hpts26k2brtqo4gbt65f3uejtb' '--with-likwid-prefix=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/likwid-5.2.2-j5cf3s3paxik43e77jdx2wvob2wp5wxt' '--with-pfft-prefix=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/pfft-1.0.8-alpha-iqo54vvmnhh2b2wnkybzmov5krzb73ds' '--with-libyaml-prefix=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/libyaml-0.2.5-tmah4brb5iwbtzpr5yisiruy4dsf76vm' '--with-elpa-prefix=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/elpa-2021.11.001-hkjeep5leg34f6ss34ud6kpr6amphp7z' '--with-nlopt-prefix=/scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/nlopt-2.7.0-majsmpk6omiciswfv2fxlpp25xvq5tn6' '--enable-cuda' '--enable-python' 'FCFLAGS=-O2 -ffree-line-length-none -fallow-argument-mismatch -fallow-invalid-boz' 'FFLAGS=O2 -ffree-line-length-none -fallow-argument-mismatch -fallow-invalid-boz'

1 error found in build log:
     111    ***************************
     112    checking for cuInit in -lcuda... no
     113    checking for nvrtcCreateProgram in -lnvrtc... yes
     114    checking for cublasCreate_v2 in -lcublas... yes
     115    checking for cufftPlan3d in -lcufft... yes
     116    checking for whether we have all the required cuda libraries... no
  >> 117    configure: error: Could not find the cuda library

See build log for details:
  /tmp/fangohr/spack-stage/spack-stage-octopus-12.1-p3jieav4smmioup2wpefbcplqiapybx2/spack-build-out.txt

Error in line 194

The error does not seem to show up in the log:

fangohr@mpsd-hpc-login2:/scratch/fangohr/mpsd-software-environments$ batcat dev-23a/sandybridge/build_toolchains_dev-23a_20230512-153715.log
<snip>
  98   │ [+] /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/a
       │ rpack-ng-3.8.0-pkcx62dghqslvftmvbddazhf3dblqquq
  99   │ [+] /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/p
       │ armetis-4.0.3-bvfeat4bq3oofns536ucnomoqla5ugy4
 100   │ [+] /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/h
       │ df5-1.12.2-sfes2d6tto2dsccvwgnc6jtj7owq272i
 101   │ [+] /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/n
       │ etlib-scalapack-2.1.0-vtf3ujukuigdknyuhokaweqfgat5iqp4
 102   │ [+] /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/f
       │ ftw-3.3.10-xjdxnqmwsjp4bvpu6thrjsu5bzt7dof2
 103   │ [+] /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/p
       │ y-mpi4py-3.1.4-rrjxx46xhqxvlreak2gf5ytdesagjlil
 104   │ [+] /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/n
       │ etcdf-c-4.9.0-jpjrglobahzmxpwruxqz7soti5qe4gyn
 105   │ [+] /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/e
       │ lpa-2021.11.001-hkjeep5leg34f6ss34ud6kpr6amphp7z
 106   │ [+] /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/p
       │ fft-1.0.8-alpha-iqo54vvmnhh2b2wnkybzmov5krzb73ds
 107   │ [+] /scratch/fangohr/mpsd-software-environments/dev-23a/sandybridge/spack/opt/spack/linux-debian11-sandybridge/gcc-11.3.0/n
       │ etcdf-fortran-4.6.0-rzab7o6o3zqqouupx4oqgknxcjn3trdz
 108   │ ==> Installing octopus-12.1-p3jieav4smmioup2wpefbcplqiapybx2
 109   │ ==> No binary for octopus-12.1-p3jieav4smmioup2wpefbcplqiapybx2 found: installing from source
 110   │ ==> Using cached archive: /scratch/fangohr/mpsd-software-environments/dev-23a/mpsd-spack-cache/_source-cache/archive/e2/e22
       │ 14e958f1e9631dbe6bf020c39f1fe4d71ab0b6118ea9bd8dc38f6d7a7959a.tar.gz
 111   │ ==> No patches needed for octopus
 112   │ ==> octopus: Executing phase: 'autoreconf'
 113   │ ==> octopus: Executing phase: 'configure'

mentioned in merge request !1 (merged)

mentioned in commit 1a01f895

Good point. I didn't notice this. The line causing the error is

subprocess.run(
                f"bash {spack_setup_script} {' '.join(install_flags)} {toolchain} | tee -a {install_log_file} 2>&1",
                shell=True,
            )

Reason, I'm redirecting the errors from the tee process and not from the bash process. It should be corrected to

subprocess.run(
                f"bash {spack_setup_script} {' '.join(install_flags)} {toolchain} 2>&1| tee -a {install_log_file} ",
                shell=True,
            )

This has been fixed in 1a01f895 - Fix the stderr not being logged

Confirm fix works in my manual test just now. Also good to have name of toolchain name in log file - I was gonna request that as well ;-).

closed

mentioned in commit 3554a57b