Christian Boulanger
experiments

Repository



Notes on Spack environments

parallel builds: https://spack.readthedocs.io/en/latest/packaging_guide.html#install-level-build-parallelism


Easybuild

What toolchains have we got?
Documentation:

history list of toolchains and versions: https://docs.easybuild.io/en/latest/Common-toolchains.html#common-toolchains-overview

list of all toolchains: https://docs.easybuild.io/en/latest/version-specific/toolchains.html

common toolchains: https://docs.easybuild.io/en/latest/Common-toolchains.html


FOSS 2021a from May 21
binutils 2.36.1
gcc 10.3.0
OpenMPI 4.1.1
FlexiBLAS 3.0.4
OpenBLAS 0.3.15
LAPack (comes with OpenBLAS)
ScaLAPACK 2.1.0
FFTW 3.3.9

Spack environments
Examples - first steps:
spack env create foss2021a
spack env activate foss2021a
-> enviroment is created in var/spack/environments/foss2021a
spack add binutils@2.36.1
The above command assumes that the environment has been activated. It should also work to use
spack -e foos2021a add binutils@2.36.1 where the environment is specified with the -e switch. (This is probably nicer for scripts as it is more explicit).
fangohr@avocado:/git/spack-hpc/var/spack/environments/foss2021a$ spack add binutils@2.36.1
==> Adding binutils@2.36.1 to environment foss2021a
fangohr@avocado:/git/spack-hpc/var/spack/environments/foss2021a$ spack add gcc@10.3.0
==> Adding gcc@10.3.0 to environment foss2021a
(etc) leads to
fangohr@avocado:~/git/spack-hpc/var/spack/environments/foss2021a$ cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs: [binutils@2.36.1, gcc@10.3.0, openmpi@4.1.1, flexiblas@3.0.4, openblas@0.3.15,
    netlib-scalapack@2.1.0, fftw@3.3.9]
  view: true
fangohr@avocado:~/git/spack-hpc/var/spack/environments/foss2021a$

Add concretization: together  (see https://spack.readthedocs.io/en/latest/environments.html#environments-concretization)

compile compiler for spack
Before we can build the environment, we have to build the compiler
that is needed. For the foss2020b toolchain, this is gcc-10.2.0:


spack compilers   # show known compilers

spack install gcc@10.2.0   # compile compiler

spack load gcc@10.2.0      # put new compiler into path

spack compiler find        # ask spack to update list of known compilers

Now the new compiler should show up in spack compilers.

build environment

spack env activate foss2020b

spack concretize    # show which concretization will be chosen

spack install       # do the install

Untested: could probably also use

spack -e foss2020b install


Questions:

I have listed the compiler itself as a package in the environment - is that required / good / bad / irrelevant?


Questions with answers


when activating the environment (spack env activate foss2020b), autoreconf doesn't show up in my path. I would have expected it to be visible in var/spack/environments/foss2020b/.spack-env/view/bin
Answer: it appears that only packages that have been requested
explicitly are visible in the environment view. I ran spack -e foss2020b add autoconf@2.6.9 followed by spack -e foss2020b install,
and then autoreconfigure shows up in the search path in the
environment.
The version I have used (2.6.9) was taken from what is installed
already. It would probably be better to just say that we want to add
autoconf and let spack figure out the rigth version.


Testing toolchain foss2020b (stuff to do here)
spack env activate foss2020b
autoreconf -i
./configure --with-blas=/home/fangohr/git/spack-hpc/var/spack/environments/foss$
make

this completes and produces a src/octopus executable. It seems to use some libraries from the host machine (check with ldd src/octopus). Ran out of time here.

Testing toolchain foss2021a
spack env activate foss2021a
autoreconf -i
# find directory of the environment's view  (there must be spack command for this, for now try `which autoreconf`)
export SPACKENV=/home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view
./configure --with-blas=$SPACKENV/lib/libopenblas.a --with-fftw-prefix=$SPACKENV --with-libxc-prefix=$SPACKENV
make

this completes and produces a src/octopus executable. It seems to use the right libraries:
fangohr@avocado:~/git/octopus-build-in-spack-env-foss2021a$ ldd src/octopus
        linux-vdso.so.1 (0x00007ffed09f4000)
        libmpfr.so.6 => /home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view/lib/libmpfr.so.6 (0x00007f42a91e5000)
        libgmp.so.10 => /home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view/lib/libgmp.so.10 (0x00007f42a916e000)
        libfftw3_omp.so.3 => /home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view/lib/libfftw3_omp.so.3 (0x00007f42a9164000)
        libfftw3.so.3 => /home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view/lib/libfftw3.so.3 (0x00007f42a8e0c000)
        libgsl.so.25 => /home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view/lib/libgsl.so.25 (0x00007f42a8b3e000)
        libgslcblas.so.0 => /home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view/lib/libgslcblas.so.0 (0x00007f42a8aff000)
        libxcf03.so.9 => /home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view/lib/libxcf03.so.9 (0x00007f42a8af7000)
        libxc.so.9 => /home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view/lib/libxc.so.9 (0x00007f42a81be000)
        libstdc++.so.6 => /home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view/lib64/libstdc++.so.6 (0x00007f42a7fe9000)
        libgfortran.so.5 => /home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view/lib64/libgfortran.so.5 (0x00007f42a7d2d000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f42a7baf000)
        libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1 (0x00007f42a7b83000)
        libgcc_s.so.1 => /home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view/lib64/libgcc_s.so.1 (0x00007f42a7b68000)
        libquadmath.so.0 => /home/fangohr/git/spack-hpc/var/spack/environments/foss2021a/.spack-env/view/lib64/libquadmath.so.0 (0x00007f42a7b1f000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f42a795a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f42a9491000)
        libgomp.so.1 => /home/fangohr/git/spack-hpc/opt/spack/linux-debian11-cascadelake/gcc-10.2.1/gcc-10.3.0-dm4mscrniwxuuec7qjaui4kkesknbggf/lib64/libgomp.so.1 (0x00007f42a7915000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f42a78f3000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f42a78eb000)

fangohr@avocado:~/git/octopus-build-in-spack-env-foss2021a$ ldd src/octopus | grep -v view
        linux-vdso.so.1 (0x00007fff60bf7000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007eff0c446000)
        libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1 (0x00007eff0c41a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007eff0c1f1000)
        /lib64/ld-linux-x86-64.so.2 (0x00007eff0dd28000)
        libgomp.so.1 => /home/fangohr/git/spack-hpc/opt/spack/linux-debian11-cascadelake/gcc-10.2.1/gcc-10.3.0-dm4mscrniwxuuec7qjaui4kkesknbggf/lib64/libgomp.so.1 (0x00007eff0c1ac000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007eff0c18a000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007eff0c182000)
fangohr@avocado:~/git/octopus-build-in-spack-env-foss2021a$


Comments on the toolchain implementation through spack here

For foss2021b
Package flexiblas@3.0.4 is missing: it leads (30 Sept 2021) to a
concretization conflict (with today's develop branch from spack,
b96d836a729e05fc717ab61dc87dbf4d596cd2c5, 30 September 2021)
See https://docs.easybuild.io/en/latest/Common-toolchains.html#common-toolchains-overview for the official requirements.

Building toolchains on a docker container
Docker containers are a nice way to test the compilation of toolchains packaged in this repositor (spack-environments) without the process interfering local installations of spack.
A Dockerfile is a recipe to build  docker image, the Dockerfile contained in the root of this repository instructs docker to build a container similar to our MPSD HPC systems, in terms of the linux distribution (Debian 11 in this case), MPSD  package mirrors and few custom packages (like Lmod v8.6). The dockerfile then also lists the following steps (in that order):

compile the toolchains specified (see targets below) using spack,
create files need by lmod to make them module loadable,
loads the newly compiled toolchain (using lmod and not spack load)
Clones the latest octopus release
Compiles the octopus using the right configuration file (based on the toolchain name)
Runs make check-short which is a command that quickly albeit 1hr long, to check that the compilation was successful.
Check the log of the check-short via a python script check_log.py to make that the failed test are in fact ignorable (mostly due to tolerance issues in the test suite).
And if all the above steps are successful, the docker container is committed to a docker image.

A make file is used as a wrapper to simplify the commands needed to build the docker image and run the container. The make file has the following targets:


image  - use "toolchain=foss2021a-mpi" and make octopus with the mpi dependencies


image-serial - use "toolchain=foss2021a-serial" and make octopus with only the serial dependencies


no-cache variants do not use docker  while building the container (useful for debugging)


run - run the latest mpi based container


run-serial - run the latest serial based container


So in principle one would just run make image (or make image-serial) on a terminal to build the container. The container can then be run as a docker image with make run (or make run-serial).
In each of the run targets, the container also mounts the current directory (the root of this repository) to the container which can be useful during development/debugging.