From d7c75a567e4ff1d128e52ca2729eae77e9f64921 Mon Sep 17 00:00:00 2001
From: Hans Fangohr <hans.fangohr@mpsd.mpg.de>
Date: Mon, 12 Jun 2023 17:47:42 +0200
Subject: [PATCH 1/8] first draft of README
---
README.rst | 137 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 137 insertions(+)
create mode 100644 README.rst
diff --git a/README.rst b/README.rst
new file mode 100644
index 0000000..e49f4dc
--- /dev/null
+++ b/README.rst
@@ -0,0 +1,137 @@
+MPSD Software installation
+==========================
+
+.. contents::
+
+This repository provides the `mpsd-software` tool which is used to install
+package sets and toolchains on the `MPSD HPC cluster
+<https://computational-science.mpsd.mpg.de/docs/mpsd-hpc.html>`__.
+
+It can also be used to install the software on other machines, such as Linux
+laptops and desktops. This can be useful to work - on a local machine - with
+the same software environment, for example to debug a problem.
+
+Note that this tool is under development, and the recommended way to install and
+use as well as the user interface may change. This document will be kept
+up-to-date in any case.
+
+Quick start
+-----------
+
+To install, for example, the ``foss2022a-serial`` toolchain:
+
+1. Clone this repository to the place where you want to install the software environment::
+
+ git clone https://gitlab.gwdg.de/mpsd-cs/mpsd-software-environments.git
+
+2. ``cd mpsd-software-environments``
+
+3. Execute the script to install the toolchain::
+
+ $> ./mpsd-software.py install dev-23a --toolchain foss2022a-serial
+
+ This will take some time.
+
+4. To see the installation status, and the required ``module use`` command line
+ to activate the created modules, try the ``status`` command::
+
+ $> ./mpsd-software status dev-23a
+
+ Installed toolchains (dev-23a):
+
+ - cascadelake
+ foss2022a-serial
+ [module use /home/fangohr/git/mpsd-software-environments/dev-23a/cascadelake/lmod/Core]
+
+
+
+Documentation
+-------------
+
+Package sets and toolchains
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- Package sets are a combination of particular versions of multiple software
+ packages (such as anaconda3, or gcc and fftw). In the way the SSU
+ Computational Science provides software on the MPSD HPC cluster and for the
+ Octopus continuous integration services, these package sets are compiled
+ together (using Spack).
+
+- Toolchains are a particular type of package sets:
+
+ - the choice of software packages (typically a compiler and scientific
+ computing libraries) and their versions follows the Easybuild toolchains
+ (such as the `FOSS toolchains
+ <https://docs.easybuild.io/common-toolchains/#common_toolchains_foss>`__).
+
+ - all packages grouped together in a toolchain can be loaded together using
+ the `module load` command.
+
+ Example: the `foss2022a-serial` tool chain provides (in spack notation)::
+
+ - gcc@11.3.0
+ - binutils@2.38+headers+ld
+ - fftw@3.3.10+openmp~~mpi
+ - openblas@0.3.20
+
+ - in addition to the Easybuild-driven choice of packages, there are
+ additional packages included in each package which support the build of
+ Octopus within these toolchains. For `foss2022a-serial` these packages
+ include::
+
+ - libxc@5.2.3 # octopus-dependencies:
+ - gsl@2.7.1
+ - sparskit@develop # 2021.06.01
+ - nlopt@2.7.0
+ - libgd@2.2.4 # 2.3.1
+ - libvdwxc@0.4.0~~mpi
+ - nfft@3.2.4
+ - berkeleygw@2.1~~mpi~scalapack
+ - python@3.9.5
+ - cgal@5.0.3 # 5.2
+ - hdf5@1.12.2~mpi
+ - etsf-io@1.0.4
+
+
+MPSD software releases
+~~~~~~~~~~~~~~~~~~~~~~
+
+As `explained in the MPSD HPC documentation
+<https://computational-science.mpsd.mpg.de/docs/mpsd-hpc.html#software>`__, we
+label software releases available on the HPC using a naming scheme of the year
+(such as ``23``) and a letter starting from ``a``. There is an exception that
+the first available software version is ``dev-23a`` (starting with ``dev-`` to
+indicate this was a development prototype).
+
+At the moment (June 2023), there is only one release (that is ``dev-23a``).
+
+For each MPSD software release, multiple toolchains and package sets are available::
+
+ $> ./mpsd-software.py available dev-23a
+
+ MPSD software release dev-23a, AVAILABLE for installation are
+ Toolchains:
+ foss2021a-cuda-mpi
+ foss2021a-mpi
+ foss2021a-serial
+ foss2022a-cuda-mpi
+ foss2022a-mpi
+ foss2022a-serial
+ Package sets:
+ global (octopus@12.1, octopus@12.1)
+ global_generic (anaconda3@2022.10)
+
+
+Prerequisites
+~~~~~~~~~~~~~
+
+What needs to be installed for the installation to succeed?
+
+Probably an ongoing topic of work. We know the following tools need to be available:
+
+- XXX TODO, add, requirements in container from CI might be best. Things like rsync, git, buildtools, Linux-headers,
+
+
+
+
+
--
GitLab
From 8b3b6223642d897af6b9d4a058f7740e6d397ee5 Mon Sep 17 00:00:00 2001
From: Hans Fangohr <hans.fangohr@mpsd.mpg.de>
Date: Fri, 16 Jun 2023 11:17:37 +0200
Subject: [PATCH 2/8] update API and repo name
---
README.rst | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/README.rst b/README.rst
index e49f4dc..e71474a 100644
--- a/README.rst
+++ b/README.rst
@@ -3,7 +3,7 @@ MPSD Software installation
.. contents::
-This repository provides the `mpsd-software` tool which is used to install
+This repository provides the ``mpsd-software`` tool which is used to install
package sets and toolchains on the `MPSD HPC cluster
<https://computational-science.mpsd.mpg.de/docs/mpsd-hpc.html>`__.
@@ -22,15 +22,15 @@ To install, for example, the ``foss2022a-serial`` toolchain:
1. Clone this repository to the place where you want to install the software environment::
- git clone https://gitlab.gwdg.de/mpsd-cs/mpsd-software-environments.git
+ git clone https://gitlab.gwdg.de/mpsd-cs/mpsd-software.git
-2. ``cd mpsd-software-environments``
+2. ``cd mpsd-software``
3. Execute the script to install the toolchain::
- $> ./mpsd-software.py install dev-23a --toolchain foss2022a-serial
+ $> ./mpsd-software.py install dev-23a foss2022a-serial
- This will take some time.
+ This will take some time (up to several hours depending on hardware).
4. To see the installation status, and the required ``module use`` command line
to activate the created modules, try the ``status`` command::
@@ -48,12 +48,14 @@ To install, for example, the ``foss2022a-serial`` toolchain:
Documentation
-------------
+More detailed documentation that goes beyond the `Quick Start`_ section.
+
Package sets and toolchains
~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Package sets are a combination of particular versions of multiple software
packages (such as anaconda3, or gcc and fftw). In the way the SSU
- Computational Science provides software on the MPSD HPC cluster and for the
+ Computational Science provides software on the MPSD HPC cluster, and for the
Octopus continuous integration services, these package sets are compiled
together (using Spack).
--
GitLab
From 7e41b6397e588121a9f34a4c64b21fc1c4ddc63a Mon Sep 17 00:00:00 2001
From: Hans Fangohr <hans.fangohr@mpsd.mpg.de>
Date: Fri, 16 Jun 2023 11:21:32 +0200
Subject: [PATCH 3/8] syntax fixes
---
README.rst | 44 ++++++++++++++++++++++----------------------
1 file changed, 22 insertions(+), 22 deletions(-)
diff --git a/README.rst b/README.rst
index e71474a..09f3cca 100644
--- a/README.rst
+++ b/README.rst
@@ -67,41 +67,41 @@ Package sets and toolchains
<https://docs.easybuild.io/common-toolchains/#common_toolchains_foss>`__).
- all packages grouped together in a toolchain can be loaded together using
- the `module load` command.
+ the ``module load`` command.
- Example: the `foss2022a-serial` tool chain provides (in spack notation)::
+ Example: the ``foss2022a-serial`` tool chain provides (in spack notation)::
- gcc@11.3.0
- binutils@2.38+headers+ld
- fftw@3.3.10+openmp~~mpi
- openblas@0.3.20
- - in addition to the Easybuild-driven choice of packages, there are
- additional packages included in each package which support the build of
- Octopus within these toolchains. For `foss2022a-serial` these packages
- include::
-
- - libxc@5.2.3 # octopus-dependencies:
- - gsl@2.7.1
- - sparskit@develop # 2021.06.01
- - nlopt@2.7.0
- - libgd@2.2.4 # 2.3.1
- - libvdwxc@0.4.0~~mpi
- - nfft@3.2.4
- - berkeleygw@2.1~~mpi~scalapack
- - python@3.9.5
- - cgal@5.0.3 # 5.2
- - hdf5@1.12.2~mpi
- - etsf-io@1.0.4
+ - in addition to the Easybuild-driven choice of packages, there are
+ additional packages included in each package which support the build of
+ Octopus within these toolchains. For ``foss2022a-serial`` these packages
+ include::
+
+ - libxc@5.2.3 # octopus-dependencies:
+ - gsl@2.7.1
+ - sparskit@develop # 2021.06.01
+ - nlopt@2.7.0
+ - libgd@2.2.4 # 2.3.1
+ - libvdwxc@0.4.0~~mpi
+ - nfft@3.2.4
+ - berkeleygw@2.1~~mpi~scalapack
+ - python@3.9.5
+ - cgal@5.0.3 # 5.2
+ - hdf5@1.12.2~mpi
+ - etsf-io@1.0.4
MPSD software releases
~~~~~~~~~~~~~~~~~~~~~~
-As `explained in the MPSD HPC documentation
-<https://computational-science.mpsd.mpg.de/docs/mpsd-hpc.html#software>`__, we
+As ``explained in the MPSD HPC documentation
+<https://computational-science.mpsd.mpg.de/docs/mpsd-hpc.html#software>``__, we
label software releases available on the HPC using a naming scheme of the year
-(such as ``23``) and a letter starting from ``a``. There is an exception that
+(such as `23``) and a letter starting from ``a``. There is an exception that
the first available software version is ``dev-23a`` (starting with ``dev-`` to
indicate this was a development prototype).
--
GitLab
From 821d7988630fd513bc1b220c25fc114810ede63e Mon Sep 17 00:00:00 2001
From: Hans Fangohr <hans.fangohr@mpsd.mpg.de>
Date: Fri, 16 Jun 2023 12:51:12 +0200
Subject: [PATCH 4/8] work in progress
---
README.rst | 46 +++++++++++++++++++++++++++++++++++++---------
1 file changed, 37 insertions(+), 9 deletions(-)
diff --git a/README.rst b/README.rst
index 09f3cca..7479d62 100644
--- a/README.rst
+++ b/README.rst
@@ -98,10 +98,9 @@ Package sets and toolchains
MPSD software releases
~~~~~~~~~~~~~~~~~~~~~~
-As ``explained in the MPSD HPC documentation
-<https://computational-science.mpsd.mpg.de/docs/mpsd-hpc.html#software>``__, we
+As ``explained in the MPSD HPC documentation <https://computational-science.mpsd.mpg.de/docs/mpsd-hpc.html#software>``__, we
label software releases available on the HPC using a naming scheme of the year
-(such as `23``) and a letter starting from ``a``. There is an exception that
+(such as ``23``) and a letter starting from ``a``. There is an exception that
the first available software version is ``dev-23a`` (starting with ``dev-`` to
indicate this was a development prototype).
@@ -112,7 +111,7 @@ For each MPSD software release, multiple toolchains and package sets are availab
$> ./mpsd-software.py available dev-23a
MPSD software release dev-23a, AVAILABLE for installation are
- Toolchains:
+ Toolchains:
foss2021a-cuda-mpi
foss2021a-mpi
foss2021a-serial
@@ -129,11 +128,40 @@ Prerequisites
What needs to be installed for the installation to succeed?
+1. ``mpsd-software.py`` installation requirements
++++++++++++++++++++++++++++++++++++++++++++++++++
+
+- a recent version of Python3 (3.9 or newer)
+
+- an environment variable MPSD_MICROARCH to point to the correct microarchitecture of the computer
+- or the ``archspec`` command in the search path that can determine this.
+ Install ``archspec`` as ``pip install archspec`` or ``pipx install archspec``.
+ If you are on a MPSD-maintained linux laptop, you can probably also use
+ `apt-get install archspec`.
+- Optional: if the ``rich`` package is available, some output will be coloured. (Debian package: `python3-rich` or `pip install rich`)
+
+2. Spack requirements
++++++++++++++++++++++
+
+To be able to compile the toolchains, one needs standard build tools (required by spack). On Debian/Ubuntu, this includes the packages:
+
+autoconf
+automake
+build-essential
+ca-certificates
+coreutils
+curl
+file
+gfortran
+git
+libtool
+lmod
+openssh-server
+python3
+rsync
+unzip
+wget
+
Probably an ongoing topic of work. We know the following tools need to be available:
- XXX TODO, add, requirements in container from CI might be best. Things like rsync, git, buildtools, Linux-headers,
-
-
-
-
-
--
GitLab
From 022eef556ba5c3a3ef8e72e43131a0ff9876f062 Mon Sep 17 00:00:00 2001
From: Hans Fangohr <hans.fangohr@mpsd.mpg.de>
Date: Fri, 16 Jun 2023 13:07:33 +0200
Subject: [PATCH 5/8] rework instructions in preparation for the new package
---
README.rst | 91 ++++++++++++++++++++++++++----------------------------
1 file changed, 43 insertions(+), 48 deletions(-)
diff --git a/README.rst b/README.rst
index 7479d62..071a8b6 100644
--- a/README.rst
+++ b/README.rst
@@ -1,5 +1,5 @@
-MPSD Software installation
-==========================
+MPSD Software manager
+=====================
.. contents::
@@ -20,28 +20,44 @@ Quick start
To install, for example, the ``foss2022a-serial`` toolchain:
-1. Clone this repository to the place where you want to install the software environment::
+1. Install this mpsd-software-manager Python package. The recommended way is to
+ use ``pipx`` to that this tool is available independent from the use of any
+ other python environments::
- git clone https://gitlab.gwdg.de/mpsd-cs/mpsd-software.git
+ pipx install git+https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager
-2. ``cd mpsd-software``
-3. Execute the script to install the toolchain::
+2. Navigate to the location in your file system where you would like to store
+ the compiled software. (Once compiled, the location cannot be changed.) For
+ example::
- $> ./mpsd-software.py install dev-23a foss2022a-serial
+ cd /home/user/mpsd-software
+
+
+3. Initiate the installation at this location using::
+
+ mpsd-software init .
+
+ (This creates a hidden file ``.mpsd-software-root`` to tag the location for
+ as the root of the installation. All compiled files, logs etc are written in
+ or below this subdirectory.)
+
+4. From the same directory, run the command to install the ``foss2022a-serial`` toolchain:
+
+ $> mpsd-software install dev-23a foss2022a-serial
This will take some time (up to several hours depending on hardware).
-4. To see the installation status, and the required ``module use`` command line
+5. To see the installation status, and the required ``module use`` command line
to activate the created modules, try the ``status`` command::
- $> ./mpsd-software status dev-23a
+ $> mpsd-software status dev-23a
Installed toolchains (dev-23a):
- cascadelake
foss2022a-serial
- [module use /home/fangohr/git/mpsd-software-environments/dev-23a/cascadelake/lmod/Core]
+ [module use /home/user/mpsd-software/dev-23a/cascadelake/lmod/Core]
@@ -108,7 +124,7 @@ At the moment (June 2023), there is only one release (that is ``dev-23a``).
For each MPSD software release, multiple toolchains and package sets are available::
- $> ./mpsd-software.py available dev-23a
+ $> mpsd-software available dev-23a
MPSD software release dev-23a, AVAILABLE for installation are
Toolchains:
@@ -128,40 +144,19 @@ Prerequisites
What needs to be installed for the installation to succeed?
-1. ``mpsd-software.py`` installation requirements
-+++++++++++++++++++++++++++++++++++++++++++++++++
-
-- a recent version of Python3 (3.9 or newer)
-
-- an environment variable MPSD_MICROARCH to point to the correct microarchitecture of the computer
-- or the ``archspec`` command in the search path that can determine this.
- Install ``archspec`` as ``pip install archspec`` or ``pipx install archspec``.
- If you are on a MPSD-maintained linux laptop, you can probably also use
- `apt-get install archspec`.
-- Optional: if the ``rich`` package is available, some output will be coloured. (Debian package: `python3-rich` or `pip install rich`)
-
-2. Spack requirements
-+++++++++++++++++++++
-
-To be able to compile the toolchains, one needs standard build tools (required by spack). On Debian/Ubuntu, this includes the packages:
-
-autoconf
-automake
-build-essential
-ca-certificates
-coreutils
-curl
-file
-gfortran
-git
-libtool
-lmod
-openssh-server
-python3
-rsync
-unzip
-wget
-
-Probably an ongoing topic of work. We know the following tools need to be available:
-
-- XXX TODO, add, requirements in container from CI might be best. Things like rsync, git, buildtools, Linux-headers,
+The ``mpsd-software-manager`` python package.
+
+- This needs a recent Python (3.9 or later).
+- Install via pip or pipx.
+
+ Pipx commands are::
+
+ - to install: ``pipx install git+https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager``
+ - to update: ``pipx upgrade mpsd-software-manager``
+ - to uninstall: ``pipx uninstall mpsd-software-manager``
+
+- Requirements to be able to run `spack <spack.readthedocs.io>`__
+
+ - Please check https://spack.readthedocs.io/en/latest/getting_started.html#system-prerequisites
+
+
--
GitLab
From 02f9ad7b7f82fedf87978b3217679cce3a69ca82 Mon Sep 17 00:00:00 2001
From: Hans Fangohr <hans.fangohr@mpsd.mpg.de>
Date: Fri, 16 Jun 2023 13:09:22 +0200
Subject: [PATCH 6/8] corrections
---
README.rst | 13 +++++++------
1 file changed, 7 insertions(+), 6 deletions(-)
diff --git a/README.rst b/README.rst
index 071a8b6..217e08a 100644
--- a/README.rst
+++ b/README.rst
@@ -24,34 +24,35 @@ To install, for example, the ``foss2022a-serial`` toolchain:
use ``pipx`` to that this tool is available independent from the use of any
other python environments::
- pipx install git+https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager
+ $ pipx install git+https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager
2. Navigate to the location in your file system where you would like to store
the compiled software. (Once compiled, the location cannot be changed.) For
example::
- cd /home/user/mpsd-software
+ $ cd /home/user/mpsd-software
3. Initiate the installation at this location using::
- mpsd-software init .
+ $ mpsd-software init
(This creates a hidden file ``.mpsd-software-root`` to tag the location for
as the root of the installation. All compiled files, logs etc are written in
or below this subdirectory.)
-4. From the same directory, run the command to install the ``foss2022a-serial`` toolchain:
+4. From the same directory, run the command to install the ``foss2022a-serial``
+ toolchain::
- $> mpsd-software install dev-23a foss2022a-serial
+ $ mpsd-software install dev-23a foss2022a-serial
This will take some time (up to several hours depending on hardware).
5. To see the installation status, and the required ``module use`` command line
to activate the created modules, try the ``status`` command::
- $> mpsd-software status dev-23a
+ $ mpsd-software status dev-23a
Installed toolchains (dev-23a):
--
GitLab
From 43e092de9d27dd502e7a2d7fd23c1cd0cb138885 Mon Sep 17 00:00:00 2001
From: Hans Fangohr <hans.fangohr@mpsd.mpg.de>
Date: Fri, 16 Jun 2023 13:11:36 +0200
Subject: [PATCH 7/8] fix link
---
README.rst | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/README.rst b/README.rst
index 217e08a..ecd8327 100644
--- a/README.rst
+++ b/README.rst
@@ -115,7 +115,7 @@ Package sets and toolchains
MPSD software releases
~~~~~~~~~~~~~~~~~~~~~~
-As ``explained in the MPSD HPC documentation <https://computational-science.mpsd.mpg.de/docs/mpsd-hpc.html#software>``__, we
+As `explained in the MPSD HPC documentation <https://computational-science.mpsd.mpg.de/docs/mpsd-hpc.html#software>`__, we
label software releases available on the HPC using a naming scheme of the year
(such as ``23``) and a letter starting from ``a``. There is an exception that
the first available software version is ``dev-23a`` (starting with ``dev-`` to
--
GitLab
From 248edc0f022c67d9d513b2b8d212736ec19b9384 Mon Sep 17 00:00:00 2001
From: Hans Fangohr <hans.fangohr@mpsd.mpg.de>
Date: Fri, 16 Jun 2023 13:13:52 +0200
Subject: [PATCH 8/8] Additions
---
README.rst | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/README.rst b/README.rst
index ecd8327..c29a590 100644
--- a/README.rst
+++ b/README.rst
@@ -150,7 +150,7 @@ The ``mpsd-software-manager`` python package.
- This needs a recent Python (3.9 or later).
- Install via pip or pipx.
- Pipx commands are::
+ Pipx commands are:
- to install: ``pipx install git+https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager``
- to update: ``pipx upgrade mpsd-software-manager``
@@ -161,3 +161,13 @@ The ``mpsd-software-manager`` python package.
- Please check https://spack.readthedocs.io/en/latest/getting_started.html#system-prerequisites
+The compilation of the ``*-mpi`` toolchains needs linux headers installed. (TODO Add more detail.)
+
+Working example
+~~~~~~~~~~~~~~~
+
+We have a continuous integration example at XXX, that shows the complete
+compilation cycle (including compilation of Octopus). (TODO)
+
+
+
--
GitLab