.gitignore

@@ -6,3 +6,4 @@ dev-23a/
 dist/
 build/
 *.egg-info/
+.mpsd-software-root

.gitlab-ci.yml

@@ -29,7 +29,7 @@ stages:
 
   - echo "Install Python3"
   - apt-get update
-  - apt-get install -y python3 python3-venv python3-rich
+  - apt-get install -y python3 python3-venv
   - python3 -m venv --help
   - python3 -m venv venv
   - source venv/bin/activate
@@ -39,13 +39,10 @@ stages:
   - echo "Install Python dependencies for running the tests"
   - pip install -U pip
   - pip --version
-  - pip install pytest black ruff archspec
-
+  - pip install .[dev]
   - echo "Diagnostics - which versions are we using"
   - python3 --version
   - pytest --version
-  - black --version
-  - ruff --version
 
   - echo "Install additional packages we need to run spack-setup.sh"
   - apt-get install -y git rsync

README.rst

@@ -22,7 +22,7 @@ To install, for example, the ``foss2022a-serial`` toolchain:
 
 1. Install this mpsd-software-manager Python package. The recommended way is to
    use ``pipx`` to that this tool is available independent from the use of any
-   other python environments::
+   other Python environments::
 
      $ pipx install git+https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager
 
@@ -33,18 +33,27 @@ To install, for example, the ``foss2022a-serial`` toolchain:
 
      $ cd /home/user/mpsd-software
 
-   Future calls of the `mpsd-software` command need to be executed from this
-   "mpsd-software-root" directory.
 
+3. Initiate the installation at this location using::
 
-3. From the same directory, run the command to install the ``foss2022a-serial``
+      $ mpsd-software init
+
+    Future calls of the `mpsd-software` command need to be executed from this
+    "mpsd-software-root" directory or in one of its subdirectories.
+
+    (The above command creates a hidden file ``.mpsd-software-root`` to tag the location for
+    as the root of the installation. All compiled files, logs etc are written in
+    or below this subdirectory.)
+
+
+4. From the same directory, run the command to install the ``foss2022a-serial``
    toolchain::
 
      $ mpsd-software install dev-23a foss2022a-serial
 
    This will take some time (up to several hours depending on hardware).
 
-4. To see the installation status, and the required ``module use`` command line
+5. To see the installation status, and the required ``module use`` command line
    to activate the created modules, try the ``status`` command::
 
      $ mpsd-software status dev-23a
@@ -55,7 +64,7 @@ To install, for example, the ``foss2022a-serial`` toolchain:
          foss2022a-serial
          [module use /home/user/mpsd-software/dev-23a/cascadelake/lmod/Core]
 
-5. To compile Octopus, source the provided configure script, for example ``foss2022a-serial-config.sh``, as
+6. To compile Octopus, source the provided configure script, for example ``foss2022a-serial-config.sh``, as
    `explained here <https://computational-science.mpsd.mpg.de/docs/mpsd-hpc.html#loading-a-toolchain-to-compile-octopus>`__).
    The configure scripts are located in ``dev-23a/spack-environments/octopus``::
 
@@ -100,7 +109,7 @@ Package sets and toolchains
       - openblas@0.3.20
 
   - in addition to the Easybuild-driven choice of packages, there are
-    additional packages included in each package which support the build of
+    additional packages included in each toolchain which support the build of
     Octopus within these toolchains. For ``foss2022a-serial`` these packages
     include::
 
@@ -151,7 +160,7 @@ Prerequisites
 
 What needs to be installed for the installation to succeed?
 
-The ``mpsd-software-manager`` python package.
+The ``mpsd-software-manager`` Python package.
 
 - This needs a recent Python (3.9 or later).
 - Install via pip or pipx.
@@ -177,6 +186,26 @@ Requirements for particular toolchains and package sets
 - ``foss*-mpi`` currently needs linux header files installed (to compile the ``knem`` package)
 - ``foss*-cuda-mpi`` (proably as `*-mpi, needs testing TODO`)
 
+Finding the Octopus configure wrapper
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For each Octopus toolchain, there is an Octopus configure wrapper available.
+The wrapper essentially calls the configure script with the right parameters,
+and library locations for the current toolchain. Once the
+toolchain is loaded, the variable ``$MPSD_OCTOPUS_CONFIGURE`` contains that
+path. The path can also be seen using the ``module show TOOLCHAIN_NAME`` command. For example::
+
+  $ mpsd-software install dev-23a foss2022a-mpi
+  $ module use ~/mpsd-software/dev-23a/cascadelake/lmod/Core
+  $ module show toolchains/foss2022a-mpi
+  ...
+  depends_on("cgal/5.0.3")
+  depends_on("hdf5/1.12.2")
+  setenv("MPSD_OCTOPUS_CONFIGURE","~/mpsd-software/dev-23a/spack-environments/octopus/foss2022a-mpi-config.sh")
+  $ module load toolchains/foss2022a-mpi
+  $ echo $MPSD_OCTOPUS_CONFIGURE
+  ~/mpsd-software/dev-23a/spack-environments/octopus/foss2022a-mpi-config.sh
+
 
 Working example
 ~~~~~~~~~~~~~~~
@@ -191,17 +220,17 @@ Octopus) using the ``foss2022a-serial`` toolchain.
 Frequently asked questions
 --------------------------
 
-- Can I install the ``mpsd-software-manager`` package in a python virtual environment?
+- Can I install the ``mpsd-software-manager`` package in a Python virtual environment?
 
   Yes. ``pipx`` is probably more convenient, but you can create your own Pyton
   virtual environment and install the ``mpsd-software-manager`` in that as a
-  regular python package::
+  regular Python package::
 
     python3 -m venv venv
     . venv/bin/activate
     pip install git+https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager
 
-  You just need to activate that python virtual environment before being able to
+  You just need to activate that Python virtual environment before being able to
   use the tool.
 
 - Does the command write anything outside the mpsd-software-root directory?
@@ -219,9 +248,10 @@ Frequently asked questions
 - How long does the compilation take?
 
   This depends on the hardware. A few hours are typical per toolchain. If a
-  second toolchain is compiled in the same MPSD software instance is likely to
-  be faster, in particular if the same compiler is used (and thus the compiler
-  does not need to be re-compiled for the second toolchain).
+  second toolchain is compiled in the same MPSD software instance and the same
+  MPSD release it is likely to be faster, in particular if the same compiler is
+  used (and thus the compiler does not need to be re-compiled for the second
+  toolchain).
 
 - How much disk storage do I need?
 
@@ -244,15 +274,3 @@ Frequently asked questions
 Development
 -----------
 Developers documentation is available at development.rst.
-
-
-.. comment:
-   Draft for additional steps for 'quickstart' once/if we have the the `init` command added.
-
-    3. Initiate the installation at this location using::
-
-         $ mpsd-software init
-
-       (This creates a hidden file ``.mpsd-software-root`` to tag the location for
-       as the root of the installation. All compiled files, logs etc are written in
-       or below this subdirectory.)

development.rst

@@ -17,3 +17,20 @@ Then every time you commit, pre-commit will run all checks defined in `.pre-comm
 you can run the pre-commit checks manually by running::
 
     pre-commit run --all-files
+
+Debugging exit codes
+--------------------
+Non zero exit codes are used to indicate that the program exited due to an error.
+There are multiple ways to debug this. You could run the program with more verbose logging::
+
+    mpsd-software -l debug ...
+
+Here is a list of exit codes and what they mean:
++-----------+------------------------------------------+----------------------------------------------------------------------------------+
+| Exit code | Reason                                   | Solution                                                                         |
++===========+==========================================+==================================================================================+
+| 10        | Call of 'archspec cpu' failed            | Please install archspec, for example via 'pipx install archspec'                 |
+| 20        | Requested package set is not available   | Use 'available' command to see list of available package_sets                    |
+| 30        | Current directory is already initialised | Check if you are in the  right directory                                         |
+| 40        | Current directory is not initialised     | Check if you are in the  right directory, if so use 'init' command to initialise |
++-----------+------------------------------------------+----------------------------------------------------------------------------------+

pyproject.toml

@@ -7,8 +7,9 @@ name = "mpsd_software_manager"
 authors = [{name = "SSU-Computational Science (Fangohr et al)", email = "ssu-cs@mpsd.mpg.de"}]
 license = {file = "LICENSE"}
 classifiers = ["License :: OSI Approved :: MIT License"]
-version = "2023.6.14"
+version = "2023.7.6"
 readme = "README.rst"
+requires-python = ">=3.9"
 dependencies = [
     "archspec",
     "rich",
@@ -17,16 +18,15 @@ dependencies = [
 mpsd-software = "mpsd_software_manager.mpsd_software:main"
 
 [project.urls]
-Home = "https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager/"
+homepage = "https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager/"
+repository = "https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager/"
 
 [project.optional-dependencies]
 dev = [
-    "black",
     "pre-commit",
     "pytest",
     "pytest-mock",
     "pytest-cov",
-    "ruff",
 ]
 
 [tool.pytest.ini_options]

tests/test_mpsd_software.py

@@ -192,7 +192,7 @@ def test_record_script_execution_summary(tmp_path):
 
 def test_install_environment_wrong_package_set(tmp_path):
     """Test exception is raised for non-existing package_set."""
-    # exits with exit code 1 when wrong package_sets are provided
+    # exits with exit code 20 when wrong package_sets are provided
     with pytest.raises(SystemExit) as e:
         mod.install_environment(
             mpsd_release="dev-23a",
@@ -200,7 +200,7 @@ def test_install_environment_wrong_package_set(tmp_path):
             root_dir=(tmp_path),
         )
     assert e.type == SystemExit
-    assert e.value.code == 1
+    assert e.value.code == 20
 
 
 def test_install_environment_wrong_mpsd_release(tmp_path):
@@ -280,7 +280,7 @@ def test_install_environment_zlib():
     # install global_generic package_set
     mod.set_up_logging(
         "WARNING",
-        mod.get_installer_log_file_path(mpsd_release_to_test, "install", root_dir),
+        mod.get_log_file_path(mpsd_release_to_test, "install", root_dir),
     )
     mod.install_environment(
         mpsd_release=mpsd_release_to_test,
@@ -333,7 +333,7 @@ def test_install_environment_zlib():
     importlib.reload(mod)
     mod.set_up_logging(
         "WARNING",
-        mod.get_installer_log_file_path(mpsd_release_to_test, "install", root_dir),
+        mod.get_log_file_path(mpsd_release_to_test, "install", root_dir),
     )
     mod.install_environment(
         mpsd_release=mpsd_release_to_test,
@@ -417,17 +417,16 @@ def test_get_available_package_sets():
     )
 
 
-def test_create_log_file_names():
+def test_create_log_file_name():
     """Test that the log file names are created correctly."""
-    create_log_file_names = mod.create_log_file_names
+    create_log_file_name = mod.create_log_file_name
     mpsd_release = "dev-23a"
-    microarch = "sandybridge"
+    microarch = mod.get_native_microarchitecture()
     date = datetime.datetime.now().replace(microsecond=0).isoformat()
     action = "install"
     package_set = "foss2021a"
     # test build_log_file_name  generation
-    build_log_file_name = create_log_file_names(
-        microarch=microarch,
+    build_log_file_name = create_log_file_name(
         mpsd_release=mpsd_release,
         date=date,
         action=action,
@@ -437,8 +436,7 @@ def test_create_log_file_names():
         build_log_file_name
         == f"{mpsd_release}_{microarch}_{date}_BUILD_{package_set}_{action}.log"
     )
-    installer_log_file_name = create_log_file_names(
-        microarch=microarch,
+    installer_log_file_name = create_log_file_name(
         mpsd_release=mpsd_release,
         date=date,
         action=action,
@@ -448,8 +446,7 @@ def test_create_log_file_names():
         == f"{mpsd_release}_{microarch}_{date}_APEX_{action}.log"
     )
     # test no build log file for incorrect action
-    build_log_file_name = create_log_file_names(
-        microarch=microarch,
+    build_log_file_name = create_log_file_name(
         mpsd_release=mpsd_release,
         date=date,
         action="status",
@@ -520,6 +517,145 @@ def test_remove_environment(tmp_path):
     # done in test_install_environment_zlib
 
 
+def test_initialise_environment(tmp_path):
+    """Test that init_file is created as expected."""
+    # test that the init file is created as expected
+    mod.initialise_environment(tmp_path)
+    init_file = tmp_path / mod.config_vars["init_file"]
+
+    assert init_file.exists()
+    # ensure "Initialising MPSD software ..." is in the log file
+    log_file = tmp_path / mod.config_vars["cmd_log_file"]
+    with open(log_file, "r") as f:
+        assert (f"Initialising MPSD software instance at {tmp_path}") in f.read()
+
+    # test that calling again results in warning and exit code 30
+    with pytest.raises(SystemExit) as pytest_wrapped_e:
+        mod.initialise_environment(tmp_path)
+    assert pytest_wrapped_e.type == SystemExit
+    assert pytest_wrapped_e.value.code == 30
+
+
+def test_get_root_dir(tmp_path):
+    """Test that the root directory is correct."""
+    with mod.os_chdir(tmp_path):
+        # test that  function exists with error 40 if root dir doesn't exist
+        with pytest.raises(SystemExit) as pytest_wrapped_e:
+            mod.get_root_dir()
+        assert pytest_wrapped_e.type == SystemExit
+        assert pytest_wrapped_e.value.code == 40
+
+        # test that initialize_environment creates the root dir
+        mod.initialise_environment(tmp_path)
+        root_dir = mod.get_root_dir()
+        assert root_dir == tmp_path
+
+        # test that root_dir from parent is detected correctly
+        sub_dir = tmp_path / "sub_dir"
+        sub_dir.mkdir()
+        with mod.os_chdir(sub_dir):
+            root_dir = mod.get_root_dir()
+            assert root_dir == tmp_path
+
+        # test that initialising in a subdirectory makes it the root dir
+        with mod.os_chdir(sub_dir):
+            mod.initialise_environment(sub_dir)
+            root_dir = mod.get_root_dir()
+            assert root_dir == sub_dir
+
+
+def test_get_available_releases():
+    res = mod.get_available_releases()
+    assert "dev-23a" in res
+    assert len(res) >= 1
+    for release in res:
+        assert isinstance(release, str)
+
+
+def test_argument_parsing_logic(mocker):
+    """Test to find errors in argparse logic.
+
+    Strategy:
+
+    In each of the tests below, we are setting the sys.argv to simulate the
+    input from the command line, and in each instance, we ensure that the
+    mocked function get the arguments as expected. The function is mocked not
+    to carry out any activity.
+
+    """
+
+    # pretend we have a rootdir defined
+    mock = mocker.patch(
+        "mpsd_software_manager.mpsd_software.get_root_dir", return_value=Path(".")
+    )
+
+    sys.argv = ["mpsd-software-tests", "init"]
+    mock = mocker.patch(
+        "mpsd_software_manager.mpsd_software.initialise_environment", return_value=None
+    )
+    with pytest.raises(SystemExit):
+        mod.main()
+    call_argument = mock.call_args[0][0]
+    assert isinstance(call_argument, Path)
+
+    ### available
+    sys.argv = ["mpsd-software-tests", "available"]
+    mock = mocker.patch(
+        "mpsd_software_manager.mpsd_software.get_available_releases", return_value=None
+    )
+    with pytest.raises(SystemExit):
+        mod.main()
+
+    sys.argv = ["mpsd-software-tests", "available", "dev-23a"]
+    mock = mocker.patch(
+        "mpsd_software_manager.mpsd_software.get_available_package_sets",
+        return_value=None,
+    )
+    mod.main()
+    call_argument = mock.call_args[0][0]
+    assert call_argument == "dev-23a"
+
+    ### prepare
+    sys.argv = ["mpsd-software-tests", "prepare", "dev-23a"]
+    mock = mocker.patch(
+        "mpsd_software_manager.mpsd_software.prepare_environment", return_value=None
+    )
+    mod.main()
+    call_argument = mock.call_args[0][0]
+    assert call_argument == "dev-23a"
+
+    ### install
+    mock = mocker.patch(
+        "mpsd_software_manager.mpsd_software.install_environment", return_value=None
+    )
+    sys.argv = ["mpsd-software-tests", "install", "dev-23a", "foss2022a-mpi"]
+    mod.main()
+    assert mock.call_args[0][0] == "dev-23a"
+    assert mock.call_args[0][1] == ["foss2022a-mpi"]
+
+    sys.argv = [
+        "mpsd-software-tests",
+        "install",
+        "23b",
+        "foss2022a-mpi",
+        "foss2022a-serial",
+    ]
+    mod.main()
+    assert mock.call_args[0][0] == "23b"
+    assert mock.call_args[0][1] == ["foss2022a-mpi", "foss2022a-serial"]
+
+    ### status
+    mock = mocker.patch(
+        "mpsd_software_manager.mpsd_software.environment_status", return_value=None
+    )
+    sys.argv = ["mpsd-software-tests", "status", "dev-23a"]
+    mod.main()
+    assert mock.call_args[0][0] == "dev-23a"
+
+    ### remove (argparse doesn't allow this yet.
+    ### Copy from 'install' when the time has come.)
+
+
 def test_interface(tmp_path):
     """Test other things (not implemented yet)."""
     pass
@@ -528,6 +664,7 @@ def test_interface(tmp_path):
     # check that the help message is printed when no arguments are provided
     # check that the help message is printed when -h is provided
     # check that the error messages are also logged to the log file
+    # check that `/` in release is handled correctly
 
 
 # other tests to add (ideally)