Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • hmenke/mpsd-software-manager
  • mpsd-cs/mpsd-software-manager
2 results
Show changes
Commits on Source (114)
14 additional commits have been omitted to prevent performance issues.
Hide whitespace changes
Inline Side-by-side
.gitignore
View file @ 30a9c4b7
......@@ -6,3 +6,4 @@ dev-23a/
dist/
build/
*.egg-info/
.mpsd-software-root
.gitlab-ci.yml
View file @ 30a9c4b7
......@@ -29,7 +29,7 @@ stages:
- echo "Install Python3"
- apt-get update
- apt-get install -y python3 python3-venv python3-rich
- apt-get install -y python3 python3-venv
- python3 -m venv --help
- python3 -m venv venv
- source venv/bin/activate
......@@ -39,13 +39,10 @@ stages:
- echo "Install Python dependencies for running the tests"
- pip install -U pip
- pip --version
- pip install pytest black ruff archspec
- pip install .[dev]
- echo "Diagnostics - which versions are we using"
- python3 --version
- pytest --version
- black --version
- ruff --version
- echo "Install additional packages we need to run spack-setup.sh"
- apt-get install -y git rsync
......
README.rst
View file @ 30a9c4b7
......@@ -22,7 +22,7 @@ To install, for example, the ``foss2022a-serial`` toolchain:
1. Install this mpsd-software-manager Python package. The recommended way is to
use ``pipx`` to that this tool is available independent from the use of any
other python environments::
other Python environments::
$ pipx install git+https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager
......@@ -33,18 +33,27 @@ To install, for example, the ``foss2022a-serial`` toolchain:
$ cd /home/user/mpsd-software
Future calls of the `mpsd-software` command need to be executed from this
"mpsd-software-root" directory.
3. Initiate the installation at this location using::
3. From the same directory, run the command to install the ``foss2022a-serial``
$ mpsd-software init
Future calls of the `mpsd-software` command need to be executed from this
"mpsd-software-root" directory or in one of its subdirectories.
(The above command creates a hidden file ``.mpsd-software-root`` to tag the location for
as the root of the installation. All compiled files, logs etc are written in
or below this subdirectory.)
4. From the same directory, run the command to install the ``foss2022a-serial``
toolchain::
$ mpsd-software install dev-23a foss2022a-serial
This will take some time (up to several hours depending on hardware).
4. To see the installation status, and the required ``module use`` command line
5. To see the installation status, and the required ``module use`` command line
to activate the created modules, try the ``status`` command::
$ mpsd-software status dev-23a
......@@ -55,7 +64,7 @@ To install, for example, the ``foss2022a-serial`` toolchain:
foss2022a-serial
[module use /home/user/mpsd-software/dev-23a/cascadelake/lmod/Core]
5. To compile Octopus, source the provided configure script, for example ``foss2022a-serial-config.sh``, as
6. To compile Octopus, source the provided configure script, for example ``foss2022a-serial-config.sh``, as
`explained here <https://computational-science.mpsd.mpg.de/docs/mpsd-hpc.html#loading-a-toolchain-to-compile-octopus>`__).
The configure scripts are located in ``dev-23a/spack-environments/octopus``::
......@@ -100,7 +109,7 @@ Package sets and toolchains
- openblas@0.3.20
- in addition to the Easybuild-driven choice of packages, there are
additional packages included in each package which support the build of
additional packages included in each toolchain which support the build of
Octopus within these toolchains. For ``foss2022a-serial`` these packages
include::
......@@ -151,7 +160,7 @@ Prerequisites
What needs to be installed for the installation to succeed?
The ``mpsd-software-manager`` python package.
The ``mpsd-software-manager`` Python package.
- This needs a recent Python (3.9 or later).
- Install via pip or pipx.
......@@ -177,6 +186,26 @@ Requirements for particular toolchains and package sets
- ``foss*-mpi`` currently needs linux header files installed (to compile the ``knem`` package)
- ``foss*-cuda-mpi`` (proably as `*-mpi, needs testing TODO`)
Finding the Octopus configure wrapper
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For each Octopus toolchain, there is an Octopus configure wrapper available.
The wrapper essentially calls the configure script with the right parameters,
and library locations for the current toolchain. Once the
toolchain is loaded, the variable ``$MPSD_OCTOPUS_CONFIGURE`` contains that
path. The path can also be seen using the ``module show TOOLCHAIN_NAME`` command. For example::
$ mpsd-software install dev-23a foss2022a-mpi
$ module use ~/mpsd-software/dev-23a/cascadelake/lmod/Core
$ module show toolchains/foss2022a-mpi
...
depends_on("cgal/5.0.3")
depends_on("hdf5/1.12.2")
setenv("MPSD_OCTOPUS_CONFIGURE","~/mpsd-software/dev-23a/spack-environments/octopus/foss2022a-mpi-config.sh")
$ module load toolchains/foss2022a-mpi
$ echo $MPSD_OCTOPUS_CONFIGURE
~/mpsd-software/dev-23a/spack-environments/octopus/foss2022a-mpi-config.sh
Working example
~~~~~~~~~~~~~~~
......@@ -191,17 +220,17 @@ Octopus) using the ``foss2022a-serial`` toolchain.
Frequently asked questions
--------------------------
- Can I install the ``mpsd-software-manager`` package in a python virtual environment?
- Can I install the ``mpsd-software-manager`` package in a Python virtual environment?
Yes. ``pipx`` is probably more convenient, but you can create your own Pyton
virtual environment and install the ``mpsd-software-manager`` in that as a
regular python package::
regular Python package::
python3 -m venv venv
. venv/bin/activate
pip install git+https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager
You just need to activate that python virtual environment before being able to
You just need to activate that Python virtual environment before being able to
use the tool.
- Does the command write anything outside the mpsd-software-root directory?
......@@ -219,9 +248,10 @@ Frequently asked questions
- How long does the compilation take?
This depends on the hardware. A few hours are typical per toolchain. If a
second toolchain is compiled in the same MPSD software instance is likely to
be faster, in particular if the same compiler is used (and thus the compiler
does not need to be re-compiled for the second toolchain).
second toolchain is compiled in the same MPSD software instance and the same
MPSD release it is likely to be faster, in particular if the same compiler is
used (and thus the compiler does not need to be re-compiled for the second
toolchain).
- How much disk storage do I need?
......@@ -244,15 +274,3 @@ Frequently asked questions
Development
-----------
Developers documentation is available at development.rst.
.. comment:
Draft for additional steps for 'quickstart' once/if we have the the `init` command added.
3. Initiate the installation at this location using::
$ mpsd-software init
(This creates a hidden file ``.mpsd-software-root`` to tag the location for
as the root of the installation. All compiled files, logs etc are written in
or below this subdirectory.)
development.rst
View file @ 30a9c4b7
......@@ -17,3 +17,20 @@ Then every time you commit, pre-commit will run all checks defined in `.pre-comm
you can run the pre-commit checks manually by running::
pre-commit run --all-files
Debugging exit codes
--------------------
Non zero exit codes are used to indicate that the program exited due to an error.
There are multiple ways to debug this. You could run the program with more verbose logging::
mpsd-software -l debug ...
Here is a list of exit codes and what they mean:
+-----------+------------------------------------------+----------------------------------------------------------------------------------+
| Exit code | Reason | Solution |
+===========+==========================================+==================================================================================+
| 10 | Call of 'archspec cpu' failed | Please install archspec, for example via 'pipx install archspec' |
| 20 | Requested package set is not available | Use 'available' command to see list of available package_sets |
| 30 | Current directory is already initialised | Check if you are in the right directory |
| 40 | Current directory is not initialised | Check if you are in the right directory, if so use 'init' command to initialise |
+-----------+------------------------------------------+----------------------------------------------------------------------------------+
pyproject.toml
View file @ 30a9c4b7
......@@ -7,8 +7,9 @@ name = "mpsd_software_manager"
authors = [{name = "SSU-Computational Science (Fangohr et al)", email = "ssu-cs@mpsd.mpg.de"}]
license = {file = "LICENSE"}
classifiers = ["License :: OSI Approved :: MIT License"]
version = "2023.6.14"
version = "2023.7.6"
readme = "README.rst"
requires-python = ">=3.9"
dependencies = [
"archspec",
"rich",
......@@ -17,16 +18,15 @@ dependencies = [
mpsd-software = "mpsd_software_manager.mpsd_software:main"
[project.urls]
Home = "https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager/"
homepage = "https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager/"
repository = "https://gitlab.gwdg.de/mpsd-cs/mpsd-software-manager/"
[project.optional-dependencies]
dev = [
"black",
"pre-commit",
"pytest",
"pytest-mock",
"pytest-cov",
"ruff",
]
[tool.pytest.ini_options]
......
src/mpsd_software_manager/mpsd_software.py
View file @ 30a9c4b7
......@@ -2,7 +2,6 @@
"""mpsd-software: tool for installation of software as on MPSD HPC."""
__version__ = "2023.6.19"
import argparse
import datetime
......@@ -16,16 +15,14 @@ from pathlib import Path
from typing import List, Tuple, Union
import re
import shutil
from functools import cache
import importlib.metadata
# If 'rich' is available ("pip install rich" or "apt-get install python3-rich"),
# then use coloured output, otherwise proceed as before
try:
import rich.logging
except ModuleNotFoundError:
rich_available = False
else:
rich_available = True
__version__ = importlib.metadata.version(__package__ or __name__)
import rich.logging
command_name = Path(sys.argv[0]).name
about_intro = f"""
Build software as on MPSD HPC.
......@@ -44,7 +41,7 @@ Build software as on MPSD HPC.
Command line usage:
$> {sys.argv[0]}
$> {command_name}
"""
......@@ -54,18 +51,22 @@ about_epilog = f"""
Examples:
1. Query what package sets and toolchains are available for installation in
1. Query what releases are available for installation
$> {command_name} available
2. Query what package sets and toolchains are available for installation in
release dev-23a
$> {sys.argv[0]} available dev-23a
$> {command_name} available dev-23a
2. Install foss2022a-serial toolchain from the dev-23a release
3. Install foss2022a-serial toolchain from the dev-23a release
$> {sys.argv[0]} install dev-23a foss2022a-serial
$> {command_name} install dev-23a foss2022a-serial
3. Check what package sets and toolchains are installed from release dev-23a
4. Check what package sets and toolchains are installed from release dev-23a
$> {sys.argv[0]} status dev-23a
$> {command_name} status dev-23a
The `status` command also displays the `module use` command needed to load
the created modules.
......@@ -82,15 +83,62 @@ config_vars = {
"metadata_tag_open": "!<meta>",
"metadata_tag_close": "</meta>!",
"spack_environments_repo": "https://gitlab.gwdg.de/mpsd-cs/spack-environments.git",
"init_file": ".mpsd-software-root",
}
def create_log_file_names(
def log_metadata(key: str, value: str) -> None:
"""Log metadata to the log file.
This function logs metadata to the log file. The metadata is
enclosed in a tag, so that it can be easily found in the log file.
logging module is used to write the metadata to the log file.
Parameters
----------
key : str
key of the metadata
value : str
value of the metadata
returns : None
"""
logging.info(
f"{config_vars['metadata_tag_open']}{key}:{value}{config_vars['metadata_tag_close']}"
)
def read_metadata_from_logfile(logfile: Union[str, Path]) -> dict:
"""Read metadata from the log file.
This function reads metadata from the log file. The metadata is
enclosed in a tag, so that it can be easily found in the log file.
Parameters
----------
logfile : str or Path
log file name
returns : dict
dictionary containing the metadata
"""
with open(logfile, "r") as f:
log_text = f.read()
# check for all data that matches the regex
# metadata_tag_open {key}:{value} metadata_tag_close
# and return a dictionary with all the matches
return {
match.group(1): match.group(2)
for match in re.finditer(
rf"{config_vars['metadata_tag_open']}(\w+):(\w+){config_vars['metadata_tag_close']}",
log_text,
)
}
def create_log_file_name(
mpsd_release: str,
microarch: str,
action: str,
date: str = call_date_iso,
package_set: str = None,
package_set: Union[str, None] = None,
) -> Union[str, None]:
"""Create log file names.
......@@ -107,14 +155,12 @@ def create_log_file_names(
----------
mpsd_release : str
MPSD software stack version
microarch : str
system architecture
date : str
date of the call ins iso format
action : str
action performed (install,remove,reinstall,prepare,status)
only install and remove are valid for build log file.
package_set : str
package_set : str or None
package_set name (only for build log file)
Returns
......@@ -123,9 +169,46 @@ def create_log_file_names(
log file name
installer_log_file_name or build_log_file_name depending on the
parameters given.
If the action is not one that changes the files on disk ( info only actions)
If the action is not one that changes the files on disk (info only actions)
then None is returned.
Examples
--------
# installer log file name for `mpsd-software install dev-23a foss2021a-mpi`
>>> create_log_file_name(
... "dev-23a",
... "install",
... "2023-07-03T12-27-52",
... )
'dev-23a_sandybridge_2023-07-03T12-27-52_APEX_install.log'
# build log file name for `mpsd-software install dev-23a foss2021a-mpi`
>>> create_log_file_name(
... "dev-23a",
... "install",
... "2023-07-03T12-27-52",
... "foss2021a-mpi",
... )
'dev-23a_sandybridge_2023-07-03T12-27-52_BUILD_foss2021a-mpi_install.log'
# installer log file name for `mpsd-software status dev-23a`
>>> create_log_file_name(
... "dev-23a",
... "status",
... "2023-07-03T12-27-52",
... )
'dev-23a_sandybridge_2023-07-03T12-27-52_APEX_status.log'
# build log file name for `mpsd-software status dev-23a` (no log file is created)
>>> create_log_file_name(
... "dev-23a",
... "status",
... "2023-07-03T12-27-52",
... "foss2021a-mpi",
... )
(None)
"""
microarch = get_native_microarchitecture()
if package_set:
# if package_set is given, then we build the build_log_file_name
if action in ["install", "remove"]:
......@@ -141,73 +224,94 @@ def create_log_file_names(
return log_file_name
def log_metadata(key: str, value: str) -> None:
"""Log metadata to the log file.
This function logs metadata to the log file. The metadata is
enclosed in a tag, so that it can be easily found in the log file.
logging module is used to write the metadata to the log file.
Parameters
----------
key : str
key of the metadata
value : str
value of the metadata
returns : None
"""
logging.info(
f"{config_vars['metadata_tag_open']}{key}:{value}{config_vars['metadata_tag_close']}"
)
def get_log_file_path(
mpsd_release: str, cmd: str, root_dir: Path, package_set: Union[str, None] = None
) -> Union[Path, None]:
"""Get log file path.
This function creates the log file paths for either the installer or
the build log files.
def read_metadata_from_logfile(logfile: Union[str, Path]) -> dict:
"""Read metadata from the log file.
If a package_set is given, then the build log file path is returned.
if no package_set is given, then the installer log file path is returned.
This function reads metadata from the log file. The metadata is
enclosed in a tag, so that it can be easily found in the log file.
If the logs folder does not exist, then it is created.
Parameters
----------
logfile : str or Path
log file name
returns : dict
dictionary containing the metadata
"""
with open(logfile, "r") as f:
log_text = f.read()
# check for all data that matches the regex
# metadata_tag_open {key}:{value} metadata_tag_close
# and return a dictionary with all the matches
return {
match.group(1): match.group(2)
for match in re.finditer(
f"{config_vars['metadata_tag_open']}(\w+):(\w+){config_vars['metadata_tag_close']}",
log_text,
)
}
mpsd_release : str
MPSD software stack version
cmd : str
command to be executed
root_dir : str
root directory of the mpsd software stack
package_set : str
package_set name (only for build log file)
Returns
-------
Path or None
log file path
installer_log_file_path or build_log_file_path depending on the
parameters given.
def get_installer_log_file_path(mpsd_release: str, cmd: str, root_dir: str) -> str:
"""Get installer log file path."""
# Get machine configs
os.environ.get("MPSD_OS", "UNKNOWN_OS")
microarch = get_native_microarchitecture()
# parse logging first
# decide the log_file_name
installer_log_name = create_log_file_names(
mpsd_release=mpsd_release, microarch=microarch, action=cmd
Examples
--------
# installer log file path for `mpsd-software install dev-23a foss2021a-mpi`
>>> get_log_file_path(
... "dev-23a",
... "install",
... Path(
... "/tmp/root_dir"
... ),
... )
PosixPath('/tmp/root_dir/dev-23a/logs/dev-23a_zen3_2023-07-03T12-28-55_APEX_install.log')
# build log file path for `mpsd-software install dev-23a foss2021a-mpi`
>>> get_log_file_path(
... "dev-23a",
... "install",
... Path(
... "/tmp/root_dir"
... ),
... "foss2021a-mpi",
... )
PosixPath('/tmp/root_dir/dev-23a/logs/dev-23a_zen3_2023-07-03T12-28-55_BUILD_foss2021a-mpi_install.log')
# installer log file path for `mpsd-software status dev-23a`
>>> get_log_file_path(
... "dev-23a",
... "status",
... Path(
... "/tmp/root_dir"
... ),
... )
PosixPath('/tmp/root_dir/dev-23a/logs/dev-23a_zen3_2023-07-03T12-28-55_APEX_status.log')
# build log file path for `mpsd-software status dev-23a` (no log file is created)
>>> get_log_file_path(
... "dev-23a",
... "status",
... Path(
... "/tmp/root_dir"
... ),
... "foss2021a-mpi",
... )
(None)
"""
log_file_name = create_log_file_name(
mpsd_release=mpsd_release,
action=cmd,
package_set=package_set,
)
log_folder = root_dir / mpsd_release / "logs"
# if the log_folder dosent exist, dont log this message if
# the command is a info-only command
if cmd not in ["status", "available"]:
if not os.path.exists(log_folder):
os.makedirs(log_folder)
installer_log_file = log_folder / installer_log_name
if log_file_name:
# if the log_folder dosent exist, create it
if not log_folder.exists():
log_folder.mkdir(parents=True)
return log_folder / log_file_name
else:
installer_log_file = None
return installer_log_file
return None
def set_up_logging(loglevel="warning", file_path=None):
......@@ -241,8 +345,7 @@ def set_up_logging(loglevel="warning", file_path=None):
1. log = logging.getLogger('')
This is the 'root' logger. It uses a RichHandler if rich is available for
output to the shell, otherwise plain text.
This is the 'root' logger.
Typical use:
......@@ -289,19 +392,13 @@ def set_up_logging(loglevel="warning", file_path=None):
logger.setLevel(0)
# the handler determines where the logs go: stdout/file
if rich_available:
# https://rich.readthedocs.io/en/stable/logging.html
shell_handler = rich.logging.RichHandler()
# rich handler provides metadata automatically:
logging_format = "%(message)s"
# for shell output, only show time (not date and time)
shell_formatter = logging.Formatter(logging_format, datefmt="[%X]")
else:
shell_handler = logging.StreamHandler()
# include line numbers in output if level is DEBUG
linenumbers = " %(lineno)4d" if log_level_numeric == logging.DEBUG else ""
logging_format = "%(asctime)s %(levelname)7s" + linenumbers + " | %(message)s"
shell_formatter = logging.Formatter(logging_format)
# We use 'rich' to provide a Handler:
# https://rich.readthedocs.io/en/stable/logging.html
shell_handler = rich.logging.RichHandler()
# rich handler provides metadata automatically:
logging_format = "%(message)s"
# for shell output, only show time (not date and time)
shell_formatter = logging.Formatter(logging_format, datefmt="[%X]")
# here we hook everything together
shell_handler.setFormatter(shell_formatter)
......@@ -334,10 +431,10 @@ def set_up_logging(loglevel="warning", file_path=None):
# create formatter 'empty' formatter
formatter = logging.Formatter("%(message)s")
# create, format and add handler for shell output
# create, format and set handler for shell output
ch = logging.StreamHandler()
ch.setFormatter(formatter)
print_log.addHandler(ch)
print_log.handlers = [ch]
# if filename provided, write output of print_log to that file, too
if file_path:
......@@ -350,8 +447,7 @@ def set_up_logging(loglevel="warning", file_path=None):
# short message
#
logging.debug(
f"Logging has been setup, loglevel={loglevel.upper()} "
+ f"{file_path=} {rich_available=}"
f"Logging has been setup, loglevel={loglevel.upper()} " + f"{file_path=}"
)
......@@ -384,9 +480,10 @@ def get_available_package_sets(mpsd_release: str) -> List[str]:
"""
logging.debug(f"get_available_package_sets({mpsd_release=})")
logging.info(f"Retrieving available package_sets for release {mpsd_release}")
print_log = logging.getLogger("print")
logging.info(f"Retrieving available package_sets for release {mpsd_release}")
# create temporary directory
tmp_dir = tempfile.TemporaryDirectory(prefix="mpsd-software-available-")
tmp_dir_path = Path(tmp_dir.name)
......@@ -552,7 +649,7 @@ def run(*args, counter=[0], **kwargs):
def record_script_execution_summary(
mpsd_release: str, root_dir: str, msg: str = None, **kwargs
root_dir: Path, msg: Union[str, None] = None, **kwargs
) -> None:
"""Log the command used to build the package_set.
......@@ -562,8 +659,6 @@ def record_script_execution_summary(
Parameters
----------
- mpsd_release : str
The name of the release to install toolchains for.
- root_dir : str
The path to the directory where the scripts are located.
- msg : str, optional
......@@ -606,10 +701,11 @@ def record_script_execution_summary(
f.write(f"{datetime.datetime.now().isoformat()}, {cmd_line}\n")
# logs script version
f.write(f"MPSD Software manager version: {__version__}\n")
f.write(
f"Spack environments branch: {spe_branch} "
f"(commit hash: {spe_commit_hash})\n"
)
if spe_branch and spe_commit_hash:
f.write(
f"Spack environments branch: {spe_branch} "
f"(commit hash: {spe_commit_hash})\n"
)
def clone_repo(
......@@ -647,7 +743,7 @@ def clone_repo(
)
if checkout_result.returncode != 0:
msg = f"Couldn't find {branch=}\n"
msg = f"Couldnt find {branch=}\n"
branches_result = run(
["git", "branch", "-a"], check=True, capture_output=True
......@@ -664,6 +760,28 @@ def clone_repo(
run(["git", "pull"], check=True, capture_output=capture_output)
def get_available_releases(print_result: bool = False) -> List[str]:
"""
Return available MPSD software release versions.
Example
-------
>>> get_available_releases()
["dev-23a"]
Notes
-----
This needs to be updated when a new version (such as 23b) is released.
"""
releases = ["dev-23a"]
print_log = logging.getLogger("print")
if print_result:
print_log.info("Available MPSD software releases:")
for release in releases:
print_log.info(f" {release}")
return releases
def get_release_info(mpsd_release: str, root_dir: Path) -> Tuple[str, str, List[str]]:
"""
Get information about the specified release.
......@@ -787,11 +905,12 @@ def prepare_environment(mpsd_release: str, root_dir: Path) -> List[str]:
mpsd_release, root_dir
)
record_script_execution_summary(
mpsd_release, root_dir, spe_branch=spe_branch, spe_commit_hash=spe_commit_hash
root_dir, spe_branch=spe_branch, spe_commit_hash=spe_commit_hash
)
return available_package_sets
@cache
def get_native_microarchitecture():
"""Return native microarchitecture.
......@@ -829,7 +948,7 @@ def get_native_microarchitecture():
msg += "Documentation of package: https://archspec.readthedocs.io/"
logging.error(msg)
sys.exit(1)
sys.exit(10)
else: # we have found archspec and executed it
if process.returncode == 0: # sanity check
microarch = process.stdout.strip()
......@@ -918,32 +1037,25 @@ def install_environment(
msg += f" in release {mpsd_release}. "
msg += "Use 'available' command to see list of available package_sets."
logging.error(msg)
sys.exit(1)
sys.exit(20)
# Install the package_sets
with os_chdir(package_set_dir):
# run spack_setup_script with the package_sets as arguments
for package_set in package_sets:
# Set the install log file name from create_log_file_names
build_log_file_name = create_log_file_names(
mpsd_release, microarch, "install", package_set=package_set
build_log_path = get_log_file_path(
mpsd_release, "install", root_dir, package_set
)
build_log_folder = release_base_dir / "logs"
build_log_path = build_log_folder / build_log_file_name
# if logs folder dosent exist, create it
if not os.path.exists(build_log_folder):
os.makedirs(build_log_folder)
logging.info(f"Installing package_set {package_set} to {package_set_dir}")
# log the command
record_script_execution_summary(
mpsd_release,
root_dir,
msg=f"installing {package_set} and logging at {build_log_path}",
)
record_script_execution_summary(
mpsd_release,
root_dir,
msg=(
f"CMD: bash {spack_setup_script} {' '.join(install_flags)} "
......@@ -1002,7 +1114,7 @@ def start_new_environment(release, from_release, target_dir):
raise NotImplementedError(msg)
def environment_status(mpsd_release: str, root_dir: Union[str, Path]) -> dict:
def environment_status(mpsd_release: str, root_dir: Path) -> Union[dict, None]:
"""Show status of release in installation.
Parameters
......@@ -1019,6 +1131,7 @@ def environment_status(mpsd_release: str, root_dir: Union[str, Path]) -> dict:
toolchain_map : dict
A dictionary containing available microarchitectures as keys and
a list of available package_sets as values for each microarchitecture.
If the release is not installed/found, None is returned.
Note: only toolchains can be reported at the moment (i.e. package_sets
such as global and global_generic are missing, even if installed).
......@@ -1041,7 +1154,7 @@ def environment_status(mpsd_release: str, root_dir: Union[str, Path]) -> dict:
# if the mpds_release directory exists but the spack repository is not fully
# cloned - indicates some kind of incomplete installation:
if not spack_dir.exists():
logging.debug(f"Looking for files in {spack_dir}")
logging.info(f"Could not find directory {spack_dir}.")
logging.error(
f"MPSD release '{mpsd_release}' has not been completely installed."
)
......@@ -1086,6 +1199,85 @@ def environment_status(mpsd_release: str, root_dir: Union[str, Path]) -> dict:
return toolchain_map
def initialise_environment(root_dir: Path) -> None:
"""Initialize the software environment.
This creates a hidden file ``.mpsd-software-root`` to tag the location for
as the root of the installation. All compiled files, logs etc are written in
or below this subdirectory.
Parameters
----------
root_dir : pathlib.Path
A Path object pointing to the current directory where the script was called.
"""
# check if the root_dir is not already initialized
init_file = root_dir / config_vars["init_file"]
if init_file.exists():
logging.error(f"Directory {str(root_dir)} is already initialised.")
sys.exit(30)
else:
# create the init file
init_file.touch()
# note the execution in the execution summary log
# create the log file and fill it with the headers
record_script_execution_summary(root_dir=root_dir)
# record the msg in the log file
record_script_execution_summary(
root_dir=root_dir,
msg=f"Initialising MPSD software instance at {root_dir}.",
)
def get_root_dir() -> Path:
"""Get the root directory of the installation.
Look for the hidden file ``.mpsd-software-root``
(defined in config_vars["init_file"])
in the current directory, or any parent directory.
If found, return the path to the root directory
of the MPSD software instance.
If not found, exit with an error message.
Returns
-------
root_dir : pathlib.Path
A Path object pointing to the root directory of the installation.
This folder contains the hidden file ``.mpsd-software-root``,
``mpsd_releases`` ( for eg ``dev-23a``) and ``mpsd-spack-cache``.
"""
# check if the root_dir is not already initialized
script_call_dir = Path.cwd()
init_file = script_call_dir / config_vars["init_file"]
if init_file.exists():
return script_call_dir
# if not, look for the init file in the parent directories
for parent_folder in script_call_dir.parents:
init_file = parent_folder / config_vars["init_file"]
if init_file.exists():
script_call_dir = parent_folder
return script_call_dir
# if not found in any parent directory, exit with an error message
logging.debug(f"Directory {str(script_call_dir)} is not a MPSD software instance.")
logging.error(
"Could not find MPSD software instance "
"in the current directory or any parent directory.\n\n"
f"The current directory is {script_call_dir}.\n\n"
"To initialise a MPSD software instance here, "
"run 'mpsd-software init'.\n\n"
f"To find the root directory of an existing MPSD software instance, look "
f"for the directory containing '{config_vars['cmd_log_file']}' "
+ f"and the hidden file '{config_vars['init_file']}'."
)
sys.exit(40)
def main():
"""Execute main entry point."""
parser = argparse.ArgumentParser(
......@@ -1109,6 +1301,7 @@ def main():
)
subparsers.required = True
list_of_cmds = [
("init", "Initialise the MPSD software instance in the current directory"),
("available", "What is available for installation?"),
("install", "Install a software environment"),
# ("reinstall", "Reinstall a package_set"),
......@@ -1137,11 +1330,22 @@ def main():
)
else:
subp.add_argument(
"release",
type=str,
help="Release version to prepare, install, reinstall or remove",
)
# most commands except need a release version
if cmd in ["install", "prepare", "reinstall", "remove", "status"]:
subp.add_argument(
"release",
type=str,
help="Release version to prepare, install, reinstall or remove",
)
elif cmd in ["available"]:
# for some commands the release version is optional
subp.add_argument(
"release",
type=str,
nargs="?",
help="Release version to prepare, install, reinstall or remove",
)
if cmd in ["install", "reinstall", "remove"]:
# "install" command needs additional documentation
package_set_help = (
......@@ -1168,22 +1372,46 @@ def main():
# Carry out the action
args = parser.parse_args()
# root dir is the place where this script is called from
root_dir = Path(os.getcwd())
# Set up logging without file handle:
# this is used in the init action and for logging the
# get_root_dir() function
set_up_logging(args.loglevel)
# Check if the action is init
# if so, call the init function and exit
if args.action == "init":
initialise_environment(Path(os.getcwd()))
sys.exit(0)
# if a release version is specified:
if args.release:
# sanity check for common mistakes in command line arguments
if args.release.endswith("/"): # happens easily with autocompletion
args.release = args.release.removesuffix("/")
logging.warning(f"Removed trailing slash from release: {args.release}")
# root_dir is the place where this MPSD software instance has its root
root_dir = get_root_dir()
# set up logging filename: we record activities that change the installation
if args.action in ["init", "install", "prepare", "reinstall", "remove"]:
log_file = get_log_file_path(
args.release,
args.action,
root_dir,
)
# some commands do not write any log_files:
elif args.action in ["available", "status"]:
log_file = None
else:
# sanity check
raise NotImplementedError(f"Should never happen: unknown {args.action=}")
set_up_logging(
args.loglevel,
get_installer_log_file_path(args.release, args.action, root_dir),
log_file,
)
# sanity check for common mistakes in command line arguments
if args.release.endswith("/"): # happens easily with autocompletion
logging.error(
f"You provided mpsd-release='{args.release}'. "
f"Did you mean '{args.release.removesuffix('/')}'?"
)
sys.exit(1)
# Check the command and run related function
if args.action == "remove":
remove_environment(args.release, root_dir, args.package_set)
......@@ -1198,7 +1426,11 @@ def main():
elif args.action == "prepare":
prepare_environment(args.release, root_dir)
elif args.action == "available":
get_available_package_sets(args.release)
if args.release:
get_available_package_sets(args.release)
else:
get_available_releases(print_result=True)
sys.exit(0)
else:
message = (
f"No known action found ({args.action=}). Should probably never happen."
......
tests/test_mpsd_software.py
View file @ 30a9c4b7
......@@ -192,7 +192,7 @@ def test_record_script_execution_summary(tmp_path):
def test_install_environment_wrong_package_set(tmp_path):
"""Test exception is raised for non-existing package_set."""
# exits with exit code 1 when wrong package_sets are provided
# exits with exit code 20 when wrong package_sets are provided
with pytest.raises(SystemExit) as e:
mod.install_environment(
mpsd_release="dev-23a",
......@@ -200,7 +200,7 @@ def test_install_environment_wrong_package_set(tmp_path):
root_dir=(tmp_path),
)
assert e.type == SystemExit
assert e.value.code == 1
assert e.value.code == 20
def test_install_environment_wrong_mpsd_release(tmp_path):
......@@ -280,7 +280,7 @@ def test_install_environment_zlib():
# install global_generic package_set
mod.set_up_logging(
"WARNING",
mod.get_installer_log_file_path(mpsd_release_to_test, "install", root_dir),
mod.get_log_file_path(mpsd_release_to_test, "install", root_dir),
)
mod.install_environment(
mpsd_release=mpsd_release_to_test,
......@@ -333,7 +333,7 @@ def test_install_environment_zlib():
importlib.reload(mod)
mod.set_up_logging(
"WARNING",
mod.get_installer_log_file_path(mpsd_release_to_test, "install", root_dir),
mod.get_log_file_path(mpsd_release_to_test, "install", root_dir),
)
mod.install_environment(
mpsd_release=mpsd_release_to_test,
......@@ -417,17 +417,16 @@ def test_get_available_package_sets():
)
def test_create_log_file_names():
def test_create_log_file_name():
"""Test that the log file names are created correctly."""
create_log_file_names = mod.create_log_file_names
create_log_file_name = mod.create_log_file_name
mpsd_release = "dev-23a"
microarch = "sandybridge"
microarch = mod.get_native_microarchitecture()
date = datetime.datetime.now().replace(microsecond=0).isoformat()
action = "install"
package_set = "foss2021a"
# test build_log_file_name generation
build_log_file_name = create_log_file_names(
microarch=microarch,
build_log_file_name = create_log_file_name(
mpsd_release=mpsd_release,
date=date,
action=action,
......@@ -437,8 +436,7 @@ def test_create_log_file_names():
build_log_file_name
== f"{mpsd_release}_{microarch}_{date}_BUILD_{package_set}_{action}.log"
)
installer_log_file_name = create_log_file_names(
microarch=microarch,
installer_log_file_name = create_log_file_name(
mpsd_release=mpsd_release,
date=date,
action=action,
......@@ -448,8 +446,7 @@ def test_create_log_file_names():
== f"{mpsd_release}_{microarch}_{date}_APEX_{action}.log"
)
# test no build log file for incorrect action
build_log_file_name = create_log_file_names(
microarch=microarch,
build_log_file_name = create_log_file_name(
mpsd_release=mpsd_release,
date=date,
action="status",
......@@ -520,6 +517,145 @@ def test_remove_environment(tmp_path):
# done in test_install_environment_zlib
def test_initialise_environment(tmp_path):
"""Test that init_file is created as expected."""
# test that the init file is created as expected
mod.initialise_environment(tmp_path)
init_file = tmp_path / mod.config_vars["init_file"]
assert init_file.exists()
# ensure "Initialising MPSD software ..." is in the log file
log_file = tmp_path / mod.config_vars["cmd_log_file"]
with open(log_file, "r") as f:
assert (f"Initialising MPSD software instance at {tmp_path}") in f.read()
# test that calling again results in warning and exit code 30
with pytest.raises(SystemExit) as pytest_wrapped_e:
mod.initialise_environment(tmp_path)
assert pytest_wrapped_e.type == SystemExit
assert pytest_wrapped_e.value.code == 30
def test_get_root_dir(tmp_path):
"""Test that the root directory is correct."""
with mod.os_chdir(tmp_path):
# test that function exists with error 40 if root dir doesn't exist
with pytest.raises(SystemExit) as pytest_wrapped_e:
mod.get_root_dir()
assert pytest_wrapped_e.type == SystemExit
assert pytest_wrapped_e.value.code == 40
# test that initialize_environment creates the root dir
mod.initialise_environment(tmp_path)
root_dir = mod.get_root_dir()
assert root_dir == tmp_path
# test that root_dir from parent is detected correctly
sub_dir = tmp_path / "sub_dir"
sub_dir.mkdir()
with mod.os_chdir(sub_dir):
root_dir = mod.get_root_dir()
assert root_dir == tmp_path
# test that initialising in a subdirectory makes it the root dir
with mod.os_chdir(sub_dir):
mod.initialise_environment(sub_dir)
root_dir = mod.get_root_dir()
assert root_dir == sub_dir
def test_get_available_releases():
res = mod.get_available_releases()
assert "dev-23a" in res
assert len(res) >= 1
for release in res:
assert isinstance(release, str)
def test_argument_parsing_logic(mocker):
"""Test to find errors in argparse logic.
Strategy:
In each of the tests below, we are setting the sys.argv to simulate the
input from the command line, and in each instance, we ensure that the
mocked function get the arguments as expected. The function is mocked not
to carry out any activity.
"""
# pretend we have a rootdir defined
mock = mocker.patch(
"mpsd_software_manager.mpsd_software.get_root_dir", return_value=Path(".")
)
sys.argv = ["mpsd-software-tests", "init"]
mock = mocker.patch(
"mpsd_software_manager.mpsd_software.initialise_environment", return_value=None
)
with pytest.raises(SystemExit):
mod.main()
call_argument = mock.call_args[0][0]
assert isinstance(call_argument, Path)
### available
sys.argv = ["mpsd-software-tests", "available"]
mock = mocker.patch(
"mpsd_software_manager.mpsd_software.get_available_releases", return_value=None
)
with pytest.raises(SystemExit):
mod.main()
sys.argv = ["mpsd-software-tests", "available", "dev-23a"]
mock = mocker.patch(
"mpsd_software_manager.mpsd_software.get_available_package_sets",
return_value=None,
)
mod.main()
call_argument = mock.call_args[0][0]
assert call_argument == "dev-23a"
### prepare
sys.argv = ["mpsd-software-tests", "prepare", "dev-23a"]
mock = mocker.patch(
"mpsd_software_manager.mpsd_software.prepare_environment", return_value=None
)
mod.main()
call_argument = mock.call_args[0][0]
assert call_argument == "dev-23a"
### install
mock = mocker.patch(
"mpsd_software_manager.mpsd_software.install_environment", return_value=None
)
sys.argv = ["mpsd-software-tests", "install", "dev-23a", "foss2022a-mpi"]
mod.main()
assert mock.call_args[0][0] == "dev-23a"
assert mock.call_args[0][1] == ["foss2022a-mpi"]
sys.argv = [
"mpsd-software-tests",
"install",
"23b",
"foss2022a-mpi",
"foss2022a-serial",
]
mod.main()
assert mock.call_args[0][0] == "23b"
assert mock.call_args[0][1] == ["foss2022a-mpi", "foss2022a-serial"]
### status
mock = mocker.patch(
"mpsd_software_manager.mpsd_software.environment_status", return_value=None
)
sys.argv = ["mpsd-software-tests", "status", "dev-23a"]
mod.main()
assert mock.call_args[0][0] == "dev-23a"
### remove (argparse doesn't allow this yet.
### Copy from 'install' when the time has come.)
def test_interface(tmp_path):
"""Test other things (not implemented yet)."""
pass
......@@ -528,6 +664,7 @@ def test_interface(tmp_path):
# check that the help message is printed when no arguments are provided
# check that the help message is printed when -h is provided
# check that the error messages are also logged to the log file
# check that `/` in release is handled correctly
# other tests to add (ideally)
......