User interface for top level install command

changed the description

created branch 1-user-interface-for-top-level-install-command to address this issue

mentioned in merge request !1 (merged)

Addition: Concretised environments should be saved when an environment is created, and placed into an appropriate place.

Thats the job of the spack_setup.sh see spack-environments#30 (closed)

The tool should also check that gpg keys are present if skipping of build cache is not enabled.

Implementing the cmd logs in this format for readability, i hope thats ok @hans.fangohr

--------------------------------------------------
2023-05-03T11:56:26.964084, ./install-mpsd-software-environment.py --release dev-23a --install ALL
Software environment installer branch: 1-user-interface-for-top-level-install-command (commit hash: 62e96e6)
Spack environments branch: None (commit hash: None)
--------------------------------------------------
2023-05-03T11:56:38.887466, ./install-mpsd-software-environment.py --release dev-23a --install ALL
Software environment installer branch: 1-user-interface-for-top-level-install-command (commit hash: 62e96e6)
Spack environments branch: None (commit hash: None)

Regarding our discussion on starting a new release: Maybe the --set-up flag is redundant. We could get the same functionality using ./install-software-environment.py --release dev-23b --from 23a and the script could create the new branch according to release whenever from is used.

For the logging: we could think about including some high-level information about the success, either for the complete call or for the top-level steps, e.g. (i) initial set-up, (ii) installation/re-installation/removal of each toolchain.

Based on the discussions today morning with @hans.fangohr and @henning.glawe, im trying to implement a system with a command/action as the first positional argument {remove,install,start-new}. So a new set of possible commands looks like:

./install-mpsd-software-environment.py --release dev-23a # Create the environment on disk
./install-mpsd-software-environment.py remove --release dev-23a # Remove the entire dev-23a environment from DEFAULT location
./install-mpsd-software-environment.py remove --release dev-23a --toolchains=foss2021a,foss2022a # from dev-23a remove toolchains foss2021a,foss2022a
./install-mpsd-software-environment.py install --release dev-23a # Install the entire dev-23a environment from DEFAULT location
./install-mpsd-software-environment.py install --release dev-23a --target-dir=/tmp/dev-23a # Install the entire dev-23a environment at /tmp/dev-23a
./install-mpsd-software-environment.py install --release dev-23a --target-dir=/tmp/dev-23a --force-reinstall# Force reinstall the entire dev-23a environment at /tmp/dev-23a
./install-mpsd-software-environment.py start-new --from-release dev-23a --to-release  dev-23b # Create a new spack environemnt dev-23b based on dev-23a

--target-dir is an argument that can be passed to all commands/actions

inconsistent, first line has no action? may be done implicitly by the install?
assuming that --release is mandatory, make it a positional argument instead of an option
if --target-dir is global option you could also move it to the root argparse from the subparsers...

The first line is in accordance with https://gitlab.gwdg.de/mpsd-cs/mpsd-software-environments/-/issues/1#prepare-installation-of-mpsd-software-environments ( see Prepare installation of MPSD software environments headding)

I would change this to a prepare action, to keep it consistent with the rest (and simplify the argparse handling)

More changes to the interface were decided over lunch:

No target dir: We will use the script dir as the folder for softwarestack base dir, and release the releases ( eg dev-23a) in this dir. so once installed it would look like so:

Fig1

/opt_mpsd/linux-debian11:$ tree
.
├── dev-23a
│   ├── install.log
│   ├── sandybridge
│   │   ├── build-040523-foss2021a.log
│   │   ├── build_cache
│   │   ├── lmod
│   │   └── spack
│   └── spack-environmnet
└── install-mpsd-software-environment.py

separate actions / sub commands for reinstall and prepre
move --release as a mandatory positional argument (less clear but easier to type)
install.log for the brief overview of command and list of steps taken inside release dir see Fig 1.
--enable-build-cache as an optional argument for install and reinstall commands.

Future ideas:

create a subcommand generate-build-cache that generates the build cache for a given release
Use spacks gpg key (spack gpg create <name> <email> and spack gpg init) to create build cache

so we are almost back at what we did in spack-environments ;)

I would move the install.log somewhere below $MPSD_MICROARCH, as the installs are done independently for each target microarch.

and as build-040523-foss2021a.log files may accumulate, it may be better to move them to a log subdir. so my idea would be something along

├── dev-23a
│   ├── sandybridge
│   │   ├── log
│   │   |   ├── install.log
|   |   |   └── build-foss2021a-%Y%m%d_%H%M%S.log
│   │   ├── build_cache
│   │   ├── lmod
│   │   └── spack
│   └── spack-environmnets
└── install-mpsd-software-environment.py

Stringifying dates that way makes ls output lexicographically sortable by time. To be discussed: move date before the build

The install.log only lists all the times a user ran the script and each time a high level summary of all the steps it took. This is independent of the microarch.
We would have at max #(toolchain) number of build-040523-foss2021a.log files, so it shouldnt be a problem.
The file name was only a quick suggestion the long name for the file would be something like "build_{toolchain}_{mpsd_spack_ver}_{time.strftime('%Y%m%d-%H%M%S')}.log"

Today talked about Possibly add a show-status or summaries command that summaries the information in install.log and build-log

mentioned in issue #2 (closed)

Having used the script in development for some tests now, I suggest to rename it from install-mpsd-software-environment.py to mpsd-software-environment.py: the install is only one use case, and it re-appears as an action later in the line.

Current state:

install-mpsd-software-environment.py install dev-23a --toolchains global
install-mpsd-software-environment.py remove dev-23a --toolchains global

Proposed version:

mpsd-software-environment.py install dev-23a --toolchains global
mpsd-software-environment.py remove dev-23a --toolchains global

User interface for top level install command

Prepare installation of MPSD software environments

Install selected or all toolchains

What to do if a directory exists already?

Possible additional features for development

Use a target directory name that deviates from the release name

Removing an existing toolchain

Starting an new software environment version

Logging

Designs

Child items ...

Activity