LIBRARY_PATH and CPATH
This issue tries to summaries the discussion about LIBRARY_PATH
and CPATH
that came up in !82 (closed). It first explains
how different tools behave and then lists reasons to set/not set the two variables and possible "solutions".
LIBRARY_PATH AND CPATH
(For simplicity we only consider gcc and initially ignore CMake.)
-
CPATH
is used bycpp
to find include directories (hence it replaces a list of-I...
) -
LIBRARY_PATH
is used by gcc in link mode (orld
directly) to find libraries (hence it replaces a list of-L...
) -
ld
by default does not add rpath to a binary - The two variables are not set by Spack by default. They are added to the modules via our custom Spack configuration (following the procedure explained in the Spack documentation).
The two variables have no effect during runtime; for library resolution at this stage LD_LIBRARY_PATH
is used:
- the binary contains a list of sonames
-
ld.so
searches for shared libraries inLD_LIBRARY_PATH
, then rpath of the binary, then system paths (defined in/etc/ld.so.conf
) -
LD_LIBRARY_PATH
cannot by default be set for all modules because it will break system functionality (e.g. wrong ncurses). It is not set in our modules (and not set by Spack by default)
What tools are affected by LIBRARY_PATH and CPATH
- gcc (and ld) directly as explained above
- autotools (providing suitably written m4 scripts, see below)
- CMake (indirectly via gcc, see below)
- probably many other build utilities (either directly or indirectly via compiler/linker)
Autotools
With M4 scripts written in suitable form it is sufficient to just activate a desired library, e.g. --enable-mpi
and the
actual include/library path is determined from the variables. (I don't know if this could also apply to setting rpath.)
In practice most m4 scripts in Octopus seem to need either --with-<library>=/path/to/library/root/lib
or --with-<library>-prefix=/path/to/library/root
explicitly and setting
CPATH/LIBRARY_PATH is not sufficient - at least that is the impression from what I have seen so far.
CMake
We assume we are searching for a library called pkg
Finding libraries
CMake searches through different directories (in order):
-
<PKG>_ROOT
(user defined) - CMAKE_PREFIX_PATH (user defined, set by Spack)
- directories inside the CMake install prefix
- system defaults (
/lib
,/usr/lib
,/share
...)
to find
-
<pkg>config.cmake
(created by the library pkg) -
Find<pkg>.cmake
(created by the consuming project, e.g. Octopus) -
<pkg>.pc
(created by the library pkg)
which in turn determines how pkg can be used.
- At build time CMake includes rpath for all libraries (with the exceptions explained below).
- At install time the user can decide whether to keep or strip out the rpath.
LIBRARY_PATH
-
CMake can detect from the linker call invoked from gcc if a library is in LIBRARY_PATH.
-
If this is the case, CMake will not add the rpath for that library and assume that the system will also make the library available at runtime (e.g. via
LD_LIBRARY_PATH
) -
This is intentional behaviour confirmed here: https://gitlab.kitware.com/cmake/cmake/-/issues/25363#note_1498508
-
This feature can e.g. be useful for cross-compilation, explained in general here: https://cmake.org/cmake/help/book/mastering-cmake/chapter/Cross%20Compiling%20With%20CMake.html (LIBRARY_PATH is not mentioned in this document)
-
Cross-compilation typically cannot work with rpath because the paths at compilation time and runtime differ.
-
It may need dummy libraries at compile time, which pretend behaviour of the runtime library (e.g. MPI support) without actually providing any functionality. These could be added to
LIBRARY_PATH
(but would not make any sense inLD_LIBRARY_PATH
)
Why should they be set?
They can make compilation of Autotools (or custom Makefile based) code more convenient. The user
would still need to manually pass the right linker flags to set the rpath correctly (-Wl,rpath...
) otherwise
libraries will not be found at runtime.
Possible solution 1
Also set LDFLAGS
in the module files to "automatically" set the rpath in binaries. This will work for all
projects including CMake based projects (tested for Octopus).
Slight disadvantage: without additional (maybe manual) work the rpath would contain all paths to all loaded modules not just the ones actually needed by the binary.
Possible solution 2
Set LIBRARY_PATH
and LD_LIBRARY_PATH
, but only for the "actual" dependencies (e.g. libxc, hdf5, ...)
and not for transient low-level dependencies (ncurses, ...).
Disadvantage: this will probably require a hand-crafted list of packages for which the variables should be set.
Possible workarond (solution 3)
(currently used on the BuildBot) When compiling a CMake project the user has to manually unset CPATH and LIBRARY_PATH
Why should they not be set?
As explained in the CMake section above setting LIBRARY_PATH
but not LD_LIBRARY_PATH
breaks CMake's assumptions.
If the variables are not set CMake (and likely other modern build tools) will automatically add the required rpaths to the binaries.
Other users (e.g. autotools without suitable m4 scripts, custom Makefiles, ...) would need to pass all required include/library paths and the linker flags to set the rpath explicitly. This would be more consistent that the current setup where LIBRARY_PATH is provided but not much help/indication is given to set the correct rpath.
For the autotools-based Octopus compilation (as mentioned above)
passing include/library paths is anyway the default for almost all libraries. Flags to set the rpath explicitly are
currently exported in our configure wrappers (in the octopus
directory in this repository).
Closing thoughts/notes
- It is not clear how many users do actually benefit from setting
LIBRARY_PATH
andCPATH
, i.e. directly or indirectly make use of it at all. - On the MPCDF clusters the variables are not set when loading modules. (I have intentionally excluded the other examples mentioned before as they seemed less relevant.)
- The MPCDF currently also does not set
CMAKE_PREFIX_PATH
but there are attempts to get this behaviour changed.