Ensure transitive dependencies for headers and libraries are always available

[ocaisa]

This addresses #5124 but is quite a major change even if just a bugfix. I believe that this was probably the original intention when introducing the related options.

Requires additional tests (and probably debating) so leaving as a draft PR for now.

[ocaisa]

This may seem quite aggressive, but it does seem to work in the cases I tried.

[ocaisa]

I think this is good enough for a review, it passes existing CI and also fixes #5143

[ocaisa]

This looks odd, but the methods available here are limited. My subdirs are full paths relative to ''

[akesandgren]

Can't we just clear the relevant env variables at start of eb and then just load modules and "all will be fine"?

[ocaisa]

Sure, but that is not what we say we support with the option --search-path-cpp-headers. @lexming can correct me, but I thought the intention there was to exclusively use the indicated approach

[lexming]

When I worked on --search-path-cpp-headers I didn't change the behaviour of EB at build time in that regard. That option only introduces the possibility to set how those paths are injected in the build environment.

The default behaviour of EB (which has not changed in ages AFAIK):

• manually inject search paths in direct dependencies with compiler/linker flags, so include folders are always passed with -I flags independently of what modules set
• the build environment is not reset, any environment variables like CPATH and LIBRARY_PATH will be present in the build environment

Now, I have not yet looked in detail the code in this PR, but it is indeed probably the case that setting --search-path-cpp-headers to an environment variable (instead of flags) is resetting that variable instead of prepending to it.

[boegel]

We should be careful with changing the logic here imho, since I expect it's likely that we would overlook some situations here and introduce regressions.

This effort was triggered by us hitting this bug in EESSI, because we were configuring EasyBuild with $EASYBUILD_MODULE_SEARCH_PATH_HEADERS and $EASYBUILD_SEARCH_PATH_CPP_HEADERS set to include_paths, so $C_INCLUDE_PATH & co are used both in module file and by EasyBuild itself when setting up the build environment.

We revised that:

• EESSI-extend: Use flags as value for EASYBUILD_SEARCH_PATH_CPP_HEADERS (to rely on $CPPFLAGS in build environment instead of redefining $C_INCLUDE_PATH & co) EESSI/software-layer-scripts#175
• add comment to EESSI-extend easyconfig file to note that module-search-path-headers and search-path-cpp-headers should not be set to same value in EasyBuild configuration EESSI/software-layer-scripts#176

That effectively defuses the situation in EESSI, which makes me think we should take our time here...