[mplegendre]

The issue was that we were dynamically creating directories in the cache when we identified a file that needed to be cached in that directory. But ld.so was also maintaining a list of exists/not-exists directory caches, which could conflict with spindle's dynamic cache creation. To reproduce we needed a library located via $ORIGIN/something, which meant the dynamic linker was searching the spindle cache for libraries. We first needed to load another library (say libm.so in some testcases) that didn't exist in that $ORIGIN/something location. ld.so would iterate through its search list, notice that $ORIGIN/something didn't exist yet and mark it not-exists. Then we needed to try to load the original library that was in $ORIGIN/something, but ld.so would no longer search that path and the library wouldn't be found. Also this didn't happen on systems where we successfully install ld.so trampoline hooks, as the interceptions of the stat() calls would fix the result of the 'does this directory exist' test in ld.so. To fix this, we now create a directory in the cache when it is first searched, even if we do not yet have files to put in it.

Also add tests, and fix issues with fork tests not working properly.
Also write spindle_mkdir to support with feature.

[nchaimov]

I've confirmed that this fixes the problem with Java and with the stc testcase in E4S.

I notice that it's creating directories in the cache even where the original directory didn't exist, not just in cases where it does. For example, after running the reproducer with spindle --noclean I see:

spindleuser@spindlenode:/tmp/commpath/cachepath/spindle.744c4f35ed7495ee$ tree
.
|-- home
|   `-- spindleuser
|       |-- 0-spindlens-dso-test-libfoo-before-libm
|       |-- Spindle-inst
|       |   `-- lib
|       |       `-- spindle
|       |           `-- 1-spindlens-dso-libspindle_audit_pipe.so
|       `-- lib
|           |-- 2-spindlens-dso-libfoo.so
|           `-- glibc-hwcaps
|               |-- x86-64-v2
|               `-- x86-64-v3
|-- lib
|   `-- x86_64-linux-gnu
`-- usr
    `-- lib
        `-- x86_64-linux-gnu
            `-- 3-spindlens-dso-libm.so.6

15 directories, 4 files

where there are now directories for /home/spindleuser/lib/glibc-hwcaps/x86-64-v2 and x86-64-v3, even though the original directories don't exist. Is that expected behavior?

[mplegendre]

Oops. Good catch. It was not supposed to do that.

[nchaimov]

I've confirmed that this fixes the problem where it was creating directories even where one didn't exist, and the E4S stc test is passing. I am now rerunning the full E4S testsuite with this PR.

[nchaimov]

The stc test is fixed, but this is introducing a new failure in the tau test that doesn't occur on the devel branch. The problem occurs when a script launches another script through a path relative to itself.

Here is a minimal reproducer for the issue:

I have this layout, with two scripts in two different directories:

test/
|-- bin
|   `-- run.sh
`-- libexec
	`-- helper.sh

bin/run.sh:

#!/bin/bash
set -x
"$(dirname "$0")/../libexec/helper.sh"

libexec/helper.sh:

#!/bin/bash
echo PASSED

On the devel branch, this works with or without Spindle:

spindleuser@spindlenode:~/test$ bash ./bin/run.sh
++ dirname ./bin/run.sh
+ ./bin/../libexec/helper.sh
PASSED

spindleuser@spindlenode:~/test$ spindle bash ./bin/run.sh
++ dirname ./bin/run.sh
+ ./bin/../libexec/helper.sh
PASSED

However, with this PR, it fails under Spindle:

spindleuser@spindlenode:~/test$ bash ./bin/run.sh
++ dirname ./bin/run.sh
+ ./bin/../libexec/helper.sh
PASSED

spindleuser@spindlenode:~/test$ spindle bash ./bin/run.sh
++ dirname ./bin/run.sh
+ ./bin/../libexec/helper.sh
./bin/run.sh: line 3: ./bin/../libexec/helper.sh: cannot execute: required file not found
spindleuser@spindlenode:~/test$

Oddly, I have to invoke the script as an argument to bash to reproduce the problem. If I run it as ./bin/run.sh, it works under Spindle:

spindleuser@spindlenode:~/test$ ./bin/run.sh
++ dirname ./bin/run.sh
+ ./bin/../libexec/helper.sh
PASSED

spindleuser@spindlenode:~/test$ spindle ./bin/run.sh
++ dirname ./bin/run.sh
+ ./bin/../libexec/helper.sh
PASSED

Here is a log of the failure:
spindle_output.spindlenode.166.txt

from which the failure appears to be:

[Client.1.175@intercept_exec.c:561] execve_wrapper - Intercepted execve on ./bin/../libexec/helper.sh
[Client.1.175@intercept_exec.c:92] shouldPropogateSpindle - Decided to propogate spindle through exec
[Client.1.175@intercept_exec.c:391] find_exec - Requesting stat on exec of ./bin/../libexec/helper.sh to validate file
[Client.1.175@lookup.c:258] get_metadata_result - Sending request for stat of ./bin/../libexec/helper.sh to server
[Server.168@ldcs_audit_server_handlers.c:380] handle_client_file_request - Server recvd query stat for /home/spindleuser/test/bin/../libexec/helper.sh.  Dir = /home/spindleuser/test/bin/../libexec, File = helper.sh
[Server.168@ldcs_audit_server_handlers.c:2411] handle_metadata_and_broadcast_file - Stating and broadcasting file /home/spindleuser/test/bin/../libexec/helper.sh
[Server.168@ldcs_audit_server_filemngt.c:490] - ERROR: Failed to create file /tmp/commpath/cachepath/spindle.e3f3d24983a27a22/home/spindleuser/test/bin/../libexec/6-spindlens-stat-helper.sh for writing: No such file or directory

[mplegendre]

Thanks for finding this.
Looks like the new spindle_mkdir is resolving paths with '../' components that are under non-existent subdirectories incorrectly. So '/foo/../bar' gets resolved inside spindle_mkdir to '/bar', and then '/bar' gets created. But the original path passed to spindle_mkdir is still set at '/foo/../bar' and '/foo' was never created. So that original path can't be used to exec() or load files.

[nchaimov]

The E4S testsuite now fully passes, but the Slurm Plugin failure looks legitimate and not some random fluke. I can reproduce it in the testing container locally. slurmd fails to start, with the plugin giving an undefined symbol: spindle_debug_output_f error:

[2026-10-30T07:17:16.761] debug:  spank: opening plugin stack /etc/slurm/plugstack.conf
[2026-10-30T07:17:16.763] error: plugin_load_from_file: dlopen(/home/slurmuser/Spindle-inst/lib/libspindleslurm.so): /home/slurmuser/Spindle-inst/lib/libspindleslurm.so: undefined symbol: spindle_debug_output_f
[2026-10-30T07:17:16.763] error: spank: /home/slurmuser/Spindle-inst/lib/libspindleslurm.so: Dlopen of plugin file failed
[2026-10-30T07:17:16.763] error: spank: /etc/slurm/plugstack.conf:1: Failed to load plugin /home/slurmuser/Spindle-inst/lib/libspindleslurm.so. Aborting.
[2026-10-30T07:17:16.763] error: slurmd initialization failed

[nchaimov]

The Slurm Plugin testsuite is passing in the CI environment now, and I've run it locally as well to confirm that the problem is no longer occurring. All E4S tests pass.