Skip to content

java under Spindle not searching RPATH for libjli #230

New issue
New issue
Closed
#238
Closed
java under Spindle not searching RPATH for libjli#230
#238

Description

@nchaimov
nchaimov
opened on Oct 7, 2026

The stc test failure in the E4S testsuite is caused by java failing to find a library under Spindle that it can find without Spindle. The stc test depends on an older version of Java (Java 11), for which the problem occurs, but it does not occur with Java 25.

The failure can be reproduced in the serial CI container:

spindleuser@spindlenode:~$ sudo apt install openjdk-11-jre
[...]
spindleuser@spindlenode:~$ sudo apt install openjdk-25-jre
[...]

Java 11 works without Spindle but can't find libjli.so under Spindle:

spindleuser@spindlenode:~$ /usr/lib/jvm/java-11-openjdk-amd64/bin/java -version
openjdk version "11.0.32.1" 2026-08-18
OpenJDK Runtime Environment (build 11.0.32.1+1-post-1ubuntu1-24.04-Ubuntu)
OpenJDK 64-Bit Server VM (build 11.0.32.1+1-post-1ubuntu1-24.04-Ubuntu, mixed mode, sharing)

spindleuser@spindlenode:~$ spindle /usr/lib/jvm/java-11-openjdk-amd64/bin/java -version
java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

Java 25 works in either case:

spindleuser@spindlenode:~$ /usr/lib/jvm/java-25-openjdk-amd64/bin/java -version
openjdk version "25.0.4.1" 2026-08-18
OpenJDK Runtime Environment (build 25.0.4.1+1-1-24.04.4-Ubuntu)
OpenJDK 64-Bit Server VM (build 25.0.4.1+1-1-24.04.4-Ubuntu, mixed mode, sharing)
spindleuser@spindlenode:~$ spindle /usr/lib/jvm/java-25-openjdk-amd64/bin/java -version
openjdk version "25.0.4.1" 2026-08-18
OpenJDK Runtime Environment (build 25.0.4.1+1-1-24.04.4-Ubuntu)
OpenJDK 64-Bit Server VM (build 25.0.4.1+1-1-24.04.4-Ubuntu, mixed mode, sharing)

Java 11 dynamic section:

spindleuser@spindlenode:~$ readelf -d /usr/lib/jvm/java-11-openjdk-amd64/bin/java

Dynamic section at offset 0x2d40 contains 31 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libjli.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../lib/jli:$ORIGIN/../lib]
[...]

Java 25 dynamic section:

spindleuser@spindlenode:~$ readelf -d /usr/lib/jvm/java-25-openjdk-amd64/bin/java

Dynamic section at offset 0x2d60 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libjli.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN:$ORIGIN/../lib]
[...]

Without Spindle, libjli is found in the first location searched, $ORIGIN/../lib/jli:

  2       2493:     file=libz.so.1 [0];  needed by /usr/lib/jvm/java-11-openjdk-amd64/bin/java [0]
  3       2493:     find library=libz.so.1 [0]; searching
  4       2493:      search path=/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/glibc-hwcaps/x86-64-v3:/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/glibc-hwcaps/x86-64-v2:/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli:/usr/lib/jvm/java-11-openj    dk-amd64/bin/../lib/glibc-hwcaps/x86-64-v3:/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/glibc-hwcaps/x86-64-v2:/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib                (RPATH from file /usr/lib/jvm/java-11-openjdk-amd64/bin/java)
  5       2493:       trying file=/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/glibc-hwcaps/x86-64-v3/libz.so.1
  6       2493:       trying file=/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/glibc-hwcaps/x86-64-v2/libz.so.1
  7       2493:       trying file=/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/libz.so.1
  8       2493:       trying file=/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/glibc-hwcaps/x86-64-v3/libz.so.1
  9       2493:       trying file=/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/glibc-hwcaps/x86-64-v2/libz.so.1
 10       2493:       trying file=/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/libz.so.1
 11       2493:      search cache=/etc/ld.so.cache
 12       2493:       trying file=/lib/x86_64-linux-gnu/libz.so.1
 13       2493:
 14       2493:     file=libz.so.1 [0];  generating link map
 15       2493:       dynamic: 0x0000719976091dd0  base: 0x0000719976077000   size: 0x000000000001b098
 16       2493:         entry: 0x0000719976077000  phdr: 0x0000719976077040  phnum:                 11
 17       2493:
 18       2493:
 19       2493:     file=libjli.so [0];  needed by /usr/lib/jvm/java-11-openjdk-amd64/bin/java [0]
 20       2493:     find library=libjli.so [0]; searching
 21       2493:      search path=/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli:/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib            (RPATH from file /usr/lib/jvm/java-11-openjdk-amd64/bin/java)
 22       2493:       trying file=/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/libjli.so
 23       2493:
 24       2493:     file=libjli.so [0];  generating link map
 25       2493:       dynamic: 0x0000719976075b28  base: 0x0000719976065000   size: 0x0000000000011120
 26       2493:         entry: 0x0000719976065000  phdr: 0x0000719976065040  phnum:                 11

With Spindle, the executable's RPATH is searched when looking for the first library, libz, but is NOT searched when looking for libjli:

   	   2487:     file=libz.so.1 [0];  needed by /usr/lib/jvm/java-11-openjdk-amd64/bin/java [0]
      2487:     find library=libz.so.1 [0]; searching
      2487:      search path=/tmp/commpath/cachepath/spindle.ebd36db3852edbd0/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/glibc-hwcaps/x86-64-v3:/tmp/commpath/cachepath/spindle.ebd36db3852edbd0/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/glibc-hwcaps/x86-64-v2:/tmp/commpath/cachepath/spindle.ebd36db3852edbd0/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli:/tmp/commpath/cachepath/spindle.ebd36db3852edbd0/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/glibc-hwcaps/x86-64-v3:/tmp/commpath/cachepath/spindle.ebd36db3852edbd0/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/glibc-hwcaps/x86-64-v2:/tmp/commpath/cachepath/spindle.ebd36db3852edbd0/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib                (RPATH from file /usr/lib/jvm/java-11-openjdk-amd64/bin/java)
      2487:       trying file=/tmp/commpath/cachepath/spindle.ebd36db3852edbd0/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/glibc-hwcaps/x86-64-v3/libz.so.1
      2487:       trying file=/tmp/commpath/cachepath/spindle.ebd36db3852edbd0/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/glibc-hwcaps/x86-64-v2/libz.so.1
      2487:       trying file=/tmp/commpath/cachepath/spindle.ebd36db3852edbd0/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/jli/libz.so.1
      2487:       trying file=/tmp/commpath/cachepath/spindle.ebd36db3852edbd0/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/glibc-hwcaps/x86-64-v3/libz.so.1
      2487:       trying file=/tmp/commpath/cachepath/spindle.ebd36db3852edbd0/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/glibc-hwcaps/x86-64-v2/libz.so.1
      2487:       trying file=/tmp/commpath/cachepath/spindle.ebd36db3852edbd0/usr/lib/jvm/java-11-openjdk-amd64/bin/../lib/libz.so.1
      2487:      search cache=/etc/ld.so.cache
      2487:       trying file=/lib/x86_64-linux-gnu/libz.so.1
      2487:
      2487:     file=libz.so.1 [0];  generating link map
      2487:       dynamic: 0x00007972581eadd0  base: 0x00007972581d0000   size: 0x000000000001b098
      2487:         entry: 0x00007972581d0000  phdr: 0x00007972581d0040  phnum:                 11
      2487:
      2487:
      2487:     file=libjli.so [0];  needed by /usr/lib/jvm/java-11-openjdk-amd64/bin/java [0]
      2487:     find library=libjli.so [0]; searching
      2487:      search cache=/etc/ld.so.cache
      2487:      search path=/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3:/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2:/usr/lib/x86_64-linux-gnu:/lib/glibc-hwcaps/x86-64-v3:/lib/glibc-hwcaps/x86-64-v2:/lib:/usr/lib/glibc-hwcaps/x86-64-v3:/usr/lib/glibc-hwcaps/x86-64-v2:/usr/lib            (system search path)
      2487:       trying file=/lib/x86_64-linux-gnu/libjli.so
      2487:       trying file=/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libjli.so
      2487:       trying file=/__not_exists/4-linux-gnu/glibc-hwcaps/x86-64-v2/libjli.so
      2487:       trying file=/__not_exists/4-linux-gnu/libjli.so
      2487:       trying file=/lib/glibc-hwcaps/x86-64-v3/libjli.so
      2487:       trying file=/__noglibc-hwcaps/x86-64-v2/libjli.so
      2487:       trying file=/__nolibjli.so
      2487:       trying file=/usr/lib/glibc-hwcaps/x86-64-v3/libjli.so
      2487:       trying file=/__not_exglibc-hwcaps/x86-64-v2/libjli.so
      2487:       trying file=/__not_exlibjli.so
      2487:
/usr/lib/jvm/java-11-openjdk-amd64/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

The reason that Java 25 works appears to be that libjli is the first library loaded there. RPATH is being searched for the first library with either Java version.

Activity

  1. nchaimov
    added a parent issue on Oct 7, 2026

  2. nchaimov commented on Oct 7, 2026

    @nchaimov
    nchaimov
    on Oct 7, 2026
    CollaboratorAuthor

    Simple reproducer:

    spindleuser@spindlenode:~$ cat foo.c
    #ifdef LIB
int foo(void)
{
   return 0;
}
#else
#include <stdio.h>
int foo(void);
int main(void)
{
   printf("Hello world!\n");
   return foo();
}
#endif
    spindleuser@spindlenode:~$ mkdir -p lib
spindleuser@spindlenode:~$ gcc -shared -fPIC -DLIB -o lib/libfoo.so foo.c
spindleuser@spindlenode:~$ gcc -o test-libm-before-libfoo foo.c -Wl,--no-as-needed -lm -Llib -lfoo -Wl,-rpath,'$ORIGIN/lib'
spindleuser@spindlenode:~$ gcc -o test-libfoo-before-libm foo.c -Wl,--no-as-needed -Llib -lfoo -lm -Wl,-rpath,'$ORIGIN/lib'

spindleuser@spindlenode:~$ ./test-libfoo-before-libm
Hello world!
spindleuser@spindlenode:~$ spindle ./test-libfoo-before-libm
Hello world!

spindleuser@spindlenode:~$ ./test-libm-before-libfoo
Hello world!
spindleuser@spindlenode:~$ spindle ./test-libm-before-libfoo
./test-libm-before-libfoo: error while loading shared libraries: libfoo.so: cannot open shared object file: No such file or directory

  3. nchaimov commented on Oct 7, 2026

    @nchaimov
    nchaimov
    on Oct 7, 2026
    CollaboratorAuthor

    This problem occurs in the CI image, but not on Tuolumne, possibly because the ld.so binary patch is enabled on Tuolumne but not in the CI images — this was the case for an earlier bug affecting library search paths in the CI images.

  4. mplegendre commented on Oct 13, 2026

    @mplegendre
    mplegendre
    on Oct 13, 2026
    Member

    Alright, I think I understand what's happening. It's related to ld-linux.so keeping a cache of what directories exist and which directories don't, which is being triggered by spindle dynamically creating directories in its cache when needed. Here's the order of events around the bug for both test cases:

    When libm is loaded before libfoo:

    • $ORIGIN is set to a location in the spindle cache with the executable
    • The linker searches for libm.so at $ORIGIN/lib/libm.so.
    • Spindle translates that cachepath to a real location, sees that file doesn't exist, and returns not-exists.
    • The linker's not-found code path checks $ORIGIN/lib and sees it doesn't exist, and removes it from the search order.
    • The linker later finds libm.so normally in /usr/lib.
    • The linker doesn't check for libfoo.so in $ORIGIN/lib because it believes that directory doesn't exist.

    When libfoo is loaded before libm:

    • $ORIGIN is set to a location in the spindle cache with the executable
    • The linker searches for libfoo.so in $ORIGIN/lib/libfoo.so.
    • Spindle translates that cachepath to a real location, sees the files exists, and mkdirs $ORIGIN/lib and creates libfoo.so in the cache.
    • The linker loads $ORIGIN/lib/libfoo.so.
    • The linker searches libm.so in $ORIGIN/lib/libm.so.
    • Spindle translates that cachepath to a real location, sees that file doesn't exist, and returns not-exists.
    • The linker's not-found code path checks $ORIGIN/lib, but this time sees it exists because of the previous libfoo.so creation.
    • The linker later finds libm.so normally in /usr/lib.

    To fix this spindle should mkdir on cachepath directories that exist in the shared file system, even if the file being searched for in that directory does not exist.

    I'll work on that.

  5. mplegendre commented on Oct 30, 2026

    @mplegendre
    mplegendre
    on Oct 30, 2026
    Member

    Fixed by #238

  6. mplegendre
    closed this as completedon Oct 30, 2026
  7. nchaimov
    linked a pull request that will close this issueFix issues with apps not finding libraries under ORIGIN/subdir structure #238on Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions