Use sandbox mode which is more robust in singularity cleanup phase

[julianmorillo]

No description provided.

[boegel]

@julianmorillo Is there any documentation that would explain why this works better?

No strong objections as long as it works, I just don't see why this would work better (though it seems to based on EESSI/dev.eessi.io-riscv#35 (comment))

[trz42]

Could we use this optionally? It's been tested on RISC-V, but we haven't seen this on other systems. So, to not cause problems on the other systems, maybe the safest way forward would be to add an option and use that where needed.

[julianmorillo]

@boegel
SingularityCE User Guide (runtime behavior changes)
SingularityCE 4.1 Admin/User notes (FUSE + sandbox fallback)

The critical mechanism here (from the documentation above) is that SIF images require mounting (via FUSE). In the documentation above it can be read: "SIF/SquashFS container images will now be mounted with squashfuse… when kernel mounts are not available."
In an unprivileged HPC environment this becomes:

• FUSE mount
• Which requires cleanup on exit --> where errors in {2025.06-001}[2025b] LLVM 20.1.8 dev.eessi.io-riscv#35 (comment) happen.

The documentation above also explains that if mounting fails or is unsuitable --> Singularity uses a sandbox: “If the FUSE mount fails, Singularity will fall back to extracting the container to a temporary sandbox.” and “fallback to extraction to a temporary sandbox directory”.
The key insight is that:

• Sandbox = extracted directory
• No mount needed
• No FUSE
• No teardown/cleanup complexity

[bedroge]

Could we use this optionally? It's been tested on RISC-V, but we haven't seen this on other systems. So, to not cause problems on the other systems, maybe the safest way forward would be to add an option and use that where needed.

This makes sense to me. The sandbox mode is probably (much) slower, especially when multiple jobs are extracting that image to a shared filesystem at the same time.

[julianmorillo]

Could we use this optionally? It's been tested on RISC-V, but we haven't seen this on other systems. So, to not cause problems on the other systems, maybe the safest way forward would be to add an option and use that where needed.

This makes sense to me. The sandbox mode is probably (much) slower, especially when multiple jobs are extracting that image to a shared filesystem at the same time.

I already made it optional through an environment variable that should be defined in site_config.sh. Would that work?