Tests crash on Linux

[renshyle]

Crashes like this are observed when running the test suite:

Restore complete (0.6s)
  Wasmtime netstandard2.0 succeeded (2.0s) → src/bin/Debug/netstandard2.0/Wasmtime.Dotnet.dll
  Wasmtime netstandard2.1 succeeded (2.2s) → src/bin/Debug/netstandard2.1/Wasmtime.Dotnet.dll
  Wasmtime net8.0 succeeded (2.3s) → src/bin/Debug/net8.0/Wasmtime.Dotnet.dll
  Wasmtime net9.0 succeeded (2.4s) → src/bin/Debug/net9.0/Wasmtime.Dotnet.dll
  Wasmtime.Tests net9.0 succeeded (0.1s) → tests/bin/Debug/net9.0/Wasmtime.Tests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.8.1+ce9211e970 (64-bit .NET 9.0.14)
[xUnit.net 00:00:00.06]   Discovering: Wasmtime.Tests
[xUnit.net 00:00:00.12]   Discovered:  Wasmtime.Tests
[xUnit.net 00:00:00.12]   Starting:    Wasmtime.Tests
thread '<unnamed>' (414349) panicked at crates/c-api/src/val.rs:325:22:
unknown wasmtime_valkind_t: 160
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[xUnit.net 00:00:00.17]     Wasmtime.Tests.Memory64AccessTests.ItThrowsForOutOfBoundsAccess [SKIP]
[xUnit.net 00:00:00.17]       Test consumes too much memory for CI
The active test run was aborted. Reason: Test host process crashed : thread '<unnamed>' (414349) panicked at crates/c-api/src/val.rs:325:22:
unknown wasmtime_valkind_t: 160
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

  Wasmtime.Tests test net9.0 failed with 1 error(s) (1.3s)
    /home/renshyle/tmp/wasmtime-dotnet/tests/bin/Debug/net9.0/Wasmtime.Tests.dll : error TESTRUNABORT: Test Run Aborted.

Build failed with 1 error(s) in 8.2s

But also other kinds of errors on other attemps, such as ones where a test crashes without any message:

Restore complete (0.6s)
  Wasmtime netstandard2.0 succeeded (2.0s) → src/bin/Debug/netstandard2.0/Wasmtime.Dotnet.dll
  Wasmtime net8.0 succeeded (2.2s) → src/bin/Debug/net8.0/Wasmtime.Dotnet.dll
  Wasmtime net9.0 succeeded (2.4s) → src/bin/Debug/net9.0/Wasmtime.Dotnet.dll
  Wasmtime netstandard2.1 succeeded (2.3s) → src/bin/Debug/netstandard2.1/Wasmtime.Dotnet.dll
  Wasmtime.Tests net9.0 succeeded (0.0s) → tests/bin/Debug/net9.0/Wasmtime.Tests.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.8.1+ce9211e970 (64-bit .NET 9.0.14)
[xUnit.net 00:00:00.06]   Discovering: Wasmtime.Tests
[xUnit.net 00:00:00.12]   Discovered:  Wasmtime.Tests
[xUnit.net 00:00:00.12]   Starting:    Wasmtime.Tests
[xUnit.net 00:00:00.17]     Wasmtime.Tests.Memory64AccessTests.ItThrowsForOutOfBoundsAccess [SKIP]
[xUnit.net 00:00:00.17]       Test consumes too much memory for CI
The active test run was aborted. Reason: Test host process crashed
  Wasmtime.Tests test net9.0 failed with 1 error(s) (1.3s)
    /home/renshyle/tmp/wasmtime-dotnet/tests/bin/Debug/net9.0/Wasmtime.Tests.dll : error TESTRUNABORT: Test Run Aborted.

Build failed with 1 error(s) in 8.2s

Reproduction steps

• Be on Linux (Windows might work too for all I know)
• git clone https://github.com/bytecodealliance/wasmtime-dotnet
• cd wasmtime-dotnet
• dotnet test Wasmtime.sln

I reproduced this on a fresh Debian 13 install in a VM.

[renshyle]

Looks like Wasmtime.Tests.MemoryAccessTests.ItGrows is failing on new Memory(Store, 1, 4), which fails in wasmtime_memorytype_new. I checked the parameters in the CPU registers as wasmtime_memorytype_new got called and they looked just fine, so doesn't look like an ABI issue. Also tried calling the same function with the same parameters from C code and it didn't segfault. That test is also not the only failing test.

[renshyle]

Ah, I see that the latest development release of Wasmtime is downloaded by default. With DevBuild disabled, v34.0.2 is downloaded and all tests pass :)
Very confusing though

[jsturtevant]

I generally agree this is confusing I had similar experience when I was first asked to take a look at getting a release working #315. It requires some changes to the way the project does its releases #315 (comment)

[renshyle]

I've gone ahead and closed my PR because upon reading that discussion, the issue is much wider than I thought and actually disabling DevBuild by default sort of reverses the problem - the problem would then occur when wasmtime-dotnet has updated to be compatible with the dev branch of wasmtime and thus is incompatible with the release.

Pinning wasmtime to a specific commit (or tag in case of releases) is what I'd personally lean towards, but because artifacts aren't stored forever on GitHub, that would require building wasmtime locally and in CI. There might also be some maintenance overhead from having to bump wasmtime revision, but perhaps it would only need to be done when wasmtime's dev branch gets a breaking change to c-api.