Description
Hi, trying to get Edge0-145B-NVFP4 running on Linux and I can't get a single token out of it.
Setup:
RTX PRO 6000 Blackwell Server Edition 96GB (sm_120), GPU passthrough into a KVM VM, 32 vCPU, 80GB RAM
Ubuntu 26.04, driver 595.84, CUDA 13.2.1, gcc 15.2
commit b7900cb, built from source since there's no Linux package. strata built with -DCMAKE_CUDA_ARCHITECTURES=120, edge0-infer with the cpu preset
The model loads fine (90-135s, ~58GB VRAM) but every request gets stuck on "reading the prompt" and after 60s the engine dies with exit code -6 and restarts. Doesn't matter what the prompt is, even a 54 token "Say hi" does it.
[strata] reading the prompt: 125 of 130 tokens, 11 s so far
...
[strata] reading the prompt: 125 of 130 tokens, 61 s so far
[strata] the engine stopped unexpectedly (exit code -6). The usual cause is running out of RAM ...
[strata] done: 0 tokens in 68 s (no speed) (error, cancel=False)
It's not RAM, nothing in dmesg, no Xid, and the GPU never went above 72C. So I guess the -6 is just the watchdog and the RAM hint is a bit misleading.
Looks a lot like #199 so I applied that patch and rebuilt. Still hangs, same spot. Then tried STRATA_LATENT_FUSED_DU=0, also still hangs. One thing that tripped me up there: setting it as an env var does nothing because the model.json that ships with 145B-NVFP4 has "STRATA_LATENT_FUSED_DU": "1" in its env block and that wins. Had to change it in model.json.
I couldn't get the actual strata stderr, even with --verbose. Looks like it goes to a temp dir that gets deleted when it exits. If you tell me the best way to run it so the log stays around I can try again.
Couple of questions:
is sm_120 supposed to work with 145B NVFP4 on Linux right now?
should I be building with 120a or 120f instead of 120?
Also small thing, the full edge0-infer build fails on gcc 15 because a few tests are missing #include (slots_test.cpp, edge0_35b/loading_test.cpp, edge0_35b/assets_test.cpp). Building just the edge0-infer target works.
Thanks!
Activity