Skip to content

[strata] Edge0-145B NVFP4 prefill hangs RTX PRO 6000 (sm_120) #200

Open

Description

@akbarharyadi

Hi, trying to get Edge0-145B-NVFP4 running on Linux and I can't get a single token out of it.

Setup:

RTX PRO 6000 Blackwell Server Edition 96GB (sm_120), GPU passthrough into a KVM VM, 32 vCPU, 80GB RAM
Ubuntu 26.04, driver 595.84, CUDA 13.2.1, gcc 15.2
commit b7900cb, built from source since there's no Linux package. strata built with -DCMAKE_CUDA_ARCHITECTURES=120, edge0-infer with the cpu preset
The model loads fine (90-135s, ~58GB VRAM) but every request gets stuck on "reading the prompt" and after 60s the engine dies with exit code -6 and restarts. Doesn't matter what the prompt is, even a 54 token "Say hi" does it.

[strata] reading the prompt: 125 of 130 tokens, 11 s so far
...
[strata] reading the prompt: 125 of 130 tokens, 61 s so far
[strata] the engine stopped unexpectedly (exit code -6). The usual cause is running out of RAM ...
[strata] done: 0 tokens in 68 s (no speed) (error, cancel=False)
It's not RAM, nothing in dmesg, no Xid, and the GPU never went above 72C. So I guess the -6 is just the watchdog and the RAM hint is a bit misleading.

Looks a lot like #199 so I applied that patch and rebuilt. Still hangs, same spot. Then tried STRATA_LATENT_FUSED_DU=0, also still hangs. One thing that tripped me up there: setting it as an env var does nothing because the model.json that ships with 145B-NVFP4 has "STRATA_LATENT_FUSED_DU": "1" in its env block and that wins. Had to change it in model.json.

I couldn't get the actual strata stderr, even with --verbose. Looks like it goes to a temp dir that gets deleted when it exits. If you tell me the best way to run it so the log stays around I can try again.

Couple of questions:

is sm_120 supposed to work with 145B NVFP4 on Linux right now?
should I be building with 120a or 120f instead of 120?
Also small thing, the full edge0-infer build fails on gcc 15 because a few tests are missing #include (slots_test.cpp, edge0_35b/loading_test.cpp, edge0_35b/assets_test.cpp). Building just the edge0-infer target works.

Thanks!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions