Repository navigation
tests, flashnext: fix main's three nightly failures - #3002
Merged
Merged
Conversation
#2585 (dfd9ac5) put `try @import("env_validate.zig").startup();` at the top of the learn, prompt and compare mains. Each of those executables also imports the flashnext module, whose cuda_lanes_load.zig imports the same file, so Zig refuses the build: "file exists in modules 'root' and 'flashnext'". The nightly's compile-flashnext-learn (test-host-ci-linux- compile) failed on it; flashnext-prompt and flashnext-compare fail the same way but no CI step compiles them. cuda_launch.zig (the flashnext module's root) now exports env_validate, and the three mains call flash.env_validate.startup(), so the file belongs to the one module. The check itself and its place before any device work are unchanged.
…t norm #2579 (4dd870b, a pipeline stage holds only its layer range) changed one pinned call in cuda_forward.zig: the input norm's add flag went from `i != 0` to `g.first + i != 0`, so a later stage's first layer adds the incoming residual as layer i of one engine does. One engine's geometry has first = 0, so its calls are the same; the commit states step, window and prompt logits equal the parent's bit for bit. The other eleven pinned calls are unchanged. The pin takes the new sequence's hash, with the change named beside it.
test-gradient-serving pins the bytes of the maintained serving kernels and launch wrappers; it stopped at the first changed file, cublaslt.zig, but fourteen differed: - #2585 (dfd9ac5, 52479d3) reads TF_FLASHNEXT_* through platform/environment.zig and refuses malformed text at startup; valid values keep their meaning (ten Flash Next files). - c274b45 on this branch takes that startup check through the flashnext module (cuda_launch.zig and the prompt and compare mains). - #2285 (52375ee) adds cuBLASLt layout and algo-config entries for Qwen3 embeddings to cublaslt.zig; existing entries are unchanged. - #2205 (c8ad987) lets tp_group.zig's RoCE gathers run at any world up to roce.max_world when every rank asks (TF_ROCE's default is unchanged), and #2789 (61a2651) ends a call failing after abort as Aborted. Each row takes its file's new hash, with the change named beside it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Main's nightly has failed three nights running on three separate things. None of them changes what the engine serves.
1.
compile-flashnext-learn: "file exists in modules 'root' and 'flashnext'"#2585 imported
env_validate.zigdirectly in the learn, prompt and compare mains. Those executables also import theflashnextmodule, which reaches the same file, so Zig refuses the build.cuda_launch.zig(the module's root) now exportsenv_validate, and the three mains callflash.env_validate.startup(). The check still runs first.flashnext-promptandflashnext-comparehad the same break; no CI step compiles them.2.
dense_serving_test: launch sequence hash#2579 (pipeline stages) changed one of the twelve pinned calls: the first input norm now keys on
g.first + i != 0. With one enginefirstis 0, so the calls are the same; the pin takes the new hash with the change named beside it. The hash changes at 4dd870b and at no other commit.3. Serving identities (
gradient-serving.tsv)The check stops at the first changed file, so
cublaslt.zighid thirteen more. Fourteen rows are re-pinned, each naming its change: #2585 (ten Flash Next files, settings parsed through the shared reader), #2285 (new cuBLASLt entries only), #2205 and #2789 (tp_group.zig: RoCE gathers at any world size when every rank asks, default unchanged; a call failing after abort ends as Aborted), and fix 1 above.Checked locally on this head, one step at a time
-Dtarget=x86_64-linux-gnu, the nightly's list): 29/29 steps.zig fmt --checkclean on the changed files; lean count unchanged (345 before and after).These two pin tests run only in the nightly, so a PR that touches a pinned file turns them red the next morning. A follow-up should run them in PR CI.