Repository navigation
Conversation
DeepSeek-V4.1's LaneHost set no .structures, so the server answered response_format, the guided_* fields and required or named tool calls with HTTP 400 (#847). The lane backend now passes each window's grammar bits to the target in Segment.masks, and before a prompt's pick it passes the first token's bits from the stream's own grammar (Target.first_mask). first_masked checks the prompt's pick against the bits the lane core asks for. With TF_DSV41_TAIL_JOIN, a joined prompt's first draft keeps only a count of held drafts, so the first round's plan has held drafts, and windows.constrain walks them with the grammar through Backend.held. Lanes.heldChain gives them: the held chain's drafts, the same tokens that `verify` puts in a window's held rows. Only constrain reads `held` here (dsv41 sets no context_copies). On the GPU target, grammar_gpu.zig fills the Forward.grammar hook. Before each forward's picks, rank 0 stages the masked rows and sends their row ids and bits to the other ranks on the plan link (op_grammar). Before a greedy or keyed pick reads "w.logits", every rank copies those rows to pinned memory, sets the logits of disallowed tokens in its vocabulary part to -inf and writes them back. Each run of adjacent masked rows is one copy each way. The pinned buffer (64 rows of the rank's part) is allocated at load, so a pick allocates nothing after rank 0 sends its op. A request without a grammar, and a row that allows every token, send and copy nothing. Tree windows give each chain row its tree row's bits; row windows stage once per forward. A nucleus row (top_p without top_k) whose grammar masks a rank's whole vocabulary part gave that rank the statistics (-inf, NaN), and keepCount sent the row to the whole-row gather. keepCount now leaves such a rank out of the sum: every logit it holds is -inf, so it adds 0. No finite sum changes, and a row without a grammar never has such a rank. With ignore_eos the server kept every end token in the reply's text, so a grammar's stop token came back at the end of the content: '{"key": "value"}<｜end▁of▁sentence｜>' on the GPUs. The lane core already ends the reply on it. The server's Generation now follows the request's grammar, from the token after `</think>` with thinking on. Under it an end token is the grammar's stop token, so it ends the reply as without ignore_eos: finish reason stop, and out of the text and the stream. The commit loop moves the grammar's phase token by token, so each end token is judged at its own phase. The stream still scans the whole landed round, and eos_phase.zig replays the phase from the round's start: an end token in the reasoning and the grammar's stop can land in one round, and judged at the round's last phase the first one would end the stream. End tokens before the grammar starts, and all end tokens without a grammar, stay in the text. draft/lanes_grammar_test.zig runs the lane engine over the CPU twin with a test grammar: greedy and keyed replies equal the twin's masked serial decode, drafted, with drafts off and four at once. A third test runs the prompts in pieces with a `tail` that leaves the last row to the first round, as the GPU target does, and joins them with one draft call or one each. A sampling_test.zig case masks one rank's whole part of two nucleus rows: the picks equal one rank's and the whole rows', and no row is gathered whole (2 before the keepCount change). grammar_gpu_test.zig covers the staged rows, the plan-link message, a rank's mask and the copy runs. hf_eos_http_test.zig sends a json_object chat request with ignore_eos on the DeepSeek-V4.1 test tokenizer, thinking off and on, whole and streamed, in rounds of every size from one token to the whole reply. Each round's reasoning and content stream with it: judged at the round's last phase, the whole-reply round streams its reasoning in 2 pieces, not 1.
The structured output section gains a line for dsv41-cuda beside GLM's CUDA backend: rank 0 sends each pick's grammar rows to the other ranks, and every rank masks its own vocabulary columns before the pick. The ignore_eos paragraph says that a grammar's stop token still ends the reply, out of the text.
The followers take the mask bits rank 0 sends, so op 28 has no bind, step or release sub-op any more. The note says so, and it no longer claims 29-31.
The new `grammar` phase covers one pick's masked rows: the copy out, the host mask and the copy back. With TF_DSV41_PHASES=sync it syncs at both ends, so with drafts off its ms_avg is the cost of one masked row.
…rallel walk The lanes grammar test now also runs its cases with first-position siblings (TF_DSV41_TREE 2-4) on the CPU twin, four at once and alone: every reply equals its masked serial decode, and the twin counts the tree windows it ran (more than 0 in each run). The test grammar gains an optional early stop and a grammar_budget hook that keeps only the stop token once a longer reply could not end within the draws left. With the budget armed and max_new below the full length, the drafted, "draft": false, tree, four-at-once and solo replies all end on the stop token within max_new and equal the serial decode under the same budget. The same grammar without the bound runs past max_new. The test grammar's matchers are independent, so it sets parallel_walk, and four constrained streams take the lane core's worker pool (#1455). The four-at-once case now checks that the pool ran.
… bits The lane core calls Backend.first_masked after the prompt pass to draw the first token among the grammar's bits. DeepSeek's lanes draw it in the pass itself: before `prefill`, `finish` or `join` they take the stream's first-token bits (Constraint.next), and the target draws the prompt's last row among them. lanes_first.zig now states this contract and first_masked checks it. The grammar takes no token between that call and Engine.drawFirst, so drawFirst passes the same bits. first_masked compares them word for word with the bits the pass drew under (error.FirstMaskChanged when they differ, and when the pass drew without bits), checks that the pick is among them (error.FirstNotMasked), and returns the pass's pick as `first` does. It runs no forward, writes no KV row and calls no drafter. Before, it only checked the pick against the bits it was given. The pass's bits are a copy in the stream's St (first_bits), freed with it. The pick stays in the pass, and is not drawn in the callback from a kept row as Gemma 4's CUDA backend does, because on DeepSeek it is the TP=2 forward's own pick: rank 0 sends the row's bits on the plan link and every rank masks its part of the vocabulary first. A later draw would need its own plan-link op and a logits row kept per slot on every rank. With TF_DSV41_TAIL_JOIN the first token comes from the first round's window, under the lane core's mask for its pending row, so first_masked is not called; when the target runs the tail row anyway (`tail` returns its pick), it is. lanes_first_test.zig drives the backend over the CPU twin with a target that runs prompts as the GPU target does: whole, in pieces, after a prefix that `begin` puts in place as a cache hit does, and joined with the tail row run. On each path, under a grammar that refuses the twin's unmasked first token, greedy and keyed: the pick equals the masked serial decode's first token; the KV rows, taps and drafter rings equal the unmasked pass's, so the bits changed only the pick; first_masked leaves them and the lanes' state as they were and adds one handle; other bits are refused with error.FirstMaskChanged, and the state stays the same. That includes bits one token away from the pass's that still allow its pick: with the pick-only check the test fails, because it takes them. A target that draws the pick without the bits is refused with error.FirstNotMasked. Through the lane engine, three replies at once whose prompts came whole, in pieces, cached, joined or tail-run equal their masked serial decode.
TF_DSV41_TREE resolves each tree window as chain windows (branches.zig), and its counts reached only the gates. Rank 0 now logs them as it logs piece and replay runs, at the first tree window, at the first one with a row its grammar narrows and at every 256th: "tree windows: N (M under a grammar), C chains, R rows run again". M counts a window only when a row's bits refuse a token, so a grammar that allows everything at that point does not count. A structured-output run with trees on can then show that tree windows ran under its grammar.
This branch made the reply's content tokens keep a trailing end token only when no grammar was on, and left server/logprobs.zig on the older rule. On main the two agreed, so the branch opened the gap: with logprobs, ignore_eos and a grammar, a streamed reply's last logprob row would carry the stop token that the content dropped. No request reaches it today. openai.zig refuses logprobs together with structured output, tools, stop strings or a thinking budget, and that is the only place a request sets logprobs. The rule still belongs in one place. eos_phase.Phase gains keepsTrailing, the negation of the condition ends applies. The content tokens and both logprob delta calls now read it from one place, beside the rule that decides whether the token ends the reply at all.
eos_phase.zig: keepsTrailing's comment on one line. forward_follow.zig: drop the history note on op_grammar.
Both are host tests with no GPU: keyed sampling, DSpark and the shared lane core, and the row and device plans.
An engine with token grammar support holds a strict tool's arguments to its schema under auto, and the reply stays free text or calls. DeepSeek-V4.1 on CUDA now has that support.
test-dsv41-device-host carries main's aot-needs test, which takes about ten minutes on a runner and hits the step's limit. It already runs nightly with a 30-minute limit, so the pull request step keeps only the draft tests. Co-authored-by: nuk3s <44327449+nuk3s@users.noreply.github.com>
6 of 9 tasks
Owner
Author
|
Device check status: blocked for now. On main, the CUDA server refuses DeepSeek-V4.1 behind its whole-model gate (#827). Its prompt path still needs captured Triton kernels, so a served main-vs-branch run can't start. The EXL3 2.9 bpw checkpoint is 197 GB, so it needs two GB10s at TP=2; one box can't hold it. This PR waits for #827, or for a two-GB10 run once both boxes have room for the checkpoint. Host tests and CI stand as they are.
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is @nuk3s's #1665, opened from our own branch because GitHub refused our push to their fork even with maintainer edits on. It lets DeepSeek-V4.1 on CUDA serve structured output through the lane core's grammar masks, with drafts on:
response_format, theguided_*fields,tool_choice: "required"or a named function, and strict tools undertool_choice: "auto". Closes #1664. The work is all theirs. Their 11 commits come across unchanged, with their authorship and the same SHAs (headbb3df5353), and the design, the runs on two DGX Sparks and the full receipt are in #1665. We added one commit on top,ci: run only test-dsv41-draft on pull requests, with @nuk3s as co-author:test-dsv41-device-hostnow carries main's aot-needs test, which outruns the pull request step's time limit, so it stays in the nightly DeepSeek suite instead. The branch sits on main79d3dc661and merges cleanly with today's main. Before it lands we still want the main, branch, branch, main run on two GB10s described in #1665. Thank you @nuk3s.