Repository navigation
Conversation
…ounded and aborted as its sum sumBlock leaves block `rank` of the two ranks' values summed in place (ncclReduceScatter), with sum's peer check, deadline, asynchronous-error wait and abort. flashnext-rank's eof-scatter and timeout-scatter modes make it the first collective a dead or silent peer fails; a mode it does not know is an error instead of a sum.
…slots by rows and gathers their sums A pass wider than any decode round or verify window (more than 64 rows) reduce-scatters y [n, 11, 2560] by row halves instead of all-reducing it, sums each rank's rows' slots with fn_moe_sum into moe_f32, then all-gathers the FP32 sums; the next writeback takes the pending_f32 form (mode 4, one slice). Every summed element is one IEEE add of the two ranks' values, as before, and fn_moe_sum is the writeback's slot chain, so the written rows keep their bytes while the exchange moves 61,440 B a row each way instead of 112,640 B. Rounds and windows keep the all-reduce. y and moe_f32 take one spare row for an odd pass's padded block. flashnext-rank's exchange mode compares the written rows, square sums and normalized rows with the all-reduce's at 65 to 8,192 rows, odd and even, with a changed-sum control, and checks that the exchange writes no row of y or moe_f32 past the spare one. flashnext-model-rank takes TF_FLASHNEXT_CHUNK's rows a pass, as its engine does, where it fixed 32 and so stopped with RankIdentity unless the variable was 32.
A folder with 1_LogitScore opens as a reranker, as on HIP: its yes and no rows of the output head read on the host at load, the score and packed-decision hooks of lane.zig on the CUDA engine, and Info's rerank labels with decides, so /v1/rerank serves it. A decision's trunk runs once a pass: its keys stay in the cache's first rows, and each branch attends to them and to its own rows from the trunk's end. The CUDA GPU check adds the reranker's gate (yes - no and the score against transformers, the packed decision equal to each prompt's run alone).
…st; the docs and file headers name the reranker Engine.run's argument checks move to checkBatch, so the trunk, strictly rising ends, the window, the vocabulary and the rows are pinned by a host test (as on HIP). docs/api.md says the native HIP and CUDA engines serve the reranker; two comments said embedding only.
… edges the cases' short common prefix never reaches A trunk of 700 tokens (past one 512-key chunk, off its multiple) with branches of 600, 513, 1 and 530 rows, cut from the longest case, packed in one decision; each branch's logits must equal its prompt's run alone, bit for bit.
…erves; the weights header names the head's stored name
After each prompt chunk the draft head runs over the chunk's rows to fill its own caches. It ran as a decode window, so its projections and its MoE took the stream tiles for up to a whole chunk of rows, re-reading each weight once per four rows. A prompt chunk now runs it on the prompt tiles (o.prefill and moe.run's prompt path), as the main layers' prompt rows already do; decode windows keep the window path. The head's prompt-row caches take the prompt tiles' sums, so its drafts can differ in the last bits; replies are verified as before and do not change.
…not from the options
…en gain of its receipt
…over Metal target.zig is the lanes' verify target: a slot's prompt, its windows of rows for several slots in one launch, keeps and rewinds, and kept prompt states (the prompt cache) whose resumed rows the reply reports as cached_tokens. boot.zig loads a release checkpoint folder and entry.zig serves it as tensorfold-dsv41 (dsv41-metal-serve). Tests serve the tiny model through LaneHost: concurrent equals solo equals serial, drafted equals draft:false, cache hits equal fresh runs, seeds and cancels.
…for the next A shared round with the plain switch at it asked no drafts and ran no head pass (core Rule.plain_streams, threaded from the family's resolved setting: 0 on CUDA, so its rounds are unchanged), and the family's speculation no longer launches a pass for a round that wide (armSpec/speculate plainRound).
…teps and the forward try's
…stays within 600 lines
…nything else by name
…treams streams decode
…er holds a rejected draft's taps
…nd the next ask catches the drafter up
Owner
Author
|
Landed rider by rider from this green tip after the build host restarted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI for a landing train: #2977 #2915 #2896 #2995 #1878 #1720 stacked on main in that order. Nothing merges from this PR: each PR above lands on its own once this tip is green, and this PR is then closed.