Repository navigation
Conversation
… the lookup The embedding kernels read the table at the id they are given and none of them bounds it, so the host check is the only one. Each path now refuses an id at or past the vocabulary before it uploads the ids, and the loaders that took the table as stored refuse one whose rows are not the vocabulary's.
… the lookup The embedding kernels read the table at the id they are given and none of them bounds it, so the host check is the only one. Each path now refuses an id at or past the vocabulary before it uploads the ids, and the loaders that took the table as stored refuse one whose rows are not the vocabulary's.
…limit grammar_budget explored a grammar's parser states on the request thread with no bound but the state cap, and each state kept a buffer sized for the whole vocabulary. Exploring now stops at a fixed count of steps and masks only the states it reached, a build still running after a minute is refused, and a state keeps only its own successors. A long bounded string reaches the step limit early: see docs/api.md.
…'s masks below it - The time limit is gone: no clock is read, so the same grammar gets the same verdict on every machine and under any load, and the cache keeps only that. - A grammar whose exploring passes the work limit is refused with HTTP 400 that names the limit. Its partial tables are dropped and a matcher armed for it keeps xgrammar's bits, so no mask comes from part of a grammar. - Below the limit exploring is main's: the count only adds up, and each parser step and token walk is the one main makes, so the masks are main's. - The limit goes from 2^31 to 2^38 steps, about 256 times what json_object takes over a 250,000-token vocabulary. - tfg_budget_ready answers 2 past the limit; tfg_budget_limits takes the work alone.
…ess holds a quarter at most 128 connections that sent nothing, or one /health each, held every slot for up to 15 minutes. From three quarters full, a connection with no request in progress closes after 5 seconds, and one remote address holds at most a quarter of the slots. Local clients share an address and are not counted by it.
…or it and trusted proxies One address's share of the connection slots (a quarter by default) now applies only once 96 of 128 are open; below that its connections are counted but never refused. --max-connections N (1-1024, default 128), --max-connections-per-address N (0 turns the share off) and --trusted-proxy ADDRESS (repeatable; exact addresses never counted, as loopback is not) set the rule. A connection with no request in progress gives its slot up only with every slot taken, so a pooled client below the limit keeps its connection. IPv6 peers are counted by their /64.
…the caller allows unverified bytes The manifest's hash is of a whole file, so a copy held in part can never be checked against it. Such a copy was mapped anyway and its bytes only counted, and a later run that pulled nothing new mapped the same bytes and counted none. A load now refuses, before it asks a peer for a byte, any file with a manifest hash that would not end up whole. A caller that wants a sparse copy sets `unverified`, and every byte mapped from such a copy is counted, whenever it was pulled. No command passes a peer to the loader yet, so nothing that ships changes.
…eep or too large is refused
A tokenizer.json's Split pattern runs on a backtracking matcher with no limit. `(?:a+)+b` never finished
on a run of 64 letters, 100,000 nested groups ran the parser off its stack, and
`(?:(?:a{1000}){1000}){1000}` compiled to 29 GB. A search now runs at most 2^20 instructions plus 2,048
for each byte of its text and then ends with RegexBudget, about 60 times what the shipped patterns need
on their worst runs. A pattern is refused past 65,536 instructions, 64 nested groups or 8 quantifiers on
one element. The tests move to regex_test.zig to keep regex.zig within 600 lines.
…p and bound only the collective Both ranks initialize under the 30 s bound and acknowledge each other before the peer fails; timeout then gives rank 0's collective 200 ms. NCCL's initialization can take longer than 200 ms, which failed the timeout mode before its collective ran, and across two hosts a peer that finished initializing first could leave before the other had, which failed eof inside the initialization instead of the collective.
…ounded and aborted as its sum sumBlock leaves block `rank` of the two ranks' values summed in place (ncclReduceScatter), with sum's peer check, deadline, asynchronous-error wait and abort. flashnext-rank's eof-scatter and timeout-scatter modes make it the first collective a dead or silent peer fails; a mode it does not know is an error instead of a sum.
…slots by rows and gathers their sums A pass wider than any decode round or verify window (more than 64 rows) reduce-scatters y [n, 11, 2560] by row halves instead of all-reducing it, sums each rank's rows' slots with fn_moe_sum into moe_f32, then all-gathers the FP32 sums; the next writeback takes the pending_f32 form (mode 4, one slice). Every summed element is one IEEE add of the two ranks' values, as before, and fn_moe_sum is the writeback's slot chain, so the written rows keep their bytes while the exchange moves 61,440 B a row each way instead of 112,640 B. Rounds and windows keep the all-reduce. y and moe_f32 take one spare row for an odd pass's padded block. flashnext-rank's exchange mode compares the written rows, square sums and normalized rows with the all-reduce's at 65 to 8,192 rows, odd and even, with a changed-sum control, and checks that the exchange writes no row of y or moe_f32 past the spare one. flashnext-model-rank takes TF_FLASHNEXT_CHUNK's rows a pass, as its engine does, where it fixed 32 and so stopped with RankIdentity unless the variable was 32.
A folder with 1_LogitScore opens as a reranker, as on HIP: its yes and no rows of the output head read on the host at load, the score and packed-decision hooks of lane.zig on the CUDA engine, and Info's rerank labels with decides, so /v1/rerank serves it. A decision's trunk runs once a pass: its keys stay in the cache's first rows, and each branch attends to them and to its own rows from the trunk's end. The CUDA GPU check adds the reranker's gate (yes - no and the score against transformers, the packed decision equal to each prompt's run alone).
…st; the docs and file headers name the reranker Engine.run's argument checks move to checkBatch, so the trunk, strictly rising ends, the window, the vocabulary and the rows are pinned by a host test (as on HIP). docs/api.md says the native HIP and CUDA engines serve the reranker; two comments said embedding only.
… edges the cases' short common prefix never reaches A trunk of 700 tokens (past one 512-key chunk, off its multiple) with branches of 600, 513, 1 and 530 rows, cut from the longest case, packed in one decision; each branch's logits must equal its prompt's run alone, bit for bit.
…erves; the weights header names the head's stored name
After each prompt chunk the draft head runs over the chunk's rows to fill its own caches. It ran as a decode window, so its projections and its MoE took the stream tiles for up to a whole chunk of rows, re-reading each weight once per four rows. A prompt chunk now runs it on the prompt tiles (o.prefill and moe.run's prompt path), as the main layers' prompt rows already do; decode windows keep the window path. The head's prompt-row caches take the prompt tiles' sums, so its drafts can differ in the last bits; replies are verified as before and do not change.
…not from the options
…en gain of its receipt
…over Metal target.zig is the lanes' verify target: a slot's prompt, its windows of rows for several slots in one launch, keeps and rewinds, and kept prompt states (the prompt cache) whose resumed rows the reply reports as cached_tokens. boot.zig loads a release checkpoint folder and entry.zig serves it as tensorfold-dsv41 (dsv41-metal-serve). Tests serve the tiny model through LaneHost: concurrent equals solo equals serial, drafted equals draft:false, cache hits equal fresh runs, seeds and cancels.
…for the next A shared round with the plain switch at it asked no drafts and ran no head pass (core Rule.plain_streams, threaded from the family's resolved setting: 0 on CUDA, so its rounds are unchanged), and the family's speculation no longer launches a pass for a round that wide (armSpec/speculate plainRound).
…teps and the forward try's
…stays within 600 lines
…nything else by name
…treams streams decode
…er holds a rejected draft's taps
…nd the next ask catches the drafter up
Owner
Author
|
Stopped at #2977. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI for a landing train: #2976 #2977 #2915 #2896 #2995 #1878 #1720 stacked on main in that order. Nothing merges from this PR: each PR above lands on its own once this tip is green, and this PR is then closed.