Repository navigation
Conversation
…client write can't hang images_generations_test's serve() returned without closing route_fd when the route finished before reading the whole request body. The client writes the full body before reading, so once the socket buffer filled its write blocked for ever. Linux's Unix socket buffer holds the bodies these tests send, so CI never saw it; on macOS test-cuda-host hung there for 20 minutes holding a build lock (twice on one tree). - serve() closes route_fd on every exit (defer), as the server does with a finished connection; deinit no longer closes it. - The client end gets a send timeout (10 s, far past the route's 2 s read timeout) and no SIGPIPE (SO_NOSIGPIPE where it exists, MSG_NOSIGNAL on send), so a closed route is an error, never a hang or a signal. - open() and ask() release the server and join the route on every error path. - New test: a request line the route refuses from the head alone, with a 4 MiB body, ends the client's write at once (well under the send timeout). test-cuda-host on macOS: 385/387 steps, 4414/4451 tests, the images tests and the new one passing; the one failure is benchmark_cli_test's attached run, which this change does not touch.
test-gradient-serving (the serving kernels' and launch wrappers' source hashes) and test-qwen-serving-host (the dense serving launch sequence) ran only in the macOS nightly. A PR that changed a pinned file therefore landed green and turned the next nightly red: 10, 11 and 12 Nov, and again now, because #2843 (3768bc0) changed zig/src/cuda/experts.zig. - no-gpu.yml runs both steps on every PR, after the goldens, so a change to a pinned file needs its re-pin in the same PR. - gradient-serving.tsv re-pins experts.zig with #2843 named beside it: the grouped EXL3 experts are launched and sized at the caller's tile and pair count; no kernel source changed. test-gradient-serving: 1/1 pass.
…zig, re-exported by cuda_ple_pool.zig
…s pages asked for first
…ges, and only EINVAL falls back to WILLNEED for good
…stays on the caller, tested by the pool's call count
The test stopped the run before it started and then expected the fake server to see no request at all. The manifest and thinking fetches race the first stop poll, so on a busy runner one or two requests landed and the test failed. It now checks what cancellation guarantees: exit 130, no POST to /submissions, no upload preview, and one sample saved with error_code "cancelled". test-benchmark-cancel-host runs it alone. Fixes #2853.
The HIP drafter implements core/lanes Drafter for chains: absorb gathers a stream's tap rows into the drafter's KV ring, hold runs the five-layer frame over a block and draws a chain with the target's head. Per-stream rings and segments, frames batched over up to 16 streams. - host tests for chain selection and position staging - tf-qwen35-test drafter: determinism, refusals, and acceptance against the plain target's greedy continuation (fails below 2.0 a round)
A frame's attention reads its own block's keys from the frame, never from the ring: the ring holds absorbed rows only, so no key of a row's window is overwritten before its attention. The ring is still sized at the window and a padded block (config.ringRows), and a host test now pins that. Frames pass no mask, so every row sees its whole block and the merge never sums zero keys. Trees stay refused until their mask keeps each row's own key.
The HIP drafter's hold now picks its chain with core/lanes/draft_tree.zig's walker instead of a copy of its chain, choice and drawn. The core Candidates take the rows apart (ids, f32 logits and f32 selector projections), as HIP's frame downloads them; CUDA's words keep their path and arithmetic. Host tests pin HIP chains on fixed rows, greedy and sampled, and an f32 projection between bf16 steps, before and after the switch. The acceptance prior by depth is one table, draft_prior.zig, that the CUDA and HIP drafters both read.
absorbFn took its first row's position from the features' offset and ignored Absorb.start. It now absorbs from start, and refuses features whose offset (where hip_lanes.zig's tap ring names the first row's position) names another one, with a host test. most_held's comment now says what its 15 is: a block of 16, the ring's padded block, but its pending row.
…uns a seeded sampled stream The acceptance floor of `tf-qwen35-test drafter` moves from 2.0 to 4.5 tokens a round, between a drafter whose attention reads its key ring one slot behind and a clean one, so that planted fault now fails the check. A seeded sampled stream (seed 7, temperature 1) follows: the target draws each prompt's continuation by the rule, the drafter's holds carry the same rule, each prompt's rounds run twice and must draft the same chains, and the line prints the acceptance and a hash of the chains.
… either order Prompts 0 and 1 run together in frames of two, in both orders, and each stream's chains must not depend on its slot. The line prints each stream's acceptance beside its one-stream run.
… has followers for When a stream's last round accepts every draft and the continuation ends right after them, its bonus token has no follower, so the absorb now declares as many rows as it passes.
prefillFn's dangling else ran a text stream's fill zero times when the stream had no readout, so a startup probe of --prompt-cache-disk or --learn-conversations failed with ProbeUnmarked. prefill.whole now runs a readout's pieces to the end in its own block and any other stream's fill once. The opening restore goes through #2802's restoreKept; when it fails, the cache reset stays fatal as on main, and only then is the restore marked failed. test-qwen35-prefill-host runs the stepped, whole and opening tests without devices.
…e read row's 64-row tile (#2849) tailPlan (xpu/prompt_tail.zig) asks core.prompt_tail whether a chunk's last layer is read: an unread chunk runs add_rmsnorm, mixer_in and attn_prep (its keys and values) on every row and stops; a read chunk runs attention, the gate, mixer_out, the post norm, the MLP and the final norm on the rows from ((T-1)/64)*64, and the head on the last row. Dump, probe and fidelity hooks, the A8 study and taps on the last layer keep the whole layer. tail_test compares the last row's logits and the keys with the tail on and off at ten lengths.
Skip the bf16 tail comparison when int8 KV layout and scales apply. Keep decode attention launch arguments unchanged and name the tail test. Refs #2849
…pth, not fifteen a round A confidence floor makes a greedy stream draft the most and leaves the cut to the head's chances. While GLM's chains stopped at three that drafted at most two steps past the cut; since #2257 they reach fifteen, so each greedy round under --mtp-confidence runs fifteen head steps and verifies only the prefix the floor keeps. Model.floor_rates keeps a greedy stream's rates' depth under a floor, as a sampled stream's already is, and the floor still cuts what the window verifies. GLM sets it for the MTP head; DFlash2's blocks and the other families keep the rule they had. GLM's facts drop `.gpu_tokens = false`, Model's default, so cuda_lanes.zig keeps to 600 lines with main's changes.
A folder with 1_LogitScore opens as a reranker, as on HIP: its yes and no rows of the output head read on the host at load, the score and packed-decision hooks of lane.zig on the CUDA engine, and Info's rerank labels with decides, so /v1/rerank serves it. A decision's trunk runs once a pass: its keys stay in the cache's first rows, and each branch attends to them and to its own rows from the trunk's end. The CUDA GPU check adds the reranker's gate (yes - no and the score against transformers, the packed decision equal to each prompt's run alone).
…st; the docs and file headers name the reranker Engine.run's argument checks move to checkBatch, so the trunk, strictly rising ends, the window, the vocabulary and the rows are pinned by a host test (as on HIP). docs/api.md says the native HIP and CUDA engines serve the reranker; two comments said embedding only.
… edges the cases' short common prefix never reaches A trunk of 700 tokens (past one 512-key chunk, off its multiple) with branches of 600, 513, 1 and 530 rows, cut from the longest case, packed in one decision; each branch's logits must equal its prompt's run alone, bit for bit.
…erves; the weights header names the head's stored name
After each prompt chunk the draft head runs over the chunk's rows to fill its own caches. It ran as a decode window, so its projections and its MoE took the stream tiles for up to a whole chunk of rows, re-reading each weight once per four rows. A prompt chunk now runs it on the prompt tiles (o.prefill and moe.run's prompt path), as the main layers' prompt rows already do; decode windows keep the window path. The head's prompt-row caches take the prompt tiles' sums, so its drafts can differ in the last bits; replies are verified as before and do not change.
…not from the options
…en gain of its receipt
…over Metal target.zig is the lanes' verify target: a slot's prompt, its windows of rows for several slots in one launch, keeps and rewinds, and kept prompt states (the prompt cache) whose resumed rows the reply reports as cached_tokens. boot.zig loads a release checkpoint folder and entry.zig serves it as tensorfold-dsv41 (dsv41-metal-serve). Tests serve the tiny model through LaneHost: concurrent equals solo equals serial, drafted equals draft:false, cache hits equal fresh runs, seeds and cancels.
…for the next A shared round with the plain switch at it asked no drafts and ran no head pass (core Rule.plain_streams, threaded from the family's resolved setting: 0 on CUDA, so its rounds are unchanged), and the family's speculation no longer launches a pass for a round that wide (armSpec/speculate plainRound).
…teps and the forward try's
…stays within 600 lines
…nything else by name
…treams streams decode
…er holds a rejected draft's taps
…nd the next ask catches the drafter up
Owner
Author
|
Conflicts with main (main moved after stacking); restacking. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI for a landing train: #2976 #2977 #2915 #2896 #2995 #1878 #1720 stacked on main in that order. Nothing merges from this PR: each PR above lands on its own once this tip is green, and this PR is then closed.