Repository navigation
Merge upstream main - #53
Merged
Merged
Conversation
…ounded and aborted as its sum sumBlock leaves block `rank` of the two ranks' values summed in place (ncclReduceScatter), with sum's peer check, deadline, asynchronous-error wait and abort. flashnext-rank's eof-scatter and timeout-scatter modes make it the first collective a dead or silent peer fails; a mode it does not know is an error instead of a sum.
…slots by rows and gathers their sums A pass wider than any decode round or verify window (more than 64 rows) reduce-scatters y [n, 11, 2560] by row halves instead of all-reducing it, sums each rank's rows' slots with fn_moe_sum into moe_f32, then all-gathers the FP32 sums; the next writeback takes the pending_f32 form (mode 4, one slice). Every summed element is one IEEE add of the two ranks' values, as before, and fn_moe_sum is the writeback's slot chain, so the written rows keep their bytes while the exchange moves 61,440 B a row each way instead of 112,640 B. Rounds and windows keep the all-reduce. y and moe_f32 take one spare row for an odd pass's padded block. flashnext-rank's exchange mode compares the written rows, square sums and normalized rows with the all-reduce's at 65 to 8,192 rows, odd and even, with a changed-sum control, and checks that the exchange writes no row of y or moe_f32 past the spare one. flashnext-model-rank takes TF_FLASHNEXT_CHUNK's rows a pass, as its engine does, where it fixed 32 and so stopped with RankIdentity unless the variable was 32.
A folder with 1_LogitScore opens as a reranker, as on HIP: its yes and no rows of the output head read on the host at load, the score and packed-decision hooks of lane.zig on the CUDA engine, and Info's rerank labels with decides, so /v1/rerank serves it. A decision's trunk runs once a pass: its keys stay in the cache's first rows, and each branch attends to them and to its own rows from the trunk's end. The CUDA GPU check adds the reranker's gate (yes - no and the score against transformers, the packed decision equal to each prompt's run alone).
…st; the docs and file headers name the reranker Engine.run's argument checks move to checkBatch, so the trunk, strictly rising ends, the window, the vocabulary and the rows are pinned by a host test (as on HIP). docs/api.md says the native HIP and CUDA engines serve the reranker; two comments said embedding only.
… edges the cases' short common prefix never reaches A trunk of 700 tokens (past one 512-key chunk, off its multiple) with branches of 600, 513, 1 and 530 rows, cut from the longest case, packed in one decision; each branch's logits must equal its prompt's run alone, bit for bit.
…erves; the weights header names the head's stored name
After each prompt chunk the draft head runs over the chunk's rows to fill its own caches. It ran as a decode window, so its projections and its MoE took the stream tiles for up to a whole chunk of rows, re-reading each weight once per four rows. A prompt chunk now runs it on the prompt tiles (o.prefill and moe.run's prompt path), as the main layers' prompt rows already do; decode windows keep the window path. The head's prompt-row caches take the prompt tiles' sums, so its drafts can differ in the last bits; replies are verified as before and do not change.
…not from the options
…en gain of its receipt
…over Metal target.zig is the lanes' verify target: a slot's prompt, its windows of rows for several slots in one launch, keeps and rewinds, and kept prompt states (the prompt cache) whose resumed rows the reply reports as cached_tokens. boot.zig loads a release checkpoint folder and entry.zig serves it as tensorfold-dsv41 (dsv41-metal-serve). Tests serve the tiny model through LaneHost: concurrent equals solo equals serial, drafted equals draft:false, cache hits equal fresh runs, seeds and cancels.
…for the next A shared round with the plain switch at it asked no drafts and ran no head pass (core Rule.plain_streams, threaded from the family's resolved setting: 0 on CUDA, so its rounds are unchanged), and the family's speculation no longer launches a pass for a round that wide (armSpec/speculate plainRound).
…teps and the forward try's
…stays within 600 lines
…nything else by name
…treams streams decode
… bf16 logits on request tf_route_logits gave each lane four 32-wide slices of k, which covers a hidden width of exactly 4096 (GLM-5.3-Flash and Step-5); at 2048 it read past each row and past the packed router. Each lane now takes hidden / 1024 slices, the same four at 4096, so those routers' arithmetic is unchanged, and `source` refuses a width that is not a multiple of 1024. `round_logits` rounds each logit to bf16 before the select, as a bf16 router matmul (MLX's) gives it.
A Metal family for `model_type: flint` (RecursiveSI/flint), served from Recursive's MLX INT4 export (RecursiveSI/flint-int4: affine 4-bit groups of 64, an 8-bit embedding) by the lane host with the prompt cache, structured output, tools and several streams. - Layers: 35 sliding (513-token window, RoPE) and 14 full (no positional encoding) GQA layers, 32 query heads over 4 key heads, per-head q/k norms and a learned sink each head; two dense SwiGLU layers, then 256 routed experts (top 6 by sigmoid score, renormalized, no score bias) beside an ungated shared expert. - Rounding: MLX's points, as RecursiveSI's rec_mlx computes the model: a bf16 residual, rms_norm as w * bf16(x * inv), SiLU rounded op by op, bf16 router logits. - Attention: prompt chunks on the tensor units (gemma4's prompt kernel over a position-major cache, the sink in the denominator); decode rows on sdpa_vector's lanes over key blocks at absolute positions, merged with the sink by MLX's second pass, so a row's bits do not depend on the rows beside it. Sliding layers keep a ring of the window and one prompt chunk. - Projections: q|k|v and each MLP's gate|up stacked into one matrix at load (one launch, the same row sums); one launch norms and rotates q and k and writes k and v; residual adds fused with the next norm. - Routing: core moe_route's logits, then a top-k pick a simdgroup a row (the core select's arithmetic without its per-Mac lists). - Serving: several streams' windows share one forward, each reply equal to its run alone; a lone stream's steps run a step ahead, drawn on the GPU (argmax, or the keyed sampler of core lanes' gpu_rule and gpu_full at Flint's vocabulary) into a token ring, and the host draws with the same sampler's twin. - tf-flint-run: greedy tokens, every position's logits, forced steps and step timing, for a reference's check, and one-row steps against windows of any width, bit for bit (`exact`).
…ac's RAM is refused, not converted
…(the fp32 form missed MLX on 1,360 of 65,280 bf16 gates, and gave 0 below -17); the router keeps MLX's fp32 form
…al_main has room under 600 lines
…p, decode and prompt attention; gates 1-4 tests, 16 planted controls, gate-5 records), and core tf_route_logits at Flint's, GLM's and Step-5's widths
…he tensor units keep (fp32 cut to 10 mantissa bits, measured on the device before the gates run), not bf16's
A recorded one-row step copied each stream's whole GDN state and conv tail for a keep of no rows, and served traffic never keeps one. Part now carries undo (default on, so warm-up, qualification and host tests keep their copy); the served queue and verify parts set it only while learning. With undo off the staged segment reads no before, recurrent.prepare leases no copy, and the two kernels skip only their copy: the conv tail's in-place update still runs. An empty keep after such a step refuses with QwenNoRollbackCopy rather than restoring bytes the step never wrote. Host: qwen35 shared, cache, pending, rounds, verify, live, prefill and the recurrent workspace steps pass; compile-cuda-tests for Linux compiles.
POST /v1/requests/{id}/cancel finds a request by the chatcmpl-, cmpl- or resp_ id its client got and sends the
engine the cancel a disconnect sends; the reply ends "cancelled" with the tokens it had, a queued one with none, and
other streams are unchanged. 404 request_not_found for an id no request holds.
The ids are held for the whole client request in openai.runNamed, and the ask rides on chat.Input, so every
generation for that request reads it (compaction's reply, Cortheon's rounds) and server-made ones never do. A
Cortheon turn stops after a cancelled round. A Responses reply ends with status "cancelled" in its
response.incomplete event, without completed_at, and a Responses socket keeps it like an incomplete one.
Who may cancel: with API keys, a full or [admin] key (the key gate refuses scoped keys, and the route checks the
matched key again). Without keys, only a caller whose socket is on loopback: the peer address from accept(), never
a header, including listener.peerText's IPv4-mapped form, which IpAddress.parse refuses. A keyless request that
carries Forwarded, X-Forwarded-For, X-Forwarded-Host, X-Real-IP, Via, CF-Connecting-IP or True-Client-IP is refused,
since a reverse proxy on the same machine makes every caller loopback. Refusals answer 403 loopback_only.
Fixes ashhart#3172
A decode window of 4 rows was captured at 16 rows, so lane_gemv loaded and stored 12 padding rows and attention launched their dead blocks. Shape.init now buckets 1 row as 1, 2-4 as 4, 5-8 as 8, and larger windows as before. Every kernel takes its live rows from the staged segment table and MMA rows are independent, so a row keeps the bits of the same one-row step. Warm-up and the cost table follow the new keys: seven buckets instead of five warm at start, and the live-width tests pin the new call counts. The shapes tests now run in the qwen35 module and get their own host step.
…links FFmpeg's x86_64 assembly addresses tables defined in C, such as ff_pw_* and ff_h264_cabac_tables, with 32-bit PC-relative relocations. Linked from the static libav*.a into a shared library, those tables are exported and so preemptible, and the link stopped on R_X86_64_PC32 errors (5,046 with gcc 13.3) although build_ffmpeg.sh passes --enable-pic. On ELF targets the library now links with a version script that exports the four tf_video_* symbols core/tfvideo.zig looks up and makes every other symbol local, so the tables bind inside the library with no text relocations. check-video-exports-linux checks this for aarch64 and x86_64 Linux with a planted table. The loader, build_ffmpeg.sh and its FFmpeg pin, and Mach-O builds are unchanged.
…e threads tf_video_frames asked FFmpeg for slice threads only, so H.264 in one slice a picture and MPEG-4 decoded on about one core. MPEG-4, and H.264 that declares no B-frames, now use up to TF_FRAME_THREADS (4) frame threads, none on one CPU. An H.264 packet goes to them only as one I or P slice, with no new extradata to change how its NAL lengths read; anything else returns the decode to main's path, as FFmpeg's frame threads conceal damaged B-frame H.264 unlike main's, with no error or mark to restart on. On a decode error, a frame marked damaged, or a packet with no frame, the decode restarts on main's path from the first undelivered index. A frame waits until every earlier packet has given an unmarked frame.
…MPEG-4, under Unreleased
…cancels The fake lane counts the tokens it hands a sink, and the test cancels only past the first, so a turn always has text so far. On a slow Windows runner the cancel reached the engine before the worker's first token: the output array was empty and the test indexed it (main red at 4cfd955). The server is unchanged.
…iles the page has dashboard_test.zig was wired only into the macOS test step, so no CI job ran it. Since ashhart#2655's landing it has expected a "Paged prefix cache" label that dashboard.html never had and nothing in /stats reports, so it failed on every Mac running zig build test. The test now runs from build.zig's shared test step on every OS, and that one string is gone; every other label it checks is on the page.
…softmax body from one source (ashhart#2251)
…eam-main # Conflicts: # build.zig # docs/recipes/README.md # zig/src/families/drama3/drama3.zig # zig/src/server/dashboard_test.zig
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I merge upstream
ashhart/TensorFoldmaininto this fork'smain(normal merge, 186 upstream commits).Conflicts I resolved:
docs/recipes/README.md,zig/src/families/drama3/drama3.zig: kept both sides (Woof and Flint recipe rows;nvfp4_test.zigandweights.zigtest imports).zig/src/server/dashboard_test.zig: kept my deletion (967ccb9 removed its source-text assertions); so inbuild.zigI kept mydrama3_quanttargets and dropped upstream's wiring of that test.zig build test test-goldenpasses on the merge.