Repository navigation
Conversation
…ig stays at 600 lines or fewer
ricursiveAI's Ripple-65B-Preview (model_type ricursive) served from the publisher's own releases as published, read in place from their shards: the NVFP4 release (ModelOpt NVFP4 MLPs, bf16 attention, 53 GiB) and the bf16 release (121 GiB). - Family in zig/src/families/ricursive and kernels in zig/kernels/metal/ricursive: PeriNorm, q/k norms, RoPE on sliding layers (1,024-key window, 3,072-slot ring), NoPE global layers every fifth, fp32 sigmoid gate, value residual, clamped SwiGLU. - NVFP4 MLPs: E2M1 nibbles, E4M3 block scales and the fp32 global, applied exactly; activations stay bf16 (the publisher's SGLang serving is W4A4; disclosed in the doc). - Exact bf16 caches; prompt chunks planned on one 2,048-row grid, so resumed == fresh. - Every kernel registered with the kernel contract: 36 pass gates 1-4, 15 controls red. - Server: replies are always read for a think block when the model decides itself whether to think (Info.reasoning_optional); a thinking budget and logprobs are refused, since Ripple's think markers are spelled from several tokens. - tools/zig/ricursive_reference.py: the float64 / bf16-point CPU reference. - docs/recipes/ripple-65b-metal.md: correctness (KL), harness speed and serving checks.
…release code Longer contexts are refused by name until a run at that length qualifies them. THIRD_PARTY_NOTICES names modeling_ricursive.py and the release's SGLang model, which the family follows for its layer order and rounding points.
…ase through the harness
… kernel contract The rc kernels follow glue.metal step for step at its bf16 rounding points: embed, add-norms, output gate, FFN activation, and q/k head norms with RoPE into rings or pages, one stream's rows or each row at its own position and stream. window_attention.cu gains the group-10 prompt instances; global decode runs through tree_attention. ricursive-contract checks them through gates 1-4 with planted controls; nvcc 13.3 records for the eight launches.
…CUDA graphs The NVFP4 release's weights and gate; Metal forward.zig's layer order on the rc kernels. Prompt chunks of one slot, and rounds of rows each at its own position and slot (several streams, verify windows), replayed from a graph of the round's row count. ripple-forward checks rounds against one-row steps, bit for bit.
…and kept states Several streams behind lanes.Backend, a slot of the arena each: shared rounds, copied drafts verified as chains, draws and logprobs by the shared cuda.sample, and the prompt cache's states. ripple-serve checks it on synthetic weights.
… gate tools run and kl drive the CUDA forward directly on one stream; decode rounds replay graphs unless --eager. compile-cuda-cli builds the CLI without a GPU. The gate tools compare kl dumps with ricursive_reference.py's logits (KL, top-1, worst rows, identical dumps, a greedy reply) and turn a chat into the kl command's ids.
… near-tie leads in the gate compare
… files in the notices
…re hooks tf_rc_tap copies a tapped layer's residual rows into a slot's ring, as dflash2.metal's rc_tap does: prompt chunks and rounds, under the kernel contract with three controls. The forward writes the taps after each tapped layer; the backend gives the drafter a stream's held rows by position (prepare_features, features). The slots' caches move to cuda_arena.zig and the forward tests' weight fixture to ripple_fixture.zig, unchanged, so each file stays under 600 lines.
…er's packed weights and the selector walk The drafter's config parse moves from dflash2_config.zig to dflash2_spec.zig, which needs no Metal, so the CUDA build shares it; dflash2_config.zig re-exports it. ashhart#2260's refusal of a clamped MLP (ffn_clamp_threshold) moves with it, and max_streams stays in dflash2_config.zig. cuda_dflash2_weights.zig reads the release through that parse and packs its projections for the lane matmul; cuda_dflash2_select.zig is the host walk over each row's top-32 candidates, as rc_df_walk.
…s under the contract ricursive_dflash2.cu runs dflash2.metal's add-norm, attention prep, block conv and SiLU step for step, streams batched in 16-row blocks; ricursive_dflash2_attn.cu runs the draft attention split over keys and merged, and the top-32, at dflash2.metal's rounding points. Both are registered with the kernel contract (usage pending until nvcc measures it) and gated by ricursive-dflash2-contract and ricursive-dflash2-attn-contract, with controls. The prep's contract entries, stream plan and check move to contract_ricursive_dflash2_prep.zig, unchanged, so each file stays under 600 lines.
…rafted replies equal plain ones The drafter runs dflash2.zig's op order over the target's taps, streams batched in 16-row blocks, and serves through the lane core's Drafted wrapper. --drafter's config.json goes through ashhart#2260's qualified parse and the CUDA kernels' gate before any stream, so its bytes are admitted with the streams'; a drafter that leaves no room for one stream is refused by name. ripple-drafted checks drafted == plain at 1, 2 and 8 streams. The config read and the gate move to cuda_dflash2_config.zig, unchanged, with a test that a clamped MLP (ffn_clamp_threshold) is refused as ashhart#2260 refuses it. THIRD_PARTY_NOTICES lists the CUDA drafter files beside dflash2.zig and dflash2.metal.
…masks The drafted wrapper always offers first_masked and forwards to the family, so a family without masks answered response_format and forced tool calls with HTTP 500. Read the family's own vtable, as hip.zig does, so the server refuses them up front with 400.
… SMs kernel-contract-limits on the Spark measured tf_rc_tap and the eight DFlash2 launches on sm_80, 86, 89, 120 and 121. Their receipts go in pending_133.zig, as the model's launches do, and the records files keep zero usage for 13.0.
|
Thanks, @jrb-ai, for the drafter, and for keeping it in the family's folder. Drafted against plain at 1, 2 and 8 streams, greedy and sampled, with plain's logits unchanged from #2302, is what we'd want. It can only land after its stack (#2231, then #2302 and #2260), so here's what to do when you rebase:
Moving the config parse into |
…shared drafts by them
calibrate() copied the measured window and lane costs into the facts but not the shared-round costs, so rows past the widest window were priced as one stream's widest window scaled by rows. At 8 Ripple streams that made every round past 16 rows look 1.5x dearer than measured, and the allocator held one draft a stream. Drafters that time no shared rounds are unchanged.
|
Thanks @jrb-ai for the resumed cell with the drafter, which is what we hoped to see. #2171 now has a yes, so this lands last of the four, after #2302 and after our 1.2.6 release. What's left:
Then we run it on a GB10. We can't promise a date.
|
|
Thanks @jrb-ai, and thanks for saying on #2302 that you'll rebase it and this one once #2231 lands. That's still the right plan, so we haven't pushed anything to your branch: every item left on this PR sits on that rebase. Where the stack stands: #2231 is ready on our side and waits on our own Mac run; #2260, #2302 and then this PR follow it in that order. So the list is the one from 10 Nov, with one pointer for the rebase:
After the rebase we'll recheck
|
|
Thanks @ashhart. The plan stands: I rebase #2302 and this PR once #2231 lands. The answer to item 2 is ready now, so here it is ahead of the rebase. Item 2, the structured-output offer. The head reads
Measured on GB10, on my local restack: #2231's head (
The base there is #2231's head, not main. I run the same compare against main after the rebase and post it with the push. Items 1 and 3 are ready on the local restack and come with the rebase: the drafter's |
|
Thanks @jrb-ai for answering item 2 ahead of the rebase, and for confirming the plan. Your reading matches main at
So the one offer that changes is Ripple's own with Your 8 of 8 token hashes and the equal structured-output reply are against #2231's head. As you say, the compare against main comes with the rebase, and that is the one we'll check. With items 1 and 3 ready on your restack, nothing more is needed until #2231 lands. After the rebase we'll recheck
|
What this changes
Before, the CUDA
ricursivefamily (#2302) served Ripple-65B-Preview without a drafter. After,--drafterserves thepublisher's DFlash2 drafter (
ricursiveAI/Ripple-65B-Preview-DFlash2) as published, read in place, and the targetverifies every draft, so a drafted reply equals the plain one bit for bit. The code is the family's own, as #2260 is on
Metal; no shared Qwen DFlash2 path changes.
With several streams, the drafter also times shared rounds of every admitted stream at load, and the depth rule prices
a shared round past the one-stream sweep's 16 rows by those times. Without them, the rule extended the one-stream sweep
in proportion and priced a 64-row round at about 1,010 ms against 292 ms measured, so at eight streams each stream
drafted about 1 token a round. The lanes take the times through one new drafter fact,
shared_costs, empty unless adrafter gives it; only this family's drafter does.
79fb36ae1) and ricursive: the publisher's DFlash2 drafter for Ripple-65B (stacked on #2231) #2260 (its commit8843987e8, here442ee0730); this PR adds the 16 commitsafter them (
5dbdd80f5to409d34ee2).atc overlap --repo ashhart/TensorFold --base 442ee0730 --json(8 Nov): incomplete (pullFilesUnavailable,timeLimit). Of 23 open PRs that share files, 2 merge clean (Native Vulkan backend for Qwen3.5-2B (opt-in): engine and kernel-contract registration (stacked on #1526) #1544, glm cuda: decode windows' EXL3 experts run on the 16-pair item plan #2044), 8 conflict and 13 were not tried in thetime limit. Metal: serve the dense Qwen3.8-27B straight from a GGUF file (Q4_K_M, Q8_0) (#1920) #2078 conflicts in
CHANGELOG.mdonly; the other seven (lfm3: serve LFM3-1.7B on CUDA: NVFP4 and FP8 weight-only, MLX 4-bit (stacked on #1513, #1492 and #1503) #1508, cuda experts: a bf16 format for a checkpoint's own bf16 experts #1630, Nemotron 3.5 Transcribe on CUDA: the language model and the audio encoder #1631, Nemotron 3.5 Transcribe: audio input, the 65,536-token window, an hour of audio, MTP drafts #1632, idol: Idol's denoiser as a native Metal family #1766, Metal: the 27B verifies 32 rows a pass on pre-M5 Macs #1841, Living Weights on Qwen4 dense, step 2: the native CUDA and Metal learner (experimental, --slide refused) #2054)each also change
zig/src/core/contract/registry.zig, the shared record list.Checks on this head
409d34ee2. The eight-stream speeds and their exactness checks ran on0c4f8c2f3; the head movesone function from it into its own file (
cuda_dflash2_shared.zig) and calls it after the window sweep, with the samerounds in the same order, and updates this PR's recipe to the new speeds. The GB10 build, the family suites and the
resumed-prompt check ran on the head. Every family's kernel contract and the limits ran on
871c2af4d, before theshared-round fix; the fix adds no kernels. The one-stream speeds and first-token times
ran on
0c8fd876a(on ricursive: Ripple-65B-Preview on native CUDA (NVFP4) #2302's6bc8c6d22). At one stream a round has at most 16 rows, so the fix does not changeit: the run on
0c4f8c2f3gave 27.0, 18.4, 10.0 and 11.4 tok/s drafted at one stream (--levels 1), against 26.1,18.4, 10.0 and 11.2 in the table.
tools/zig/lean_new.py).80a085e27(the head less the recipe's numbers),zig build test test-kernel-contract test-golden fuzz-cortheon test-ricursive-cuda-host:every step passes except one test, which also fails on ricursive: Ripple-65B-Preview on native CUDA (NVFP4) #2302's head and on main:
exl3_load's "a scan that fails anallocation frees every group prefix it held".
zig fmt --check build.zig zigand
git diff --check: clean.GB10,
tf-cuda-teston the head:ricursive-contract83 passed and its 27 planted controls failed as planted;ricursive-dflash2-contract36 and 19;ricursive-dflash2-attn-contract11 and 6;ripple-forward40 passed, 0failed;
ripple-serve9 and 0;ripple-drafted10 and 0.zig build kernel-contract -Dsm=121for everyfamily, on
871c2af4d: 592 kernels pass gates 1-4, 0 fail, 8 known findings; 369 controls red as planted, 0 missed; 417 limitproblems against ricursive: Ripple-65B-Preview on native CUDA (NVFP4) #2302's 372. The 45 more are the nine new launches on five SMs, each "no compiled usage
recorded": like ricursive: Ripple-65B-Preview on native CUDA (NVFP4) #2302's launches, their nvcc 13.3 receipts are in
pending_133.zigand their 13.0 records stayzero. Apart from the limit count, the
SUMMARYline and the known findings equal ricursive: Ripple-65B-Preview on native CUDA (NVFP4) #2302's.launch records fit each). Draft trees (several drafts a stream): not done; chain verify only, as on Metal.
Receipt
ricursiveAI/Ripple-65B-Preview-NVFP4at99eea4477a001a41b86914e41feb4b2ed86105e5,ricursiveAI/Ripple-65B-Preview-DFlash2at35609b3be3dd08a822b3ba1257c1ea5dd6f87632.ricursivefamily (the taps after layers 1, 20, 39, 58 and 77, andthe drafter); ricursive: the publisher's DFlash2 drafter for Ripple-65B (stacked on #2231) #2260's backend-independent DFlash2 config parse, read unchanged; the contract registry and the CUDA
kernel list (new entries only); the drafted lanes and the CUDA calibration (
core/lanes/drafter.zig,core/lanes/drafted.zig,native/cuda_startup_costs.zig), which take a drafter'sshared_costswhen it gives them.Metal and the shared DFlash2 code are not reached. The other CUDA families' drafters give no
shared_costs, so theirrounds are priced as before.
--drafteragainst the same binary without it, 22 of 22replies equal on
871c2af4d(greedy and sampled at fixed seeds, at 1, 2 and 8 streams).bench_concurrent.py --levels 1,8 --alone --serialwith the drafter on0c4f8c2f3(256 tokens, 3 reps, T=1.0 and 0): 0 unequal in 144 comparisons, each replyagainst itself alone (108) and against
"draft": false(36). Cold 2,048- and 8,192-token prompts: the drafted reply equals the plain one.seeds, thinking off, up to 256 tokens). Each reply resumed from the prompt cache (49 to 2,303 cached rows) equals its
fresh reply and its
"draft": falsereply, alone and with eight streams at once: 8 of 8 conversations, 24 requestsresumed at a row above 0.
(prompt chunks of 2,048 and 333 rows, and one-row decode), and its greedy reply's token hash is the same.
--drafter(256 tokens, median of 3; code is a raw text completion, prose achat reply). One stream from
tools/bench_openai.py; eight streams together fromtools/bench_concurrent.py --levels 8on0c4f8c2f3, counting all eight streams' tokens:Before the shared-round fix (
3065161da), the eight-stream drafted cells were 49.5, 42.8, 40.3 and 35.2 tok/s.Prompt speed, one stream, cold prompts (
tools/bench_cache.py --require-cold, median of 3): time to first token,and prefill as prompt tokens over that time:
the drafter follows.
Weaknesses and open questions
drafter rows that round (
shared_drafts.budgets), so code may run on short copies in place of the drafter. Notmeasured: the server has no copy counters.
420 ms at 128 (the drafter's load-time times). That is a lead for kernel tuning, not part of this PR.
PR does not tune. With
--drafterthe first token comes 0.11 s later at 2,048 tokens and 0.19 s later at 8,192(2.8% and 1.2%), so the checklist's last item is not met for prompts.
Kernel contract
ricursive, 10 inricursive dflash2, 3 inricursive dflash2 attention); none replaced.SUMMARYline and exit status: CUDA (GB10), each exit 0:SUMMARY ricursive: 22 kernels, 22 pass gates 1-4, 0 fail; 27 controls: 27 red as planted, 0 missedSUMMARY ricursive dflash2: 10 kernels, 10 pass gates 1-4, 0 fail; 19 controls: 19 red as planted, 0 missedSUMMARY ricursive dflash2 attention: 3 kernels, 3 pass gates 1-4, 0 fail; 6 controls: 6 red as planted, 0 missedMetal: ricursive: the publisher's DFlash2 drafter for Ripple-65B (stacked on #2231) #2260's.
zig build fatbins kernel-contract-fatbins -Dsm=80,86,89,120,121,then
tf-cuda-test kernel-contract-limits: the nine new launchespass on all five SMs (45 of 45, each at its nvcc 13.3 receipt); the run's total stays at ricursive: Ripple-65B-Preview on native CUDA (NVFP4) #2302's 677 problems, all
inherited.
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.Authored by JRB AI in a private capacity. This contribution does not represent the views or positions of any organisation.