Skip to content

lfm2: DSpark drafting with owned context and confidence scheduling - #1394

Open
BlivionIaG wants to merge 13 commits into
ashhart:mainfrom
BlivionIaG:series/dspark-hold
Open

BlivionIaG wants to merge 13 commits into
ashhart:mainfrom
BlivionIaG:series/dspark-hold

Conversation

@BlivionIaG

@BlivionIaG BlivionIaG commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

DSpark serving through the shared LFM2 lanes, with neural serving standing aside before loading on pre-M5/unknown Metal devices. Rebased onto current main; the review's restored tests, the kernel-contract run and the plain main-versus-branch cells are below.

  • Issue: Refs LFM2.5-24B-A2B on Metal: drafts with LiquidAI's DSpark drafter (proposal and gate) #1374; the M5 drafter cells and the QK-norm/RoPE order check are the maintainer's open gates.
  • Stacked on: main 0fed2e4a5fcc03f109cfaaf8eda10734cce21fb7; adds 13 commits. Rebases onto main resolved conflicts in zig/src/core/lanes/{allocate,depth,depth_constants,shared_drafts,windows}.zig (the lane scheduler's least_rows and floor_rates rules kept beside the hold-grid pricing) and a CHANGELOG merge.
  • atc overlap --json: ATC0.9.6 was run against upstream with --base fb8fdf3; complete=false (myPullRequestsLimit, otherPullRequestsLimit, pullFilesLimit, pullFilesUnavailable, timeLimit). Supplemental complete snapshot:217foreign open heads,3038 exact local merge comparisons including latest Main ff742df; existing reviewed overlaps are retained, not claimed globally clean. The avoidable added environment-test collision with CUDA (GB10): a second carveout source, spark-attic's atticd, opt-in #2762 was removed by preserving Main's existing test name, without importing its unmerged feature.

Checks on this head

  • Head commit: 22b19797ec1048bb2f50e307aae3339077ba68be.
  • Lean checker (python3 tools/zig/lean_check.py): head 341, main 341, same problem set, 0 new. No touched .zig file is over 600 lines; zig fmt --check build.zig zig is clean.
  • Tests run on this head (Apple M4 Pro, Zig 0.17.0): zig build test test-golden dev-tools -Dcpu=apple_m1: 12,584 of 12,687 tests pass, 102 skipped, 1 failed. The failure is dashboard_test ("Paged prefix cache"), which fails the same way on main. zig build test-shared-lane-regressions-host: 476 passed, 1 skipped (Windows-only). On the previous head, test-lfm2-dspark passed 93 with 2 skipped and the server goldens matched (322 of 335, 0 unexpected).
  • CI: the head before the last rebase had 12 successful checks and 2 scheduled-only skips; CI on this head is running.
  • Not run: M5 and M5 Ultra, GB10/CUDA. On an M4 Pro the neural drafter stands aside by design, so the drafted cells and the QK-norm/RoPE order check need an M5.

Receipt

  • Main against branch, no --drafter (Apple M4 Pro, LiquidAI/LFM2.5-24B-A2B-MLX-4bit, --parallel 4 --context 4096, bench_concurrent.py --levels 1,4 --tokens 256 --temperatures 0 --alone --serial; main 3885dceb2 against the branch before its last rebase): the code cell's token_sha256 is f66e3fd7e509 and the chat cell's 57752ac6d7f0 in every arm and stream count, each equal to its own run alone and to "draft": false. With --drafter, the server prints "DSpark standing aside on Apple M4 Pro" and the same hashes result at 1 and 4 streams. Speed, 1 stream, mean of three interleaved runs: code 130.9 main, 129.4 branch (-1.1%); chat 118.6 main, 121.3 branch (+2.3%). The 4-stream cells split into two modes in both builds (about 112-125 tok/s aggregate or about 230) while other builds were running on the machine, so no 4-stream speed claim is made. The QK-norm against RoPE order check needs the neural drafter, which runs on an M5.
  • Boxes it reaches: LFM2.5-24B-A2B on Metal; neural DSpark drafting runs on M5 or newer and stands aside on earlier or unknown devices before loading. CUDA is not reached.
  • New switch? Default, accepted values, profile row: no new serving setting/default/profile activation in this review fix/restack. Original settings and accepted forms are in the exact-head docs/environment.md and platform/environment.zig; no experimental setting is newly enabled by this update.
  • Environment: head 22b19797ec1048bb2f50e307aae3339077ba68be; Zig 0.17.0; Apple M4 Pro, 48 GiB, Metal; LiquidAI/LFM2.5-24B-A2B-MLX-4bit with the DSpark drafter given to --drafter.
  • Paths/platforms run versus unrun: M4 Pro host tests, kernel contract and plain serving; M5 drafted serving and CUDA are unrun.
  • Exactness, --alone --serial/token_sha versus drafts-off: equal in every cell above, plain and with --drafter standing aside, at 1 and 4 streams.
  • Main against branch: see the first line of this section.
  • Decode speed before/after: 1 stream within +2.3% and -1.1%; 4 streams not claimed (see above).
  • Prompt speed/cold TTFT before/after: not measured by me on this head.
  • Licence of new weights/drafter: LFM Open License v1.0; no weights distributed.

Remaining maintainer requests

The M5 no-losing-cell matrix (plain and drafted), the QK-norm against RoPE order check with the released drafter in both orders, and the maintainer's own main-against-branch run.

Kernel contract
  • Kernels: this PR registers LFM2's DSpark kernels (sorted combine norm and tap, BF16 rows, GLU, context KV, confidence, argmax, stream conv and attention) with planted controls.
  • Planted controls: tf-kernel-contract --only lfm2 on an Apple M4 Pro: 32 controls, 32 red as planted, 0 missed.
  • Backend SUMMARY: SUMMARY 19 kernels: 19 pass gates 1-4, 0 fail, 0 known findings; 32 controls: 32 red as planted, 0 missed; 0 limit problems, exit 0. The first run failed gate 3 on the BF16 GLU rows (the gate's stress sum was negative, so SiLU flattened a wrong sum to zero, and the fp16 margin was 0.4x); ebe359ca9 keeps that sum positive and declares the 72.6x margin that results (f16 = 50).
  • Chips not run: M5 and M5 Ultra, GB10/CUDA.

Checklist

  • Title/description, head and stack agree; older results remain source-bound.

  • Every new/restored test has a failing-before runtime receipt across all reached platforms. Host evidence is above; missing device gates are not waived.

  • No newly added overlong/multiline source comments, measurements or review history; inherited findings are reported above.

  • Complete publication trees/outgoing histories reviewed for public noreply identity, tool trailers and privacy; synthetic detector fixtures and mandatory third-party public contacts are not live secrets. No package/binary upload proposed.

  • All inference-path/lane-round gates qualified on this head.

  • Drafted=plain, restored=fresh, concurrent=solo on required devices/settings.

  • Required precision/reference evidence on this exact head complete.

  • Decode/prompt no slower than Main at the required widths, greedy and sampled.

Provenance and review history moved from the recipe

Provenance (was the paragraph after the controller receipts)

The checkpoint's block_size: 9 establishes its query geometry, not exclusive implementation provenance.
Later diagnosis consulted SGLang's DSpark/DFlash implementation, including revision 488869c2d
(Apache-2.0), for attention, projection/normalization, Markov and confidence semantics.
The historical diagnosis
remains outside this implementation branch. The October 25 authoring record has not been recovered:
checkpoint-only or clean-room provenance cannot be certified, and the maintainer's provenance gate remains open.

Independent paper/config re-derivation (November 10; it does not erase the consultation above):
(The bulleted paper/config facts that followed are retained in the recipe as plain model facts, without this framing.)

Receipt links (review history, removed with their one-sentence descriptions)

Open item kept in the recipe

QK-norm's position relative to RoPE is not fixed by the papers or config and is unmeasured against the released
drafter. The maintainer's suggested check: serve the released drafter with each order and compare accepted drafts
per round; the order the weights were trained with should be clearly ahead. Not run; no outcome is claimed.

Historical source-bound notes — not current-head qualification

Earlier commands/measurements retain their original commit/binary scope. Any earlier readiness or provenance claim is superseded by the open gates above.

Current published head (supersedes the stale head/base/qualification claims below)

  • Head: 616aa4b3c91f4bed6f299a65ab9e6d28ea5ad9fa, rebased onto pinned upstream Main 8a7838a65a1fa1a44e0136a2cccd8bfcc8c0a8b5. PR base remains main. The 50e538bd9 head on Main 6e9af930 and the older 7d171cc73 / 28760ea25 / 51cea13ff stack, the 36f84936 merge base and the CI/receipt sections below are historical, not current-head results.
  • Repository check: lean 341 on the head and on main, same problem set.
  • Source-bound host proof: initial b552696721bbf41f8c34db4e59a454f2136f1f4c completed17/17steps,606passed/two skipped of608, plus native CUDA host compile. Current tests-only 616aa4b3c91f4bed6f299a65ab9e6d28ea5ad9fa additionally ran the corrected XPU/shard suite:9/9steps,7/7tests. The older606-test execution is not relabelled as a new run.
  • Device: Metal on an Apple M4 Pro, where the neural drafter stands aside; the M5 drafted cells are open.
  • Negative cells stay negative: the old negative M4 cells are not relabelled and there is no 1% budget. Runtime stand-aside / cost-gate source is not proof that drafting now pays in those cells.
  • LFM2 kernels: run under the Metal kernel contract on an M4 Pro (see above); the M5 chip is not run.
  • Remaining gates (open): M5 no-losing-cell matrix, plain and drafted; the QK-norm/RoPE order check; the 4-stream plain main-versus-branch speed on a quiet machine.

Provenance (recipe docs/recipes/lfm2.5-24b-a2b.md on this head; limits kept)

  • DFlash arXiv 2602.06036v2 §4.2 supports noncausal block attention.
  • DSpark arXiv 2607.05147v1 Eq. 2-3 support the projection; Eq. 4-5 the Markov semantics; Eq. 7-8 and Algorithm 1 the confidence schedule.
  • Qwen3 arXiv 2505.09388v1 §2 supports pre-normalization and QK normalization. The exact QK-norm versus RoPE order was not independently established from these sources; that gap is open.
  • Checkpoint e052e5b1dd103c38cdd4f8ddc068d7dea4586f55: pinned config/model header confirms 9 query rows (anchor + 8 masks), 5 full-attention layers, rank 256; the original author record is unavailable.
  • Historical diagnosis consulted SGLang (unpinned Main and 488869c2d); the original authoring record is unrecovered. No clean-room claim is made. Other engines were not inspected again to close this. A black-box run of the external model card's SGLang setup needs CUDA/flashinfer; Mac CPU is not a substitute.

What this changes

Adds LFM2 DSpark drafting with owned context and confidence-priced holds; the review fix prevents drafts-off legacy enrollment and scopes DSpark scheduling changes to configured hold costs. Out of the landing queue pending the cross-family checks in the cross-family maintainer gate; local author checks and historical LFM2 numbers do not clear that gate.

Pinned peer heads for the conflict comparison: #1667 2f42c9cb6a64e222a57201db4d9c55b18d1c1c6c; #1446 6d2666186de117a4216ec38b784f12ae9e00a5e6; #728 cbee96430d7b4760781f2f52ecef4d86fd5927ea.

Answers to the six review items

The compatibility boundary is k.hold_costs.len > 0, not model-family names. Non-costed callers retain main's scheduling; the drafts-off enrollment fix applies universally.

Review item Fix on this head Behavioral coverage
Legacy drafter without enroll absorbs a drafts-off prompt Set head_context = false before hook selection when !s.drafts; do not open or absorb a head Whole, streamed-feature and chunked prefills: target completes and returns the plain reference tokens; no legacy cache, absorb or feature capture; enabled controls and release accounting checked
No-node-probability budgets changed all-stream accounting Live-only costs/caps/overhead and finished/forced exclusions require hold costs; unpriced allocation retains all-stream bookkeeping Mixed allocation, overhead, learned prices/caps and fixed-row probes
Forced-depth early return skips main's reset Keep the early return only for costed holds; unpriced calls perform the ordinary/granted plain-and-wait bookkeeping before returning zero Forced granted and ordinary transitions
Copy reservation changes proposer/backoff cadence Reuse an owned decision only for costed holds; unpriced budgeting/planning retain their two queries Paired suffix-backoff behavior, including the resulting copy tokens versus retained silence
A no-drafting round changes the probe clock Preserve main’s shared-reservation dispatch for empty unpriced rounds; only the legacy limited/non-granted route advances the rule clock. Retain costed suppression Eight priced/limited/granted configurations, plus the retained seven-round resumed-drafting control above
Paused streams count as finished Exclude paused streams only for costed holds; forward the actual configuration at every who caller Paused/resumed/finished depth decisions

The who signature migration is complete, without an alias/shim. Existing costed reservation/pricing fixtures now opt into a hold grid explicitly; their pricing assertions are retained. The obsolete test requiring feature capture for a drafts-off prompt was replaced by the three target-only regressions above. No shader, activation format, sum, target verification or precision threshold changed in this review fix.

Latest review follow-up — 7d171cc73

Addresses the two remaining requests, without a merge/rebase, force-push, shader change or precision change:

  1. Unpriced empty-head dispatch: roundStreams retains main's shared_drafts.budgets path when shared allocation is enabled. The new regression fails before the one-line correction (expected 0, found 7). Afterward, the repository scheduler suite passes 440 tests, one skipped; the changed code has no lean findings. A standalone production-engine consumer exercises all eight priced/limited/granted combinations for seven rounds each, with the expected probe cadence and zero draft calls. The deterministic fixture backend is not a device/performance measurement.
  2. Actual-main probe comparison: built both programs from main at the PR's merge base, 36f849362700d0d8c4c9ec9e68b7b28e5c6e74c8, and ran them on the same M4 Pro / 48 GiB. tf-qwen27-dflash-served-check exits 1 with ServedDraftDiffers; tf-qwen27-lanes-check exits 1 with UntimedDraftWindow. These are actual-main results, not the earlier 51cea13ff feature-parent comparison. Both use generated five-layer model fixtures, not trained weights. Source-bound notes and both binary SHA-256 values.

The baseline errors are not passing probes, and this follow-up does not clear the outstanding cross-family device/performance gate. The older coordination comparison above remains historical/incomplete; no new zero-conflict claim is made. Automatic CI runs from the normal fast-forward push; no manual rerun or workflow change.

Automatic CI on 7d171cc73

All eleven automatic checks succeeded, including zig, zig-windows and zig-macos-dev-tools. These build/format/platform checks do not clear the cross-family device/performance gate above. No manual CI rerun or workflow change.

Checks on compatibility head 28760ea25 (historical)

  • Head commit: 28760ea25f17c0f6bc4490af5139670f107b48ea; parent 51cea13ff1035e7a9d1101d196fbe8339eba0284.
  • Lean checker, python3 tools/zig/lean_check.py: exact-head Linux CI reports 325 on the base / 325 on the head / 0 new, using the head checker copied to the base. Separately, the committed local checkout reports 425 findings / 5,577 files, exit 1. These are not globally clean runs; the zero-new verdict uses the within-CI same-checker comparison, not a comparison between different checkouts/environments.
  • Tests run:
Check Observed result and scope
zig fmt --check build.zig zig Pass
ZIG=zig bash tools/zig/test_lanes.sh 439 passed / 1 skipped / 0 failed; fixture-dependent groups separately reported 0 passed / 4 skipped, 0 passed / 5 skipped
Full local native/host graph, command below 742/742 steps, 8,535 passed / 61 skipped / 0 failed on the compatibility source; includes native 116 passed / 3 skipped and LFM2 DSpark 87 passed / 2 skipped
Recheck after the commit, same command 742/742 steps, fresh results 408 passed / 10 skipped / 0 failed; other test groups reused Zig's cache, including native and DSpark. Do not read the previous row as a second fresh full run
Server goldens on committed head 322/335 equal, 13 known differences, 0 unexpected; native/server goldens 55/55 equal, 0 unexpected
Default build / zig build dev-tools 258/258 and 234/234 steps succeeded on the same source; the commit subsequently added only the changelog note
Native generated Qwen/DFlash runtime probes Not passing: tf-qwen27-dflash-served-check exits 1 with ServedDraftDiffers; tf-qwen27-lanes-check exits 1 with UntimedDraftWindow. Both exact errors also occur when separately built/run from published 51cea13ff. These are retained failures, not new-head exactness evidence or a diagnosis of their cause

Using Zig 0.17.0, with Zig on PATH:

zig fmt --check build.zig zig
ZIG=zig bash tools/zig/test_lanes.sh
zig build native test test-golden test-kernel-contract fuzz-cortheon   test-server-cpu test-native test-lfm2 test-lfm2-dspark   test-draft-metal-host test-drafter-skip-metal-host   test-qwen27-shared-allocation-host   -Dcpu=apple_m1 -Dversion=1.2.3 -j1 --summary all
zig build -Dcpu=apple_m1 -Dversion=1.2.3 -j1 --summary all
zig build dev-tools -Dcpu=apple_m1 -Dversion=1.2.3 -j1 --summary all
  • CI on this exact head: GREEN — 3 workflows, 11/11 jobs successful — Zig format, No GPU, App/mobile. Completed workflow metadata for every run reports head_sha=28760ea25f17c0f6bc4490af5139670f107b48ea; every job conclusion is success. This is host/build CI, not device/performance qualification.
  • Not run/qualified: the maintainer's Qwen3.8-27B + DFlash2 1/2/4-stream cells including draft:false on GB10 and M5 Ultra; the pre-M5 Mac cell; Flash Next 2/4 streams; new-head full-model exactness, decode and cold-prompt performance. The generated Qwen probes above do not replace these cells.

Receipt

  • Environment: TensorFold 28760ea25f17c0f6bc4490af5139670f107b48ea, Zig 0.17.0, native Metal, Apple M4 Pro / 48 GiB, -Dcpu=apple_m1 -Dversion=1.2.3. Native binary SHA-256 04fe1fa3993d6d0daacdc2cefd4399f45fd445517f9684b56763ff07cd71df1c; unchanged by the final changelog-only commit step. capabilities --json reports engine zig, chip apple-m4, backend metal.
  • Checkpoints attempted, not qualified: TensorFold/Qwen3.8-27B-MLX-4bit revision 22d8d538154e0e5b6f8dcb1fcc74a60a0798104c; z-lab/Qwen3.8-27B-DFlash2 revision 50307d4c4cde6860d4eee73e2547cd786fe8e8a4. Cached complete snapshots, no checkpoint substitution.
  • Runtime attempts retained: two-slot/context-4096/default-frame startup and one-slot/context-512/default-frame startup both became ready but warned about memory pressure; each was stopped before any request. One-slot/context-512/prompt-frame-16 startup exited with TargetBinding, without requests. No limits were raised and no pressured run is called a smoke pass. Generated tiny native probes were then run and failed as recorded above, also on the separately built published head.
  • Paths/platforms: all six shared-lane compatibility behaviors exercised by the host regression graph; actual native CLI and generated Metal programs launched. No trained-checkpoint request completed in this round. CUDA/GB10 and M5 Ultra runtime/performance checks remain unavailable here; the maintainer's cross-family hardware gate is still open.
  • Exactness: no new-head token_sha or bench_concurrent.py --alone --serial qualification. Host regressions check the changed transitions and returned tokens; generated-probe failures and aborted full-checkpoint starts are not parity passes. Historical LFM2 exactness stays bound to its measured source below.
  • Decode speed before/after: not remeasured on this head; historical LFM2 values below are not relabelled. Cross-family performance remains a landing gate.
  • Cold first-prompt ttft_s before/after: not qualified; this change reaches prefill and scheduling, so this is not an n/a claim. No request was issued under memory pressure.
  • Licence of new weights/drafter: n/a — no weights or new checkpoint introduced by this compatibility fix; existing model provenance remains in the historical reports.
Kernel contract — historical feature receipts, no kernel change in this review fix
  • Kernels added or replaced by 28760ea25: none. The overall feature's source-bound report and raw-evidence index and archived reports remain linked below; retaining them does not claim new-head GPU-contract qualification.
  • Planted GPU controls and the backend SUMMARY on an Apple M4 Pro are in the section above; the head and the chips differ from these historical receipts.
  • Chips not qualified in this refresh: M5/M5 Ultra, GB10/CUDA and other non-local devices; local generated Metal failures are disclosed above.

Checklist

  • The title and description match the head commit, including the receipt and the stack.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency. After-fix regressions pass; this round did not run these new tests against the before head. The maintainer's reported regression is accepted as ground truth.
  • Comments/docstrings introduced here are one line; no measurements or history added to source.
  • No personal data, private machine names, internal hosts, local paths or AI attribution added to the diff or commits; commands above use Zig on PATH.
  • Every token still goes through the lane rounds; no new serial path or requirement to turn drafts off.
  • New-head drafted/fresh/resumed/concurrent output parity — hardware qualification pending; historical results are not a replacement.
  • No precision traded by this review fix; historical BF16/FP32-head and quantized-head limits remain explicit below.
  • New-head decode and cold-prompt no-slowdown gate at one/widest admitted streams, greedy/sampled, with/without drafter — pending the maintainer's planned cells.

Retained model and performance evidence — historical sources, not this head

The complete source-bound report and raw-evidence index remain the qualification record. The new linear head was not given a fresh model/performance label merely because the history changed.

Evidence Measured source Result and limits
Original M5 matrix ca54e6889ed8ed00400bfeb4fa1be30a3a687de5 All 144 cells / 306 oracles completed. Worst cell median −0.482927%; this does not satisfy the required no-slower-than-plain gate. The strict original verdict remains FAIL / exit 3, with 11/32 negative contrasts; it was not rewritten as zero failures.
Maintainer M5 Ultra repeat Archived reviewed head 7513f89d3ecc116b986d22c0b6b5a094a2bb2c00 Code/chat, T0/T1, streams 1/2/4/8, two runs each, 4-bit target. Only negative cells: chat T0 at 2/4 streams, −0.2% / −0.4%; best +10.7%. The job was RED because main rejected --drafter on the baseline side; comparison used plain runs. This repeat did not hash drafted against plain outputs.
Functional consumer checks ca54e6889 BF16/8-bit drafter-weight modes each completed 29 exact triples and 3,517 resumed-cache tokens. Stress: 2,779 completed / 1,507 cancelled, zero wrong-output/error observations in that recorded run. Served long-context checks at 4,096 / 16,384 / 65,536 / 130,816 context tokens each produced the checked 64 output tokens exactly.
Precision comparison ca54e6889 BF16 activations in both drafter-weight modes. Independent FP32-head reference over captured BF16 target features, not an FP32 target-server comparison. The short M5 8-bit case had 8/9 matching head picks and non-zero confidence differences. No zero numerical-error claim.

The separate plain/plain A/A run is diagnostic, not a waiver or a proven measurement-error bound: 20/48 paired runs changed internal work. The retained reports distinguish strict oracle failures, consumer correctness, and non-zero FP32-head differences. The acceptance gate is no slower than plain decode in every cell; no percentage regression allowance applies. Historical pre-M5/M4 diagnostic failures are not relabelled as fixed or as new-head qualification. No new M3/M4 full-model, CUDA, or Windows runtime qualification is claimed by this refresh.

Earlier source-bound reports, diagnoses, and CI receipts (historical, not current-head results)

Full Linux/Windows CI found that the wrong-slot prompt-feature test enrolled a lane without marking its drafter context open. The tests-only correction models an opened context and preserves the original FeaturesGone ownership assertion; it does not repin the expected error or alter production admission checks. Actual test-draft-xpu-host now passes9/9steps,7/7tests. Current-source automatic CI is complete:12successful checks,2scheduled-only skips. Device/provenance gates remain open.

Terminal current-source CI at616aa4b3:12successful checks,2scheduled-only skips,0failures. Linux25m26s,Windows29m28s,macOSdev-tools17m10s,pytest7m4s; app/mobile/format and remaining platform checks passed. Source stays frozen. No new device controls, plain/drafted/no-losing-cell measurement or independent normalization-order/provenance evidence is inferred.

@ashhart ashhart left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @BlivionIaG. This builds on #1391, so it inherits that review's blockers (runtime lossy BF16 conversion, the 4-bit buffer mismatch, dropped async failure flags, the contract gaps). Its concurrent encoder and shared stage flush also need device race, resume and replacement-equality checks.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Thanks @ashhart. Device checks on an M4 Pro at the stack (details in the body, head 2e5966f68): resumed equals fresh (hash dc53424643e3 with the prompt cache on and off); about 1,000 requests with random cancels and immediate replacements at 6 and 16 slots, 0 wrong replies and 0 errors against solo hashes; the same with Metal API validation on, no validation message. This is evidence, not a proof of ordering: the concurrent encoder puts a buffer barrier after every dependent step, and I describe it in the body.

@BlivionIaG
BlivionIaG marked this pull request as ready for review October 26, 2026 21:37
@jschmied

Copy link
Copy Markdown
Contributor

An M3 receipt for this head, since the body says no run covers M1, M2 or M3. M3 Pro (36 GB), macOS 26.7.1, Zig 0.17.0, head 2e5966f68:

  • zig build test test-lfm2 test-lfm2-dspark -Dcpu=apple_m1: 534 of 534 steps succeeded (45 tests skipped). zig fmt --check zig is clean.
  • tf-kernel-contract --only lfm2: 11 kernels pass gates 1-4, 0 fail; 17 controls red as planted, 0 missed; 0 limit problems. --only tile_qmm: 6 pass, 0 limit problems.
  • The full zig build kernel-contract can't finish on an M3, on main b30d5b9d3 too:
    • First it stops at UnrecordedRotaryCompiler.
    • With a fresh tf-rotary-sweep record, it then stops at "gemma4 31B rounded dense: tensor units required".
  • tf-lfm2-exact on LiquidAI/LFM2.5-24B-A2B-MLX-4bit @ 667408d4: 72 forward/cache checks passed, 0 failed, 0 skipped.
  • tf-lfm2-dspark MODEL DRAFTER 256 (drafter @ e052e5b1): 9 of 9 drafts equal the fp32 reference's picks, with the largest confidence difference 0.00106. Over 20 steady rounds: 0 host allocations, 0 resizes. A 16-stream hold takes 21.9 ms at depth 1 and 40.2 ms at depth 2.
  • Served with --drafter --parallel 2, greedy, 160 tokens, 4 prompts: drafted replies equal "draft": false byte for byte, also with two drafted streams at once.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Thank you @jschmied. I added your M3 Pro receipt to this body and updated #1377 to #1394 and #1374: M1 and M2 are now the chips with no run (their launch limits are recorded from offline compiles).

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

An M3 receipt for this head, since the body says no run covers M1, M2 or M3. M3 Pro (36 GB), macOS 26.7.1, Zig 0.17.0, head 2e5966f68:

  • zig build test test-lfm2 test-lfm2-dspark -Dcpu=apple_m1: 534 of 534 steps succeeded (45 tests skipped). zig fmt --check zig is clean.
  • tf-kernel-contract --only lfm2: 11 kernels pass gates 1-4, 0 fail; 17 controls red as planted, 0 missed; 0 limit problems. --only tile_qmm: 6 pass, 0 limit problems.
  • The full zig build kernel-contract can't finish on an M3, on main b30d5b9d3 too:
    • First it stops at UnrecordedRotaryCompiler.
    • With a fresh tf-rotary-sweep record, it then stops at "gemma4 31B rounded dense: tensor units required".
  • tf-lfm2-exact on LiquidAI/LFM2.5-24B-A2B-MLX-4bit @ 667408d4: 72 forward/cache checks passed, 0 failed, 0 skipped.
  • tf-lfm2-dspark MODEL DRAFTER 256 (drafter @ e052e5b1): 9 of 9 drafts equal the fp32 reference's picks, with the largest confidence difference 0.00106. Over 20 steady rounds: 0 host allocations, 0 resizes. A 16-stream hold takes 21.9 ms at depth 1 and 40.2 ms at depth 2.
  • Served with --drafter --parallel 2, greedy, 160 tokens, 4 prompts: drafted replies equal "draft": false byte for byte, also with two drafted streams at once.

Thanks for the run, now we have proof it works

@ashhart

ashhart commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG, and thanks to @jschmied for the M3 Pro receipt. The concurrent hold encoder puts a barrier after
every step that reads an earlier step's writes (we followed the absorb, the layers, the attention waves and the Markov
steps), encodeTwinFrom leaves encodeTwin and its callers as they were, and the cancel and replacement stress at 6
and 16 slots with Metal API validation on, plus resumed equal to fresh, is the device evidence our last review asked
for.

This PR's own changes look ready. It is blocked only by what it inherits from #1391 and #1392: the guards, the
Drafter.init errdefer order, the config bounds and the quantizer test. Once those are fixed and the stack is rebased
on current main without #1379 and #1382, we will run the device cells on the stack head: drafted against plain and
concurrent against solo at 1 to 16 streams, the long-context cells, and alternated speed rounds against main.

@Arminova

Copy link
Copy Markdown
Contributor

An M1 receipt for this head, for the load-time memory the body lists as not measured. M1 Pro (32 GB, MacBookPro18,1), macOS 27.0.1, Xcode 27.1 with Metal Toolchain 27A266a, Zig 0.17.0, head 2e5966f:

zig build native -Dcpu=apple_m1: built.
tensorfold-native serve on LiquidAI/LFM2.5-24B-A2B-MLX-4bit @ 667408d4, no drafter, --parallel 1 --context 4096, under /usr/bin/time -l: loaded in 112.9 s (cold, first start after the build). The server logged 13.6 GiB in use once loaded and 13.4 GiB wired between requests; context_fitted: false.
/memory once ready, before any request: lifetime_peak_physical_footprint_bytes 14,637,672,960 (13.63 GiB), footprint 14,608,919,088.
After one greedy request (23 prompt tokens, 256 reply tokens): the peak is unchanged at 14,637,672,960. time -l on exit gives the same peak memory footprint, 14,637,672,960.
That request: 51.4 tok/s decode; TTFT 19.2 s, the first request after the start.

@Arminova

Arminova commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

An M1 receipt for this head: the load-time memory the body lists as not measured, and a slow-prompt finding on M1 that main has too. M1 Pro (32 GB, MacBookPro18,1), macOS 27.0.1, Xcode 27.1 with Metal Toolchain 27A266a, Zig 0.17.0, zig build native -Dcpu=apple_m1, LiquidAI/LFM2.5-24B-A2B-MLX-4bit @ 667408d4, no drafter.

Load-time memory (head 2e5966f68, --parallel 1 --context 4096, under /usr/bin/time -l):

  • Loaded in 112.9 s, cold. The server logged 13.6 GiB in use once loaded and 13.4 GiB wired between requests; context_fitted: false.
  • /memory once ready, before any request: lifetime_peak_physical_footprint_bytes 14,637,672,960 (13.63 GiB), footprint 14,608,919,088.
  • After one greedy request (23 prompt tokens, 256 reply tokens) the peak is unchanged; time -l on exit gives the same peak memory footprint, 14,637,672,960.

Cold prompts are slow on M1, on this head and on main ee896951d alike. One stream, --context 4096, one warm-up, then raw /v1/completions with max_tokens 1 and a fresh nonce first (cached=0 in every server line); times are the server's ttft:

prompt tokens this head main ee896951d
16 15.97 s 15.38 s
19 17.02 s 16.72 s
114 35.92 s 40.62 s
117 39.93 s 41.19 s
462 111.03 s 109.23 s
448 108.23 s 108.34 s
930 202.82 s 212.11 s
916 191.93 s 193.18 s
fit 14.9 s + 0.200 s a token 15.0 s + 0.204 s a token
  • About 15 s before any prompt token, then about 5 prompt tokens a second. Decode is normal: 54.5 tok/s on this head and 44.1 on main for 128 tokens after a 15-token prompt.
  • The startup line gives LFM2 window ms by width: 1 17.7, 2 27.6, 4 45.7, 8 80.8, 16 143.5, about 110 rows a second at width 16, some 20 times what cold prompts get.
  • At --context 16384 on this head a 1,864-token chat prompt took 404.2 s; a following ~8k-token prompt did not finish, and the server ignored SIGINT and SIGTERM during it and needed SIGKILL (seen once).

@ashhart

ashhart commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Thanks @Arminova for the M1 Pro receipt. The load peak (13.63 GiB without a drafter) is the number the body was missing. The slow cold prompts show on main too, so this stack didn't cause them. The long prompt that ignored SIGTERM looks like the same whole-prompt pass, which has no cancel point on main either. We'll look at it with #1377.

@BlivionIaG, nothing has been pushed to #1390-#1394 since our last comments, so where things stand is unchanged. This PR's own changes are fine. What still blocks the stack:

Once that's in, we'll run these on the stack head: drafted against plain and concurrent against solo at 1 to 16 streams, the long-context cells, and alternated speed rounds against main.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

@ashhart Following up on your last checklist: the cumulative fixes are now pushed here, at 98f5cc72b. This PR targets main directly; the older intermediate PR branches have not been rewritten.

  • Rebuilt on main dd0a9cc5e, excluding the rejected lanes: a shared round drafts only while drafting measures faster (Model.shared_guard) #1379/lanes: a lone stream holds drafts only while its drafted rounds measure faster #1382 guard commits. Removed the shared/lone guards and their recipe descriptions, and restored the unrelated native/metal.zig comment.
  • Fixed Drafter.init failure cleanup ordering, added allocation-failure coverage, validated rank/theta bounds, and widened the quantizer arithmetic so 64 * bits does not overflow in Debug. BF16 is the default, 8-bit is opt-in, and 4-bit is refused. Corrected the recipe and included load-peak evidence.
  • Source 099455490 has the M5 drafted/plain and concurrent/solo, sampled/resumed-cache, cancellation/replacement, 8-bit, and 65,536/130,816-token equality checks. LFM kernel contracts: 11 pass, all 17 negative controls red. Independent drafter reference: 9/9; 20 steady rounds with zero host allocations/resizes. The new M4 kernel/reference checks pass, but memory pressure prevented repeating the served qualification there; older-chip receipts remain explicitly historical.
  • The 128-cell, four-rotation M5 performance comparison shows 16-stream gains of 10.96% code / 8.86% prose over the earlier PR, with four-stream losses of 1.93% / 0.42%. The task owner accepted that tradeoff. Candidate plain four-stream prose is also 2.04% below main; we are not claiming gains everywhere.

Receipts, reproducer and limitations.

The final allocation-only follow-up removes duplicate probability and held-grammar buffers without changing arithmetic or decoding policy. Its shared-lane/native checks pass (341 tests, three skipped), and the separate 64-cell M5 before/after comparison completed with matching token hashes and throughput changes between -0.45% and +0.68%. That comparison is additional evidence, not a repeat of the full hardware suite on the final commit.

All nine CI checks are green on 98f5cc72b, and GitHub reports no merge conflicts. Could you re-review this cumulative head and run your maintainer device cells when available?

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

@ashhart The intermediate branches are now rebuilt too, rather than leaving fixes only at the cumulative tip. All are on current main d33dcfc76, without the rejected #1379/#1382 ancestry:

No M5 was accessed for this restack. New M4 Pro / 48 GiB checks of the final integration:

  • Selected LFM2 contracts: 11 kernels pass, all 17 controls red, zero limit problems. Independent reference: 9/9 picks, worst confidence difference 0.00106; 20 steady rounds, zero host allocations/resizes.
  • Context 4096, parallel capacity 16: cached/resumed equals fresh, with 3517 reused tokens. Under Metal API validation, stress at 6 slots/60 s gives 120 completed +79 cancelled, and 16 slots/90 s gives 250 completed +132 cancelled; zero wrong replies and zero errors. All 29 sampled cases equal plain/solo/concurrent, and eight 8-bit cells at 1/4/8/16 streams across code/prose equal their plain and solo oracles. All five servers shut down gracefully, with no memory-pressure admission warning.
  • Each rebuilt head passes its scoped checks and global formatting; zero new lean findings. Final scoped checks: 355 passed / three skipped, 73/73 steps. Default build: 239/239 steps.

Reproduction, raw hashes/results, actual load peaks and limitations. CI is running on the new heads; these local results do not stand in for pending CI or your review.

The previously measured 1–2% slowdown at four/eight streams is still open against your performance requirement. Restacking does not fix or waive it, and the task owner’s accepted tradeoff is not presented as your approval. No new performance comparison or 65k/130k long-context run was made in this M4 replay; earlier receipts keep their original source scope. Could you re-review the propagated correctness/base fixes, with that remaining performance item kept explicit?

@ashhart

ashhart commented Oct 30, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG. A status note on this stack (#1381, #1390 to #1393 and this PR). We ran the default path, with no --drafter, on an M5 Ultra against main. It is clean: 0 of 20 prompt-test rows differ, prefill from 2k to 64k is within 0.1%, and the 1, 4, 8 and 16 stream sweeps are within 0.4%. What is left is on our side: the drafted cells (drafted replies equal to "draft": false and solo, and drafted tok/s at least the no-drafter tok/s at 4, 8 and 16 streams). They need the DSpark drafter on our test machine. We'll post those numbers here when they're done; nothing is needed from you for now.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Thanks @ashhart. Item 1 is back on 591833a39: run() in drafted_test.zig again asserts that the wrapped backend passes first_before_draft and penalties through from the target, for both modes. test-shared-lane-regressions-host passes 458 of 459 (one Windows-only skip); setting the target's penalties to false fails two of those tests, so the lines discriminate. zig fmt --check build.zig zig is clean and lean reports 386 problems, none new.

The QK-norm order experiment and the plain-main numbers are still open. The branch now conflicts with main in allocate.zig, depth.zig, shared_drafts.zig and windows.zig, from your bb267e639 (learned row cost below the table, least_rows); folding that into the hold-grid pricing needs care, so I haven't rebased yet.

@ashhart

ashhart commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG for putting the first_before_draft and penalties checks back on 591833a39, and for holding the rebase until the fold is right. Where it stands:

  1. Rebase onto today's main. The four conflicts (core/lanes/allocate.zig, depth.zig, shared_drafts.zig, windows.zig) all come from one main commit, bb267e6: a learned row cost may now fall below the table, and a learning round reaches the width it measures. The fold needs to keep three things:
    • HeadCost.least_rows: allocateDrafts doesn't price a round until it has least_rows rows, and a round that can't reach them takes every node on offer. Your restructured loop needs the same, on each pass of the outer loop.
    • rowTerm: a negative learned row cost is bounded so a round never prices below its streams' plain width. Your loop adds head.row_ms * (rows - live) directly; it should go through rowTerm. The same bound belongs in Base.forwardMs (hold_price.zig:149-152), which now carries what main's payHolds prices with @max(extra, plain - verify) (depth.zig).
    • The allocateDrafts callers in depth.zig, shared_drafts.zig and windows.zig pass .least_rows = row_cost.leastRows(...) beside .row_ms, as main's do.
      LFM2 doesn't learn row costs, so its numbers shouldn't move. GLM does, so main's two new tests in allocate.zig (the measuring round, and rows cheaper than the table) and the row_cost.zig tests need to pass on the folded loop.
  2. QK-norm versus RoPE order: still open. Serving the released drafter with each order and comparing accepted drafts per round settles it.
  3. Numbers: still open. Plain main against this branch without --drafter at 1 and 4 streams (token_sha and tok/s), plus the M4 Pro run with --drafter set matching plain.

CI hasn't run on 591833a39 because the branch doesn't merge with main. It will run once the branch is rebased.

  • Synthena (Ash's Agent)

BlivionIaG added a commit to BlivionIaG/TensorFold that referenced this pull request Nov 12, 2026
@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Rebased onto main 3885dceb2 (11 commits, linear; GitHub now shows it mergeable): 9bc1d1f0e. Two commits conflicted, both in the lane scheduler against your bb267e639:

  • allocate.zig: HeadCost keeps the hold fields and gains least_rows. allocateDrafts keeps the hold-grid and mixed-host structure and applies your rules inside it: a measuring round reaches least_rows while the grid price still accumulates, every row term goes through rowTerm, and the out-of-reach case takes every node. With no hold costs, no host quotes and no least_rows it reduces to main's loop. Your two tests are kept as they were.
  • payHolds: it now runs through hold_offers.pay, so your bound (a learned fall never prices a round under its plain width) moved into hold_price.Base.forwardMs, which every hold-pricing path calls. hold_price.canExplore uses the same bound.
  • depth.zig, shared_drafts.zig, windows.zig: least_rows is passed wherever a HeadCost is built. Commit 2 renamed live to cost_streams, which extraMs and leastRows now take.
  • New tests: hold_row_cost_test.zig covers holds combined with a learned fall and with least_rows; the unbounded term would give {1,0} where the bound gives {2,2}.

zig fmt is clean, lean reports 345 problems on the branch and on main, and the problem sets are identical. test-shared-lane-regressions-host passes 473 of 474 (one Windows-only skip; main is 360), test_lanes.sh passes, test-lfm2-dspark passes 93 with 2 skipped, and the server goldens match (322 of 335, no unexpected). The full zig build test has one failure, dashboard_test ("Paged prefix cache"), which fails the same way on main: its string is in the test and not in dashboard.html. The QK-norm experiment and the plain-main numbers are still open.

BlivionIaG pushed a commit to BlivionIaG/TensorFold that referenced this pull request Nov 12, 2026
@ashhart

ashhart commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG for the rebase onto 3885dceb2 and for holding it until the fold was right. CI is green on 225d192811. We read the fold against main's bb267e639, and it keeps all three rules:

  • least_rows: allocate.zig:113-114 and :164-165 skip pricing until a round reaches least_rows, on every pass of the outer loop, and :175 takes every node when the width is out of reach.
  • rowTerm: every rate in allocateDrafts goes through it, and hold_price.zig:151-153 (Base.forwardMs) applies the same bound, so payHolds through hold_offers.pay and canExplore never price a round under its plain width.
  • The callers: depth.zig:349-353, shared_drafts.zig:72-73 and windows.zig:347 pass least_rows beside row_ms. Without priced holds, cost_streams is the stream count, so GLM's rounds price as main's do.

You're right about dashboard_test. Main's #2655 added "Paged prefix cache" to the test but not to dashboard.html. That's ours to fix on main, not this PR's.

Two things are still open, both as before:

  1. QK-norm versus RoPE order. Serve the released drafter with each order and compare accepted drafts per round.
  2. Plain numbers on your M4 Pro. Main against this branch without --drafter at 1 and 4 streams (token_sha and tok/s), and the same run with --drafter set, which should match plain.

Once those are in, we'll run our own cells on this head. We'll fold the two CHANGELOG lines into one at landing.

  • Synthena (Ash's Agent)

Linearize the remaining reviewed DSpark series onto main 21ea8de, preserving target-owned context, confidence and physical hold pricing, pre-M5 stand-aside and the existing correctness coverage. Keep current main's native fixtures and both CLI audit modules. The prior reviewed history remains preserved at 7513f89.
…m and fused BF16 GLU rows

lfm2_combine_sorted_norm and lfm2_combine_sorted_norm_tap are checked bit for bit against lfm2_combine_sorted, lfm2_norm and the tap copy on order-sensitive picks stored by expert; lfm2_bf16_glu_rows against two lfm2_bf16_rows projections and lfm2_swiglu. Each has source-edit controls that must fail the shapes gate.
BlivionIaG added a commit to BlivionIaG/TensorFold that referenced this pull request Nov 13, 2026
BlivionIaG pushed a commit to BlivionIaG/TensorFold that referenced this pull request Nov 13, 2026
@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Rebased again, now 4f5467761 on main 0fed2e4a5: Bizuayeu's floor_rates change (682c82d27) conflicted in depth.zig and depth_constants.zig, so both rules are kept beside the hold-grid pricing. Lean is 341 on the head and on main (same set), and the full suite passes except dashboard_test, which fails on main too.

I ran the Metal kernel contract on the M4 Pro (tf-kernel-contract --only lfm2). It found one real failure: gate 3 on the BF16 GLU rows, because the stress sum was negative, SiLU flattened a wrong sum to zero, and fp16 sums missed by 0.4x. ebe359ca9 keeps the sum positive and declares the 72.6x margin that gives. Now: SUMMARY 19 kernels: 19 pass gates 1-4, 0 fail; 32 controls: 32 red as planted, 0 missed; 0 limit problems.

Plain main (3885dceb2) against the branch, no --drafter, 1 and 4 streams (--alone --serial): the code and chat token_sha256 are the same in every arm, and with --drafter the server says it stands aside on the M4 Pro and the hashes are the same. 1-stream speed is within -1.1% and +2.3%. My 4-stream cells split into two speed modes in both builds while other builds ran on the machine, so I'm not claiming a 4-stream speed.

The QK-norm against RoPE order check and the M5 cells need the neural drafter, which runs on an M5.

@BlivionIaG

BlivionIaG commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Commit identity fixed on all 13 commits: author and committer are now BlivionIaG <15323295+BlivionIaG@users.noreply.github.com>. Two commits and every committer line had a malformed noreply address from a local git config. Trees and dates are unchanged, so the content is the same as before; only the hashes moved. The head is now 22b19797e (it was 4f5467761), so use it for any run.

@ashhart

ashhart commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG. The identity correction, restored assertions and M4 kernel-contract result are accounted for. The plain hashes and named stand-aside check help too; thank you for leaving the noisy four-stream speed unclaimed.

Two evidence gaps remain: the released drafter’s QK-norm/RoPE comparison, and a quiet four-stream main/branch timing. The M5 drafter matrix also remains open. Current main now conflicts in zig/src/native/metal.zig, so that merge needs updating before the final checks. Green host CI alone cannot clear these device gates.

  • Synthena (Ash's Agent)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants