Repository navigation
lfm2: DSpark drafting with owned context and confidence scheduling - #1394
BlivionIaG wants to merge 13 commits into
Conversation
ashhart
left a comment
There was a problem hiding this comment.
Thanks @BlivionIaG. This builds on #1391, so it inherits that review's blockers (runtime lossy BF16 conversion, the 4-bit buffer mismatch, dropped async failure flags, the contract gaps). Its concurrent encoder and shared stage flush also need device race, resume and replacement-equality checks.
|
Thanks @ashhart. Device checks on an M4 Pro at the stack (details in the body, head |
|
An M3 receipt for this head, since the body says no run covers M1, M2 or M3. M3 Pro (36 GB), macOS 26.7.1, Zig 0.17.0, head
|
Thanks for the run, now we have proof it works |
|
Thanks @BlivionIaG, and thanks to @jschmied for the M3 Pro receipt. The concurrent hold encoder puts a barrier after This PR's own changes look ready. It is blocked only by what it inherits from #1391 and #1392: the guards, the |
|
An M1 receipt for this head, for the load-time memory the body lists as not measured. M1 Pro (32 GB, MacBookPro18,1), macOS 27.0.1, Xcode 27.1 with Metal Toolchain 27A266a, Zig 0.17.0, head 2e5966f: zig build native -Dcpu=apple_m1: built. |
|
An M1 receipt for this head: the load-time memory the body lists as not measured, and a slow-prompt finding on M1 that main has too. M1 Pro (32 GB, MacBookPro18,1), macOS 27.0.1, Xcode 27.1 with Metal Toolchain 27A266a, Zig 0.17.0, Load-time memory (head
Cold prompts are slow on M1, on this head and on main
|
|
Thanks @Arminova for the M1 Pro receipt. The load peak (13.63 GiB without a drafter) is the number the body was missing. The slow cold prompts show on main too, so this stack didn't cause them. The long prompt that ignored SIGTERM looks like the same whole-prompt pass, which has no cancel point on main either. We'll look at it with #1377. @BlivionIaG, nothing has been pushed to #1390-#1394 since our last comments, so where things stand is unchanged. This PR's own changes are fine. What still blocks the stack:
Once that's in, we'll run these on the stack head: drafted against plain and concurrent against solo at 1 to 16 streams, the long-context cells, and alternated speed rounds against main. |
2e5966f to
017919f
Compare
|
@ashhart Following up on your last checklist: the cumulative fixes are now pushed here, at
Receipts, reproducer and limitations. The final allocation-only follow-up removes duplicate probability and held-grammar buffers without changing arithmetic or decoding policy. Its shared-lane/native checks pass (341 tests, three skipped), and the separate 64-cell M5 before/after comparison completed with matching token hashes and throughput changes between -0.45% and +0.68%. That comparison is additional evidence, not a repeat of the full hardware suite on the final commit. All nine CI checks are green on |
98f5cc7 to
f9de08d
Compare
|
@ashhart The intermediate branches are now rebuilt too, rather than leaving fixes only at the cumulative tip. All are on current main
No M5 was accessed for this restack. New M4 Pro / 48 GiB checks of the final integration:
Reproduction, raw hashes/results, actual load peaks and limitations. CI is running on the new heads; these local results do not stand in for pending CI or your review. The previously measured 1–2% slowdown at four/eight streams is still open against your performance requirement. Restacking does not fix or waive it, and the task owner’s accepted tradeoff is not presented as your approval. No new performance comparison or 65k/130k long-context run was made in this M4 replay; earlier receipts keep their original source scope. Could you re-review the propagated correctness/base fixes, with that remaining performance item kept explicit? |
|
Thanks @BlivionIaG. A status note on this stack (#1381, #1390 to #1393 and this PR). We ran the default path, with no |
|
Thanks @ashhart. Item 1 is back on The QK-norm order experiment and the plain-main numbers are still open. The branch now conflicts with main in |
|
Thanks @BlivionIaG for putting the
CI hasn't run on
|
591833a to
9bc1d1f
Compare
|
Rebased onto main
|
|
Thanks @BlivionIaG for the rebase onto
You're right about Two things are still open, both as before:
Once those are in, we'll run our own cells on this head. We'll fold the two CHANGELOG lines into one at landing.
|
Linearize the remaining reviewed DSpark series onto main 21ea8de, preserving target-owned context, confidence and physical hold pricing, pre-M5 stand-aside and the existing correctness coverage. Keep current main's native fixtures and both CLI audit modules. The prior reviewed history remains preserved at 7513f89.
…m and fused BF16 GLU rows lfm2_combine_sorted_norm and lfm2_combine_sorted_norm_tap are checked bit for bit against lfm2_combine_sorted, lfm2_norm and the tap copy on order-sensitive picks stored by expert; lfm2_bf16_glu_rows against two lfm2_bf16_rows projections and lfm2_swiglu. Each has source-edit controls that must fail the shapes gate.
…GLU contract comment
…d states its fp16 margin
ebe359c to
4f54677
Compare
|
Rebased again, now I ran the Metal kernel contract on the M4 Pro ( Plain main ( The QK-norm against RoPE order check and the M5 cells need the neural drafter, which runs on an M5. |
4f54677 to
22b1979
Compare
|
Commit identity fixed on all 13 commits: author and committer are now |
|
Thanks @BlivionIaG. The identity correction, restored assertions and M4 kernel-contract result are accounted for. The plain hashes and named stand-aside check help too; thank you for leaving the noisy four-stream speed unclaimed. Two evidence gaps remain: the released drafter’s QK-norm/RoPE comparison, and a quiet four-stream main/branch timing. The M5 drafter matrix also remains open. Current main now conflicts in
|
What this changes
DSpark serving through the shared LFM2 lanes, with neural serving standing aside before loading on pre-M5/unknown Metal devices. Rebased onto current main; the review's restored tests, the kernel-contract run and the plain main-versus-branch cells are below.
0fed2e4a5fcc03f109cfaaf8eda10734cce21fb7; adds 13 commits. Rebases onto main resolved conflicts inzig/src/core/lanes/{allocate,depth,depth_constants,shared_drafts,windows}.zig(the lane scheduler'sleast_rowsandfloor_ratesrules kept beside the hold-grid pricing) and a CHANGELOG merge.atc overlap --json: ATC0.9.6 was run against upstream with --base fb8fdf3; complete=false (myPullRequestsLimit, otherPullRequestsLimit, pullFilesLimit, pullFilesUnavailable, timeLimit). Supplemental complete snapshot:217foreign open heads,3038 exact local merge comparisons including latest Main ff742df; existing reviewed overlaps are retained, not claimed globally clean. The avoidable added environment-test collision with CUDA (GB10): a second carveout source, spark-attic's atticd, opt-in #2762 was removed by preserving Main's existing test name, without importing its unmerged feature.Checks on this head
22b19797ec1048bb2f50e307aae3339077ba68be.python3 tools/zig/lean_check.py): head 341, main 341, same problem set, 0 new. No touched.zigfile is over 600 lines;zig fmt --check build.zig zigis clean.zig build test test-golden dev-tools -Dcpu=apple_m1: 12,584 of 12,687 tests pass, 102 skipped, 1 failed. The failure isdashboard_test("Paged prefix cache"), which fails the same way on main.zig build test-shared-lane-regressions-host: 476 passed, 1 skipped (Windows-only). On the previous head,test-lfm2-dsparkpassed 93 with 2 skipped and the server goldens matched (322 of 335, 0 unexpected).Receipt
--drafter(Apple M4 Pro,LiquidAI/LFM2.5-24B-A2B-MLX-4bit,--parallel 4 --context 4096,bench_concurrent.py --levels 1,4 --tokens 256 --temperatures 0 --alone --serial; main3885dceb2against the branch before its last rebase): the code cell'stoken_sha256isf66e3fd7e509and the chat cell's57752ac6d7f0in every arm and stream count, each equal to its own run alone and to"draft": false. With--drafter, the server prints "DSpark standing aside on Apple M4 Pro" and the same hashes result at 1 and 4 streams. Speed, 1 stream, mean of three interleaved runs: code 130.9 main, 129.4 branch (-1.1%); chat 118.6 main, 121.3 branch (+2.3%). The 4-stream cells split into two modes in both builds (about 112-125 tok/s aggregate or about 230) while other builds were running on the machine, so no 4-stream speed claim is made. The QK-norm against RoPE order check needs the neural drafter, which runs on an M5.22b19797ec1048bb2f50e307aae3339077ba68be; Zig 0.17.0; Apple M4 Pro, 48 GiB, Metal;LiquidAI/LFM2.5-24B-A2B-MLX-4bitwith the DSpark drafter given to--drafter.--drafterstanding aside, at 1 and 4 streams.Remaining maintainer requests
The M5 no-losing-cell matrix (plain and drafted), the QK-norm against RoPE order check with the released drafter in both orders, and the maintainer's own main-against-branch run.
Kernel contract
tf-kernel-contract --only lfm2on an Apple M4 Pro: 32 controls, 32 red as planted, 0 missed.SUMMARY 19 kernels: 19 pass gates 1-4, 0 fail, 0 known findings; 32 controls: 32 red as planted, 0 missed; 0 limit problems, exit 0. The first run failed gate 3 on the BF16 GLU rows (the gate's stress sum was negative, so SiLU flattened a wrong sum to zero, and the fp16 margin was 0.4x);ebe359ca9keeps that sum positive and declares the 72.6x margin that results (f16 = 50).Checklist
Title/description, head and stack agree; older results remain source-bound.
Every new/restored test has a failing-before runtime receipt across all reached platforms. Host evidence is above; missing device gates are not waived.
No newly added overlong/multiline source comments, measurements or review history; inherited findings are reported above.
Complete publication trees/outgoing histories reviewed for public noreply identity, tool trailers and privacy; synthetic detector fixtures and mandatory third-party public contacts are not live secrets. No package/binary upload proposed.
All inference-path/lane-round gates qualified on this head.
Drafted=plain, restored=fresh, concurrent=solo on required devices/settings.
Required precision/reference evidence on this exact head complete.
Decode/prompt no slower than Main at the required widths, greedy and sampled.
Provenance and review history moved from the recipe
Provenance (was the paragraph after the controller receipts)
The checkpoint's
block_size: 9establishes its query geometry, not exclusive implementation provenance.Later diagnosis consulted SGLang's DSpark/DFlash implementation, including revision
488869c2d(Apache-2.0), for attention, projection/normalization, Markov and confidence semantics.
The historical diagnosis
remains outside this implementation branch. The October 25 authoring record has not been recovered:
checkpoint-only or clean-room provenance cannot be certified, and the maintainer's provenance gate remains open.
Independent paper/config re-derivation (November 10; it does not erase the consultation above):
(The bulleted paper/config facts that followed are retained in the recipe as plain model facts, without this framing.)
Receipt links (review history, removed with their one-sentence descriptions)
records actual cache, sampled and cancellation workflows on both devices; it is not a performance pass.
queued serving and its limits: correctly declining a known cohort is not a complete performance pass.
record the constructed timing cases and source hashes; they are not device-throughput qualification.
Open item kept in the recipe
QK-norm's position relative to RoPE is not fixed by the papers or config and is unmeasured against the released
drafter. The maintainer's suggested check: serve the released drafter with each order and compare accepted drafts
per round; the order the weights were trained with should be clearly ahead. Not run; no outcome is claimed.
Historical source-bound notes — not current-head qualification
Earlier commands/measurements retain their original commit/binary scope. Any earlier readiness or provenance claim is superseded by the open gates above.
Current published head (supersedes the stale head/base/qualification claims below)
616aa4b3c91f4bed6f299a65ab9e6d28ea5ad9fa, rebased onto pinned upstream Main8a7838a65a1fa1a44e0136a2cccd8bfcc8c0a8b5. PR base remainsmain. The50e538bd9head on Main6e9af930and the older7d171cc73/28760ea25/51cea13ffstack, the36f84936merge base and the CI/receipt sections below are historical, not current-head results.b552696721bbf41f8c34db4e59a454f2136f1f4ccompleted17/17steps,606passed/two skipped of608, plus native CUDA host compile. Current tests-only616aa4b3c91f4bed6f299a65ab9e6d28ea5ad9faadditionally ran the corrected XPU/shard suite:9/9steps,7/7tests. The older606-test execution is not relabelled as a new run.Provenance (recipe
docs/recipes/lfm2.5-24b-a2b.mdon this head; limits kept)e052e5b1dd103c38cdd4f8ddc068d7dea4586f55: pinned config/model header confirms 9 query rows (anchor + 8 masks), 5 full-attention layers, rank 256; the original author record is unavailable.488869c2d); the original authoring record is unrecovered. No clean-room claim is made. Other engines were not inspected again to close this. A black-box run of the external model card's SGLang setup needs CUDA/flashinfer; Mac CPU is not a substitute.What this changes
Adds LFM2 DSpark drafting with owned context and confidence-priced holds; the review fix prevents drafts-off legacy enrollment and scopes DSpark scheduling changes to configured hold costs. Out of the landing queue pending the cross-family checks in the cross-family maintainer gate; local author checks and historical LFM2 numbers do not clear that gate.
main; linear feature commit51cea13ff1035e7a9d1101d196fbe8339eba0284, then compatibility fix28760ea25, then shared-dispatch follow-up7d171cc73. No new merge commit or force-push; measured/history sources remain accessible.atc overlap --json(historical check at28760ea25): ran against the main merge base36f849362700d0d8c4c9ec9e68b7b28e5c6e74c8. Incomplete, not clean:myPullRequestsLimit,pullFilesLimit,pullFilesUnavailable,timeLimit. Manual fallback fully paginated the file lists of 189 open PRs at pinned heads. Only hip: Qwen3.8-Flash-Next on gfx1151, a reference branch for review (to be split) #1667, EXL3 Flash Next: draft lane guard for offloaded experts (8/8) #1446 and MiMo-V2.6-Flash on CUDA, on two GB10s with its DFlash drafter #728 share the six primary lane source paths. Existing peer conflicts remain; this fix adds two conflict paths against EXL3 Flash Next: draft lane guard for offloaded experts (8/8) #1446 (engine.zig,lanes.zig), with none added against hip: Qwen3.8-Flash-Next on gfx1151, a reference branch for review (to be split) #1667 or MiMo-V2.6-Flash on CUDA, on two GB10s with its DFlash drafter #728. These are disclosed integration work, not resolved/green coordination. No other feature branch was edited.Pinned peer heads for the conflict comparison: #1667
2f42c9cb6a64e222a57201db4d9c55b18d1c1c6c; #14466d2666186de117a4216ec38b784f12ae9e00a5e6; #728cbee96430d7b4760781f2f52ecef4d86fd5927ea.Answers to the six review items
The compatibility boundary is
k.hold_costs.len > 0, not model-family names. Non-costed callers retain main's scheduling; the drafts-off enrollment fix applies universally.enrollabsorbs a drafts-off prompthead_context = falsebefore hook selection when!s.drafts; do not open or absorb a headwhocallerThe
whosignature migration is complete, without an alias/shim. Existing costed reservation/pricing fixtures now opt into a hold grid explicitly; their pricing assertions are retained. The obsolete test requiring feature capture for a drafts-off prompt was replaced by the three target-only regressions above. No shader, activation format, sum, target verification or precision threshold changed in this review fix.Latest review follow-up —
7d171cc73Addresses the two remaining requests, without a merge/rebase, force-push, shader change or precision change:
roundStreamsretains main'sshared_drafts.budgetspath when shared allocation is enabled. The new regression fails before the one-line correction (expected 0, found 7). Afterward, the repository scheduler suite passes 440 tests, one skipped; the changed code has no lean findings. A standalone production-engine consumer exercises all eight priced/limited/granted combinations for seven rounds each, with the expected probe cadence and zero draft calls. The deterministic fixture backend is not a device/performance measurement.36f849362700d0d8c4c9ec9e68b7b28e5c6e74c8, and ran them on the same M4 Pro / 48 GiB.tf-qwen27-dflash-served-checkexits 1 withServedDraftDiffers;tf-qwen27-lanes-checkexits 1 withUntimedDraftWindow. These are actual-main results, not the earlier51cea13fffeature-parent comparison. Both use generated five-layer model fixtures, not trained weights. Source-bound notes and both binary SHA-256 values.The baseline errors are not passing probes, and this follow-up does not clear the outstanding cross-family device/performance gate. The older coordination comparison above remains historical/incomplete; no new zero-conflict claim is made. Automatic CI runs from the normal fast-forward push; no manual rerun or workflow change.
Automatic CI on
7d171cc73All eleven automatic checks succeeded, including zig, zig-windows and zig-macos-dev-tools. These build/format/platform checks do not clear the cross-family device/performance gate above. No manual CI rerun or workflow change.
Checks on compatibility head
28760ea25(historical)28760ea25f17c0f6bc4490af5139670f107b48ea; parent51cea13ff1035e7a9d1101d196fbe8339eba0284.python3 tools/zig/lean_check.py: exact-head Linux CI reports 325 on the base / 325 on the head / 0 new, using the head checker copied to the base. Separately, the committed local checkout reports 425 findings / 5,577 files, exit 1. These are not globally clean runs; the zero-new verdict uses the within-CI same-checker comparison, not a comparison between different checkouts/environments.zig fmt --check build.zig zigZIG=zig bash tools/zig/test_lanes.shzig build dev-toolstf-qwen27-dflash-served-checkexits 1 withServedDraftDiffers;tf-qwen27-lanes-checkexits 1 withUntimedDraftWindow. Both exact errors also occur when separately built/run from published51cea13ff. These are retained failures, not new-head exactness evidence or a diagnosis of their causeUsing Zig 0.17.0, with Zig on
PATH:zig fmt --check build.zig zig ZIG=zig bash tools/zig/test_lanes.sh zig build native test test-golden test-kernel-contract fuzz-cortheon test-server-cpu test-native test-lfm2 test-lfm2-dspark test-draft-metal-host test-drafter-skip-metal-host test-qwen27-shared-allocation-host -Dcpu=apple_m1 -Dversion=1.2.3 -j1 --summary all zig build -Dcpu=apple_m1 -Dversion=1.2.3 -j1 --summary all zig build dev-tools -Dcpu=apple_m1 -Dversion=1.2.3 -j1 --summary allhead_sha=28760ea25f17c0f6bc4490af5139670f107b48ea; every job conclusion issuccess. This is host/build CI, not device/performance qualification.draft:falseon GB10 and M5 Ultra; the pre-M5 Mac cell; Flash Next 2/4 streams; new-head full-model exactness, decode and cold-prompt performance. The generated Qwen probes above do not replace these cells.Receipt
28760ea25f17c0f6bc4490af5139670f107b48ea, Zig 0.17.0, native Metal, Apple M4 Pro / 48 GiB,-Dcpu=apple_m1 -Dversion=1.2.3. Native binary SHA-25604fe1fa3993d6d0daacdc2cefd4399f45fd445517f9684b56763ff07cd71df1c; unchanged by the final changelog-only commit step.capabilities --jsonreports enginezig, chipapple-m4, backendmetal.TensorFold/Qwen3.8-27B-MLX-4bitrevision22d8d538154e0e5b6f8dcb1fcc74a60a0798104c;z-lab/Qwen3.8-27B-DFlash2revision50307d4c4cde6860d4eee73e2547cd786fe8e8a4. Cached complete snapshots, no checkpoint substitution.TargetBinding, without requests. No limits were raised and no pressured run is called a smoke pass. Generated tiny native probes were then run and failed as recorded above, also on the separately built published head.token_shaorbench_concurrent.py --alone --serialqualification. Host regressions check the changed transitions and returned tokens; generated-probe failures and aborted full-checkpoint starts are not parity passes. Historical LFM2 exactness stays bound to its measured source below.ttft_sbefore/after: not qualified; this change reaches prefill and scheduling, so this is not an n/a claim. No request was issued under memory pressure.Kernel contract — historical feature receipts, no kernel change in this review fix
28760ea25: none. The overall feature's source-bound report and raw-evidence index and archived reports remain linked below; retaining them does not claim new-head GPU-contract qualification.SUMMARYon an Apple M4 Pro are in the section above; the head and the chips differ from these historical receipts.Checklist
PATH.Retained model and performance evidence — historical sources, not this head
The complete source-bound report and raw-evidence index remain the qualification record. The new linear head was not given a fresh model/performance label merely because the history changed.
ca54e6889ed8ed00400bfeb4fa1be30a3a687de57513f89d3ecc116b986d22c0b6b5a094a2bb2c00--drafteron the baseline side; comparison used plain runs. This repeat did not hash drafted against plain outputs.ca54e6889ca54e6889The separate plain/plain A/A run is diagnostic, not a waiver or a proven measurement-error bound: 20/48 paired runs changed internal work. The retained reports distinguish strict oracle failures, consumer correctness, and non-zero FP32-head differences. The acceptance gate is no slower than plain decode in every cell; no percentage regression allowance applies. Historical pre-M5/M4 diagnostic failures are not relabelled as fixed or as new-head qualification. No new M3/M4 full-model, CUDA, or Windows runtime qualification is claimed by this refresh.
Earlier source-bound reports, diagnoses, and CI receipts (historical, not current-head results)
Full Linux/Windows CI found that the wrong-slot prompt-feature test enrolled a lane without marking its drafter context open. The tests-only correction models an opened context and preserves the original
FeaturesGoneownership assertion; it does not repin the expected error or alter production admission checks. Actualtest-draft-xpu-hostnow passes9/9steps,7/7tests. Current-source automatic CI is complete:12successful checks,2scheduled-only skips. Device/provenance gates remain open.Terminal current-source CI at616aa4b3:12successful checks,2scheduled-only skips,0failures. Linux25m26s,Windows29m28s,macOSdev-tools17m10s,pytest7m4s; app/mobile/format and remaining platform checks passed. Source stays frozen. No new device controls, plain/drafted/no-losing-cell measurement or independent normalization-order/provenance evidence is inferred.