Skip to content

fix(dsv41): retain complete packed prefix backing in prompt snapshots - #56

Draft
beastllama wants to merge 1 commit into
bertholomus:deepseek-v41-zig-tp2from
beastllama:fix/dsv41-independent-prefix-snapshots
Draft

beastllama wants to merge 1 commit into
bertholomus:deepseek-v41-zig-tp2from
beastllama:fix/dsv41-independent-prefix-snapshots

Conversation

@beastllama

Copy link
Copy Markdown

Root cause: DeepSeek prompt snapshots retain fixed slot state while their packed prefix planes still depend on reusable working-pool rows. Evidence: the public e100ee8 source plus the CPU overwrite regression restores 0xa5 overwritten bytes where the saved first packed byte is 0x00; the same regression passes with this change.

Each production snapshot now owns one complete device buffer per rank: the existing window/raw state followed by initialized packed KV/index code and scale planes. Its position-dependent byte count is charged through the existing Store callback before allocation. The cache budget and working-pool capacity are unchanged; this does not make additional active long reservations fit.

Backed snapshots remain eligible after physical overwrite, do not pin old working extents, and restore from their independent buffer for both same-base and relocated resumes. The public refusal, snapResumesFn, and snapLiveFn APIs retain their existing replay/resume-grid rules. Resident-only synthetic placement controls remain available with backed=false; production saves always use backed=true.

Save/restore frames carry ID, slot, base, position, and full byte length. Followers validate the complete view and charge and reject inconsistent records. Local allocation/copy failures clean up before peer dispatch and before pointer publication; uncertain delivery and follower snapshot failures terminate the rank so the existing peer watch stops the pair. This is not an acknowledged fail-soft protocol for asymmetric follower OOM.

All participating head/follower binaries must come from the same updated build. Snapshot frame payloads changed under the existing opcodes; mixed old/new binaries are unsupported.

This is a separate focused native backing port. It does not modify draft #54 (cache reservation) or #55 (per-job ContextFull backoff). The private selected source also has a preceding cached-cut-in guard, but this public base has no cut_in_limit scheduler API; this port does not introduce that separate private scheduler feature. Independent backing removes the production source-residency dependency directly.

Validation on installed Zig 0.17.0, ReleaseSafe, CPU-only:

  • Public baseline: overwrite regression fails on the first packed byte (0x00 expected, 0xa5 found).
  • Candidate: five distinct focused tests pass (two model tests are imported/repeated by the lanes root). They cover same-base/relocated restoration, two modeled-rank byte patterns, ratio-one/two indexed/non-indexed layouts, fixed raw/ring state, untouched tails, geometry/byte accounting, cache eligibility after overwrite, real live-pool exhaustion, frame roundtrip/truncation, and local save-refusal cleanup without publishing a dangling pointer or consuming a wire step.
  • Six distinct existing CPU controls pass, including four resident-only placement/shared-prefix controls, the stepped-prefill control, and its imported attention-pick control. Repeated matches are counted once; the unmatched a state filter ran zero tests and is excluded.
  • Full DeepSeek-family semantic compilation passes with the dsv41.zig test root and -fno-emit-bin. This is compilation, not execution of the full family suite.
  • zig build native -Doptimize=safe -j2 --summary all fails before source compilation with the existing WSL checking cache failed: manifest_stat Unexpected limitation. No full native/GPU build pass is claimed and no repeat was attempted.
  • git diff --check passes. zig fmt --check fails on both untouched public preimages and candidate files; existing unrelated formatting is preserved.

No production process, model input, weights, precision, context, slot count, sampling, or configuration changed during this port. Saved private deployment performance measurements are not qualification of this public branch. GPU copy ordering/parity, real peer failure handling, and end-to-end pressure behavior still require native testing; this is submitted as a draft for review.

@beastllama

Copy link
Copy Markdown
Author

Follow-up from our two-Spark deployment: the v0.3.0 recipe preview includes a separate recency fix for the compact Store in our pinned stack. An older covering snapshot could make a recently reused short checkpoint a preferred eviction victim. The recipe patch adds a last-use check and keeps the same 1 GiB snapshot budget.

In one SparkDash retention sequence, all ten recalls passed. After two budget evictions, the short replay reused 75,776 of 76,048 input tokens and reached first token in 473 ms. A separate 32K recall control passed 16/16 with zero stale values. These are scoped results from the published setup.

The packed-prefix backing change in this PR is unchanged. The newer upstream cache policy needs its own reproduction before a recency port. Thanks to the TensorFold contributors, MiaAI-Lab, and the SparkDash contributors whose work this builds on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant