gemma-4-12b-it (UD-Q2_K_XL): spec decode and plain decode part at one token on a counting prompt #901

Before you start

• I have read the FAQ and my problem is not answered there.
• I have read the Roadmap and this is not already planned there.
• I have searched existing issues and found no duplicate.
• I am on the latest release, or on a freshly rebuilt main when building from source.

What happened

The prompt "Write the numbers from 1 to 300 in order, separated by commas, and nothing more."
(35 tokens in, 1024 out, temperature 0) served from unsloth/gemma-4-12b-it-GGUF
(gemma-4-12b-it-UD-Q2_K_XL.gguf) with the mtp-gemma-4-12b-it-BF16.gguf assistant:

• plain decode and --spec-algo mtp diverge at exactly one token: plain emits
  ... 183, 184, 115, 186 ..., spec emits ... 183, 184, 185, 186 ... (first differing byte 813
  of the streamed text; the fork is at token-id level, not detokenization)
• both runs are 1024 tokens and identical everywhere else; 24/25 prompts on the same suite are
  byte-identical between the two modes
• reproducible: seen in recorded runs from Nov 10 and again on the PR feat(gemma4): speculative decoding with the assistant draft #895 head (6ce1992, merged
  as 58b3ad7); the plain baseline itself is stable (two independent plain runs byte-identical
  25/25)
• not universal: E4B (BF16) and 26B-A4B spec runs are byte-identical to their own plain
  baselines on all 25 prompts - only this 12B quant shows the split
• the odd one out may be plain, not spec: the correct continuation of the sequence is 185, and
  the recorded 26B-A4B plain run also emits 185 there. This does not prove which side is
  canonical (different model), but it inverts the usual "the draft corrupted it" reading

Suspicion: an fp16 near-tie at the fork token where the verify-row and decode-row logits
(GEMM/attention shapes differ) order two candidates differently - the same rounding-tie class as
#649, handled for the DS-FP4 transcription check in #668. The fork sits around absolute position
~844, inside the 1024-token sliding window, so the out-of-window free is not active there.

Experiments that would decide it (none run yet):

1. batch-width probe: plain server with --max-running-requests 4, four concurrent copies of the prompt
   • if plain's own output changes with batch shape, plain is not a canonical reference
2. truncated-prefix continuation at the fork on both modes (confirming the prefix re-tokenizes
   to the same context first)
3. --spec-num-steps 1 on spec - if the divergence disappears, the mechanism is in multi-step
   verify row construction
4. log the top-2 logit gap of the fork token in both modes

Deferred here from PR #895 per review ("the 12B tie is unrelated; a separate issue is fine").

How did you install FreeToken

Built from source

FreeToken version

58b3ad7 (merge of PR #895); reproduced on its head 6ce1992 before the squash. The Nov 10 recordings predate the branch.

OS

Ubuntu

OS details

Ubuntu, kernel 6.8.0-136-generic, Python 3.10.19, torch 2.14.1+cu130.

GPU and driver

RTX 5090 32GB (GPU 0, the serving GPU), driver 580.95.05. Second GPU in the machine: RTX 5060 Ti 16GB, idle during the runs.

CPU and system RAM

Ryzen 9 7950X3D, 96 GiB.

Checkpoint

• target: https://huggingface.co/unsloth/gemma-4-12b-it-GGUF - file gemma-4-12b-it-UD-Q2_K_XL.gguf - draft: the mtp-gemma-4-12b-it-BF16.gguf sidecar shipped alongside it (unsloth MTP/ release; upstream equivalent is google/gemma-4-12B-it-assistant after convert_hf_to_gguf.py)

Command

Spec run (plain is the same minus the two spec flags):


CUDA_VISIBLE_DEVICES=0 ft serve \
  --model-path <download-dir>/gemma-4-12b-it-UD-Q2_K_XL.gguf \
  --port 10101 --text-model-only --max-running-requests 1 --enable-timings-report \
  --num-tokens 16384 --served-model-name 12b \
  --spec-algo mtp \
  --spec-draft-model <download-dir>/mtp-gemma-4-12b-it-BF16.gguf


then one greedy completion, `"max_tokens": 1024, "temperature": 0.0`, of the prompt above.

Full log

Diff from the 25-prompt comparison (spec on merged `58b3ad7` vs the recorded plain baseline):


the counting prompt: len 1024 vs 1024, first diff at byte 813
      spec  : '79, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 1'
      plain : '79, 180, 181, 182, 183, 184, 115, 186, 187, 188, 189, 190, 1'


Per-mode logits/decode logs: the boot and streaming logs of both runs (attached). The
plain-vs-plain and cross-model comparisons quoted above come from the recorded baseline runs;
their raw per-prompt outputs are available on request.

Anything else

Related, same fp16/bf16 rounding-tie class: #649 (DS-FP4 transcription check failing on a
rounding tie) and #668 (the tie-budget fix; still open). If this is confirmed as a near-tie, the
resolution is presumably documentation or a tie-aware check rather than a correctness fix - both
outputs may be legitimate roundings of the same underlying logits.