A local-inference tok/s number is not a decode rate.
If any speculative decoding is active, your throughput depends on how much of the output was already in the prompt. Same model, same box, same flag:
199 tok/s and 88 tok/s are the same 117B MoE, same weights, same serving
flags, matched output lengths. The only difference is whether the answer was
copyable from context.
Quote the first number and you are advertising your speculator. Quote the second and you are describing your model. Both are real. A benchmark that doesn't say which one it measured isn't reporting a decode rate.
This repo is the ~200-line script that measures the difference.
Standard library only, no dependencies. Works against any OpenAI-compatible endpoint (llama.cpp, ollama, vLLM, TGI, ...).
python3 edit_vs_prose_bench.py \
--base http://127.0.0.1:8080/v1 \
--model your-model \
--label baseline \
--out baseline.jsonOutput:
edit_shaped rep0: 206.44 tok/s, 235 tok, valid=True
prose_control rep0: 28.71 tok/s, 200 tok, valid=True
...
copyability_speedup: 7.19
Reading the ratio
| ratio | meaning |
|---|---|
| ~1.0 | the stack does not exploit copyability — no context-copy speculation active |
| >>1.0 | it does, and a headline number taken from copyable work describes the speculator, not the model |
Two prompts embed the same 40-line Python file:
- edit-shaped — return the entire file with one constant changed. Output is almost entirely copyable from context.
- prose control — write ~160 words on the file's design tradeoffs, no quoting. Same context, nothing copyable.
Matched output length (~230 tokens), temperature 0, cold run discarded, p50 of
warm runs. Decode rate is measured as completion_tokens / (total - ttft) so
prefill doesn't contaminate it.
1. Validity assertions. A model that shortcuts the edit ("unchanged, see above") posts a spectacular tok/s for work it never did. Every run asserts the output contains the class, all four methods, the new constant, and not the old one. Failures are surfaced, not silently averaged in.
2. Cache discipline. N-gram caches persist across requests. At temperature 0 a repeated prompt feeds the cache its own prior output. Run the copyable case first and it poisons your control — ours climbed from 33% to 65% acceptance across three reps and quietly inflated the denominator. Restart the server between arms and run the control first. If your control gets faster across reps, it's contaminated.
Measured on one NVIDIA GB10 (121 GB unified), gpt-oss 117B MoE, MXFP4:
| stack | edit-shaped | novel prose |
|---|---|---|
| ollama | 49.9 | 70.3 |
| llama.cpp, no speculation | 65.3 | 90.1 |
llama.cpp + --spec-type ngram-mod |
199 | 88.4 |
| llama.cpp + EAGLE3 (BF16 draft head) | 73.6 | 40.0 |
- Context-copy n-gram speculation is worth 4x on edit-shaped work, with no draft model and no training. Prose stays flat, which is the proof it's copy-exploitation rather than a general speedup. Draft acceptance on edit work runs 90-100% at a mean draft length of 58-65 tokens.
- The trained draft head loses to the dumb mechanism. EAGLE3's mean accepted length is only 2.0-3.7 tokens, so its own forward pass eats the gain: +13% on edits while prose collapses 56%. A mixed workload nets out behind.
- Coding agents are the best case for this — they are nearly pure copy-with-modification. Chat and drafting workloads get close to nothing.
All figures are first-encounter (a new file each time). Repeating the same file pushes it past 400 tok/s, which is real and useless to publish.
Single-box measurements, n=3 warm runs per cell. Numbers from different batteries are not interchangeable; only within-battery ratios are compared. The 4x has not been re-verified end-to-end through a production agent harness.
MIT