multi-token-prediction

PyTorch implementations of modern open-source LLM architectures (Llama, Qwen, DeepSeek, Gemma, GPT-OSS, Kimi, and more) — written from scratch for readability and learning, based on Sebastian Raschka's LLM Architecture Gallery.

llama.cpp fork with TurboQuant WHT-rotated KV cache & weight compression + Gemma 4 MTP and Qwen 3.6 NextN speculative decoding (+30-50% throughput).

llama.cpp fork for significantly improved performance on Ampere (especially RTX 3090 / 3090 Ti): example: 95+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx

[NeurIPS 2026] DMax: Aggressive Parallel Decoding for dLLMs

Qwen4-Exp (Qwen3.8-Flash-Next) SWA + MTP + PLE-on-iswa — bounded deep-context decode w/ long-range recall, TBQ4 4.25 bpv KV, DSpark + RotorQuant. Main (master) = Qwen4-Exp build; the Qwen35 SWA hybrid lives on the qwen35 branch. RTX 4090.

Serving Qwen3.8-27B-FP8 on a single DGX Spark (GB10): 7.88 to 58.5 tok/s single-stream from decode strategy alone, weights untouched. Speculative decoding and prefix caching benchmarked, plus DFlash 2 — the only Qwen3.8-27B build that can serve it under vLLM.

Curated collection of research on the limitations of next-token prediction and methods that go beyond it.

Run Hermes Agent + Claude Code locally on llama.cpp — zero API costs. A 4h / 7M-token session that would have cost $94 on Claude Opus 4.7

Deploy Qwen3.6-35B-A3B (Q4_K_XL) + MTP speculative decoding on a single NVIDIA L4 24GB — GCP g2-standard-8 — via the official llama.cpp Docker image. Decode-optimized to ~91–99 tok/s (min ~91 chat, max ~99 math), lossless (full GPU residency + ECC-off).

Rust + CUDA inference engine for Crow on one RTX 5090: GLM-5.3-Flash, Qwen3.8-Flash-Next (MoE) and the dense Qwen3.8-27B as CNQ containers, own kernels, MTP speculative decoding, vision, OpenAI-compatible serve. Windows and Linux.

From-scratch PyTorch: frontier LLM techniques as of 2026-Q1 — the Muon optimizer and Multi-Token Prediction, plus a base BPE tokenizer. Self-contained, self-checking modules.

Faithful from-scratch reimplementation of DeepSeek-V3 (MLA + aux-loss-free MoE + MTP + speculative decoding), ~412M params, Chinchilla-optimal 8.4B-token training on a single A100 80GB.

Qwen3.8-27B on Ampere (RTX 3090 / 3090 Ti): TurboQuant+ llama.cpp fork with SM86 kernel and memory work, 245K context with the model's own MTP head. Model: sjakek/Qwen3.8-27B-ATX-4-XS-GGUF on Hugging Face.

MTP draft-head research harness for Qwen3.5-9B. Verdict: function cuts (pruning, low-rank, distillation) pay a 10-19pt off-distribution tax fidelity cuts do not — the zero-training trimmed-vocab NVFP4 recipe wins at 1.8-2.7x end-to-end in memra. Negative-result ledgers included.

Preregistered, DOI-archived llama.cpp study: Qwen3.8-27B on one RTX 3090 (sm_86). draft-mtp at --spec-draft-n-max 2 is +59.8% [+57.0, +62.8] server-reported decode over 25 purposive prompts; draft-dflash (PR #27342) +51.9%, not separated from it. Decode energy -37.1%, uncalibrated. 23-25 of 25 prompts diverge from serial greedy at 1600 tokens.

A controlled 180-run study of DeepSeek-inspired MLA, sparse MoE routing, V3-style load balancing, and multi-token prediction under constrained compute.

Reproducible Qwen3.8-27B MTPLX benchmark on Apple M1 Max

Benchmarks Multi-Token Prediction in llama.cpp against standard autoregressive baselines. Measures tokens per second, perplexity delta, and speculative acceptance rate across model sizes.

统一 C6 对称性 AI 基础设施工具包 - 路由+MTP+量化+六角注意力+纠错+流水线 | TaiChi Matrix M1-M6 Infrastructure

Some benchmarks on running speculative decoding with Qwen-3.6-35B-A3B with EAGLE3, DFlash and MTP.