Software engineer working on local LLM inference on AMD hardware. I mostly work on the HIP backend of TensorFold, on a Ryzen AI Max+ 395 (Strix Halo, gfx1151).
Every change I ship keeps the output byte-identical: the same tokens and the same logits. Each comes with receipts:
- alternated served A/B runs;
- kernel traces;
- the project's kernel contract, with launch records taken from the release toolchain.
Highlights:
- #1694: wide stream rounds reuse cached weights.
- Qwen3.8-27B serving is +31% faster at 8 streams and +47% at 16 on gfx1151, with the same tokens.
- An independent run on RDNA2 (gfx1030) measured +11.6% at 8 streams.
- #1663: the matrix-core prompt tile without selects and divisions.
- Cold prefill of Qwen3.8-27B is 32% faster at 2K and 24.5% faster at 64K (TensorFold 1.2.0).
- #1932: the planned decode score.
- HIP decode is 7.5% to 18.9% faster at 32K and 64K context on Qwen3.6-35B-A3B and Qwen3.8-27B (1.2.2).
- #1855: narrowest stream columns.
- Qwen3.6-35B-A3B chat is about 18% faster with drafts on, and Qwen3.8-27B about 5% (1.2.2).
- #2455: sampled streams draft with the graphed head chain.
- Sampled serving of Qwen3.6-35B-A3B with MTP drafts is 4.1% to 11.6% faster on gfx1151, with the same tokens.
- #2582: @davidtai's prompt DeltaNet kernels from #1667, carried onto main with gfx1151 receipts.
- Prompts take 5-6% less time on Qwen3.8-27B and 6-10% less on Qwen3.6-35B-A3B, with the same bytes.
- #1830: shared rounds timed at the served streams.
- Multi-stream drafting no longer drops into a slower mode (fixes #1695, 1.2.2).
| PR | Change | Release |
|---|---|---|
| #1542 | opt-in -Dhip-relax-branches for gfx1100 under clang 21 |
1.2.0 |
| #1648 | banded prefill WMMA GEMM: row blocks share weights | 1.2.0 |
| #1663 | matrix tile inner loop free of selects and divisions | 1.2.0 |
| #1685 | prompt GEMMs write the activation type directly | 1.2.1 |
| #1788 | prompt attention keeps its probabilities in registers, 6 waves | 1.2.1 |
| #1830 | shared rounds timed at the served streams and row buckets | 1.2.2 |
| #1855 | narrowest stream columns at one- and two-row launches | 1.2.2 |
| #1932 | decode score: four keys a warp on long spans | 1.2.2 |
| #1961 | decode softmax stats and partial sums load sixteen ahead | 1.2.2 |
| #2007 | decode router reads the bf16 router rows the model holds | 1.2.3 |
| #2099 | prompt residual add and post-mixer norm in one launch | 1.2.3 |
| #2117 | prompt conv writes q, k and v to their own rows | 1.2.3 |
| #1694 | wide stream rounds reuse cached weights | 1.2.3 |
| #2233 | decode records re-derived with the release hipcc (HIP 7.16) | 1.2.3 |
| #2247 | decode tail issues a row's loads before its branches | 1.2.4 |
| #2374 | resumed == fresh checks start their fresh runs from nothing | 1.2.4 |
| #2314 | DFlash2 part 1: tap a drafter's layers into each stream's ring | 1.2.5 |
| #2455 | sampled streams draft with the graphed head chain | 1.2.5 |
| #2477 | DFlash2 part 2: drafter config and affine-4 weights | 1.2.5 |
| #2652 | shared rows timed with a token a row, lone windows on one token | 1.2.6 |
| #2661 | shared rows' token stride coprime to the vocabulary's span | 1.2.6 |
| #2087 | skewed-row matrix tile, three blocks a WGP on gfx1151 | 1.2.6 |
| #2582 | @davidtai's prompt DeltaNet kernels, carried with gfx1151 receipts | 1.2.6 |
| #2674 | CUDA's DFlash2 drafter config refuses a tap layer named twice | 1.2.6 |
| #2545 | DFlash2 part 3: drafter kernels, each with a planted control | 1.2.6 |
| #2706 | GPU test of Qwen3.8-Flash-Next's fused prompt prep through the layer, bit for bit | 1.2.7 |
In review:
- #2726: DFlash2 part 4, the drafter's engine for the qwen3_5 target.
- #2878: the routed prompt tile takes K-parallel passes for experts of few rows; short cold prompts of Qwen3.6-35B-A3B start up to 22% sooner.
In progress: DFlash2 speculative decoding on HIP for Qwen3.8-27B, as six chained PRs. Parts 1 to 3 are merged (#2314, #2477, #2545); parts 1 and 2 ship in 1.2.5 and part 3 in 1.2.6, and part 4 is in review. Measured locally, the whole chain decodes 2.5x faster in code and 2x in chat at one stream, with the same tokens.