[LZ-QWQ]

ft bench bw --model builds block-FP8 scale banks without the loader's padding. For Qwen3.8-Flash-Next-FP8 (H2560/I640), the 200 B down_scale rows disable fused multi-bank copy; the fallback then rejects the 400 B gate_up_scale rows. The command exits 0 with null gather/overlap bandwidths. Hybrid with auto fetch (-1) consequently falls back to a fixed fetch cap of 1; remaining misses execute on CPU.

Reuse fp8_block_scale_pad in both calibration bank builders, matching production strides and byte counts. down_scale becomes 240 B and fused copy works; the aligned H2048/I512 layout is unchanged. Users must rerun calibration after updating; serving reads the saved profile without running the benchmark.

Five-device validation against main 27740fec: the original command produces three null gather/overlap fields on every device; the fix restores them. Gather GB/s after: 4090 25.11, 5090 53.70, W7900D 27.15, R9700 36.85, H100 44.54. Both geometries pass byte-exact permuted expert copies; the existing padded-scale CPU numerical reference passes on all five devices.

Model A/B switches the profiles generated by main/fix with identical serving code and settings: Qwen3.8-Flash-Next FP8, DFlash 2 k=7, Hybrid-auto, C1/TP1, 32,768 input/256 output, 2,048 expert slots, 12 CPU workers, graphs, PLE disk, naive KV40,960/chunk8,192. AB then BA, three fixed prompts per launch: six formal measurements/arm, excluding smoke and a separate 32K warmup; no outlier removal.

All speeds below are request-level medians in token/s; Decode includes [min,max]. Change is the ratio of Decode medians minus one. Decode excludes prefill and the first emitted batch; Generate includes prefill.

GPU	Decode before	Decode after	Change	Generate before → after
RTX 4090 48GB	21.96 [11.43, 23.61]	28.98 [24.11, 32.15]	+31.9%	6.54 → 6.86
RTX 5090	19.30 [18.21, 19.94]	40.54 [33.36, 42.45]	+110.1%	10.50 → 14.66
W7900D	18.28 [17.43, 18.50]	24.97 [22.24, 28.20]	+36.6%	5.63 → 6.16
R9700	11.28 [1.24, 15.36]	15.63 [0.95, 20.54]	+38.6%	5.32 → 6.13
H100 80GB HBM3	19.39 [18.91, 21.64]	40.00 [36.86, 41.06]	+106.3%	10.95 → 15.48

R9700 has large stalls: pooled Decode is 4.80→4.40 token/s despite its higher median. RTX4090 has a 137.76s TTFT outlier; pooled Generate is 5.19→4.65. All samples are retained. Profiles change CPU/GPU expert placement and greedy outputs differ; these results do not establish answer accuracy or an equal-output speedup.

Environment and reproduction

FreeToken0.2.2; Ubuntu24.04 containers, Triton3.8.0, Transformers5.18.0. CUDA13.0: Python3.12.3/PyTorch2.14.1+cu130. ROCm10.1.0: Python3.13.13/PyTorch2.14.0+rocm10.1.0 (HIP build7.16.26385).

GPU	CPU	Host driver
RTX4090 48GB	Xeon Gold6554S	590.48.01
RTX5090	EPYC9554	615.71.09
W7900D	EPYC9334	6.14.14
R9700	EPYC9334	7.1.9.31600000
H100 80GB HBM3	EPYC9654	575.57.08

H100 host: Ubuntu22.04; image-provided forward-compat libcuda580.95.05. Same per-device environment for both arms.

Run on each source version:

python -m freetoken.moe.benchbw --model /models/Qwen3.8-Flash-Next-FP8 --cpu-threads 12 --out /validation/profile.json

For a custom output file, set FREETOKEN_BENCHBW_PATH to it when serving. benchbw-fp8-validation.zip contains the measured model driver, fixed token IDs, per-device commands, raw request records, and an independently verified statistics script.