Repository navigation
Conversation
…DA, Qwen3.5-2B Metal and Nemotron Parse HIP
The server dropped logit_bias from its field list, so a request sending it got unbiased draws. It is now parsed
(decimal token IDs inside the tokenizer's vocabulary, finite values from -100 to 100, at most 300 entries, sorted,
duplicates refused; exactly -100 is a ban) and carried on the request and lane stream beside the penalties. An
engine without Info.logit_bias refuses a non-empty object by name, even one of zeros; {} and null keep working, and
Engine.submit refuses a list the engine did not declare.
On Qwen3.8-27B and Qwen3.5-MoE CUDA each kept row's (row, token) cell takes one amount, the penalty less the bias
(+inf for a ban), so a cell has one writer; a new tf_bias_rows launch subtracts in fp32 and rounds once to bf16 as
tf_penalize_rows does, and writes -inf for a ban whatever the cell held. tf_penalize_rows is unchanged and still
runs for rounds with no bias. A kernel set without tf_bias_rows does not offer the field. The first token is drawn
on the host from the cached row with the same arithmetic, and a masked verify row draws from the processed row,
skipping a ban's -inf. Qwen3.5-2B on Metal applies it in every host draw; Nemotron Parse's HIP host draw applies it
after repetition and the penalties, and a banned forced end token fails the draw rather than being emitted.
Every wrapper (CUDA host, drafter, recorder) copies Backend.logit_bias. Model discovery publishes the flag, so the
frozen /v1/models answers gain "logit_bias": false. tf_bias_rows has a pending contract record: its resources are
still to be measured on nvcc 13.0.88, 13.2 and 13.3 for the five targets before a release.
ashhart
force-pushed
the
lead/logit-bias
branch
from
November 14, 2026 03:31
830b57b to
d3126e2
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3110.
logit_biasadds each listed token's value to its logit before every draw, and -100 bans the token. Supported on Qwen3.8-27B and Qwen3.5-MoE on CUDA, Qwen3.5-2B on Metal and Nemotron Parse on HIP. The bias applies on each row's draw, so drafted replies equal undrafted ones. Every other engine now refuses a non-emptylogit_biaswith a 400 instead of silently ignoring it, and/v1/modelsnames the capability per engine.Bias lookup is a binary search over the request's sorted entries. A bias token past the model head's vocabulary is refused before launch. On CUDA the bias runs as a small kernel,
tf_bias_rows, before the draw. The shared sampling rule was refactored and checked equal to main for every temperature the server accepts.Tests: the refusal test fails on main (expected 400, got 200, because main ignored the field);
test-server-logit-bias-host24;test-logit-bias-host18 + 16; a draw-equality test;test-cuda-host393/393 steps; HIP 78/78; goldens updated for the new capability field; CUDA cross-compiles pass.Device check pending (GB10, Qwen3.8-27B with MTP and DFlash2):
tf-cuda-test kernel-contract tf_bias_rows, where the real kernel passes and the planted control fails. Served checkerzig/tests/server/logit_bias_served.py: no bias equals main, a banned token never appears, biased drafted equals undrafted, concurrent equals solo, an unbiased neighbour is unchanged. No-bias decode level with main; first-token latency of a 300-entry bias reported.tf_bias_rowsregister records move out of pending before a release.