Skip to content

fix(tokenizer): render the developer role as system for templates without it - #904

Open
coco-alen wants to merge 1 commit into
FlashML-org:mainfrom
coco-alen:fix/chat-developer-role
Open

coco-alen wants to merge 1 commit into
FlashML-org:mainfrom
coco-alen:fix/chat-developer-role

Conversation

@coco-alen

Copy link
Copy Markdown
Contributor

Closes #646

/v1/chat/completions passed a developer message (sent by Codex and other current OpenAI clients) straight to the chat template. Templates that don't name the role handle it badly, and two of the three failures are silent:

template developer on main
nvidia/Qwen3.8-Flash-Next-NVFP4 TemplateError: Unexpected message role. (400)
Qwen/Qwen3-0.6B dropped: the instruction never reaches the model
Qwen/Qwen3-Coder-30B-A3B-Instruct rendered as `<

TokenizeManager._render now renders developer as system when the chat template does not mention developer. Templates that name the role (gpt-oss) get it unchanged. The DSV4 encoder path returns before this and keeps its own handling. The mapping copies the message, so the request is not mutated. /v1/responses already folds developer into the system message, so this only changes the chat completions path.

Not covered here: a request with both a system and a developer message now behaves like one with two system messages. Templates that require a single leading system message (Qwen3.8-Flash-Next) still reject it, now with System message must be at the beginning. Merging the two would be a separate change.

Testing

RTX 6000 Ada (48 GB), driver 595.99.02, torch 2.14.1+cu130, transformers 5.18, Python 3.12. Tokenizer-only check, no model weights: TokenizeManager.render_prompt with [{"role": "developer", "content": "Answer in French."}, {"role": "user", "content": "hi"}] against each checkpoint's real tokenizer and chat template:

tokenizer main this branch
nvidia/Qwen3.8-Flash-Next-NVFP4 TemplateError: Unexpected message role. same prompt as the system spelling
Qwen/Qwen3-0.6B developer text missing from the prompt same prompt as the system spelling
Qwen/Qwen3-Coder-30B-A3B-Instruct `< im_start
openai/gpt-oss-20b same prompt as system unchanged
  • New tests/tokenizer/test_tokenize.py::test_developer_role_renders_as_system_unless_the_template_names_it. With the fix reverted, the case for a template without developer fails ({'role': 'developer', ...} != {'role': 'system', ...}). With the fix, both cases pass.
  • pytest tests/tokenizer tests/server: 755 passed, 24 skipped (needs_weights).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

/v1/chat/completions returns 400 for the OpenAI developer role

1 participant