Repository navigation
server: a warm pass keeps the state the next turn resumes (chunk cut, tool-result opener, reasoning) - #2686
server: a warm pass keeps the state the next turn resumes (chunk cut, tool-result opener, reasoning)#2686ashhart wants to merge 10 commits into
Conversation
The warm pass took the plain chunk starts, while chat requests add a cut before the conversation's own text (the system prefix). On a planned family the pass then kept a state built from other chunks than the next request's fresh run. One helper, chat.requestChunks, now gives both the same starts and cut. Requests that don't warm are unchanged.
…esult The warm pass always rendered the next turn as a user message. When the reply called tools, the next request opens with a tool message, and a template whose tool opener differs from its user opener (Kimi's, for one) left the warmed prefix off that request, so the pass was never resumed. The probe now opens as the first call's result when the reply made calls. Replies without calls warm as before.
…ient got it The warm pass rendered the reply as role, content and tool calls, without the reasoning_content the client received. A client that sends its reasoning back, and a template that renders past reasoning, then made the next request differ from the warmed prefix at the reply. One replyMessage now builds the whole reply's message and the warm pass's alike; the message cortheon remembers is unchanged.
…has no ** on strings
…se history sent reasoning back A client that drops reasoning from the replies it sends back (most do) would miss a warm pass that rendered it, and on a first turn there is no history to tell, so the pass leaves it out as main does. Tests cover a first turn, a client that keeps reasoning and one that drops it.
9f6de3d to
3094e15
Compare
|
Rebased onto today's main (head The warm pass now keeps the reply's reasoning only when the request's own earlier replies came back with
The server host tests pass on this head (1,330 passed, 7 skipped). Still owed before landing: the served tool-loop cell on Qwen3.8-27B and on Flash Next on Metal, main against this branch, once with a client that keeps reasoning and once with one that drops it. It reports token hashes per turn, and
|
|
Restacked: this branch now carries the four follow-up commits on top (
|
What this changes
After a reply, the server runs a warm pass: it renders the conversation as the client's next turn will send it, and keeps that prompt state, so the next request resumes instead of prefilling the reply again. Three gaps made the warmed state differ from the one the next request needs, so agent tool loops missed it and re-prefilled:
chat.requestChunks, now gives both the same starts and cut.reasoning_content, while a client sends back the message it received, reasoning included. On templates that render past reasoning, the history then differed from the next request's. The warm pass now renders the same message the client got (openai.replyMessage, which the whole-body reply uses too, so responses are unchanged).A warm pass only changes which prompt states are kept. No reply's tokens change.
Checks
zig/src/server/warm_turns_test.zig. Each test asserts that the warmed history is a prefix of the next request's prompt (and, for the cut, that the chunk starts below it are equal). Each fails when its fix is reverted (each control checked).zig build testpasses on macOS on this head.zig fmt --check, the lean check (no new problems), and files at most 600 lines.warmNextruns on both reply paths). The warm pass being cancelled while other streams are active is a separate change.Not yet run
A served multi-turn tool-loop cell, main against this branch (token hashes per turn,
cached_tokensand first-token time on the next turn). It runs before landing.