Repository navigation
Replay recorded model replies: resume interrupted rollouts or play a recording back - #2749
Closed
parkerpettit wants to merge 12 commits into
Closed
parkerpettit wants to merge 12 commits into
parkerpettit wants to merge 12 commits into
Conversation
A rollout given recorded traces (ModelContext.replay, RunRequest.replay, or Agent(replay=...)) continues under the recorded trace id. The interception server answers each model request whose prompt matches the recording with the recorded reply, before any upstream call, and the harness runs the tool calls again, which rebuilds the runtime and program state. A prompt matches its exact recorded path or, failing that, the path with the same structure. The first unmatched request goes to the live model. ModelCall.replayed records how each answered call matched. Each dialect can now render a vf Response as its native response, and the Anthropic and Responses dialects stream responses with several blocks or items.
This was referenced Oct 2, 2026
Structural matching now requires every non-assistant message to differ from its recording only in volatile runs (uuids, hashes, hostnames, durations, clock times, epoch timestamps); any other difference samples live. Every branch that fits the prompt stays a candidate until a message tells siblings apart, so parallel subagents with near-identical prompts resolve correctly. Tools compare order-insensitively with volatile runs masked. A failed attempt frees its recording, so an agent or episode retry replays it again under a fresh trace id. A lookup or rendering failure falls back to the live model. Replay is exported from verifiers.v1.
…verbatim replay Each request now looks up the recorded reply that followed its last assistant message, then (resume) compares the request with that one recorded prompt: identical, or differing only in volatile runs, with images compared exactly. This replaces the tree search and matches the same replies. Verbatim replay (Replay(..., verbatim=True)) skips the comparison, never samples live, stops the rollout with replay_ended when the recording runs out, and claims a recording by agent alone, so a recorded seat can drive a seat of another task. Hex ids delimited by -, _ or / are masked as volatile.
# Conflicts: # verifiers/v1/dialects/chat.py
Resume replays a recorded reply only while the prompt is exactly its recorded prompt and samples live from the first prompt that differs. Playback (formerly verbatim) replays every reply whatever the prompt and tags a reply whose prompt differs as forced. The volatile-text masking and similarity matching are gone.
A live trace that lost a delta could raise while its replay was built, failing the resumed rollout on every later resume instead of running it live.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replay recorded model replies through the harness. Two modes:
resume(default): continue an interrupted rollout. Replay recorded replies while each prompt is exactly its recorded prompt. Sample live from the first prompt that differs.playback: play a recording back, on any task. Replay every recorded reply whatever the prompt, never sample live, and stop withreplay_endedwhen the recording runs out. Use it to reproduce deliverables from a trace, or to drive one seat of a multi-agent environment.The harness runs every replayed tool call for real. Files, processes, REPL kernels, and MCP tool state are therefore rebuilt in the fresh box.
Entry points:
RunRequest.replay(env server, resume).vf.Agent(config, replay=vf.Replay(traces, mode=...)).setup():agents.solver.ctx = dataclasses.replace(agents.solver.ctx, replay=vf.Replay(traces, mode="playback")).Docs: a "Replaying traces" section in
docs/v1/evaluation.md.Use cases
uv run eval --resumecontinues each cut-off rollout from its last committed call instead of starting it over.How
ModelCall.replayedisexact(prompt equal to the recorded prompt),forced(playback onto a prompt that differs), orNone(sampled live). A replay error falls back to the live model in resume, and ends a playback withreplay_ended.Testing
Live, with
gpt-5.6-lunaat temperature 1.0. Each rollout is interrupted after N model calls, then resumed in a fresh box.forced, files rebuilt).replay_endedat the end of a cut recording, then scored.--resumeafter SIGKILL: bash on prime and rlm on docker, 6 rollouts each. Every rollout kept its trace id, replayed exactly its committed calls, and scored 1.0.proposer_solver: a bash proposer and 2 null solvers): playback of the proposer seat on another task, playback of the whole episode, resume with only the proposer recorded, and an env that plays back the proposer fromsetup(). Recorded seats replayed every call, the other seats ran live, and every solver got the recorded problem.exactin one branch, then finished live.Limits
Note
Add replay of recorded model replies to resume interrupted rollouts
Replayclass in replay.py that claims a recorded trace by agent name and task key, then serves the next recorded assistant response for an exact or structural prompt match.ModelCalltrace.Anthropic,Chat,Responses) gain aresponse_to_wireserializer so replayed generic responses can be returned in each provider's native JSON, and streaming event generation now handles multiple content blocks and non-text items such as tool calls.ModelContextfor local runs, or passed as areplaysfield inRunRequestfor remote episodes, validated in the serve handler in serve/server.py.end_turn— checkRollout.__init__in rollout.py and the SSE serializers in the dialect files.Macroscope summarized fe67615. (Automatic summaries will resume when PR exits draft mode or review begins).