Skip to content

Replay recorded model replies: resume interrupted rollouts or play a recording back - #2749

Closed
parkerpettit wants to merge 12 commits into
feat/dialect-response-wirefrom
feat/trace-replay
Closed

parkerpettit wants to merge 12 commits into
feat/dialect-response-wirefrom
feat/trace-replay

Conversation

@parkerpettit

@parkerpettit parkerpettit commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

What

Replay recorded model replies through the harness. Two modes:

  • resume (default): continue an interrupted rollout. Replay recorded replies while each prompt is exactly its recorded prompt. Sample live from the first prompt that differs.
  • playback: play a recording back, on any task. Replay every recorded reply whatever the prompt, never sample live, and stop with replay_ended when the recording runs out. Use it to reproduce deliverables from a trace, or to drive one seat of a multi-agent environment.

The harness runs every replayed tool call for real. Files, processes, REPL kernels, and MCP tool state are therefore rebuilt in the fresh box.

Entry points:

  • RunRequest.replay (env server, resume).
  • vf.Agent(config, replay=vf.Replay(traces, mode=...)).
  • An env plays back one seat from setup(): agents.solver.ctx = dataclasses.replace(agents.solver.ctx, replay=vf.Replay(traces, mode="playback")).

Docs: a "Replaying traces" section in docs/v1/evaluation.md.

Use cases

  • Long-horizon evals. A rollout that runs for hours survives a crash, a preemption, or Ctrl-C. uv run eval --resume continues each cut-off rollout from its last committed call instead of starting it over.
  • Rescoring after fixing a verifier. When a reward runs tests or reads files in the sandbox, a trace alone cannot be rescored. Playback rebuilds each rollout's sandbox from its trace without calling the model, then the fixed tests run against it.
  • Debugging multi-agent envs. Play back every seat of a failed episode to get the same run again with no model calls, while you fix the env's scoring or orchestration.
  • Reproducing artifacts. Playback runs a trace's tool calls again in a fresh box and rebuilds its files and deliverables, on the trace's task or on another one.
  • Resampling from a point. Cut a recording after any call and resume it: the recorded prefix replays exactly, and the rollout samples live from the cut. Use it to study the variance at one decision, or to debug a failure.

How

  • Claim. A new rollout claims a free recording of its agent (on the same task, in resume) and runs under the recording's trace id. A failed attempt frees its recording, so its retry replays the recording again under a fresh id.
  • Position. Replay serves the recorded reply that followed the prompt's last assistant message.
  • Match. In resume, the prompt and tools must equal the reply's recorded prompt and tools. Playback serves the reply anyway.
  • Record. ModelCall.replayed is exact (prompt equal to the recorded prompt), forced (playback onto a prompt that differs), or None (sampled live). A replay error falls back to the live model in resume, and ends a playback with replay_ended.

Testing

Live, with gpt-5.6-luna at temperature 1.0. Each rollout is interrupted after N model calls, then resumed in a fresh box.

  • Resume: bash (prime sandbox) and rlm replayed every recorded call, then continued live and finished correctly. rlm includes two parallel subagents, and a subagent tree five levels deep (10 of 10 calls).
  • Retry: a resumed bash attempt failed at scoring. The retry replayed again and scored 1.0.
  • Playback:
    • It reproduced a deliverable on a different task (5 of 5 calls forced, files rebuilt).
    • It ignored injected user messages.
    • It left an unbound seat live.
    • It stopped with replay_ended at the end of a cut recording, then scored.
  • prime-rl --resume after SIGKILL: bash on prime and rlm on docker, 6 rollouts each. Every rollout kept its trace id, replayed exactly its committed calls, and scored 1.0.
  • Other harnesses, from earlier recordings replayed against this matcher: claude_code, hermes_agent, mini_swe_agent, null, and pi replay every call. codex replays until a prompt shows the wall-clock time. openclaw, prime_agent, and terminus_2 put a session id or a screen capture in the first prompt, so they resume live.
  • Multi-agent (proposer_solver: a bash proposer and 2 null solvers): playback of the proposer seat on another task, playback of the whole episode, resume with only the proposer recorded, and an env that plays back the proposer from setup(). Recorded seats replayed every call, the other seats ran live, and every solver got the recorded problem.
  • Other models: claude_code with glm-5.3, qwen3.6-35b-a3b (with thinking blocks), and qwen3-coder-next replayed 3 of 3 calls exact in one branch, then finished live.
  • Unit suite: green.

Limits

  • Tool calls run again, side effects included.
  • Eval only. The train client needs token ids.
  • Resume stops replaying at the first prompt that differs from the recording, even if only an id or a timestamp differs.
  • rlm puts its session id, which is the trace id, in its prompts. A retry runs under a fresh id, so it samples live. Exact rlm replay needs stable ids in nano-rlm.
  • Replayed calls keep their recorded usage, so a resumed trace reports the whole rollout's tokens.

Note

Add replay of recorded model replies to resume interrupted rollouts

  • Adds a Replay class in replay.py that claims a recorded trace by agent name and task key, then serves the next recorded assistant response for an exact or structural prompt match.
  • The interception server in server.py checks replay before contacting the live model; training clients and unmatched requests fall back to live sampling. Replayed calls skip response interceptors and record a match classification on each ModelCall trace.
  • All dialects (Anthropic, Chat, Responses) gain a response_to_wire serializer so replayed generic responses can be returned in each provider's native JSON, and streaming event generation now handles multiple content blocks and non-text items such as tool calls.
  • Replay traces can be attached via ModelContext for local runs, or passed as a replays field in RunRequest for remote episodes, validated in the serve handler in serve/server.py.
  • Behavioral Change: replay-enabled rollouts reuse the claimed recorded trace ID instead of a new one, and response streaming no longer assumes a single text block ending in end_turn — check Rollout.__init__ in rollout.py and the SSE serializers in the dialect files.

Macroscope summarized fe67615. (Automatic summaries will resume when PR exits draft mode or review begins).

A rollout given recorded traces (ModelContext.replay, RunRequest.replay,
or Agent(replay=...)) continues under the recorded trace id. The
interception server answers each model request whose prompt matches the
recording with the recorded reply, before any upstream call, and the
harness runs the tool calls again, which rebuilds the runtime and program
state. A prompt matches its exact recorded path or, failing that, the path
with the same structure. The first unmatched request goes to the live
model. ModelCall.replayed records how each answered call matched.

Each dialect can now render a vf Response as its native response, and the
Anthropic and Responses dialects stream responses with several blocks or
items.
Structural matching now requires every non-assistant message to differ from
its recording only in volatile runs (uuids, hashes, hostnames, durations,
clock times, epoch timestamps); any other difference samples live. Every
branch that fits the prompt stays a candidate until a message tells siblings
apart, so parallel subagents with near-identical prompts resolve correctly.
Tools compare order-insensitively with volatile runs masked.

A failed attempt frees its recording, so an agent or episode retry replays
it again under a fresh trace id. A lookup or rendering failure falls back to
the live model. Replay is exported from verifiers.v1.
…verbatim replay

Each request now looks up the recorded reply that followed its last
assistant message, then (resume) compares the request with that one
recorded prompt: identical, or differing only in volatile runs, with images
compared exactly. This replaces the tree search and matches the same
replies. Verbatim replay (Replay(..., verbatim=True)) skips the comparison,
never samples live, stops the rollout with replay_ended when the recording
runs out, and claims a recording by agent alone, so a recorded seat can
drive a seat of another task. Hex ids delimited by -, _ or / are masked as
volatile.
@parkerpettit parkerpettit changed the title Replay recorded model replies to resume interrupted rollouts Replay recorded model replies: resume interrupted rollouts or drive a seat verbatim Oct 2, 2026
# Conflicts:
#	verifiers/v1/dialects/chat.py
@parkerpettit
parkerpettit changed the base branch from main to feat/dialect-response-wire October 2, 2026 23:47
Resume replays a recorded reply only while the prompt is exactly its recorded
prompt and samples live from the first prompt that differs. Playback (formerly
verbatim) replays every reply whatever the prompt and tags a reply whose prompt
differs as forced. The volatile-text masking and similarity matching are gone.
@parkerpettit parkerpettit changed the title Replay recorded model replies: resume interrupted rollouts or drive a seat verbatim Replay recorded model replies: resume interrupted rollouts or play a recording back Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant