Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,8 @@ jobs:
tests: tests/integration/test_user_echo.py
- behavior: Resume Tool
tests: tests/integration/test_resume_tool.py
- behavior: Checkpoint Restore
tests: tests/integration/test_checkpoint.py
- behavior: User Sim
tests: tests/integration/test_user_sim.py
- behavior: Agent User
Expand Down
12 changes: 12 additions & 0 deletions docs/agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,3 +72,15 @@ max_turn_tokens = 4096
skip_on_truncated = true
skip_on_timeout = true
```

## Retries and checkpoints

`[env.agent.retries]` sets a retry budget per error type: `rules` are checked in order, and the first rule where all fields match sets the budget for that error; unmatched errors use the top-level `max_retries` budget.

Rollout checkpointing is on by default: a retry restores the newest snapshot of the box and continues the trace from there, instead of starting over. The runtime takes the best snapshot it supports, and each checkpoint records its type. It needs a runtime that takes snapshots (prime) and a harness whose new process can continue the rollout: one that resumes on the transcript (`SUPPORTS_RESUME`: null, bash, browser-use) or an ACP harness whose agent loads its session from files (`LOADS_SESSIONS`: claude-code, codex, rlm). Elsewhere a retry starts over, as it does when a restore fails before the restored attempt's first new turn. Every snapshot is deleted once the run is final. Set `--no-env.agent.checkpoint` to turn it off.

```toml
[env.agent.checkpoint]
interval = 10 # snapshot at the 10th, 20th, ... model turn; shorter rollouts take none
keep = "last" # snapshots kept while the rollout runs: "last", or every k-th plus the newest
```
1 change: 1 addition & 0 deletions docs/harnesses.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ Class-level capability flags declare which optional features your harness implem
- `APPENDS_SYSTEM_PROMPT`: passes `TaskData.system_prompt` as a separate system message; when `False`, `resolve_prompt` prepends it to the first user message instead.
- `SUPPORTS_MCP`: indicates that `launch` wires `mcp_urls` into the program.
- `SUPPORTS_RESUME`: indicates that a multi-turn exchange can restart from the conversation so far (programs with native session state should override `resume` or `session` instead). Note that live mid-turn steering requires a `HarnessSession` that implements `steer`; conversation resume support alone is not enough (see [message delivery](agent.md#messages-during-an-active-turn)).
- `LOADS_SESSIONS` (ACP harnesses): indicates that the pinned agent implements ACP `session/load`, so a new process can continue a rollout in a box restored from a snapshot (`agent.checkpoint`). A harness with `SUPPORTS_RESUME` needs nothing more: a restore relaunches it on the transcript. Keep per-rollout state in the box (for example under `/tmp/vf-<harness>-runs/<trace id>`), so the snapshot holds it.
- `SUPPORTS_SKILLS`: indicates that the harness calls `install_skills` to copy configured skills into the directory where the program discovers them.
- `SUPPORTS_TOOL_INTERCEPTION`: indicates that the harness checks tool calls against the rollout's tool gate (`tool_interception_url`) before executing them.

Expand Down
133 changes: 133 additions & 0 deletions tests/integration/fixtures/checkpoint_restore_v1.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
"""checkpoint-restore: an agentic task whose first attempt dies mid-rollout, so the retry
restores the box's newest snapshot instead of starting over.

`setup` stamps a nonce into the box that the prompt never reveals; the task asks the model
for two tool calls that build on each other (the second needs the first's token, so they
take two turns), and the reward reads the second one's file back out of the box. The
tools are colocated: they run in the box, so their files are in its snapshots, and any
harness with MCP tools can drive them. A request `@intercept` raises a `ProviderError`
once the first attempt has made two turns — after the snapshot of turn 2 exists. The
restored attempt continues from that snapshot in a new harness process: it never sees
the injected failure again (it fires once per task), finds the earlier calls' files, and
finishes. Needs a harness and a runtime that both support the snapshot type
(`SUPPORTS_<TYPE>_SNAPSHOTS`; today prime's filesystem snapshots).
"""

import shutil
import uuid
from pathlib import Path

import verifiers.v1 as vf
from verifiers.v1.errors import SandboxError
from verifiers.v1.runtimes import Runtime

MARKER = "marker.txt"
STEP_ONE = "step1.txt"
TOKEN = "token.txt"
DONE = "done.txt"
FAIL_AFTER_TURNS = 2


class RestoreState(vf.State):
workdir: str = ""
"""The box's working directory, where `setup` put the marker."""


class StepsToolset(vf.Toolset[vf.ToolsetConfig, RestoreState]):
TOOL_PREFIX = "steps"

@vf.tool
def copy_marker(self) -> str:
"""Copy the marker file to the step-one file. Returns the token `write_done`
needs."""
workdir = Path(self.state.workdir)
shutil.copyfile(workdir / MARKER, workdir / STEP_ONE)
token = uuid.uuid4().hex[:8]
(workdir / TOKEN).write_text(token)
return f"copied; token={token}"

@vf.tool
def write_done(self, token: str) -> str:
"""Write the step-one file's content to the done file. Needs the token that
`copy_marker` returned."""
workdir = Path(self.state.workdir)
if not (workdir / TOKEN).is_file() or token != (workdir / TOKEN).read_text():
return "wrong token: call copy_marker first and pass the token it returns"
shutil.copyfile(workdir / STEP_ONE, workdir / DONE)
return "written"


class CheckpointRestoreTaskConfig(vf.TaskConfig):
tools: vf.ToolsetConfig = vf.ToolsetConfig(colocated=True)


class CheckpointRestoreConfig(vf.TasksetConfig):
task: CheckpointRestoreTaskConfig = CheckpointRestoreTaskConfig()


class CheckpointRestoreTask(
vf.Task[vf.TaskData, RestoreState, CheckpointRestoreTaskConfig]
):
failed = False
"""Whether the injected failure fired: once per task, so a retry from scratch
(checkpoints off) also gets through."""

@classmethod
def toolsets(cls, config: CheckpointRestoreTaskConfig) -> list[vf.Toolset]:
return [StepsToolset(config.tools)]

async def setup(self, trace: vf.Trace, runtime: Runtime) -> None:
nonce = uuid.uuid4().hex
trace.info["nonce"] = nonce
await runtime.write(MARKER, nonce.encode())
result = await runtime.run(["pwd"], {})
trace.state.workdir = result.stdout.strip()

@vf.intercept
def fail_once(self, request: vf.Request, trace: vf.Trace) -> None:
# Request hooks also judge each proposed tool call after a turn commits, over
# the branch plus an empty probe result; fail only as a real turn goes upstream
# (every tool returns text, so a real result is never empty).
last = request.messages[-1]
probe = isinstance(last, vf.ToolMessage) and not last.content
if trace.num_turns >= FAIL_AFTER_TURNS and not probe and not self.failed:
self.failed = True
# A 4xx: harness SDKs end the turn at once instead of retrying it until the
# rollout's deadline.
raise vf.ProviderError("injected mid-rollout failure", status_code=400)

@vf.reward(weight=1.0)
async def copied_marker(self, trace: vf.Trace, runtime: Runtime) -> float:
try:
content = (await runtime.read(DONE)).decode(errors="replace")
except (SandboxError, OSError, ValueError):
return 0.0
return float(content.strip() == trace.info["nonce"])


class CheckpointRestoreTaskset(
vf.Taskset[CheckpointRestoreTask, CheckpointRestoreConfig]
):
def load(self) -> list[CheckpointRestoreTask]:
return [
CheckpointRestoreTask(
vf.TaskData(
idx=0,
prompt=(
"Use the tools of the `steps` MCP server. Make one tool call "
"per turn and wait for each result before the next.\n"
"1. Call `copy_marker`.\n"
"2. Call `write_done` with the token `copy_marker` returned.\n"
"3. Reply with the single word: done."
),
),
self.config.task,
)
]


__all__ = ["CheckpointRestoreTaskset"]


if __name__ == "__main__":
StepsToolset.run()
1 change: 1 addition & 0 deletions tests/integration/fixtures/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ build-backend = "hatchling.build"
[tool.hatch.build]
include = [
"agent_user.py",
"checkpoint_restore_v1.py",
"counter_tool.py",
"echo_acp_resume.py",
"echo_tool.py",
Expand Down
8 changes: 7 additions & 1 deletion tests/integration/grid.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@

A harness without a capability the behavior needs is collected as a skip with its
reason (`Harness.EXECUTES_CODE`, `SUPPORTS_MCP`, `SUPPORTS_SKILLS`, resume support via
`SUPPORTS_RESUME` or a native session).
`SUPPORTS_RESUME` or a native session). A `snapshots` need asks the runtime for
`SUPPORTS_FILESYSTEM_SNAPSHOTS` and the harness for `Harness.can_restore`.

A runtime without a feature the behavior needs (`RuntimeBackend.SUPPORTS_*`, or a
`NetworkPolicyConfig` for a network policy) skips the cell the same way. That a runtime
Expand Down Expand Up @@ -65,8 +66,10 @@
"expose": "port exposure",
"compose": "Compose services",
"services": "native service VMs",
"snapshots": "snapshots",
}


# Harness dialects that cannot run against the CI model yet.
# TODO: drop each entry once its fix lands.
CI_MODEL_GAPS = {
Expand Down Expand Up @@ -144,6 +147,7 @@ def supports(runtime: str, feature: str) -> bool:
"user": "SUPPORTS_USER_SWITCHING",
"expose": "SUPPORTS_PORT_EXPOSURE",
"compose": "SUPPORTS_COMPOSE",
"snapshots": "SUPPORTS_FILESYSTEM_SNAPSHOTS",
}[feature]
return getattr(runtime_cls(runtime), flag)

Expand Down Expand Up @@ -173,6 +177,8 @@ def unsupported(
return f"{name} has no native skill support"
if "resume" in needs and not resumes(cls):
return f"{name} cannot resume an exchange"
if "snapshots" in needs and not cls.can_restore():
return f"{name} cannot restore a snapshot"
runtime_needs = (*(("container",) if cls.NEEDS_CONTAINER else ()), *needs)
for feature in runtime_needs:
if feature in RUNTIME_FEATURES and not supports(runtime, feature):
Expand Down
167 changes: 167 additions & 0 deletions tests/integration/test_checkpoint.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
"""A retry restores the box's newest snapshot (checkpoint-restore-v1: the first attempt
dies as turn 3 goes upstream; the retry restores the newest snapshot, keeps the turns
before it, and finishes in a new harness process).

Every harness that can continue in a new process (`Harness.can_restore`) runs it on
prime, the one runtime with snapshots; the others are collected as skips. The retention policies run
on one harness: what they keep or delete does not depend on it. Every case checks that
the run leaks no snapshot: each one its boxes took is deleted on the platform once the
run is final."""

import pytest
from checkpoint_restore_v1 import FAIL_AFTER_TURNS
from grid import grid
from prime_sandboxes import AsyncSandboxClient
from skill_codeword import SKILL

from verifiers.v1.harness import RESTORE_PROMPT
from verifiers.v1.runtimes import PrimeBackend
from verifiers.v1.types import content_text

RETRIES = {"max_retries": 0, "rules": [{"type": "ProviderError", "max_retries": 1}]}
NEEDS = ("snapshots", "mcp")


def checkpoint_env(**checkpoint) -> dict:
return {"agent": {"retries": RETRIES, "checkpoint": checkpoint}}


@pytest.fixture
def taken(monkeypatch) -> list[str]:
"""The ids of every snapshot the run's boxes take, in every attempt: also the
ones a restore rewinds past and the ones that never become restorable."""
ids: list[str] = []
snapshot = PrimeBackend.snapshot

async def recording(self):
result = await snapshot(self)
ids.append(result.id)
return result

monkeypatch.setattr(PrimeBackend, "snapshot", recording)
return ids


async def platform_state(snapshot_id: str) -> str | None:
"""The snapshot's state on prime, or None once it is gone."""
async with AsyncSandboxClient() as client:
try:
return (await client.get_checkpoint(snapshot_id)).state
except Exception: # noqa: BLE001 - a deleted snapshot is not found
return None


async def assert_no_leaks(taken: list[str]) -> None:
"""Every snapshot the run took is deleted."""
states = {snapshot_id: await platform_state(snapshot_id) for snapshot_id in taken}
leaked = {
snapshot_id: state
for snapshot_id, state in states.items()
if state not in (None, "DELETING")
}
assert leaked == {}


@pytest.mark.parametrize("harness,harness_runtime", grid(needs=NEEDS), indirect=True)
async def test_retry_restores_checkpoint(run, taken, harness, harness_runtime):
"""The box restores to the state before the turn that failed, the turns up to
there stay sampled, the task's `setup` (a nonce the prompt never reveals) is not
rerun, and the harness is told about the restore — the restored attempt builds on
the earlier tool call's file. A snapshot every turn, so the failure at turn 3
restores a snapshot with turns behind it."""
trace = await restored_trace(run, harness, harness_runtime, interval=1)
assert trace.num_turns >= FAIL_AFTER_TURNS
await assert_no_leaks(taken)


@pytest.mark.parametrize(
"harness,harness_runtime", grid(["bash"], needs=NEEDS), indirect=True
)
async def test_a_short_rollout_takes_no_snapshot(run, taken, harness, harness_runtime):
"""With the default interval (every 10th turn), a rollout that fails at turn 3
has taken no snapshot: its retry starts over and still succeeds."""
(episode,) = await run(
"checkpoint-restore-v1",
harness=harness,
runtime=harness_runtime,
env=checkpoint_env(),
max_turns=12,
max_tokens=4096,
)
(trace,) = episode.traces
assert trace.ok, [error.message for error in trace.errors]
assert trace.reward == 1.0
assert [error.type for error in trace.errors] == ["ProviderError"]
assert trace.attempt == 2
assert not trace.has_restored
assert trace.checkpoints == []
assert taken == []


@pytest.mark.parametrize(
"harness,harness_runtime",
grid(["rlm"], needs=(*NEEDS, "skills")),
indirect=True,
)
async def test_rlm_with_skills_restores(run, taken, harness, harness_runtime):
"""rlm keeps its session in a per-run skill home when it has skills: the retry
must load the session from the same home the restored box holds."""
config = harness.model_copy(update={"skills": [SKILL]})
trace = await restored_trace(run, config, harness_runtime, interval=1)
assert trace.num_turns >= FAIL_AFTER_TURNS
await assert_no_leaks(taken)


@pytest.mark.parametrize("keep", ["last", 2], ids=["keep-last", "keep-every-2nd"])
@pytest.mark.parametrize(
"harness,harness_runtime", grid(["bash"], needs=NEEDS), indirect=True
)
async def test_checkpoint_retention(run, taken, harness, harness_runtime, keep):
"""Snapshots the policy drops are deleted while the rollout runs, without breaking
the newest one: the retry still restores it. Every snapshot is deleted once the
run is final."""
trace = await restored_trace(run, harness, harness_runtime, interval=1, keep=keep)
assert all(checkpoint.deleted for checkpoint in trace.checkpoints)
await assert_no_leaks(taken)


async def restored_trace(run, harness, harness_runtime, **checkpoint):
(episode,) = await run(
"checkpoint-restore-v1",
harness=harness,
runtime=harness_runtime,
env=checkpoint_env(**checkpoint),
max_turns=12,
max_tokens=4096,
)
(trace,) = episode.traces
assert trace.ok, [error.message for error in trace.errors]
assert trace.reward == 1.0
assert [error.type for error in trace.errors] == ["ProviderError"]
# The newest snapshot before the failed turn boots the retry's box.
assert trace.agent.runtime is not None
restored_from = trace.agent.runtime.checkpoint
assert restored_from is not None
assert [c.id for c in trace.checkpoints if c.restored] == [restored_from]
assert trace.has_restored
assert trace.attempt == 2
# Each checkpoint records what its snapshot captured.
assert {c.snapshot.type for c in trace.checkpoints} == {"filesystem"}
# The error belongs to the first attempt, in the agent stage.
assert [(error.attempt, error.stage) for error in trace.errors] == [(1, "agent")]
# The restored process made no side request of its own (it would start a second
# root), e.g. a model probe on session load.
assert sum(node.parent is None for node in trace.nodes) == 1
# The restored process got the restore notice as a user turn.
assert any(
node.message.role == "user"
and RESTORE_PROMPT in content_text(node.message.content)
for node in trace.nodes
)
# Every turn kept has its call; the dropped turns' calls stay, unlinked, so usage
# counts them.
assert [call.node for call in trace.calls if call.node is not None] == [
i for i, node in enumerate(trace.nodes) if node.sampled
]
assert any(call.node is None for call in trace.calls)
return trace
Loading
Loading