Skip to content

fix(task_step): give snapshot_agent_state's save the transfer timeout, including stored steps - #687

Draft
andrewpark-scaleai wants to merge 2 commits into
mainfrom
andrew-park/rel-snapshot-transfer-timeout
Draft

andrewpark-scaleai wants to merge 2 commits into
mainfrom
andrew-park/rel-snapshot-transfer-timeout

Conversation

@andrewpark-scaleai

@andrewpark-scaleai andrewpark-scaleai commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Why

The goal is that every failure of an agent run leaves enough in the logs to debug it, and that code heals what it can. This PR is an interim fix for snapshot_agent_state saves that time out on large workspaces. It moves the wall; it does not remove the cause. The cause is that the save gzips the whole workspace on one core, which gains nothing on large incompressible media. A separate PR stops that compression, and another makes the save report its size and phase durations.

The step's default wait was 120 s. That breaks the order object_transfer.py documents: agent-env must wait longer than the SDK's stall budget for one upload (3 attempts of 30 s connect plus 60 s idle, plus 3 s, so 273 s). With 120 s, agent-env gives up while the agent is still inside a legitimate upload retry. A call that moves objects gets TRANSFER_TIMEOUT_SECONDS (600 s) everywhere else.

A new default alone does not reach existing tasks. to_dict wrote timeout_seconds into every stored step, and from_dict read the stored value back. So every step saved through the framework, and every template that copied the old default, stores a literal 120.

What changed

  • SnapshotAgentStateTaskStep.DEFAULT_TIMEOUT_SECONDS is now TRANSFER_TIMEOUT_SECONDS (600 s).
  • timeout_seconds is None unless set. to_dict writes it only when set, and the save falls back to the class default at run time.
  • from_dict reads a stored 120 as unset, so stored steps pick up the current default.

Design decisions

  • A stored 120 means "default". The stored data cannot tell the old default from an author who chose 120: the field was always written, and 120 is the only default it ever had. Reading it as unset costs a deliberate 120 at most 480 s more wait on a stuck agent. A longer wait never fails a save that 120 s would pass. Honoring the stored 120 would leave every existing task on the broken budget. An author who wants a short budget can set any other value, which round-trips unchanged.
  • Other values are kept. A stored value below 273 s still breaks the order above, but an author chose it, so this PR does not clamp it.
  • The timeout log moved. An earlier revision of this PR also logged snapshot_save_timeout. That log now lives in the save telemetry PR, which logs the success path too, so the two PRs do not conflict.

Scope

  • Covers snapshot_agent_state only. Periodic captures in snapshot_series pass their own deadline and are unchanged.
  • Leaves the root cause (compression) and the missing save telemetry to their own PRs. A fixed wait still fails a large enough workspace on a slow sandbox. A deadline that tracks progress needs a protocol change.

Blast Radius

Steps that omit timeout_seconds or store 120 now wait up to 600 s instead of 120 s. A stuck agent holds a worker slot 480 s longer before the step fails. Older releases load a step without the key with their own default, so stored tasks stay loadable both ways.

Verification

  • .venv/bin/python -m pytest tst/unit/task_step/test_snapshot_agent_state.py -q (the uv dev venv): 19 passed. Without the second commit's source change, 2 fail: the stored dict still holds timeout_seconds, and a stored 120 is kept.
  • Related suites (test_agent_state_capture.py, object_transfer_test.py, test_prompt_agent_snapshots.py, test_deploy_agent_snapshot_objects.py, tst/unit/task/episode, importer_entities_test.py, snapshot_step_output_test.py): 310 passed.
  • After deploy: a run_task_step failed line for step_type=snapshot_agent_state with an httpx.ReadTimeout should show about 600 s elapsed, not 120 s. Saves of workspaces under a few GB should stop failing on the wait.

Ubuntu added 2 commits November 15, 2026 07:46
…ault timeout

to_dict wrote timeout_seconds into every stored step, so a step saved
before the new default kept 120 s. Write the key only when a timeout is
set, and read a stored 120, the old default, as unset.
@andrewpark-scaleai
andrewpark-scaleai force-pushed the andrew-park/rel-snapshot-transfer-timeout branch from 1b478ff to ad1a09e Compare November 15, 2026 07:47
@andrewpark-scaleai andrewpark-scaleai changed the title fix(task_step): give snapshot_agent_state's save the transfer timeout by default fix(task_step): give snapshot_agent_state's save the transfer timeout, including stored steps Nov 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant