Repository navigation
fix(task_step): give snapshot_agent_state's save the transfer timeout, including stored steps - #687
Draft
andrewpark-scaleai wants to merge 2 commits into
Draft
andrewpark-scaleai wants to merge 2 commits into
andrewpark-scaleai wants to merge 2 commits into
Conversation
added 2 commits
November 15, 2026 07:46
…ault timeout to_dict wrote timeout_seconds into every stored step, so a step saved before the new default kept 120 s. Write the key only when a timeout is set, and read a stored 120, the old default, as unset.
andrewpark-scaleai
force-pushed
the
andrew-park/rel-snapshot-transfer-timeout
branch
from
November 15, 2026 07:47
1b478ff to
ad1a09e
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The goal is that every failure of an agent run leaves enough in the logs to debug it, and that code heals what it can. This PR is an interim fix for
snapshot_agent_statesaves that time out on large workspaces. It moves the wall; it does not remove the cause. The cause is that the save gzips the whole workspace on one core, which gains nothing on large incompressible media. A separate PR stops that compression, and another makes the save report its size and phase durations.The step's default wait was 120 s. That breaks the order
object_transfer.pydocuments: agent-env must wait longer than the SDK's stall budget for one upload (3 attempts of 30 s connect plus 60 s idle, plus 3 s, so 273 s). With 120 s, agent-env gives up while the agent is still inside a legitimate upload retry. A call that moves objects getsTRANSFER_TIMEOUT_SECONDS(600 s) everywhere else.A new default alone does not reach existing tasks.
to_dictwrotetimeout_secondsinto every stored step, andfrom_dictread the stored value back. So every step saved through the framework, and every template that copied the old default, stores a literal 120.What changed
SnapshotAgentStateTaskStep.DEFAULT_TIMEOUT_SECONDSis nowTRANSFER_TIMEOUT_SECONDS(600 s).timeout_secondsisNoneunless set.to_dictwrites it only when set, and the save falls back to the class default at run time.from_dictreads a stored 120 as unset, so stored steps pick up the current default.Design decisions
snapshot_save_timeout. That log now lives in the save telemetry PR, which logs the success path too, so the two PRs do not conflict.Scope
snapshot_agent_stateonly. Periodic captures insnapshot_seriespass their own deadline and are unchanged.Blast Radius
Steps that omit
timeout_secondsor store 120 now wait up to 600 s instead of 120 s. A stuck agent holds a worker slot 480 s longer before the step fails. Older releases load a step without the key with their own default, so stored tasks stay loadable both ways.Verification
.venv/bin/python -m pytest tst/unit/task_step/test_snapshot_agent_state.py -q(the uv dev venv): 19 passed. Without the second commit's source change, 2 fail: the stored dict still holdstimeout_seconds, and a stored 120 is kept.test_agent_state_capture.py,object_transfer_test.py,test_prompt_agent_snapshots.py,test_deploy_agent_snapshot_objects.py,tst/unit/task/episode,importer_entities_test.py,snapshot_step_output_test.py): 310 passed.run_task_step failedline forstep_type=snapshot_agent_statewith anhttpx.ReadTimeoutshould show about 600 s elapsed, not 120 s. Saves of workspaces under a few GB should stop failing on the wait.