Skip to content

feat(session): write a turn to disk while it runs - #1374

Open
LivXue wants to merge 5 commits into
mainfrom
feat/incremental_session_persistence
Open

LivXue wants to merge 5 commits into
mainfrom
feat/incremental_session_persistence

Conversation

@LivXue

@LivXue LivXue commented Nov 20, 2026

Copy link
Copy Markdown
Member

Summary

A turn used to reach the session file once, at its end. A process killed at its budget -- which is what a benchmark or a channel turn does when the budget runs out -- left only the question behind, and nothing could be recovered from the attempt.

This change writes the turn as it happens:

  • Rows are appended as they are produced: before a tool call runs (so a kill during the tool leaves a call that can be repaired) and at the start of every iteration (which files the previous round's results). The record comes from the turn log, not from the window a model call saw, so a compaction mid-turn cannot file a summary as the turn's history.
  • A call that never got its result is repaired on read: read_session_file appends an [interrupted] tool result for it. A strict provider rejects a history whose tool_calls have no matching tool message, and the repair belongs on the way in -- the file keeps saying what actually happened. The same helper now serves the cancelled-turn path, so there is one implementation.
  • Each row carries a stable _row_id, minted by the turn log when it records the row.
  • The end-of-turn stamps travel as one record. replied_via_tool and the hook observers (which ACP reads as _meta, including the hold marker) land after the rows are on disk, so they are written as one {"_type": "turn_meta", "patches": [{"row_id", "keys"}]} record that the reader merges in file order. Patching by position was tried and does not work: the record can be shorter than the list that was written.
  • A row already on disk is never written twice, and the save boundary is read from the ids actually written rather than from a count. A continuation can merge its row into the step it continues and leave the record shorter than when it was written, which is what made a count unreliable.
  • The turn log gained adopt_appended: a list is adopted only when it is the record grown at the end. A window pass's shrunken copy is a view of the record, and following it drops rows the turn produced.

The session file format gains two record types (_row_id on rows, and the turn_meta record). Every reader goes through read_session_file, which applies the patches; the replay, export and ACP paths read through it.

Type

  • Feature
  • Fix
  • Docs
  • CI / tooling
  • Refactor
  • Other

Verification

  • uv run pytest tests/test_agent_loop_session_persistence.py tests/test_agent_loop_empty_recovery.py tests/test_session_manager.py -q -> 650 passed

  • uv run pytest tests/test_agent_loop_session_persistence.py tests/test_agent_loop_iteration_hooks.py tests/test_agent_loop_dead_end.py tests/test_agent_loop_dead_end_retry.py tests/test_session_manager.py tests/test_acp_methods.py -q -> 421 passed

  • uv run ruff check and uv run ruff format --check pass on every touched file.

  • New tests: a tool round is on disk before the next call; an assistant call with no result is repaired when the file is read; the closing answer is filed exactly once and no row is written under two ids; a turn_meta record restates a row by its identifier.

  • Relevant tests pass locally

  • Relevant lint / type checks pass locally

  • User-facing docs or screenshots are updated when needed

Risk

  • The session file now carries _row_id on rows and turn_meta records. Older readers that build history from raw rows would see the extra key (transports send an allowlist, so it does not reach a provider); this change updates the one reader, read_session_file, and everything else reads through it.

  • A turn now appends several times instead of once. The cost is one locked append per step; it was not benchmarked here beyond the suite's timing, which did not move.

  • If the process is killed with SIGKILL mid-step, the last streamed text is still lost. That is the remaining gap this change narrows but does not close.

  • Rollback: revert the commit. Sessions written meanwhile stay readable: turn_meta is ignored by older readers and _row_id is an extra key.

  • Security impact considered

  • Backward compatibility considered

  • Rollback path is clear for risky changes

Related Issues

N/A

A turn was written once, at its end, so a process killed at its budget left
only the question behind and nothing could be recovered from the attempt.

Rows now reach the file as the turn produces them: before a tool call runs, and
at the start of every iteration. A row whose tool call never got an answer is
repaired when a transcript is read -- as an [interrupted] tool result -- rather
than left for a strict provider to reject.

Each row carries a stable row id. The stamps a turn files at its end, the
reply's provenance and the hook observers, land after those rows are already
on disk, so they travel as one turn_meta record that restates the row by id;
the reader merges it in file order. A row already on disk is never written a
second time, and the save boundary is read from the ids that were written
rather than from a count, because a continuation can merge its row into the
step it continues and leave the record shorter than it was.

The turn log gained adopt_appended: a list is adopted only when it is the
record grown at the end. A window pass's shrunken copy is a view of the record,
not the record, and following it would drop rows the turn produced.

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: recovery scaffolding must stay transient, and interrupted tool-call repair must preserve provider-valid ordering across restarts.

I reviewed github/main...HEAD, the surrounding turn-log/session writer and provider serialization paths, relevant callers, the branch history, backward compatibility, and the repository rules in AGENTS.md, CONTEXT-MAP.md, and CONTEXT.md. I also checked that the added tests were not weakened; they pass but do not cover either failing lifecycle below.

Verification:

  • uv run pytest tests/test_agent_loop_session_persistence.py -q -n 0 - 4 passed
  • uv run pytest tests/test_agent_loop_broken_turn.py tests/test_agent_loop_session_stamps.py tests/test_session_manager.py tests/test_session_revision.py tests/test_session_turn_shape.py -q -n 0 - 311 passed
  • uv run pytest tests/test_agent_loop_dead_end_retry.py -q -n 0 - 45 passed
  • Direct reproductions confirmed a _recovery_synthetic row is written to JSONL and a crash/resume/save/restart history reloads as assistant tool call, user, assistant, then synthetic tool result.

The first combined test attempt produced 29 setup errors because the fresh environment lacked the workspace raven_everos package. After uv sync --all-packages, the commands above passed.

Comment thread raven/agent/loop/turn_path.py Resolved
Comment thread raven/session/manager.py
keys = patch.get("keys")
if index is not None and isinstance(keys, dict):
messages[index].update(keys)
messages = repair_dangling_tool_calls(messages)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: place interrupted results at the matching call boundary while preserving disk accounting.

repair_dangling_tool_calls() appends every repair to the end. That works only while the interrupted call is the file tail. In the ordinary crash recovery lifecycle, the first reload appends a synthetic result in memory, _persisted_count counts it although it is intentionally absent from disk, and the next save appends the resumed user/assistant rows without that result. A second restart then reloads assistant(tool_calls), user, assistant, tool, which strict chat providers reject because the tool result no longer immediately follows its call. The added test already constructs a later assistant row but asserts only that a tool row exists, not its provider-valid position.

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: the two previously reported persistence defects remain unchanged.

The new commit only makes the two session-row scans tolerate session-like objects without a messages attribute. I reviewed that delta and the full resulting diff, affected callers and provider serialization, branch history, backward compatibility, tests, and the repository rules in AGENTS.md, CONTEXT-MAP.md, and CONTEXT.md. It does not change incremental persistence's unfiltered log.messages() call or the end-appending interrupted-call repair, so both existing threads still stand.

Verification on this head: uv run pytest tests/test_agent_loop_session_persistence.py tests/test_agent_loop_broken_turn.py tests/test_agent_loop_session_stamps.py tests/test_agent_loop_dead_end_retry.py tests/test_agent_loop_turn_record.py tests/test_session_manager.py tests/test_session_revision.py tests/test_session_turn_shape.py -q -n 0 passed, 381 tests. The crash/resume/save/restart reproduction still reloads the rows as assistant tool call, user, assistant, then tool result.

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: interrupted tool-call repair must preserve provider-valid ordering across resume and restart.

I reviewed the new delta and the full resulting diff, affected session and provider callers, branch history, backward compatibility, test changes, and the repository rules in AGENTS.md, CONTEXT-MAP.md, and CONTEXT.md. The row-ID cleanup and added late-stamp mappings are coherent, and the changed test still verifies the metadata patch while additionally checking that the filing key stays internal.

I withdrew and resolved my recovery-scaffolding finding after refuting it against _save_turn's existing final gate. The interrupted-call finding still reproduces: after crash, resume/save, and restart, the loaded order is assistant tool call, user, assistant, then tool result. At this fourth review round I am keeping that item blocking because all three convergence conditions hold: this change introduces the read-time repair/accounting mismatch; interruption followed by resuming and later restarting is ordinary operation; and once strict providers reject the malformed history, the affected session has no in-session recovery path short of discarding or manually editing its history.

Verification: uv run pytest tests/test_agent_loop_session_persistence.py tests/test_agent_loop_broken_turn.py tests/test_agent_loop_session_stamps.py tests/test_agent_loop_dead_end_retry.py tests/test_agent_loop_turn_record.py tests/test_session_manager.py tests/test_session_revision.py tests/test_session_turn_shape.py -q -n 0 passed, 381 tests. Direct checks confirmed synthetic recovery rows are not persisted and the interrupted-call ordering failure remains.

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: the existing interrupted tool-call ordering defect remains.

This delta only defers incremental filing at picture-bearing rows until the window replaces the picture with its durable note. I reviewed that change and the full resulting diff, affected window/session/provider callers, branch history, backward compatibility, test coverage, and the repository rules in AGENTS.md, CONTEXT-MAP.md, and CONTEXT.md; I found no new high-bar issue in the image boundary. The open interrupted-call thread is untouched, its lifecycle still reproduces on this head, and the convergence conditions stated in the prior review remain satisfied.

Verification: uv run pytest tests/test_agent_loop_session_persistence.py tests/test_agent_loop_broken_turn.py tests/test_agent_loop_session_stamps.py tests/test_agent_loop_dead_end_retry.py tests/test_agent_loop_turn_record.py tests/test_agent_window_parts.py tests/test_read_file_image.py tests/test_session_manager.py tests/test_session_revision.py tests/test_session_turn_shape.py -q -n 0 passed, 520 tests, with two unrelated Pydantic warnings from Ollama TypedDict schemas.

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: the existing interrupted tool-call ordering defect remains.

This revision extracts the picture-row boundary into _incremental_upto and adds focused coverage for that helper and TurnLog.adopt_appended; those tests accurately preserve the implemented behavior. I reviewed the delta and full resulting diff, affected callers, history, backward compatibility, test changes, and the repository rules in AGENTS.md, CONTEXT-MAP.md, and CONTEXT.md. I found no new high-bar issue, but the open blocker is untouched and still reproduces on this head; the previously stated convergence conditions remain satisfied.

Verification: uv run pytest tests/test_agent_loop_session_persistence.py tests/test_agent_loop_broken_turn.py tests/test_agent_loop_session_stamps.py tests/test_agent_loop_dead_end_retry.py tests/test_agent_loop_turn_record.py tests/test_agent_window_parts.py tests/test_read_file_image.py tests/test_session_manager.py tests/test_session_revision.py tests/test_session_turn_shape.py -q -n 0 passed, 522 tests, with two unrelated Pydantic warnings from Ollama TypedDict schemas.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants