Repository navigation
Conversation
A saved turn replaced its inline image parts with [image], so the next turn replayed a placeholder where the picture had been. The provider's cached prefix broke at that message and every later turn repaid the prompt behind it. The save now archives the exact data URI in _image_refs[].url on the message. The display content keeps the [image] placeholder, and the replay projection expands the archived string before the provider call. A missing or malformed URL leaves the placeholder; remote URLs are untouched; token estimates include the archived picture. This keeps the session self-contained: deleting the session deletes its images, and an exported session carries them. The cost is a larger session JSONL and a linear session load for image-heavy transcripts; the trajectory index is unaffected because it reads the span log and bounded artifact previews, not the session file.
gloryfromca
left a comment
There was a problem hiding this comment.
No blockers; this can merge as far as I am concerned.
Reviewed the full diff and the surrounding save, replay, provider projection, session-resume, image-withdrawal, token-accounting, and context-compaction paths. I also checked the repository rules and canonical terminology, commit history, backward compatibility for old or malformed stored messages, test changes for weakening, and the Runtime architecture boundary. The private refs stay out of provider and transcript wire projections, while replay reconstructs only matching placeholders and gracefully leaves malformed refs untouched.
Verification:
uv run pytest tests/test_read_file_image.py -x- 116 passeduv run pytest tests/test_context_history_strategies.py tests/test_context_runtime_context.py tests/test_context_compacting_runtime.py tests/test_agent_window_parts.py tests/test_rpc_session.py -x- 413 passeduv run python -m scripts.check_source_language github/main...HEAD- passedgit diff --check github/main...HEAD- passed
The first focused test attempt stopped during fixture setup because the fresh default uv environment omitted the declared raven_everos workspace package; after uv sync --all-packages, the unchanged test command passed.
There was a problem hiding this comment.
Blocking: replayed pictures skip the gates that bound and route live pictures, so user photos accumulate in every request until the endpoint refuses, and a model or transport that cannot take an image is sent one on every later turn.
replayed_content now runs expand_images on every history message (runtime_context.py:68), with no input about the request it feeds. The two things that keep pictures in check were built on base's premise that a picture lives for one turn, and neither sees a replayed one the way it should:
-
The standing image window never counts a user picture.
window_imagesonly collectsrole == "tool"or_attached_imagemessages (shrink.py:250-255); its docstring says why: "A picture the user sent is the subject of the turn rather than a result, and stays". That held while a user picture was filed as[image]. With_image_refson the user branch (turn_path.py:4402-4415), every photo from every earlier turn is sent again andimageWindowBudgetBytes(12,000,000 by default) never applies to them. When the endpoint then refuses for size, the refusal ladder closes the window withany_role(harness/memory.py:425-433) and withdraws every picture, including the one attached to the question being asked.without_picturespops the refs only from the live copy, so the session file still has them and the next turn repeats the refusal. -
Capability gates apply only to the current message.
_supports_visiongates the user's new attachment (render.py:1040) and_supports_image_tool_resultgates new tool results (organ_glue.py:92-103). An archived picture is replayed after a session switches to a model without vision, or to a transport that cannot carry images inrole=tool, so each later turn first sends a request base never sent, gets refused, and retries. With a refusal worded outside the patterns inproviders/base.py, the turn fails instead.
Measured on head 6d816d9 merged onto main fc0a30f against base fc0a30f, real AgentLoop._process_message, stub providers, pyt.sh with scratch homes:
case base head
14 turns, each attaching one ~1 MB photo 1 picture / request (1.0 MB) turn N carries N pictures (14.0 MB at turn 14)
session file after 14 turns 25,259 bytes 14,053,015 bytes
same, endpoint refuses requests over 12 MB 0 refusals, picture kept turns 12-14: 1 refused call each, then
answered WITHOUT the picture just sent
tool picture on turn 1, then switch to a turns 2-3: 0 images turns 2-3: image sent at [3,'tool'],
model without vision refused, retried without it
same, switch to a transport without turns 2-3: 0 images turns 2-3: image in role=tool, refused,
tool-result images then demoted to a user message
The cache goal does not need either behaviour. A shape that keeps it: let the window count replayed user pictures from earlier turns (exempting only the current turn's question), and make the replay projection honour the same two capability verdicts the live path uses, leaving the placeholder when the receiving model or transport cannot take the picture. That keeps the bytes identical for the requests that can carry them, which is where the cached prefix matters.
Not blocking, for the same revision: raven/templates/TOOLS.md:23 still tells the model "An image is visible for the turn that read it. Later turns see a placeholder".
There was a problem hiding this comment.
Blocking: archived replay must honor the existing image-window and capability gates before merge.
This holds. I reproduced the first path directly: fourteen archived user messages with 1,000,000-byte base64 payloads project to 14,000,000 bytes and 14 images, then window_images(..., budget=12_000_000) returns (0, 0) and leaves all 14 images because ordinary role="user" messages are outside its candidate set. The second path follows from the same unconditional projection: HistoryTrimmer.history_from_ids calls replayed_content without the selected model or transport capability, so an old image is reconstructed after the current-message capability decisions have already been made.
The unchanged image suite still passes (116 tests), which confirms the current coverage does not exercise these replay gates. I am updating my own stance to blocking; I am not resolving this thread because it is yours and remains open.
gloryfromca
left a comment
There was a problem hiding this comment.
Blocking: archived replay must honor the existing image-window and capability gates before merge.
The new review evidence invalidates my earlier clean stance. I reproduced archived user-image accumulation past the configured 12,000,000-byte budget: 14,000,000 replayed bytes remained untouched because the standing window excludes ordinary user messages. Archived images are also reconstructed without the selected model's vision verdict or the transport's tool-image capability.
Verification this turn:
- deterministic replay/window reproduction - 14 images and 14,000,000 bytes remained;
window_imageschanged and withdrew zero uv run pytest tests/test_read_file_image.py -x- 116 passed, showing the regression is not covered by the current suite
I replied on the existing blocker rather than duplicating it as a new finding.
Summary
A saved turn replaced its inline image parts with
[image], so the next turn replayed a placeholder where the picture had been. The provider's cached prefix broke at that message, and every later turn repaid the prompt behind it.The save now archives the exact data URI in
_image_refs[].urlon the saved message. The display content keeps the[image]placeholder, and the replay projection expands the archived string before the provider call. The private field is dropped by the provider allowlist and by the session view, so transcript views and the wire payload keep their existing shape. A missing or malformed URL leaves the placeholder; remote URLs are untouched; token estimates include the archived picture.The session becomes self-contained: deleting the session deletes its images, and an exported session carries them. The trade-off is a larger session JSONL and a linear session load for image-heavy transcripts. A synthetic benchmark with 40 images of 1.5MB each measured an 84MB JSONL and a 219ms cold read, versus 19KB and 0.26ms for a side-store reference. The trajectory index is unaffected because it reads the span log and bounded artifact previews, not the session JSONL.
Type
Verification
No user-facing docs or screenshots are needed: the display content and the RPC session view are unchanged.
Risk
Security: the session JSONL now carries the image bytes in the private
_image_refsfield. That field is dropped by the provider allowlist and by the session RPC view, but an exported session file now contains the pictures. Session files should be handled with the same care as the images themselves.Backward compatibility: sessions saved before this change have no
_image_refs, so their replay keeps the existing[image]placeholder. Sessions saved after it are read by older binaries as plain[image]placeholders because the private field is unknown to them.Rollback: revert this commit. Sessions already written keep the archived field, which older code ignores, so replay falls back to the placeholder instead of failing.
Related Issues
N/A