Skip to content

fix(agent): archive inline images so replayed turns keep the cache - #1376

Open
LivXue wants to merge 1 commit into
mainfrom
fix/replay_inline_images
Open

LivXue wants to merge 1 commit into
mainfrom
fix/replay_inline_images

Conversation

@LivXue

@LivXue LivXue commented Nov 20, 2026

Copy link
Copy Markdown
Member

Summary

A saved turn replaced its inline image parts with [image], so the next turn replayed a placeholder where the picture had been. The provider's cached prefix broke at that message, and every later turn repaid the prompt behind it.

The save now archives the exact data URI in _image_refs[].url on the saved message. The display content keeps the [image] placeholder, and the replay projection expands the archived string before the provider call. The private field is dropped by the provider allowlist and by the session view, so transcript views and the wire payload keep their existing shape. A missing or malformed URL leaves the placeholder; remote URLs are untouched; token estimates include the archived picture.

The session becomes self-contained: deleting the session deletes its images, and an exported session carries them. The trade-off is a larger session JSONL and a linear session load for image-heavy transcripts. A synthetic benchmark with 40 images of 1.5MB each measured an 84MB JSONL and a 219ms cold read, versus 19KB and 0.26ms for a side-store reference. The trajectory index is unaffected because it reads the span log and bounded artifact previews, not the session JSONL.

Type

  • Fix
  • Feature
  • Docs
  • CI / tooling
  • Refactor
  • Other

Verification

  • Relevant tests pass locally
  • Relevant lint / type checks pass locally
  • User-facing docs or screenshots are updated when needed

No user-facing docs or screenshots are needed: the display content and the RPC session view are unchanged.

uv run --frozen --all-extras pytest tests/test_read_file_image.py -q -p no:randomly -n 0
# 115 passed, 1 skipped (the skip is a root-only chmod case)

uv run --frozen --all-extras pytest tests/test_context_runtime_context.py tests/test_context_turn_scratch.py tests/test_context_compacting_runtime.py tests/test_rpc_session.py tests/test_token_estimation.py tests/test_agent_window_parts.py tests/test_agent_loop_turn_record.py tests/test_agent_loop_token_budget.py -q -p no:randomly -n 0
# 511 passed

uv run --frozen --all-extras ruff check raven/utils/image_store.py raven/utils/tokens.py raven/context_engine/runtime_context.py raven/agent/loop/turn_path.py raven/agent/window/images.py raven/context_engine/segments/render.py raven/agent/loop/_shared.py tests/test_read_file_image.py
uv run --frozen --all-extras ruff format --check raven/utils/image_store.py raven/utils/tokens.py raven/context_engine/runtime_context.py raven/agent/loop/turn_path.py raven/agent/window/images.py raven/context_engine/segments/render.py raven/agent/loop/_shared.py tests/test_read_file_image.py
uv run --frozen --all-extras ty check raven/utils/image_store.py raven/utils/tokens.py raven/context_engine/runtime_context.py raven/agent/loop/turn_path.py raven/agent/window/images.py raven/context_engine/segments/render.py raven/agent/loop/_shared.py
# all pass

make lint-imports
# 10 kept, 0 broken

make check-source-language
make check-large-files
git diff --check origin/main..HEAD
# all pass

Risk

  • Security impact considered
  • Backward compatibility considered
  • Rollback path is clear for risky changes

Security: the session JSONL now carries the image bytes in the private _image_refs field. That field is dropped by the provider allowlist and by the session RPC view, but an exported session file now contains the pictures. Session files should be handled with the same care as the images themselves.

Backward compatibility: sessions saved before this change have no _image_refs, so their replay keeps the existing [image] placeholder. Sessions saved after it are read by older binaries as plain [image] placeholders because the private field is unknown to them.

Rollback: revert this commit. Sessions already written keep the archived field, which older code ignores, so replay falls back to the placeholder instead of failing.

Related Issues

N/A

A saved turn replaced its inline image parts with [image], so the next turn replayed a placeholder where the picture had been. The provider's cached prefix broke at that message and every later turn repaid the prompt behind it.

The save now archives the exact data URI in _image_refs[].url on the message. The display content keeps the [image] placeholder, and the replay projection expands the archived string before the provider call. A missing or malformed URL leaves the placeholder; remote URLs are untouched; token estimates include the archived picture.

This keeps the session self-contained: deleting the session deletes its images, and an exported session carries them. The cost is a larger session JSONL and a linear session load for image-heavy transcripts; the trajectory index is unaffected because it reads the span log and bounded artifact previews, not the session file.
@LivXue
LivXue requested review from 0xKT and gloryfromca November 20, 2026 11:31

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers; this can merge as far as I am concerned.

Reviewed the full diff and the surrounding save, replay, provider projection, session-resume, image-withdrawal, token-accounting, and context-compaction paths. I also checked the repository rules and canonical terminology, commit history, backward compatibility for old or malformed stored messages, test changes for weakening, and the Runtime architecture boundary. The private refs stay out of provider and transcript wire projections, while replay reconstructs only matching placeholders and gracefully leaves malformed refs untouched.

Verification:

  • uv run pytest tests/test_read_file_image.py -x - 116 passed
  • uv run pytest tests/test_context_history_strategies.py tests/test_context_runtime_context.py tests/test_context_compacting_runtime.py tests/test_agent_window_parts.py tests/test_rpc_session.py -x - 413 passed
  • uv run python -m scripts.check_source_language github/main...HEAD - passed
  • git diff --check github/main...HEAD - passed

The first focused test attempt stopped during fixture setup because the fresh default uv environment omitted the declared raven_everos workspace package; after uv sync --all-packages, the unchanged test command passed.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: replayed pictures skip the gates that bound and route live pictures, so user photos accumulate in every request until the endpoint refuses, and a model or transport that cannot take an image is sent one on every later turn.

replayed_content now runs expand_images on every history message (runtime_context.py:68), with no input about the request it feeds. The two things that keep pictures in check were built on base's premise that a picture lives for one turn, and neither sees a replayed one the way it should:

  1. The standing image window never counts a user picture. window_images only collects role == "tool" or _attached_image messages (shrink.py:250-255); its docstring says why: "A picture the user sent is the subject of the turn rather than a result, and stays". That held while a user picture was filed as [image]. With _image_refs on the user branch (turn_path.py:4402-4415), every photo from every earlier turn is sent again and imageWindowBudgetBytes (12,000,000 by default) never applies to them. When the endpoint then refuses for size, the refusal ladder closes the window with any_role (harness/memory.py:425-433) and withdraws every picture, including the one attached to the question being asked. without_pictures pops the refs only from the live copy, so the session file still has them and the next turn repeats the refusal.

  2. Capability gates apply only to the current message. _supports_vision gates the user's new attachment (render.py:1040) and _supports_image_tool_result gates new tool results (organ_glue.py:92-103). An archived picture is replayed after a session switches to a model without vision, or to a transport that cannot carry images in role=tool, so each later turn first sends a request base never sent, gets refused, and retries. With a refusal worded outside the patterns in providers/base.py, the turn fails instead.

Measured on head 6d816d9 merged onto main fc0a30f against base fc0a30f, real AgentLoop._process_message, stub providers, pyt.sh with scratch homes:

case                                         base                        head
14 turns, each attaching one ~1 MB photo     1 picture / request (1.0 MB) turn N carries N pictures (14.0 MB at turn 14)
  session file after 14 turns                25,259 bytes                14,053,015 bytes
same, endpoint refuses requests over 12 MB   0 refusals, picture kept     turns 12-14: 1 refused call each, then
                                                                          answered WITHOUT the picture just sent
tool picture on turn 1, then switch to a     turns 2-3: 0 images          turns 2-3: image sent at [3,'tool'],
  model without vision                                                    refused, retried without it
same, switch to a transport without          turns 2-3: 0 images          turns 2-3: image in role=tool, refused,
  tool-result images                                                      then demoted to a user message

The cache goal does not need either behaviour. A shape that keeps it: let the window count replayed user pictures from earlier turns (exempting only the current turn's question), and make the replay projection honour the same two capability verdicts the live path uses, leaving the placeholder when the receiving model or transport cannot take the picture. That keeps the bytes identical for the requests that can carry them, which is where the cached prefix matters.

Not blocking, for the same revision: raven/templates/TOOLS.md:23 still tells the model "An image is visible for the turn that read it. Later turns see a placeholder".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: archived replay must honor the existing image-window and capability gates before merge.

This holds. I reproduced the first path directly: fourteen archived user messages with 1,000,000-byte base64 payloads project to 14,000,000 bytes and 14 images, then window_images(..., budget=12_000_000) returns (0, 0) and leaves all 14 images because ordinary role="user" messages are outside its candidate set. The second path follows from the same unconditional projection: HistoryTrimmer.history_from_ids calls replayed_content without the selected model or transport capability, so an old image is reconstructed after the current-message capability decisions have already been made.

The unchanged image suite still passes (116 tests), which confirms the current coverage does not exercise these replay gates. I am updating my own stance to blocking; I am not resolving this thread because it is yours and remains open.

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: archived replay must honor the existing image-window and capability gates before merge.

The new review evidence invalidates my earlier clean stance. I reproduced archived user-image accumulation past the configured 12,000,000-byte budget: 14,000,000 replayed bytes remained untouched because the standing window excludes ordinary user messages. Archived images are also reconstructed without the selected model's vision verdict or the transport's tool-image capability.

Verification this turn:

  • deterministic replay/window reproduction - 14 images and 14,000,000 bytes remained; window_images changed and withdrew zero
  • uv run pytest tests/test_read_file_image.py -x - 116 passed, showing the regression is not covered by the current suite

I replied on the existing blocker rather than duplicating it as a new finding.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants