Repository navigation
fix(agent): archive inline images so replayed turns keep the cache #1376
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
LivXue
wants to merge
1
commit into
main
Choose a base branch
from
fix/replay_inline_images
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+459
−37
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,153 @@ | ||
| """The archived image refs a session replays. | ||
|
|
||
| A message's display content keeps ``[image]`` in place of an inline picture: the | ||
| bytes would otherwise bloat every transcript view and the session's text. But | ||
| the next turn has to send the same picture, character for character, or the | ||
| provider's cached prefix breaks at it and everything after it is prefilled | ||
| again. | ||
|
|
||
| The save pass therefore archives the exact data URI in a private | ||
| ``_image_refs`` field beside the placeholder. The replay pass reads it back, so | ||
| what reaches the provider is the picture that was sent; ``content`` stays the | ||
| display form. The provider allow-list drops the refs, and the session view | ||
| never sees them. | ||
|
|
||
| Archiving trades session-load time for exact replay: the session loader parses | ||
| every line, so an image-heavy session opens more slowly than one holding only a | ||
| reference. The bytes stay in this private field, never in the display content, | ||
| so transcript views stay small. | ||
|
|
||
| A malformed reference leaves the placeholder: the turn still runs, and only the | ||
| cache entry behind the picture is lost. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from collections.abc import Callable | ||
| from typing import Any | ||
|
|
||
| from raven.utils.images import estimate_content_part_tokens, image_block, is_inline_image | ||
|
|
||
| IMAGE_REFS_KEY = "_image_refs" | ||
| """References filed beside a saved message's image placeholders. | ||
|
|
||
| Each entry carries ``at`` (the block index), ``url`` (the exact data URI), | ||
| ``tokens`` and the part's non-URL fields. Dropped by the provider allow-lists; | ||
| read only by the replay projection and the token estimator. | ||
| """ | ||
|
|
||
| IMAGE_PLACEHOLDER = "[image]" | ||
| """What the display content holds in place of an inline picture.""" | ||
|
|
||
|
|
||
| def capture_images( | ||
| content: Any, | ||
| *, | ||
| drop: Callable[[Any], bool] | None = None, | ||
| ) -> tuple[Any, list[dict[str, Any]]]: | ||
| """Replace inline pictures with placeholders, archiving their exact data URIs. | ||
|
|
||
| A non-list comes back unchanged. ``drop`` removes blocks before the index a | ||
| reference records, so a caller that also strips a runtime block gets indices | ||
| into the list it actually stores. | ||
| """ | ||
| if not isinstance(content, list): | ||
| return content, [] | ||
| kept: list[Any] = [] | ||
| refs: list[dict[str, Any]] = [] | ||
| for part in content: | ||
| if drop is not None and drop(part): | ||
| continue | ||
| if not is_inline_image(part): | ||
| kept.append(part) | ||
| continue | ||
| ref = _ref_for_part(part) | ||
| kept.append({"type": "text", "text": IMAGE_PLACEHOLDER}) | ||
| if ref is not None: | ||
| ref["at"] = len(kept) - 1 | ||
| refs.append(ref) | ||
| return kept, refs | ||
|
|
||
|
|
||
| def expand_images(content: Any, message: dict[str, Any]) -> Any: | ||
| """``content`` with each recorded placeholder replaced by the picture it stood for. | ||
|
|
||
| A message without references comes back as it is. A reference whose URL is | ||
| malformed leaves the placeholder. | ||
| """ | ||
| refs = message.get(IMAGE_REFS_KEY) | ||
| if not isinstance(content, list) or not isinstance(refs, list): | ||
| return content | ||
| expanded = list(content) | ||
| for ref in refs: | ||
| if not isinstance(ref, dict): | ||
| continue | ||
| at = ref.get("at") | ||
| if not isinstance(at, int) or not 0 <= at < len(expanded): | ||
| continue | ||
| placeholder = expanded[at] | ||
| if not ( | ||
| isinstance(placeholder, dict) | ||
| and placeholder.get("type") == "text" | ||
| and placeholder.get("text") == IMAGE_PLACEHOLDER | ||
| ): | ||
| continue | ||
| rebuilt = _rebuild_part(ref) | ||
| if rebuilt is not None: | ||
| expanded[at] = rebuilt | ||
| return expanded | ||
|
|
||
|
|
||
| def image_tokens(message: dict[str, Any]) -> int: | ||
| """The image weight a saved message's references still cost the prompt.""" | ||
| refs = message.get(IMAGE_REFS_KEY) | ||
| if not isinstance(refs, list): | ||
| return 0 | ||
| total = 0 | ||
| for ref in refs: | ||
| if isinstance(ref, dict): | ||
| tokens = ref.get("tokens") | ||
| if isinstance(tokens, int) and not isinstance(tokens, bool) and tokens > 0: | ||
| total += tokens | ||
| return total | ||
|
|
||
|
|
||
| def _ref_for_part(part: dict[str, Any]) -> dict[str, Any] | None: | ||
| image_url = part.get("image_url") | ||
| url = image_url.get("url") if isinstance(image_url, dict) else None | ||
| if not isinstance(url, str) or not url.startswith("data:image/"): | ||
| return None | ||
| ref: dict[str, Any] = {"url": url, "tokens": estimate_content_part_tokens(part) or 0} | ||
| top = {key: value for key, value in part.items() if key not in ("type", "image_url")} | ||
| url_extra = {key: value for key, value in image_url.items() if key != "url"} | ||
| if top: | ||
| ref["part"] = top | ||
| if url_extra: | ||
| ref["image_url"] = url_extra | ||
| return ref | ||
|
|
||
|
|
||
| def _rebuild_part(ref: dict[str, Any]) -> dict[str, Any] | None: | ||
| url = ref.get("url") | ||
| if not isinstance(url, str) or not url.startswith("data:image/"): | ||
| return None | ||
| rebuilt: dict[str, Any] = dict(image_block(url)) | ||
| extra = ref.get("part") | ||
| if isinstance(extra, dict): | ||
| rebuilt = {**rebuilt, **{key: value for key, value in extra.items() if key not in ("type", "image_url")}} | ||
| extra_url = ref.get("image_url") | ||
| if isinstance(extra_url, dict): | ||
| rebuilt["image_url"] = { | ||
| **rebuilt["image_url"], | ||
| **{key: value for key, value in extra_url.items() if key != "url"}, | ||
| } | ||
| return rebuilt | ||
|
|
||
|
|
||
| __all__ = [ | ||
| "IMAGE_PLACEHOLDER", | ||
| "IMAGE_REFS_KEY", | ||
| "capture_images", | ||
| "expand_images", | ||
| "image_tokens", | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Blocking: replayed pictures skip the gates that bound and route live pictures, so user photos accumulate in every request until the endpoint refuses, and a model or transport that cannot take an image is sent one on every later turn.
replayed_contentnow runsexpand_imageson every history message (runtime_context.py:68), with no input about the request it feeds. The two things that keep pictures in check were built on base's premise that a picture lives for one turn, and neither sees a replayed one the way it should:The standing image window never counts a user picture.
window_imagesonly collectsrole == "tool"or_attached_imagemessages (shrink.py:250-255); its docstring says why: "A picture the user sent is the subject of the turn rather than a result, and stays". That held while a user picture was filed as[image]. With_image_refson the user branch (turn_path.py:4402-4415), every photo from every earlier turn is sent again andimageWindowBudgetBytes(12,000,000 by default) never applies to them. When the endpoint then refuses for size, the refusal ladder closes the window withany_role(harness/memory.py:425-433) and withdraws every picture, including the one attached to the question being asked.without_picturespops the refs only from the live copy, so the session file still has them and the next turn repeats the refusal.Capability gates apply only to the current message.
_supports_visiongates the user's new attachment (render.py:1040) and_supports_image_tool_resultgates new tool results (organ_glue.py:92-103). An archived picture is replayed after a session switches to a model without vision, or to a transport that cannot carry images inrole=tool, so each later turn first sends a request base never sent, gets refused, and retries. With a refusal worded outside the patterns inproviders/base.py, the turn fails instead.Measured on head 6d816d9 merged onto main fc0a30f against base fc0a30f, real
AgentLoop._process_message, stub providers,pyt.shwith scratch homes:The cache goal does not need either behaviour. A shape that keeps it: let the window count replayed user pictures from earlier turns (exempting only the current turn's question), and make the replay projection honour the same two capability verdicts the live path uses, leaving the placeholder when the receiving model or transport cannot take the picture. That keeps the bytes identical for the requests that can carry them, which is where the cached prefix matters.
Not blocking, for the same revision:
raven/templates/TOOLS.md:23still tells the model "An image is visible for the turn that read it. Later turns see a placeholder".There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Blocking: archived replay must honor the existing image-window and capability gates before merge.
This holds. I reproduced the first path directly: fourteen archived user messages with 1,000,000-byte base64 payloads project to 14,000,000 bytes and 14 images, then
window_images(..., budget=12_000_000)returns(0, 0)and leaves all 14 images because ordinaryrole="user"messages are outside its candidate set. The second path follows from the same unconditional projection:HistoryTrimmer.history_from_idscallsreplayed_contentwithout the selected model or transport capability, so an old image is reconstructed after the current-message capability decisions have already been made.The unchanged image suite still passes (116 tests), which confirms the current coverage does not exercise these replay gates. I am updating my own stance to blocking; I am not resolving this thread because it is yours and remains open.