Repository navigation
feat: Resolve attachments a message carries by reference - #715
Draft
danny-avila wants to merge 1 commit into
Draft
danny-avila wants to merge 1 commit into
danny-avila wants to merge 1 commit into
Conversation
A message can now hold an attachment as LangChain's `source_type: 'id'` data block. `RunConfig.resolveAttachment` supplies the content while a provider request is prepared, as the block the serving provider accepts, so graph state and checkpoints keep the reference and never hold the bytes. The resolver reaches the agent, fallback and summarization requests and child graphs. Remote runs refuse a config that sets one.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An attachment that a host puts into a message lives in graph state for the whole run, so everything that serializes state pays for it again: each checkpoint write, and each trace span that records state. For a Bedrock document the cost is worst, because the bytes are a
Buffer: measured on 4.2.10, a 1 MiB document serializes to 3.0x its size throughJSON.stringifyand 11.9x throughHumanMessage.toJSON, the path a checkpointer takes (LibreChat-AI/LibreChat#18381).This adds a way for a message to carry the attachment as a reference. The reference is LangChain's
source_type: 'id'data block. A host passesRunConfig.resolveAttachment, and while a provider request is prepared the SDK asks it for the content of each reference, as the block the serving provider accepts. The result goes into that request only. Graph state and checkpoints keep the reference.Nothing changes for a host that does not pass a resolver, and
idblocks a resolver does not recognize (provider-side file ids) are left as they are.This is the SDK half. The host half, where LibreChat keeps the bytes for the run and emits references, is not written yet, and it needs this published first.
Related to LibreChat-AI/LibreChat#18381.
How it works
The graph is already the request context, so the resolver is one more optional member on
ProviderRequestContext. It is plumbed the wayproviderTextProtectionis:RunConfigto the graph, to child graphs throughSubagentExecutor, and to the summarization request, which had no context before.The resolver is synchronous. The host is expected to read and validate an attachment before the run starts, where a failure can still be reported as an attachment error, and keep the bytes for the run; resolution is then a lookup.
docs/attachment-references.mdstates the contract.Two limits of this version:
createRemoteSdkRunrefuses a config that sets one, so a host cannot send references to a worker that would pass them to a provider unresolved.Performance: without a resolver the Bedrock target still returns before scanning, and other targets do the same single pass as before. With one, a Bedrock target gains that pass. Not measured: wall time on a long history.
The commit hook reformatted existing lines in
src/run.ts; the change there is four lines.Testing
npx jest src/llm/prepareProviderRequest.test.ts src/graphs/__tests__/Graph.attachments.test.ts src/runtime/files.spec.ts— 41 tests pass.nulldrops the block and an attachment-only turn stays valid; a provider-native document beside a reference is still reprojected. A run-level case passes the resolver throughRun.create, checks what the model received, and reads the checkpoint back to confirm it holds the reference and not the payload. A remote run with a resolver is refused.src/llm,src/summarization,src/graphsandsrc/tools/subagentpassed.npm run typecheck, the declaration build,test:circular-depsandtest:light-entriespass; ESLint is clean on the touched files.Not run: live provider calls.