Skip to content

feat: Resolve attachments a message carries by reference - #715

Draft
danny-avila wants to merge 1 commit into
mainfrom
danny-avila/attachment-handles
Draft

danny-avila wants to merge 1 commit into
mainfrom
danny-avila/attachment-handles

Conversation

@danny-avila

Copy link
Copy Markdown
Collaborator

Summary

An attachment that a host puts into a message lives in graph state for the whole run, so everything that serializes state pays for it again: each checkpoint write, and each trace span that records state. For a Bedrock document the cost is worst, because the bytes are a Buffer: measured on 4.2.10, a 1 MiB document serializes to 3.0x its size through JSON.stringify and 11.9x through HumanMessage.toJSON, the path a checkpointer takes (LibreChat-AI/LibreChat#18381).

This adds a way for a message to carry the attachment as a reference. The reference is LangChain's source_type: 'id' data block. A host passes RunConfig.resolveAttachment, and while a provider request is prepared the SDK asks it for the content of each reference, as the block the serving provider accepts. The result goes into that request only. Graph state and checkpoints keep the reference.

Nothing changes for a host that does not pass a resolver, and id blocks a resolver does not recognize (provider-side file ids) are left as they are.

This is the SDK half. The host half, where LibreChat keeps the bytes for the run and emits references, is not written yet, and it needs this published first.

Related to LibreChat-AI/LibreChat#18381.

How it works

prepareProviderRequest
  projectAttachmentsForProvider(messages, provider, resolve)   # one pass, copy on write
    id block, resolver present -> resolve(reference, { provider, modelId, projectionMode })
      undefined -> block left as it is
      null      -> block dropped; an emptied message gets the omission notice
      block     -> replaces the reference, sent as returned
    anything else -> existing projection (skipped for a Bedrock target, as before)
  measure(preparedMessages)                                    # sees the resolved content

The graph is already the request context, so the resolver is one more optional member on ProviderRequestContext. It is plumbed the way providerTextProtection is: RunConfig to the graph, to child graphs through SubagentExecutor, and to the summarization request, which had no context before.

The resolver is synchronous. The host is expected to read and validate an attachment before the run starts, where a failure can still be reported as an attachment error, and keep the bytes for the run; resolution is then a lookup. docs/attachment-references.md states the contract.

Two limits of this version:

  • The model call receives the resolved block, so a callback that records model input (a trace's generation span) still sees the content once. Spans that record state see the reference.
  • A remote run does not carry the resolver. createRemoteSdkRun refuses a config that sets one, so a host cannot send references to a worker that would pass them to a provider unresolved.

Performance: without a resolver the Bedrock target still returns before scanning, and other targets do the same single pass as before. With one, a Bedrock target gains that pass. Not measured: wall time on a long history.

The commit hook reformatted existing lines in src/run.ts; the change there is four lines.

Testing

  • npx jest src/llm/prepareProviderRequest.test.ts src/graphs/__tests__/Graph.attachments.test.ts src/runtime/files.spec.ts — 41 tests pass.
  • Cases added: the resolved block reaches the request and the measurement while the source message keeps the reference; each target is resolved for its own provider and the result is not filtered again; an unrecognized reference leaves the message object untouched, with and without a resolver; null drops the block and an attachment-only turn stays valid; a provider-native document beside a reference is still reprojected. A run-level case passes the resolver through Run.create, checks what the model received, and reads the checkpoint back to confirm it holds the reference and not the payload. A remote run with a resolver is refused.
  • Five of these fail with the request-preparation change reverted.
  • Before the remote-run guard was added, 40 suites (1,291 tests) across src/llm, src/summarization, src/graphs and src/tools/subagent passed.
  • npm run typecheck, the declaration build, test:circular-deps and test:light-entries pass; ESLint is clean on the touched files.

Not run: live provider calls.

A message can now hold an attachment as LangChain's `source_type: 'id'` data
block. `RunConfig.resolveAttachment` supplies the content while a provider
request is prepared, as the block the serving provider accepts, so graph
state and checkpoints keep the reference and never hold the bytes.

The resolver reaches the agent, fallback and summarization requests and child
graphs. Remote runs refuse a config that sets one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant