You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf: reuse prepared requests and token estimates per model iteration - #704
With auto-compaction enabled, a below-threshold model iteration prepares and tokenizes the same input twice, or three times when routing changes the context window. Reuse the prepared context/text request body and its matching local estimate within one model attempt, reducing full preparation, local tokenization and memory retrieval to one pass.
This PR is based directly on upstream main at 5862db79; it contains no #687 / #701 commits and can be reviewed and merged independently.
Correctness
Budget previews defer prompt-date/cache-state commitment. Adoption validates input, UTC date, session state, extension prompts and project rules, commits once, and preserves pending cache resets and instruction events.
Compaction candidates and changed histories/tools invalidate reuse. Full checkpoints rebuild when the system date must refresh. Routed budget checks use the actual prepared request; changed routed input is recounted.
Cache only local estimates in an iteration-owned map. Provider counts, output reservations and route-scoped usage calibration retain their existing behavior.
Re-read file-backed media before sending and keep one plan-mode reminder. Final review reproduced and fixed three additional cases: input mutation during asynchronous validation, cancellation through a new adoption signal, and retrying a failed memory preview during formal preparation.
Validation
Local environment: macOS arm64, Node.js 22.23.4. Added 24 regression tests.
npm run build and npm run build:web: passed.
Context, agent-loop, session, router and model suites: 394 passed, no failures/skips.
npm run test:checkpoints: 78 passed, no failures/skips; overlaps some session coverage.
Web CI test scope with the workflow's existing exclusions and two workers: 248 files / 2,199 tests passed.
git diff --check: passed.
The full backend run (--test-concurrency 4 --test-force-exit --test-timeout 60000) is not entirely green: 911 passed, 0 failed, 7 cancelled, 2 skipped out of 920. All seven cancellations are unchanged tests/network/fetch.spec.ts unref'ed-timer cases; the same seven names and event-loop-resolution error reproduce on unmodified 5862db79. One loaded run also hit the existing approximately one-second FunASR MCP response timeout; its isolated file passed 2/2 and the final full run without competing checks did not reproduce it.
Also align the existing token-count endpoint test's obsolete strict: true expectation with the execution serializer's existing strict: false. The old assertion failure was reproduced on unmodified main; no serializer behavior changed.
Measured performance
Real AgentLoop/context/instruction discovery/tokenizer/policy; 30 tool schemas and approximately 5 KiB of rules. Deterministic history and identical fixture paths/token counts across both implementations. Each case ran 25 times; discard the first and report the median of the remaining 24 (200 total samples). Runs were sequential without concurrent builds/tests.
Local input tokens
Case
Before
After
Time reduction
3,845
Unrouted
3.983 ms
2.680 ms
32.7%
20,894
Unrouted
16.417 ms
9.591 ms
41.6%
144,726
Unrouted
105.338 ms
60.460 ms
42.6%
20,894
Routed context-window change
24.267 ms
10.545 ms
56.5%
Ordinary iterations reduced full preparation/tokenization/memory retrieval from 2 to 1, routed iterations from 3 to 1. Rule-file freshness checks remain enabled.
These measure local time from entering loop.run to entering router.execute, with immediate mock memory/model execution, a 2M-token budget (routed to 1M) and no compaction, network or workspace scans. They are not online-provider or end-to-end first-token speedups.
Closing this PR. Current main (cbded119, v2026.11.10) already reuses one prepared context for the budget check and the model request. That behavior landed separately in be9575a (perf(context): prepare one route-aware snapshot per request attempt, 2026-11-07), on a different API from reusePreparedContext.
What main does now:
DefaultContextRuntime.prepareContextAttempt builds one snapshot per attempt. prepareForModelAttempt exposes it and defers the cache-generation, prompt-date, and InstructionsLoaded commits until the request is actually sent.
prepareModelRequestAttempt in src/agent/loop/prepareModelRequest.ts projects that snapshot per provider/model. Memory retrieval, instruction discovery, and artifact context run once. Media materialization is reused for the same message list.
AgentLoop.createCandidatePreparation keeps that preparation for the pre-routing budget check, the post-routing budget check, and the request handed to the router. A changed route is reprojected from the same snapshot instead of preparing the context again.
tests/agent/loop/request-attempt.spec.ts covers it: an unchanged attempt counts memory, instructions, and artifacts once, and a route change still does not repeat those passes.
This branch conflicts with that design in AgentLoop.ts, DefaultContextRuntime.ts, and the token-accounting types. Replaying reusePreparedContext onto current main would add a second reuse protocol beside prepareForModelAttempt.
estimateRequestInput can still tokenize the same request body twice inside one iteration, once for the budget and once for calibration. That is not the two-to-three full preparations tracked by #688.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #688.
With auto-compaction enabled, a below-threshold model iteration prepares and tokenizes the same input twice, or three times when routing changes the context window. Reuse the prepared context/text request body and its matching local estimate within one model attempt, reducing full preparation, local tokenization and memory retrieval to one pass.
This PR is based directly on upstream
mainat5862db79; it contains no #687 / #701 commits and can be reviewed and merged independently.Correctness
Validation
Local environment: macOS arm64, Node.js 22.23.4. Added 24 regression tests.
npm run buildandnpm run build:web: passed.npm run test:checkpoints: 78 passed, no failures/skips; overlaps some session coverage.git diff --check: passed.The full backend run (
--test-concurrency 4 --test-force-exit --test-timeout 60000) is not entirely green: 911 passed, 0 failed, 7 cancelled, 2 skipped out of 920. All seven cancellations are unchangedtests/network/fetch.spec.tsunref'ed-timer cases; the same seven names and event-loop-resolution error reproduce on unmodified5862db79. One loaded run also hit the existing approximately one-second FunASR MCP response timeout; its isolated file passed 2/2 and the final full run without competing checks did not reproduce it.Also align the existing token-count endpoint test's obsolete
strict: trueexpectation with the execution serializer's existingstrict: false. The old assertion failure was reproduced on unmodified main; no serializer behavior changed.Measured performance
Real AgentLoop/context/instruction discovery/tokenizer/policy; 30 tool schemas and approximately 5 KiB of rules. Deterministic history and identical fixture paths/token counts across both implementations. Each case ran 25 times; discard the first and report the median of the remaining 24 (200 total samples). Runs were sequential without concurrent builds/tests.
Ordinary iterations reduced full preparation/tokenization/memory retrieval from 2 to 1, routed iterations from 3 to 1. Rule-file freshness checks remain enabled.
These measure local time from entering
loop.runto enteringrouter.execute, with immediate mock memory/model execution, a 2M-token budget (routed to 1M) and no compaction, network or workspace scans. They are not online-provider or end-to-end first-token speedups.Evidence and reproduction