Skip to content

perf: reuse prepared requests and token estimates per model iteration - #704

Closed
knqiufan wants to merge 1 commit into
OpenBMB:mainfrom
knqiufan:codex/perf-688-request-reuse
Closed

knqiufan wants to merge 1 commit into
OpenBMB:mainfrom
knqiufan:codex/perf-688-request-reuse

Conversation

@knqiufan

Copy link
Copy Markdown
Contributor

Closes #688.

With auto-compaction enabled, a below-threshold model iteration prepares and tokenizes the same input twice, or three times when routing changes the context window. Reuse the prepared context/text request body and its matching local estimate within one model attempt, reducing full preparation, local tokenization and memory retrieval to one pass.

This PR is based directly on upstream main at 5862db79; it contains no #687 / #701 commits and can be reviewed and merged independently.

Correctness

  • Budget previews defer prompt-date/cache-state commitment. Adoption validates input, UTC date, session state, extension prompts and project rules, commits once, and preserves pending cache resets and instruction events.
  • Compaction candidates and changed histories/tools invalidate reuse. Full checkpoints rebuild when the system date must refresh. Routed budget checks use the actual prepared request; changed routed input is recounted.
  • Cache only local estimates in an iteration-owned map. Provider counts, output reservations and route-scoped usage calibration retain their existing behavior.
  • Re-read file-backed media before sending and keep one plan-mode reminder. Final review reproduced and fixed three additional cases: input mutation during asynchronous validation, cancellation through a new adoption signal, and retrying a failed memory preview during formal preparation.

Validation

Local environment: macOS arm64, Node.js 22.23.4. Added 24 regression tests.

  • npm run build and npm run build:web: passed.
  • Context, agent-loop, session, router and model suites: 394 passed, no failures/skips.
  • npm run test:checkpoints: 78 passed, no failures/skips; overlaps some session coverage.
  • Web CI test scope with the workflow's existing exclusions and two workers: 248 files / 2,199 tests passed.
  • git diff --check: passed.

The full backend run (--test-concurrency 4 --test-force-exit --test-timeout 60000) is not entirely green: 911 passed, 0 failed, 7 cancelled, 2 skipped out of 920. All seven cancellations are unchanged tests/network/fetch.spec.ts unref'ed-timer cases; the same seven names and event-loop-resolution error reproduce on unmodified 5862db79. One loaded run also hit the existing approximately one-second FunASR MCP response timeout; its isolated file passed 2/2 and the final full run without competing checks did not reproduce it.

Also align the existing token-count endpoint test's obsolete strict: true expectation with the execution serializer's existing strict: false. The old assertion failure was reproduced on unmodified main; no serializer behavior changed.

Measured performance

Real AgentLoop/context/instruction discovery/tokenizer/policy; 30 tool schemas and approximately 5 KiB of rules. Deterministic history and identical fixture paths/token counts across both implementations. Each case ran 25 times; discard the first and report the median of the remaining 24 (200 total samples). Runs were sequential without concurrent builds/tests.

Local input tokens Case Before After Time reduction
3,845 Unrouted 3.983 ms 2.680 ms 32.7%
20,894 Unrouted 16.417 ms 9.591 ms 41.6%
144,726 Unrouted 105.338 ms 60.460 ms 42.6%
20,894 Routed context-window change 24.267 ms 10.545 ms 56.5%

Ordinary iterations reduced full preparation/tokenization/memory retrieval from 2 to 1, routed iterations from 3 to 1. Rule-file freshness checks remain enabled.

These measure local time from entering loop.run to entering router.execute, with immediate mock memory/model execution, a 2M-token budget (routed to 1M) and no compaction, network or workspace scans. They are not online-provider or end-to-end first-token speedups.

Evidence and reproduction

@knqiufan

Copy link
Copy Markdown
Contributor Author

Closing this PR. Current main (cbded119, v2026.11.10) already reuses one prepared context for the budget check and the model request. That behavior landed separately in be9575a (perf(context): prepare one route-aware snapshot per request attempt, 2026-11-07), on a different API from reusePreparedContext.

What main does now:

  • DefaultContextRuntime.prepareContextAttempt builds one snapshot per attempt. prepareForModelAttempt exposes it and defers the cache-generation, prompt-date, and InstructionsLoaded commits until the request is actually sent.
  • prepareModelRequestAttempt in src/agent/loop/prepareModelRequest.ts projects that snapshot per provider/model. Memory retrieval, instruction discovery, and artifact context run once. Media materialization is reused for the same message list.
  • AgentLoop.createCandidatePreparation keeps that preparation for the pre-routing budget check, the post-routing budget check, and the request handed to the router. A changed route is reprojected from the same snapshot instead of preparing the context again.
  • tests/agent/loop/request-attempt.spec.ts covers it: an unchanged attempt counts memory, instructions, and artifacts once, and a route change still does not repeat those passes.

This branch conflicts with that design in AgentLoop.ts, DefaultContextRuntime.ts, and the token-accounting types. Replaying reusePreparedContext onto current main would add a second reuse protocol beside prepareForModelAttempt.

estimateRequestInput can still tokenize the same request body twice inside one iteration, once for the budget and once for calibration. That is not the two-to-three full preparations tracked by #688.

Closing #688 as completed by be9575a.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: Reuse prepared requests and token estimates during auto-compaction checks

1 participant