Claude Code ToolSearch loads re-miss the prompt cache on translated Chat routes; ocx claude now hits this by default #7963

Client or integration

Claude Code

Area

Tools / MCP / web search

Summary

On a translated Chat Completions route (DeepSeek, GLM, …), each deferred-tool load through Claude Code's ToolSearch makes the next upstream request re-read almost the whole conversation instead of reusing the prompt cache. This applies to built-ins such as WebFetch and NotebookEdit, and to MCP tools. The cache recovers on the following request and breaks again at the next load.

Two parts of the outbound request change on every load:

1. src/adapters/tool-catalog-nudge.ts lists every current tool name in the system message ("Valid tool names for this turn are exactly …"). This sentence is the first differing byte, about 6 kB into the system message, so only the first ~1.1k tokens stay cached.
2. The loaded definition is added to the top-level tools array (toolsToResponses ignores defer_loading). Chat providers render that array ahead of the messages.

#7452 / #7453 fixed the same mechanism for Responses destinations that accept additional_tools. Chat routes have no equivalent. The Responses opt-in does not cover DeepSeek either, because DeepSeek's Responses API ignores unsupported input item types.

Affected sessions:

• Claude Desktop Code-tab sessions in first-party/picker mode. Claude Code keeps tool search on there.

• ocx claude sessions on Claude Code v2.1.304 or later. Claude Code now enables tool search behind a gateway by default (Tool search through an LLM gateway). ocx claude injects nothing when claudeCode.toolSearch is unset.

  The ocx claude should set ENABLE_TOOL_SEARCH=true: Claude Code disables MCP tool deferral behind a non-Anthropic base URL, so every tool schema ships in full #4838 comment in src/cli/claude.ts and the troubleshooting entry MCP tool schemas fill the context on turn one still describe the old behaviour: tool search off behind a gateway. The same comment says running a translated route with tool search on is "worse than the problem", and that is now the default state.

Expected

• A tool load on a Chat route does not invalidate the conversation prefix, or the limitation is documented with a supported mitigation.
• ocx claude makes an explicit choice for translated routes now that the client default has changed. One option is injecting ENABLE_TOOL_SEARCH=false when toolSearch is unset. The other is documenting the new behaviour and its cache cost.

Proposed fix (Chat routes)

An opt-in for direct Chat routes, modelled on #7453: the same selector rules, applied at the same physical-adapter boundary, with byte-identical requests when off.

• Keep one stable function in the catalog from the first request, e.g. use_loaded_tool {name, arguments}. Keep deferred tools out of it, so the catalog and the nudge never change.
• Append a loaded tool's definition as text to the ToolSearch result that loaded it. At that point this is the end of the transcript.
• Rewrite use_loaded_tool calls in the response stream back to the real tool before Claude Code sees them. Re-encode history the same way on every request, so earlier bytes stay identical.

This is the pattern the MCP client guide recommends for prompt caching with dynamic tools: one stable call_tool({name, args}) tool, so the tools array never changes. Unlike Codex code mode (#2106), Claude inbound does not enforce the undeclared-tool guard. A model that calls a loaded tool directly instead of through the wrapper still gets the call executed by Claude Code.

A working prototype with tests exists locally; the measurements are below. I can open a PR if this direction is acceptable.

Reproduction

Via the CLI. Use Claude Code 2.1.304 or later, with claudeCode.toolSearch and ENABLE_TOOL_SEARCH unset.

1. ocx claude -p "Turn 1: run echo hello with Bash" --model deepseek/deepseek-flash --output-format json
2. Resume the session with: "use ToolSearch to load the WebFetch tool, then call WebFetch on https://example.com".
3. Compare usage.cachedInputTokens in usage.jsonl for the requests before and after the load.

Via Desktop. Pick a routed Chat model in the Code-tab picker, warm the session with a few tool turns, request a ToolSearch load, and compare the same counters. OPENCODEX_CACHE_DEBUG=1 shows the tool count growing by one.

Version

opencodex 2.94.0 (npm). The code paths above are unchanged on dev at 3ea89653f. Claude Code 2.1.306–2.1.312.

Operating system

Windows 11 (two machines)

Provider and model

Official DeepSeek deepseek-flash (Chat), and a third-party Chat gateway serving deepseek-v4.1-flash and glm-5.3-flash.

Logs or error output

Raw upstream counters from usage.jsonl (cachedInputTokens / inputTokens).

ocx claude, Claude Code 2.1.312, consecutive requests:

in=22,167  cached=22,016
in=22,541  cached= 1,536   <- first request after ToolSearch loaded WebFetch
in=22,623  cached=22,400

Desktop picker, DeepSeek official, consecutive requests:

in=58,086  cached=57,728   tools=51
in=59,551  cached= 7,680   tools=52   <- after ToolSearch select:WebFetch
in=59,846  cached=59,520   tools=52

Controlled 8-turn script with two tool loads (WebFetch, NotebookEdit) in a Desktop-identified session. Hit rate is for the main stream, excluding the first request:

All task turns were correct in every run, including tool calls made through the wrapper. Latency stayed within run-to-run noise.

Two long Desktop sessions (23 turns, about 2.5 h, 2–3 tool loads each):

• Each load dropped the cache to about 6–8k tokens, while the conversation was 88k–124k tokens at the time.
• Removing only those misses would have cut total cost by about 17% at the gateway's list prices. Sessions without tool loads are unaffected.
• Idle gaps of up to 65 minutes caused no misses, so this is not TTL-related.

Screenshots and supporting files

Related:

• Translated Messages → Responses tool discovery rewrites the initial catalog and drops prompt-cache reuse #7452, fix(claude): preserve append-only client tool discovery on opted-in Responses routes #7453: same mechanism, fixed for Responses.
• ocx claude should set ENABLE_TOOL_SEARCH=true: Claude Code disables MCP tool deferral behind a non-Anthropic base URL, so every tool schema ships in full #4838, feat(claude): opt-in ENABLE_TOOL_SEARCH injection for launched Claude Code sessions #5029: the launcher tool-search setting.
• fix: preserve tool safety on translated Messages fallbacks #7919: eager degradation of defer_loading on unsupported wires.
• [BUG] Large MCP ToolSearch batches invalidate the whole conversation cache, contradicting the documented tool-search guarantee anthropics/claude-code#83756: large ToolSearch batches also rewrite the cache on Anthropic's own API.
• fix(claude): keep prompt cache when tool_search discovers the first deferred tool router-for-me/CLIProxyAPI#6560: an anchor-tool fix for the Anthropic-upstream variant.
• MCP Client Best Practices, section Interaction with Prompt Caching.

Measurement scripts and raw counters are available on request.

Redacted configuration

{
  "providers": {
    "deepseek": { "adapter": "openai-chat", "baseUrl": "https://api.deepseek.com" },
    "<third-party-gateway>": { "adapter": "openai-chat" }
  },
  "providerContextCaps": { "deepseek": 1000000, "<third-party-gateway>": 1000000 },
  "claudeCode": { "desktopMode": "first-party", "intercept": { "picker": true } }
}

claudeCode.toolSearch, appendOnlyToolLoadingModels and ENABLE_TOOL_SEARCH are unset.

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.