Skip to content

fix(models): preserve token limits without persisting fallbacks - #697

Open
knqiufan wants to merge 3 commits into
OpenBMB:mainfrom
knqiufan:codex/fix-685-model-token-limits
Open

knqiufan wants to merge 3 commits into
OpenBMB:mainfrom
knqiufan:codex/fix-685-model-token-limits

Conversation

@knqiufan

@knqiufan knqiufan commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Opening model settings could display protocol defaults instead of the model's limits, and saving unrelated options persisted those defaults as explicit capability overrides. For example, deepseek-chat could fall from its catalog limits of 1,048,576 context / 393,216 output tokens to 128,000 / 32,768 after saving the dialog.

Closes #685.

Rebased onto main @ 9d8f1d0b (2026-11-03). Upstream landed adjacent fixes while this PR was open (3a6c4b39 preserve catalog limits when enabling models, 5410f5eb share engine token limits across setup flows, c390dd53 retire hidden agent context overrides, d57f2aac/5aecccf7 model-test dialog and save scoping). They are kept as-is; the conflicts were resolved by combining both sides:

  • ui/src/shared/catalogProviders.ts keeps its upstream role (picker presentation + modelTokenLimits.ts) while model lists and protocol defaults are derived from the engine catalog.
  • ModelSettingsModal.tsx keeps upstream's saveBeforeTest / blockingTask support and adds per-field token-source display plus edit tracking.
  • ProviderCard.tsx keeps upstream's startModelTest and catalog seeding when enabling a model, and resolves displayed limits through the new resolveTokenLimits chain.

Changes

  • Share a browser-safe provider field normalizer and resolve context/output independently: explicit user override → provider metadata → model catalog (including aliases) → protocol default.
  • Retain model-list token fields through the server and client, follow Anthropic/Gemini pagination, enrich official DashScope discovery, and query Ollama /api/show only for the opened model. Ollama num_ctx wins over architecture context; input and output limits are never added together.
  • Cache validated provider limits separately in $PILOT_HOME/cache/provider-model-limits.json, scoped by provider, protocol, and endpoint. Runtime config parsing reads the cache offline, and successful updates request a gateway reload.
  • Show each displayed limit's source, preserve edits during asynchronous lookups, offer automatic-value reset, and save only values differing from the non-user resolution.
  • Recognize legacy pairs matching both protocol defaults when more specific values differ. This does not rewrite YAML on read; explicit per-field tokenLimitOverrides markers allow users to deliberately pin those same numbers again.

Implementation and migration details: docs/model-token-limits.md.

Note on protocol defaults

The engine now uses a single 262_144 / 32_768 protocol default. Legacy-pair recovery is defined against the current protocol defaults, so configurations persisted by older builds with 128_000 / 32_768 are treated as explicit user values unless tokenLimitOverrides marks them. Keep that behavior for now, or extend the legacy list with the old pairs — worth deciding during review.

Validation

Run locally with Node 22.23.2 on Windows 11:

Check Result
Engine TypeScript check and UI TypeScript check Passed
tests/model/token-limits.spec.ts 8 passed
UI model-pool suites (ui/src/components/settings/view/modelPool, catalogProviders, modelListApi, modelTokenLimits) 93 passed
Server config route (server/routes/config.test.js, 162 tests) Passed
npm run build Passed

@knqiufan
knqiufan force-pushed the codex/fix-685-model-token-limits branch from 8b73d32 to 8e7c196 Compare November 3, 2026 06:20
@knqiufan
knqiufan force-pushed the codex/fix-685-model-token-limits branch from 8e7c196 to dca786e Compare November 3, 2026 13:13
@xhd0728

xhd0728 commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the fix. The token-limit resolution and provider metadata support are useful, but I would hold this PR until the cache-update regression below is fixed.

In src/model/metadata/store.ts:51, each newly discovered model replaces its entire cached limits object. An ID-only model-list entry normalizes to {}, so refreshing the list discards limits previously obtained from a more detailed lookup.

I reproduced this with an isolated profile and a synthetic Ollama model:

  1. Cache maxContextTokens: 8192, as returned by /api/show.
  2. Refresh /v1/models, which returns the same model ID without token-limit fields.
  3. Parse the runtime configuration again: the context limit becomes the protocol default, 262144, instead of retaining 8192.

The model-list handler also requests a gateway reload, so this can cause runtime context accounting to substantially exceed the model's configured window.

Please merge validated fields into the existing per-model limits, preserving known fields when a response omits them. Add regressions for the show → list → runtime-config round trip and for partial metadata updates that contain only one limit.

Web and Desktop CI are both green, but the current tests do not cover this sequence. This needs a fix before merging.

An ID-only model list replaced the cached limit record, so runtime parsing
fell back to the protocol window.
@knqiufan

knqiufan commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @xhd0728 . Cache writes now merge validated limit fields into the existing per-model record, and a response that omits a field keeps the last known value.

Regressions cover the show → list → runtime-config round trip, and a partial update that contains only one validated limit. Both are in tests/model/token-limits.spec.ts.

Main deletes the standalone lookup import. The model-limits route still needs it, and the merged handler was throwing before it could return limits.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug]Model token limits fall back to protocol defaults and can be saved over the real limits

2 participants