Repository navigation
[Bug]: the oversized-run token fallback applies to the whole text, so one long run inflates the entire message #599
Description
Product or interface: Source build or repository tooling
Version: source commit main (3c21163c), Node.js v26.7.0, pnpm 9.12.0
Platform: macOS
OS version and architecture: macOS 26.6.2 (25G83), arm64
Issue area: Context management / token estimation
What happens
packages/agent-modules/context-manager/src/token-estimator.ts falls back to a UTF-8
byte upper bound when a text contains an oversized alphanumeric run — but the fallback
covers the whole text, not just the oversized run.
const MAX_EXACT_TOKENIZER_CHARS = 4_096;
function hasOversizedAlphanumericRun(text: string): boolean { /* any run > 4096 */ }
function estimateTextTokensUpperBound(text: string): number {
return Buffer.byteLength(text, 'utf8');
}
// inside estimateTextTokens():
if (text.length > MAX_EXACT_TOKENIZER_CHARS && hasOversizedAlphanumericRun(text)) {
return estimateTextTokensUpperBound(text);
}The class comment describes the intent as "a UTF-8 byte upper bound for oversized
word runs" — a bound for the run. The implementation charges the byte length of the
entire string instead, so one long blob inflates everything around it.
Measured with gpt-tokenizer's o200k_base (the estimator's own encoder) as the
reference:
| sample | local estimate | real BPE | ratio |
|---|---|---|---|
| base64 blob, 60 KB | 60,000 | 7,500 | 8.00 |
| single hash string, 9 KB | 9,000 | 1,125 | 8.00 |
| 20 KB of normal Chinese text + 5 KB base64 | 42,501 | 10,901 | 3.90 |
| 20 KB of normal Chinese text, no blob | 8,401 | 8,401 | 1.00 |
| a real source file | 19,254 | 19,254 | 1.00 |
| real minified JavaScript | 50,806 | 50,806 | 1.00 |
Minified JavaScript does not trigger it, because punctuation breaks the run. A base64
payload, a hash, a long URL or an unbroken identifier does.
The blast radius is the point. From a real session, one text block of 63,998
characters contained a single run of 4,218 characters — 6.6% of the block:
text block: 63,998 chars, longest run 4,218 chars
local estimate (byte bound) = 63,998 real BPE = 24,013 ratio 2.67
→ 39,985 tokens of inflation from one text block
Had the bound applied to the run alone, the inflation would have been roughly 3,000
tokens. Because it applies to the whole block, it is about 13× larger than the
comment describes.
Why it matters
The estimate is not only displayed — it decides when the context is compacted.
local-context-footprint.ts takes Math.max(local estimate, provider usage anchor),
so an inflated local estimate wins over the provider's own number and compression
triggers early. The direction is safe (over-estimating cannot cause a hard 4xx), but
the cost is that a conversation loses history long before the window is actually
full.
On a local session store (2,489 sessions, 16,100 files, 4.49 GB) scanned for
[A-Za-z0-9]{4097,}:
- 53 files (0.33%) contain such a run; after de-duplicating per session, 21
sessions (0.84%); - in those sessions, 4 actually trip the estimator fallback, and the inflation is
1.8% of provider-reported usage in aggregate, up to 45% in the worst
single session (one 64 KB text block holding a 4.2 KB run, in a 106-message
conversation).
That is not large enough to explain the report we are chasing, but it is a real
over-estimate with no test pinning it: upperBound, oversized, MAX_EXACT and
byteLength appear nowhere in packages/agent-modules/context-manager/test/.
Steps to reproduce
import { countTokens as o200k } from 'gpt-tokenizer/model/gpt-4o';
const blob = 'A'.repeat(60_000); // one 60 KB run
const text = '中文上下文内容。'.repeat(2000) + blob; // 20 KB of text + the blob
const localEstimate = Buffer.byteLength(text, 'utf8'); // what the estimator returns
const realTokens = o200k(text, { allowedSpecial: 'all' });
console.log(localEstimate, realTokens, localEstimate / realTokens);
// 108000 17500 6.17The same text without the blob keeps a ratio of 1.00.
What would help
- Apply the byte upper bound to the oversized run only, and count the rest of the
string with the normal encoder — the estimate then stays close to the real count
while keeping the bound where it was intended. - Or count the oversized run with the encoder too, and keep the byte bound only as a
fallback for the cases where the encoder actually fails.
Either way, a test that pins the behaviour for a string containing one long run would
keep this from drifting again.
Activity
UncleMoc commentedon Nov 11, 2026
Two clarifications on how this sits next to #600, plus one path we did not mention.
The over-estimate is on the fallback path. If the compaction trigger takes the provider's own number as its primary source — as
ContextManageralready does, see #600 — then this fallback only decides the threshold when the provider cannot be asked. That makes it less visible, not less wrong: it is the only number left at exactly the moment the provider is unreachable.The estimator lives in one place only —
packages/agent-modules/context-manager/src/token-estimator.ts.packages/local-runtime/src/context/token-estimator.tsis a thin wrapper that imports from@mavis/context-manager, so the same fallback is reached from the v1 runtime too, not only from v2.context-usage-calibration.tsimports the same package, so the calibration path can also see the inflated number.Two further notes for whoever picks this up:
estimateMessagesis also whatestimateContextTokensfalls back to, so any caller with no counter injected sees the same inflation.UncleMoc commentedon Nov 11, 2026
Correction: the sample size in this issue was wrong
Re-scanning after the review in #598 §9, the prevalence figures in the body do not hold. The mechanism, the reproduction and the conclusion are unchanged; the numbers around them were overstated.
What the body says vs. what the data shows. The body claims 53 files / 21 sessions (0.84%) and "1.8% of provider-reported usage in aggregate". The corrected figures:
[A-Za-z0-9]{4097,}run;The "1.8% of provider-reported usage in aggregate" figure was computed over a denominator that included transcripts which never tripped the fallback at all. The honest statement is the narrower one: 4 sessions in ~2,360, inflating those sessions by up to 166% of their real BPE count, with 0.72% of transcripts containing a run at all.
What is unchanged. The defect itself is exactly as described:
estimateTextTokensUpperBound()charges the UTF-8 byte length of the whole string, not of the oversized run, while the class comment describes the bound as being for "oversized word runs". The reproduction is unchanged and still needs no special setup — one long run in a block is enough. The 63,998-character real sample still inflates by 39,985 tokens (2.67x) against a real BPE count of 24,013. There is still no test pinning the behaviour.We also agree with the reporter's read in #598 §9 that this should be fixed and accepted separately from the trigger-threshold work: it is a real over-estimate, but on our data it is not the main cause of compaction firing early.
A detail that will bite anyone re-running the measurement
Not a defect in this issue's subject, but it cost us a wrong number once and it is easy to hit: when accumulating the local estimate over a transcript, the
compactionSummarymessage's own tokens have to be included. The summary is the root of the context after a compaction, so leaving it out shifts the running count down by however long the summary is — in our case enough to change a reported ratio by 0.02-0.05 and make the result look like a different measurement.