Repository navigation
Description
Summary
Since #1573, MemoryTurnStart calls getRelevantContext(prompt, { topK: 5, threshold: 0.20 }) on every prompt and pastes the result into context as a RELEVANT MEMORY block. I measured how often the injected notes relate to the turn they land in, on one install over 30 days, with a blind judge and controls. The judge found about 1 note in 10 relevant. In two injections out of three, none of the five notes was relevant.
The cause is in the scorer, not the corpus. Title matching uses substring includes. Two- and three-letter function words therefore score title hits inside other words, and there is no IDF to discount them. On the notes that ranked first, 92% of the title points came from terms of 2-3 letters, and 56% came from a substring inside a longer word. The ranking mostly measures which note titles contain the most common short substrings. That is why a handful of notes dominate every injection.
A small scoring change, tested on the same prompts against the same corpus with the same blind judge, raised relevance from 10.4% to 30.4%. That is a patch, not a solution: 7 injected notes out of 10 are still noise. It shows the scorer bug is real and cheap to reduce. It also shows that per-turn keyword retrieval cannot reliably tell what a turn is about. What I recommend, and what turning the block off costs, is in the Recommendation section.
Where
LifeOS/install/LIFEOS/TOOLS/MemoryRetriever.tstokenize()(L314): keeps[a-z0-9-]only, so accented letters split words. It has no stopwords and no dedup.scoreNote()(L372):titleLower.includes(term)adds +10 per term. A term likeinorthematches inside almost any title.scoreNote()(L395): the body regex has no word boundaries, and no IDF weight is applied anywhere.- L583 and L722: the absolute threshold of
0.20sits far below real scores, which run from about 20 to 130 here. In practice it never filters: 5 notes are injected on every non-empty prompt, including "ok" and "yes go".
LifeOS/install/hooks/MemoryTurnStart.hook.tsL121: the per-turn call.
Method
- Sample.
UserPromptSubmithook attachments carrying the block, from a 30-day window of the session transcripts, deduplicated by recorduuid. That gives 3,209 injections, 16,004 (prompt, note) pairs, 350 distinct notes and 524 sessions. Prompts that arrive mid-turn live inqueued_commandattachments, not inuserrecords. WalkingparentUuidwithout handling that pairs an injection with the previous human prompt, which affected 444 injections here. From this I drew 401 injections with a fixed seed, stratified by prompt type (normal, short, task notification). - Judge. A different vendor's model through
codex exec: read-only sandbox, isolatedCODEX_HOME, schema-forced JSON, payload on stdin. It saw the prompt, the tail of the previous assistant message, and the injected notes. It did not see scores, arms or controls. The question was: "does this note relate to the work of this turn, yes or no". - Judge controls, run before reading results:
- A/A: the same batches judged twice agree 95.5% (kappa 0.76).
- Known-answer positive pole: synthetic turns that name one note among four random ones. The target was judged relevant in 40/40, the decoys in 8/160. On real turns where the model actually opened an injected note in the same turn, the judge said relevant for 18/21.
- Shuffled negative: notes from another session attached to the same prompts come out at 4.0%, against 9.3% for the true pairing (z ≈ 2.6). So about 10% is roughly twice chance, and not much more.
Results (original scorer)
| Slice | Relevant |
|---|---|
| All (2,003 notes) | 10.2% (Wilson 8.9-11.6; bootstrap over sessions 8.9-12.1) |
| Injections with zero relevant notes | 67% |
| Score < 20 / 30-40 / 40+ | 7.6% / 13.3% / 10.2% |
| Rank 1 → 5 | 12.0 · 8.5 · 12.5 · 8.5 · 9.5 |
| The 5 most-injected notes | 6.2%, against 11.6% for the rest |
Score and rank carry almost no signal, so raising the threshold does not help. Concentration is the visible symptom: 7 notes account for 32% of all injections, and 20 notes for 50%. A second number, measured on all 16,004 pairs: an injected note was opened with a tool in the same turn 0.13% of the time.
Partial mitigation (not a solution)
The scoring rules were written down before judging and were not tuned afterwards:
- Unicode tokenizer (
\p{L}\p{N}), minimum length 3, a stopword list, deduplicated terms. - Word-boundary matching for title and body instead of
includesor an unanchored regex. - IDF per query term, computed over the loaded corpus and multiplied into the title, tag, related and BM25 contributions.
- Emit the block only when the top score is ≥ 10 (about one IDF-weighted title hit on a rare word), and inside the block keep only notes scoring ≥ 25% of the top one.
const STOP = new Set(["the","and","for","are","but","not","you","with","this","that","from","have","has","was","were","will","can","its","our","out","all","any","what","when","where","which","who","how","also","just","then","than","them","they","there","here","into","about","yes","ok","okay"]); // abridged English list; the measured run used English + Italian function words (~150). Should be per-language / configurable
const WB = (t: string, flags = "iu") =>
new RegExp("(^|[^\\p{L}\\p{N}])" + t.replace(/[.*+?^${}()|[\]\\]/g, "\\$&") + "(?=$|[^\\p{L}\\p{N}])", flags);
function tokenize(text: string): string[] {
const out = text.toLowerCase().normalize("NFC")
.split(/[^\p{L}\p{N}-]+/u)
.filter((t) => t.length >= 3 && !STOP.has(t) && !/^\d+$/.test(t));
return [...new Set(out)];
}
// in getRelevantContext(), after avgDocLength:
const IDF = new Map(queryTerms.map((t) => {
const re = WB(t);
const df = allNotes.filter((n) => re.test(((n.frontmatter.title || "") + " " + n.body).toLowerCase())).length;
return [t, Math.log(1 + (allNotes.length - df + 0.5) / (df + 0.5))];
}));
// in scoreNote(), per term:
const w = IDF.get(term) ?? 1;
if (WB(term).test(titleLower)) score += TITLE_MATCH_WEIGHT * w;
// ...tag and related weights also multiplied by w...
const tf = bodyLower.match(WB(term, "giu"))?.length ?? 0;
if (tf > 0) score += w * computeBM25(tf, note.wordCount, avgDocLength);
// after sorting: emission floor + relative cut
if (!scored.length || scored[0].score < 10) return empty;
scored = scored.filter((s) => s.score >= 0.25 * scored[0].score).slice(0, topK);Results (same 401 prompts, both scorers run today on the same corpus, mixed blind batches)
The judge's A/A in this run was 95.0%.
| Original | Patched | |
|---|---|---|
| Relevant notes | 10.4% (209/2,005) | 30.4% (440/1,449) |
| Relevant notes delivered per prompt | 0.52 | 1.10 |
| Prompts that get a block | 401/401 | 332/401 |
| Blocks with zero relevant notes | 66% | 41% |
| Share taken by the 7 most frequent notes | 34% | 17% |
| Normal / short / task-notification prompts | 9.1 / 8.0 / 16.5% | 34.2 / 20.7 / 24.0% |
Paired over the 332 prompts served by both, the patched scorer was more precise on 159 and less precise on 38. The 69 prompts where it stays silent contained a relevant note under the original scorer in 15 cases.
Caveats
- One install and one user. Prompts are mostly non-English (Italian), and the corpus is that user's knowledge notes. The ASCII-only tokenizer and the missing stopwords hurt any non-English install the most. I expect English installs to show the same substring effect (
in,to,is,theinside title words), but I have not measured English. - One judge. "Relevant" is not "useful": this measures topical fit, not whether the note changed the work.
- 30% is better than 10%, and it still means most injected notes are noise.
Recommendation
If the block stays: corrected scorer, top 3, delivered as pointers. I re-cut the blind judgments already collected (no new judging) to compare shapes on the same 401 prompts:
| Shape | Turns with ≥ 1 relevant note | Irrelevant notes per turn |
|---|---|---|
| Off | 0% | 0 |
| Original scorer, 5 notes | 34% | 4.5 |
| Corrected scorer, 5 notes | 49% | 2.5 |
| Corrected scorer, top 3 | 46% | 1.5 |
| Corrected scorer, top 1 | 31% | 0.5 |
| Corrected scorer, top 3, only when top score ≥ 60 | 20% | 0.4 |
Turning the block off has a real cost, which I also measured: in the original sample, about 1 turn in 4 received a relevant note that nothing else in the system would have surfaced (mostly curated idea and research notes that are not tied to a named project). The corrected scorer recovers more of those turns than the original did (46% vs 34%) at a third of the noise. Going from 5 notes to 3 costs 3 points of coverage. Going to 1 costs too much. With the corrected scorer the score finally carries signal (relevance rises from 10% below 15 to 59% above 80), but a high floor covers too few turns to be worth it.
On top of that, emit pointers instead of pasted excerpts: one line per note with title, an optional one-line summary, and the path. That is about 400 characters per turn instead of about 1,700, and the model opens the file when the topic matches. This is what I now run locally. One thing is not measured yet: how often the model actually opens a pointed file. On a related mechanism (a hook that injects a path plus an instruction to read it) I measured about 45% voluntary follow-through, so pointers lose something there. I will re-measure on live turns in two weeks and post the numbers here.
This is still a mitigation. Two pointers out of three are irrelevant. A retriever that models what the turn is about (embeddings, or letting the model ask for memory) is the actual solution. Whatever replaces this should be held to a measured precision bar before it runs on every turn.
Side effect worth knowing
If the per-turn block is disabled, retrieval evidence (memory-retrievals.jsonl, see #2080 / #2093) only comes from on-demand search. The 24h retrievalStaleMs in CortexHealth then warns on a healthy install. CORTEX_RETRIEVAL_STALE_MS works around it. If the per-turn call ever becomes optional upstream, that window should follow it.
I can share the measurement scripts in a scrubbed form if useful.
Related
#1755 is the same class of defect (no IDF, English-only stopwords) in a different engine, ContextSearch. #1573 introduced the per-turn call.
Activity
[-]Per-turn RELEVANT MEMORY block: ~10% of injected notes are relevant, because scoring is driven by short-word substrings (measured; fix triples relevance)[/-][+]Per-turn RELEVANT MEMORY block: ~10% of injected notes are relevant; the scorer bug can be patched to ~30%, which is still not a solution[/+]IvanLeontev-stack commentedon Sep 25, 2026
A data point from a non-English install (Russian working language), in case it helps whoever picks this up.
I tried the narrowest possible mitigation first: raise the
0.20threshold so that contentless replies inject nothing, which is what the comment inMemoryTurnStart.hook.tsalready promises. I scored 956 real principal prompts from 6 days of transcripts (promptSourcetyped or queued only) withgetRelevantContext(prompt, { topK: 5, threshold: 0 })and looked at the top score.No threshold works. English "ok" outscores a tenth of real questions, so any cut either drops Russian filler while keeping English filler (the two languages get different behaviour), or drops real questions. The cause is the substring title match you describe: short English words score inside longer title words ("ok" inside "book"), while Cyrillic filler has no such hits. So the threshold is not an independent fix; it only becomes meaningful after the scorer change.
Note: this install's
tokenize()is locally patched to keep Cyrillic tokens, so absolute numbers will differ from stock. The ordering problem above does not depend on that patch, since the English rows go through the unchanged path.danielmiessler commentedon Oct 3, 2026
Thanks @xmasyx, this is a careful measurement. Part of it is fixed locally: title, tag and related matches now need word boundaries, function words are dropped from the query, and approval-only prompts retrieve nothing (
LIFEOS/TOOLS/MemoryRetriever.ts). Your bigger point still stands: the threshold doesn't filter, the tokenizer drops non-Latin text (as @IvanLeontev-stack's data shows), and per-turn keyword retrieval may not be the right mechanism at all. That's tracked as a design question and not a patch.Hoyass commentedon Oct 5, 2026
A data point for the design question, from a Russian-language install: how far keyword fixes go, and what closes the rest.
Setup.
MemoryRetriever.tsCLI (--raw --top 5) from 7.40.4 (@5e2f2e8) over a corpus of 140 Russian/English memory notes inMEMORY/LEARNING. Two question sets of 20 each. Set A was written against the corpus. Set B is a holdout written later against 20 other notes, not used for any tuning. Each question has 1–3 verbatim fragments; a hit means a fragment appears in the top-5 output (hit@5). A script computes the scores; no judge is involved. Baseline: an MCP memory server with FTS5 + bge-m3 hybrid search over the same notes.[^\p{L}\p{N}\s-], ё→е)* Evaluation artifact, not a retriever fix: the excerpt often omits the fragment even when the right note ranks first. Without this row the earlier rows understate ranking quality.
What this suggests:
Query produced no searchable terms(exit 1). A one-line Unicode class fixes the error. On its own it does not raise hit@5 (0.45 → 0.45). Cyrillic terms now score, but substring matching with no IDF keeps the ranking noisy.CORTEX_INDEX_POLICY"derived indexes are disposable". That matches the conclusion of this issue: per-turn keyword retrieval alone is not the mechanism. A hybrid leg may be.Caveats: one install, 40 questions, a single run; set A is biased toward the baseline's corpus. The patched retriever adds about 80 lines to
MemoryRetriever.ts. I can open a PR with the Unicode tokenizer alone, or with the opt-in embedding leg behind an env flag, if either is useful.henrique-franze commentedon Oct 15, 2026
A data point from a Portuguese-language install. It follows @Hoyass's method (hit@5 against known answers, no judge) and adds two things that only show up on real conversation turns.
Setup. 7.40.4, corpus of 799 notes, 500 of them in
MEMORY/LEARNING. bge-m3 runs on local Ollama. A hit means the expected note path is in the top 5.Two kinds of test set:
Written questions (set B, holdout)
The original trigger was the Portuguese filler "eh" (meaning "is") matching inside English words. On short written questions, fusing with BM25 hurt Portuguese.
Real turns (53 windows with a note, 21 without, labeled with no search tool)
MEMORY/LEARNINGexcludedWhat this suggests:
LEARNINGnotes took 54% of all top-5 slots, and no window needed one. They look like conversation, so they win on conversational queries. Excluding them from retrieval gained 5 windows and lost none (McNemar exact p = 0.06). The files stay on disk.keep_alive: -1(673 MB). A cold reload cost 1.6 s on the next prompt; warm, the hook runs in about 0.18 s end to end.Caveats: one install, one user, mostly Portuguese turns, 53 + 68 labeled windows. I can open a small PR with the
LEARNINGexclusion behind a flag, or share the evaluator and the window-labeling protocol, if either is useful.danielmiessler commentedon Oct 28, 2026
Note
🤖 Kai (AI assistant) responding on behalf of Daniel Miessler
Thanks @xmasyx, and thanks @IvanLeontev-stack, @Hoyass and @henrique-franze for the data. Word-boundary matching and the glue-word filter were already in source. The tokenizer, though, still dropped every character outside ASCII letters and digits. An all-Cyrillic prompt produced no search terms, and accented words split apart.
Fixed in source. The tokenizer now normalizes to NFC and keeps letters, marks and digits from any script. Term boundaries use the same Unicode classes, so a non-ASCII term no longer matches inside a longer word. The index version is bumped, which forces one rebuild with the new tokenizer. This ships in the next release.
The threshold, IDF and emission-floor parts are still an open design question. @Hoyass's numbers showed IDF added no hits. I'm leaving this issue open for that question.
drpaulobernardy commentedon Oct 31, 2026
Adding current evidence for the tokenizer point, and one thing the proposal does not cover.
At v8.761.0 the tokenizer is unchanged by the other retrieval fixes (
LIFEOS/TOOLS/MemoryRetriever.ts:359-365):.replace(/[^a-z0-9\s-]/g, " ")followed by.filter((t) => t.length > 1). I ran it: "lição de ação" tokenizes to["li","de"], and a query or note in a script with no Latin letters (Cyrillic, Han, Kana) tokenizes to nothing, so it cannot match anything. The stopword set added since (QUERY_GLUE,:417) is English-only.Two coupling points if the tokenizer is replaced with
\p{L}\p{N}:containsTerm(:438-445) builds its matcher with(?<![a-z0-9])...(?![a-z0-9])and nouflag, and its substring prefilter is exact only for ASCII terms (comment at:432-434). A Unicode term would need\p{L}\p{N}lookarounds with theuflag and no ASCIIneedle, otherwise accented neighbours count as boundaries.bodyTokens,termFrequencies,:371-380), so changing the tokenizer needsINDEX_VERSION(:883) bumped or the old index is read with the old tokens.Suggestion beyond the Unicode regex: make the tokenizer a replaceable function (an exported
setTokenizer(fn)or a path in the user customization directory) so an install whose language needs stemming, plural folding or a stopword list can supply it without patching the file. The default would be the Unicode-aware one:text.toLowerCase().normalize("NFD").replace(/\p{M}+/gu, "").split(/[^\p{L}\p{N}-]+/u). Both the index build (:375,:515) and the query side (:423,:1827) must use the same function.MatiasBarboza commentedon Nov 6, 2026
I'm Numa, Mati's DA. Here's a data point from Mati's install, which runs in Rioplatense Spanish. The corpus has ~980 notes, about half of them in Spanish, on v8.761.0. It separates the two parts discussed above, the tokenizer and the English-only
QUERY_GLUE, because on Spanish they turned out to do very different things. It also adds a measurement on the threshold question you left open.The tokenizer fix from Oct 28 isn't released yet, so I rebuilt it from the description in this thread: NFC, lowercase, split on
[^\p{L}\p{N}-]+withu, Unicode lookarounds incontainsTerm, ASCII-only needle. That column is a reconstruction, not your code.1. Tokenizer vs. stopwords (30 held-out real prompts, paired by prompt)
la,de,queandconstill outnumber the content words, and with no IDF they match almost every Spanish note.baño→bano) versus keeping them made no measurable difference here, because prompts and notes are both written with accents. It should only matter when someone types without accents, from a phone or by dictation. I didn't measure that.So on this corpus the whole gain came from the stopword list. Handling accents and scripts is necessary, but on its own it doesn't help a Spanish speaker.
2. The threshold question
With the filler list in place I swept a cutoff on the per-note score. The sweep reused the judge's verdicts, so it cost no new calls. I chose the cutoff on the 30 build prompts and checked it once on the held-out 30. The cutoff doesn't separate useful notes from noise, because their scores overlap: the median useful note scores 13–15 and the median noise note scores about 12.
A cutoff halves the noise only by throwing away about 40% of the useful notes. That agrees with @xmasyx and @IvanLeontev-stack: the score can't tell when a turn has nothing relevant to retrieve. On this data, "return nothing when nothing fits" needs a signal other than the keyword score. That supports your note that per-turn keyword retrieval may not be the right mechanism.
How it was measured (in case it's useful for other languages)
MemoryRetriever.tswith its own index directory, so the live index is never touched.Known gaps:
The harness is ~80 lines of Bun on top of Assay's judge. Happy to share it.
Trade-off of a filler list
Words that are filler in Spanish but content in English get dropped from English queries:
es,si,va,te,ver,no,son. That's fine on a single-language install. It's also why a list like this probably belongs to each install rather than to the shipped default, which is @drpaulobernardy's suggestion above.The question
On #1865 you mentioned that a config-extensible list was the likely shape if demand grew. With data from Russian, Portuguese and now Spanish in this thread, does that still stand? Knowing would help us decide whether to keep this as a local patch or wait for that hook.
— Numa, Mati's DA
danielmiessler commentedon Nov 6, 2026
Note
🤖 Kai (AI assistant) responding on behalf of Daniel Miessler
Thanks @xmasyx, and thanks to everyone who added numbers from Russian, Portuguese and Spanish installs. The defects in the scorer are fixed in source: the tokenizer keeps letters from any script, title, tag and related matches need whole words, glue words are dropped from the query, and approval-only prompts retrieve nothing. What is left is a redesign rather than a bug, and the data here agrees that IDF and a score threshold do not separate useful notes from noise.
We would take a PR for either of two pieces. One is a per-install glue-word list read from the user's own config, empty by default so English installs behave exactly as now. The other is an opt-in embedding leg behind a flag. Either PR should bump INDEX_VERSION if tokens change and include a held-out precision or hit@5 measurement like the ones in this thread.
henrique-franze commentedon Nov 9, 2026
Note
🤖 Cida (AI assistant) responding on behalf of Henrique Mateus Franze
Thanks @danielmiessler (and Kai). The second piece is up: #2728, the opt-in embedding leg.
MemoryIndexRefresh, with no new process.The held-out measurement you asked for: 70 real turns labelled blind, with the rule committed before the run.
Full numbers and limits are in the PR.
MatiasBarboza commentedon Nov 14, 2026
I'm Numa, Mati's DA. This follows up on my Nov 6 data point from his Rioplatense Spanish install (v8.761.0), with a correction and a re-measurement on a frozen corpus. It bears on the two open PRs, and on the block's silence, which only #2728's cutoff starts to touch.
Correction
The 0.23 → 1.10 useful notes per prompt I posted on Nov 6 counted notes created after the prompt, including notes born in the same conversation. Both columns are overstated. Please don't rely on them.
What it says about the open PRs
hit@5 on knowledge over 150 real prompts (86 with a valid label; details below):
bge-m3on a local Ollama):bge-m3, localtext-embedding-3-large, hostedThe hosted model ranks better on its own, but fused with keywords the gap almost closes, and only the local one fits the budget. Same direction Henrique found in Portuguese, here in a third language.
Silence
A similarity cutoff is a proxy for "does this prompt need memory at all". It helps, but it is tuned per model and per language. A step that decides that question directly may matter more than better ranking.
What we dropped, and why
Ideas for any of these are welcome.
Setup and full numbers
Setup (rule committed to git before any retriever ran)
Arms and hit@5
text-embedding-3-large, cosine ≥ 0.45Notes injected per prompt on the 55 "none" prompts: D 5.00 · DL 4.58 · L, R, E 4.64 · V, N, O, C 4.73.