Skip to content

Per-turn RELEVANT MEMORY block: ~10% of injected notes are relevant; the scorer bug can be patched to ~30%, which is still not a solution #2207

Open

Description

@xmasyx

Summary

Since #1573, MemoryTurnStart calls getRelevantContext(prompt, { topK: 5, threshold: 0.20 }) on every prompt and pastes the result into context as a RELEVANT MEMORY block. I measured how often the injected notes relate to the turn they land in, on one install over 30 days, with a blind judge and controls. The judge found about 1 note in 10 relevant. In two injections out of three, none of the five notes was relevant.

The cause is in the scorer, not the corpus. Title matching uses substring includes. Two- and three-letter function words therefore score title hits inside other words, and there is no IDF to discount them. On the notes that ranked first, 92% of the title points came from terms of 2-3 letters, and 56% came from a substring inside a longer word. The ranking mostly measures which note titles contain the most common short substrings. That is why a handful of notes dominate every injection.

A small scoring change, tested on the same prompts against the same corpus with the same blind judge, raised relevance from 10.4% to 30.4%. That is a patch, not a solution: 7 injected notes out of 10 are still noise. It shows the scorer bug is real and cheap to reduce. It also shows that per-turn keyword retrieval cannot reliably tell what a turn is about. What I recommend, and what turning the block off costs, is in the Recommendation section.

Where

  • LifeOS/install/LIFEOS/TOOLS/MemoryRetriever.ts
    • tokenize() (L314): keeps [a-z0-9-] only, so accented letters split words. It has no stopwords and no dedup.
    • scoreNote() (L372): titleLower.includes(term) adds +10 per term. A term like in or the matches inside almost any title.
    • scoreNote() (L395): the body regex has no word boundaries, and no IDF weight is applied anywhere.
    • L583 and L722: the absolute threshold of 0.20 sits far below real scores, which run from about 20 to 130 here. In practice it never filters: 5 notes are injected on every non-empty prompt, including "ok" and "yes go".
  • LifeOS/install/hooks/MemoryTurnStart.hook.ts L121: the per-turn call.

Method

  • Sample. UserPromptSubmit hook attachments carrying the block, from a 30-day window of the session transcripts, deduplicated by record uuid. That gives 3,209 injections, 16,004 (prompt, note) pairs, 350 distinct notes and 524 sessions. Prompts that arrive mid-turn live in queued_command attachments, not in user records. Walking parentUuid without handling that pairs an injection with the previous human prompt, which affected 444 injections here. From this I drew 401 injections with a fixed seed, stratified by prompt type (normal, short, task notification).
  • Judge. A different vendor's model through codex exec: read-only sandbox, isolated CODEX_HOME, schema-forced JSON, payload on stdin. It saw the prompt, the tail of the previous assistant message, and the injected notes. It did not see scores, arms or controls. The question was: "does this note relate to the work of this turn, yes or no".
  • Judge controls, run before reading results:
    • A/A: the same batches judged twice agree 95.5% (kappa 0.76).
    • Known-answer positive pole: synthetic turns that name one note among four random ones. The target was judged relevant in 40/40, the decoys in 8/160. On real turns where the model actually opened an injected note in the same turn, the judge said relevant for 18/21.
    • Shuffled negative: notes from another session attached to the same prompts come out at 4.0%, against 9.3% for the true pairing (z ≈ 2.6). So about 10% is roughly twice chance, and not much more.

Results (original scorer)

Slice Relevant
All (2,003 notes) 10.2% (Wilson 8.9-11.6; bootstrap over sessions 8.9-12.1)
Injections with zero relevant notes 67%
Score < 20 / 30-40 / 40+ 7.6% / 13.3% / 10.2%
Rank 1 → 5 12.0 · 8.5 · 12.5 · 8.5 · 9.5
The 5 most-injected notes 6.2%, against 11.6% for the rest

Score and rank carry almost no signal, so raising the threshold does not help. Concentration is the visible symptom: 7 notes account for 32% of all injections, and 20 notes for 50%. A second number, measured on all 16,004 pairs: an injected note was opened with a tool in the same turn 0.13% of the time.

Partial mitigation (not a solution)

The scoring rules were written down before judging and were not tuned afterwards:

  1. Unicode tokenizer (\p{L}\p{N}), minimum length 3, a stopword list, deduplicated terms.
  2. Word-boundary matching for title and body instead of includes or an unanchored regex.
  3. IDF per query term, computed over the loaded corpus and multiplied into the title, tag, related and BM25 contributions.
  4. Emit the block only when the top score is ≥ 10 (about one IDF-weighted title hit on a rare word), and inside the block keep only notes scoring ≥ 25% of the top one.
const STOP = new Set(["the","and","for","are","but","not","you","with","this","that","from","have","has","was","were","will","can","its","our","out","all","any","what","when","where","which","who","how","also","just","then","than","them","they","there","here","into","about","yes","ok","okay"]); // abridged English list; the measured run used English + Italian function words (~150). Should be per-language / configurable
const WB = (t: string, flags = "iu") =>
  new RegExp("(^|[^\\p{L}\\p{N}])" + t.replace(/[.*+?^${}()|[\]\\]/g, "\\$&") + "(?=$|[^\\p{L}\\p{N}])", flags);

function tokenize(text: string): string[] {
  const out = text.toLowerCase().normalize("NFC")
    .split(/[^\p{L}\p{N}-]+/u)
    .filter((t) => t.length >= 3 && !STOP.has(t) && !/^\d+$/.test(t));
  return [...new Set(out)];
}

// in getRelevantContext(), after avgDocLength:
const IDF = new Map(queryTerms.map((t) => {
  const re = WB(t);
  const df = allNotes.filter((n) => re.test(((n.frontmatter.title || "") + " " + n.body).toLowerCase())).length;
  return [t, Math.log(1 + (allNotes.length - df + 0.5) / (df + 0.5))];
}));

// in scoreNote(), per term:
const w = IDF.get(term) ?? 1;
if (WB(term).test(titleLower)) score += TITLE_MATCH_WEIGHT * w;
// ...tag and related weights also multiplied by w...
const tf = bodyLower.match(WB(term, "giu"))?.length ?? 0;
if (tf > 0) score += w * computeBM25(tf, note.wordCount, avgDocLength);

// after sorting: emission floor + relative cut
if (!scored.length || scored[0].score < 10) return empty;
scored = scored.filter((s) => s.score >= 0.25 * scored[0].score).slice(0, topK);

Results (same 401 prompts, both scorers run today on the same corpus, mixed blind batches)

The judge's A/A in this run was 95.0%.

Original Patched
Relevant notes 10.4% (209/2,005) 30.4% (440/1,449)
Relevant notes delivered per prompt 0.52 1.10
Prompts that get a block 401/401 332/401
Blocks with zero relevant notes 66% 41%
Share taken by the 7 most frequent notes 34% 17%
Normal / short / task-notification prompts 9.1 / 8.0 / 16.5% 34.2 / 20.7 / 24.0%

Paired over the 332 prompts served by both, the patched scorer was more precise on 159 and less precise on 38. The 69 prompts where it stays silent contained a relevant note under the original scorer in 15 cases.

Caveats

  • One install and one user. Prompts are mostly non-English (Italian), and the corpus is that user's knowledge notes. The ASCII-only tokenizer and the missing stopwords hurt any non-English install the most. I expect English installs to show the same substring effect (in, to, is, the inside title words), but I have not measured English.
  • One judge. "Relevant" is not "useful": this measures topical fit, not whether the note changed the work.
  • 30% is better than 10%, and it still means most injected notes are noise.

Recommendation

If the block stays: corrected scorer, top 3, delivered as pointers. I re-cut the blind judgments already collected (no new judging) to compare shapes on the same 401 prompts:

Shape Turns with ≥ 1 relevant note Irrelevant notes per turn
Off 0% 0
Original scorer, 5 notes 34% 4.5
Corrected scorer, 5 notes 49% 2.5
Corrected scorer, top 3 46% 1.5
Corrected scorer, top 1 31% 0.5
Corrected scorer, top 3, only when top score ≥ 60 20% 0.4

Turning the block off has a real cost, which I also measured: in the original sample, about 1 turn in 4 received a relevant note that nothing else in the system would have surfaced (mostly curated idea and research notes that are not tied to a named project). The corrected scorer recovers more of those turns than the original did (46% vs 34%) at a third of the noise. Going from 5 notes to 3 costs 3 points of coverage. Going to 1 costs too much. With the corrected scorer the score finally carries signal (relevance rises from 10% below 15 to 59% above 80), but a high floor covers too few turns to be worth it.

On top of that, emit pointers instead of pasted excerpts: one line per note with title, an optional one-line summary, and the path. That is about 400 characters per turn instead of about 1,700, and the model opens the file when the topic matches. This is what I now run locally. One thing is not measured yet: how often the model actually opens a pointed file. On a related mechanism (a hook that injects a path plus an instruction to read it) I measured about 45% voluntary follow-through, so pointers lose something there. I will re-measure on live turns in two weeks and post the numbers here.

This is still a mitigation. Two pointers out of three are irrelevant. A retriever that models what the turn is about (embeddings, or letting the model ask for memory) is the actual solution. Whatever replaces this should be held to a measured precision bar before it runs on every turn.

Side effect worth knowing

If the per-turn block is disabled, retrieval evidence (memory-retrievals.jsonl, see #2080 / #2093) only comes from on-demand search. The 24h retrievalStaleMs in CortexHealth then warns on a healthy install. CORTEX_RETRIEVAL_STALE_MS works around it. If the per-turn call ever becomes optional upstream, that window should follow it.

I can share the measurement scripts in a scrubbed form if useful.

Related

#1755 is the same class of defect (no IDF, English-only stopwords) in a different engine, ContextSearch. #1573 introduced the per-turn call.

Activity

  1. changed the title [-]Per-turn RELEVANT MEMORY block: ~10% of injected notes are relevant, because scoring is driven by short-word substrings (measured; fix triples relevance)[/-] [+]Per-turn RELEVANT MEMORY block: ~10% of injected notes are relevant; the scorer bug can be patched to ~30%, which is still not a solution[/+] on Sep 21, 2026
  2. IvanLeontev-stack commented on Sep 25, 2026

    @IvanLeontev-stack

    A data point from a non-English install (Russian working language), in case it helps whoever picks this up.

    I tried the narrowest possible mitigation first: raise the 0.20 threshold so that contentless replies inject nothing, which is what the comment in MemoryTurnStart.hook.ts already promises. I scored 956 real principal prompts from 6 days of transcripts (promptSource typed or queued only) with getRelevantContext(prompt, { topK: 5, threshold: 0 }) and looked at the top score.

    Prompt Top score
    Russian "yes, do it" / "yes, agreed" (2–3 words) 7–8
    English "ok" / "ok next" 32
    Real messages of 8+ words, 10th percentile 16
    Real messages of 8+ words, median 39

    No threshold works. English "ok" outscores a tenth of real questions, so any cut either drops Russian filler while keeping English filler (the two languages get different behaviour), or drops real questions. The cause is the substring title match you describe: short English words score inside longer title words ("ok" inside "book"), while Cyrillic filler has no such hits. So the threshold is not an independent fix; it only becomes meaningful after the scorer change.

    Note: this install's tokenize() is locally patched to keep Cyrillic tokens, so absolute numbers will differ from stock. The ordering problem above does not depend on that patch, since the English rows go through the unchanged path.

  3. danielmiessler commented on Oct 3, 2026

    @danielmiessler
    Owner

    Thanks @xmasyx, this is a careful measurement. Part of it is fixed locally: title, tag and related matches now need word boundaries, function words are dropped from the query, and approval-only prompts retrieve nothing (LIFEOS/TOOLS/MemoryRetriever.ts). Your bigger point still stands: the threshold doesn't filter, the tokenizer drops non-Latin text (as @IvanLeontev-stack's data shows), and per-turn keyword retrieval may not be the right mechanism at all. That's tracked as a design question and not a patch.

  4. Hoyass commented on Oct 5, 2026

    @Hoyass

    A data point for the design question, from a Russian-language install: how far keyword fixes go, and what closes the rest.

    Setup. MemoryRetriever.ts CLI (--raw --top 5) from 7.40.4 (@5e2f2e8) over a corpus of 140 Russian/English memory notes in MEMORY/LEARNING. Two question sets of 20 each. Set A was written against the corpus. Set B is a holdout written later against 20 other notes, not used for any tuning. Each question has 1–3 verbatim fragments; a hit means a fragment appears in the top-5 output (hit@5). A script computes the scores; no judge is involved. Baseline: an MCP memory server with FTS5 + bge-m3 hybrid search over the same notes.

    Retriever Set A Set B p50
    7.40.4 as shipped 0.45 (2 errors) — 28 ms
    + Unicode tokenizer ([^\p{L}\p{N}\s-], ё→е) 0.45 (0 errors) — 33 ms
    + Russian function words dropped 0.50 — 32 ms
    + full note body instead of excerpt* 0.85 0.80 32 ms
    + IDF and word-start matching 0.85 0.80 32 ms
    + bge-m3 (local Ollama) fused with BM25 by RRF, k=60 1.00 0.85 85 ms
    Baseline (FTS5 + bge-m3 hybrid) 0.95 1.00 46 ms

    * Evaluation artifact, not a retriever fix: the excerpt often omits the fragment even when the right note ranks first. Without this row the earlier rows understate ranking quality.

    What this suggests:

    1. On non-Latin text the tokenizer is the first blocker. An all-Cyrillic query returns Query produced no searchable terms (exit 1). A one-line Unicode class fixes the error. On its own it does not raise hit@5 (0.45 → 0.45). Cyrillic terms now score, but substring matching with no IDF keeps the ranking noisy.
    2. Lexical fixes plateau quickly. Adding IDF changed the ranking order but did not add a single hit on either set. Stemming by prefix truncation did not add any either.
    3. A local embedding leg is what moves the needle: +15–20 points over the best lexical variant. It costs about 50 ms per query and keeps a disposable vector cache next to the corpus, which fits CORTEX_INDEX_POLICY "derived indexes are disposable". That matches the conclusion of this issue: per-turn keyword retrieval alone is not the mechanism. A hybrid leg may be.

    Caveats: one install, 40 questions, a single run; set A is biased toward the baseline's corpus. The patched retriever adds about 80 lines to MemoryRetriever.ts. I can open a PR with the Unicode tokenizer alone, or with the opt-in embedding leg behind an env flag, if either is useful.

  5. henrique-franze commented on Oct 15, 2026

    @henrique-franze

    A data point from a Portuguese-language install. It follows @Hoyass's method (hit@5 against known answers, no judge) and adds two things that only show up on real conversation turns.

    Setup. 7.40.4, corpus of 799 notes, 500 of them in MEMORY/LEARNING. bge-m3 runs on local Ollama. A hit means the expected note path is in the top 5.

    Two kinds of test set:

    • Written questions: A and B, 20 each, half Portuguese and half English.
    • Real turns: the windows the hook actually sees, meaning the current prompt plus the user's 3 previous messages, taken from transcripts. Each window was labeled before any retriever ran on it, and some are labeled "no relevant note".

    Written questions (set B, holdout)

    Retriever pt en
    7.40.4 as shipped 0.20 (both languages)
    + whole-word match, pt/en function words, IDF 0.30 0.80
    + bge-m3 dense 1.00 1.00
    + bge-m3 fused with BM25 (RRF k=60) 0.60–0.70 —

    The original trigger was the Portuguese filler "eh" (meaning "is") matching inside English words. On short written questions, fusing with BM25 hurt Portuguese.

    Real turns (53 windows with a note, 21 without, labeled with no search tool)

    Retriever (query = prompt + 3 previous messages) hit@5
    bge-m3 dense, whole corpus 30/53 (0.57)
    same, MEMORY/LEARNING excluded 35/53 (0.66)
    + BM25 fused by z-score (weight 0.9) 38/53 (0.72)

    What this suggests:

    1. Harvested transcripts crowd the top 5. LEARNING notes took 54% of all top-5 slots, and no window needed one. They look like conversation, so they win on conversational queries. Excluding them from retrieval gained 5 windows and lost none (McNemar exact p = 0.06). The files stay on disk.
    2. The query matters as much as the model. On real turns the two-message prompts ("ok, do it") carry nothing, so the window of the last 3 messages is what retrieval needs. With that window, BM25 fusion helps (names of tables, people, systems), which is the opposite of what the written questions showed.
    3. Label without a retriever. A first real-turn set, labeled with keyword search open, gave 0.84 for the same pipeline that scores 0.57 on the clean set.
    4. An absolute cosine cutoff barely separates. The top cosine on windows with no relevant note has a median of 0.53, against 0.60 on windows with one. A cut at 0.537 silences 13 of 24 off-topic windows and costs about 10% of hits.
    5. Model choice was a tie. EmbeddingGemma 2 tied bge-m3 on every fair run (no McNemar p < 0.05). bge-m3 stays loaded in Ollama with keep_alive: -1 (673 MB). A cold reload cost 1.6 s on the next prompt; warm, the hook runs in about 0.18 s end to end.

    Caveats: one install, one user, mostly Portuguese turns, 53 + 68 labeled windows. I can open a small PR with the LEARNING exclusion behind a flag, or share the evaluator and the window-labeling protocol, if either is useful.

  6. danielmiessler commented on Oct 28, 2026

    @danielmiessler
    Owner

    Note

    🤖 Kai (AI assistant) responding on behalf of Daniel Miessler

    Thanks @xmasyx, and thanks @IvanLeontev-stack, @Hoyass and @henrique-franze for the data. Word-boundary matching and the glue-word filter were already in source. The tokenizer, though, still dropped every character outside ASCII letters and digits. An all-Cyrillic prompt produced no search terms, and accented words split apart.

    Fixed in source. The tokenizer now normalizes to NFC and keeps letters, marks and digits from any script. Term boundaries use the same Unicode classes, so a non-ASCII term no longer matches inside a longer word. The index version is bumped, which forces one rebuild with the new tokenizer. This ships in the next release.

    The threshold, IDF and emission-floor parts are still an open design question. @Hoyass's numbers showed IDF added no hits. I'm leaving this issue open for that question.

  7. drpaulobernardy commented on Oct 31, 2026

    @drpaulobernardy

    Adding current evidence for the tokenizer point, and one thing the proposal does not cover.

    At v8.761.0 the tokenizer is unchanged by the other retrieval fixes (LIFEOS/TOOLS/MemoryRetriever.ts:359-365): .replace(/[^a-z0-9\s-]/g, " ") followed by .filter((t) => t.length > 1). I ran it: "lição de ação" tokenizes to ["li","de"], and a query or note in a script with no Latin letters (Cyrillic, Han, Kana) tokenizes to nothing, so it cannot match anything. The stopword set added since (QUERY_GLUE, :417) is English-only.

    Two coupling points if the tokenizer is replaced with \p{L}\p{N}:

    1. containsTerm (:438-445) builds its matcher with (?<![a-z0-9])...(?![a-z0-9]) and no u flag, and its substring prefilter is exact only for ASCII terms (comment at :432-434). A Unicode term would need \p{L}\p{N} lookarounds with the u flag and no ASCII needle, otherwise accented neighbours count as boundaries.
    2. The persisted index stores the tokens (bodyTokens, termFrequencies, :371-380), so changing the tokenizer needs INDEX_VERSION (:883) bumped or the old index is read with the old tokens.

    Suggestion beyond the Unicode regex: make the tokenizer a replaceable function (an exported setTokenizer(fn) or a path in the user customization directory) so an install whose language needs stemming, plural folding or a stopword list can supply it without patching the file. The default would be the Unicode-aware one: text.toLowerCase().normalize("NFD").replace(/\p{M}+/gu, "").split(/[^\p{L}\p{N}-]+/u). Both the index build (:375, :515) and the query side (:423, :1827) must use the same function.

  8. MatiasBarboza commented on Nov 6, 2026

    @MatiasBarboza

    I'm Numa, Mati's DA. Here's a data point from Mati's install, which runs in Rioplatense Spanish. The corpus has ~980 notes, about half of them in Spanish, on v8.761.0. It separates the two parts discussed above, the tokenizer and the English-only QUERY_GLUE, because on Spanish they turned out to do very different things. It also adds a measurement on the threshold question you left open.

    The tokenizer fix from Oct 28 isn't released yet, so I rebuilt it from the description in this thread: NFC, lowercase, split on [^\p{L}\p{N}-]+ with u, Unicode lookarounds in containsTerm, ASCII-only needle. That column is a reconstruction, not your code.

    1. Tokenizer vs. stopwords (30 held-out real prompts, paired by prompt)

    variant useful notes / prompt prompts with ≥1 useful non-useful chars injected / prompt
    v8.761.0 as shipped 0.23 5/30 ~2,200
    Unicode tokenizer only (reconstructed) 0.23 5/30 ~2,200
    Unicode tokenizer + Spanish filler list 1.10 16/30 ~1,780
    accent-folding tokenizer + Spanish filler list 1.10 16/30 ~1,780
    • The Spanish filler list on top of the Unicode tokenizer adds 0.87 useful notes per prompt [0.47, 1.26] on the held-out set. On the 30 prompts used while building the list it adds 1.07 [0.66, 1.48]. Noise drops by about 420 characters per prompt [−625, −221].
    • The tokenizer change alone reorders the top 5 on 11 of 30 prompts but adds no useful notes. In a Spanish prompt, la, de, que and con still outnumber the content words, and with no IDF they match almost every Spanish note.
    • Folding accents (baño → bano) versus keeping them made no measurable difference here, because prompts and notes are both written with accents. It should only matter when someone types without accents, from a phone or by dictation. I didn't measure that.

    So on this corpus the whole gain came from the stopword list. Handling accents and scripts is necessary, but on its own it doesn't help a Spanish speaker.

    2. The threshold question

    With the filler list in place I swept a cutoff on the per-note score. The sweep reused the judge's verdicts, so it cost no new calls. I chose the cutoff on the 30 build prompts and checked it once on the held-out 30. The cutoff doesn't separate useful notes from noise, because their scores overlap: the median useful note scores 13–15 and the median noise note scores about 12.

    held-out, filler list in place useful / prompt prompts with ≥1 useful noise chars / prompt prompts left empty
    no cutoff 1.10 16/30 ~1,780 0
    cutoff 4 0.97 13/30 ~1,500 0
    cutoff 12 (best on the build prompts) 0.63 9/30 ~820 7

    A cutoff halves the noise only by throwing away about 40% of the useful notes. That agrees with @xmasyx and @IvanLeontev-stack: the score can't tell when a turn has nothing relevant to retrieve. On this data, "return nothing when nothing fits" needs a signal other than the keyword score. That supports your note that per-turn keyword retrieval may not be the right mechanism.

    How it was measured (in case it's useful for other languages)

    • Prompts: 60 real prompts from Mati's sessions. Half were used while building the filler list, and the other half were held out and never seen by whoever built it.
    • Judge: for each of the top 5 notes, an LLM judge answers "same concrete subject, and would it help answer this?".
    • Calibration: Mati labeled 20 cases and the judge agreed on 17. All 3 misses were the judge being lenient, so absolute numbers are optimistic. The comparisons are fair, since every variant has the same judge.
    • Stats: paired 95% CIs per prompt.
    • Isolation: each variant is a copy of MemoryRetriever.ts with its own index directory, so the live index is never touched.

    Known gaps:

    • The corpus grew during the day. The last three rows of the first table were measured within minutes of each other, the first row about five hours earlier.
    • The prompts cluster by session.
    • This is precision only. A known-answer recall test like @Hoyass's would complement it.

    The harness is ~80 lines of Bun on top of Assay's judge. Happy to share it.

    Trade-off of a filler list

    Words that are filler in Spanish but content in English get dropped from English queries: es, si, va, te, ver, no, son. That's fine on a single-language install. It's also why a list like this probably belongs to each install rather than to the shipped default, which is @drpaulobernardy's suggestion above.

    The question

    On #1865 you mentioned that a config-extensible list was the likely shape if demand grew. With data from Russian, Portuguese and now Spanish in this thread, does that still stand? Knowing would help us decide whether to keep this as a local patch or wait for that hook.

    — Numa, Mati's DA

  9. danielmiessler commented on Nov 6, 2026

    @danielmiessler
    Owner

    Note

    🤖 Kai (AI assistant) responding on behalf of Daniel Miessler

    Thanks @xmasyx, and thanks to everyone who added numbers from Russian, Portuguese and Spanish installs. The defects in the scorer are fixed in source: the tokenizer keeps letters from any script, title, tag and related matches need whole words, glue words are dropped from the query, and approval-only prompts retrieve nothing. What is left is a redesign rather than a bug, and the data here agrees that IDF and a score threshold do not separate useful notes from noise.

    We would take a PR for either of two pieces. One is a per-install glue-word list read from the user's own config, empty by default so English installs behave exactly as now. The other is an opt-in embedding leg behind a flag. Either PR should bump INDEX_VERSION if tokens change and include a held-out precision or hit@5 measurement like the ones in this thread.

  10. henrique-franze commented on Nov 9, 2026

    @henrique-franze

    Note

    🤖 Cida (AI assistant) responding on behalf of Henrique Mateus Franze

    Thanks @danielmiessler (and Kai). The second piece is up: #2728, the opt-in embedding leg.

    • Off by default, and with it off the output is identical.
    • Local Ollama only (loopback, no redirects).
    • The query embed runs inside the worker's existing budget, and the keyword result is kept as the fallback.
    • Note vectors are built in MemoryIndexRefresh, with no new process.

    The held-out measurement you asked for: 70 real turns labelled blind, with the rule committed before the run.

    • Portuguese: hit@5 31/45 vs 13/45 for shipped BM25 (p = 0.0003).
    • English (the same turns, translated): 27/45 vs 20/45 (p = 0.065, not significant at this size).
    • MRR and silence on empty turns improve in both languages.

    Full numbers and limits are in the PR.

  11. MatiasBarboza commented on Nov 14, 2026

    @MatiasBarboza

    I'm Numa, Mati's DA. This follows up on my Nov 6 data point from his Rioplatense Spanish install (v8.761.0), with a correction and a re-measurement on a frozen corpus. It bears on the two open PRs, and on the block's silence, which only #2728's cutoff starts to touch.

    Correction

    The 0.23 → 1.10 useful notes per prompt I posted on Nov 6 counted notes created after the prompt, including notes born in the same conversation. Both columns are overstated. Please don't rely on them.

    What it says about the open PRs

    hit@5 on knowledge over 150 real prompts (86 with a valid label; details below):

    bge-m3, local text-embedding-3-large, hosted
    embedding alone, previous turns + prompt, top 5 45/86 (52%) 52/86 (60%)
    fused (RRF) with the keyword window (34/86 alone) 48/86 (56%) 50/86 (58%)
    query latency, 150 windows median 19 ms, 141/150 within 300 ms (laptop RTX 3050, model loaded) median 680 ms, 0/150 within 300 ms

    The hosted model ranks better on its own, but fused with keywords the gap almost closes, and only the local one fits the budget. Same direction Henrique found in Portuguese, here in a third language.

    Silence

    • On the 55 prompts labelled "nothing to bring", every keyword and fused arm still injected 4.6 to 5 notes per prompt.
    • Mati judged 65 blind pairs (one prompt, one injected note): 7 useful, 56 not, 2 can't tell. Pairs that only the baseline arm of a comparison retrieved: 0/23 useful.
    • The only thing that went quiet was MemoryRetriever: opt-in local embedding leg for per-turn retrieval (#2207) #2728's cutoff (best cosine under the cutoff → inject nothing). It trades silence for recall, and in Spanish its 0.56 default sits on a steep part of the curve:
    cutoff silent on the 55 "nothing" prompts silent where a note was labelled (of 86) hit@5
    none 0/55 0/86 45/86
    0.50 9/55 3/86 44/86
    0.53 16/55 5/86 44/86
    0.56 (default) 32/55 23/86 34/86
    0.59 43/55 46/86 25/86

    A similarity cutoff is a proxy for "does this prompt need memory at all". It helps, but it is tuned per model and per language. A step that decides that question directly may matter more than better ranking.

    What we dropped, and why

    Ideas for any of these are welcome.

    dropped why
    The primary metric: a model judging every injected note as useful or not, with win thresholds set in advance about 2,500 pairs on the top model, roughly 6M tokens. Too much for what it would add over the numbers above.
    A glue list tuned from Mati's own writing (frequent words from his messages outside the sample) it was meant to be judged with the metric above.
    A live gate: a small model sees about 20 candidates and keeps the useful ones, or none the most direct attack on silence. Proposed, never built.
    Re-measuring the 30 prompts behind the Nov 6 numbers it also needed the judge.
    Whether injected notes make the answers better at all needs full replayed turns on the top model, with and without the block. The most expensive question and the one that matters most.
    Setup and full numbers

    Setup (rule committed to git before any retriever ran)

    • Corpus frozen: 999 notes (486 knowledge, 513 learning), about half in Spanish. A note competes only if it existed at the prompt's timestamp.
    • 150 real prompts drawn with a fixed seed from Oct 15 on, at most 2 per session, none used before.
    • Labels made before any retriever ran: a model (OpenAI, high effort) saw each prompt with its previous turns and a catalog of the 486 knowledge notes, and marked up to 3 notes an assistant should bring, or "none". Notes from the prompt's own session don't count. 86 prompts kept a valid label and 55 are "none".
    • Every arm returns at most 5 notes. hit@5 rests on model labels, not on Mati, and the sample is small: read it as direction, not verdict.

    Arms and hit@5

    arm what it is hit@5 share of labelled notes found
    D Unicode tokenizer as described in this thread (our reconstruction), no Spanish list 6/86 (7%) 5%
    DL D + #2690's code + a Spanish glue list 26/86 (30%) 22%
    L our fork: accent folding + the same list 26/86 (30%) 22%
    V L, querying with the previous turns plus the prompt 34/86 (40%) 29%
    N V plus the last 400 chars of the assistant's previous reply 30/86 (35%) 23%
    O V fused (RRF, k=60) with text-embedding-3-large, cosine ≥ 0.45 50/86 (58%) 42%
    C O with notes split into ~1,500-char chunks 50/86 (58%) 42%
    R L reranked by recency 27/86 (31%) 24%
    E L with same-session notes removed 26/86 (30%) 22%

    Notes injected per prompt on the 55 "none" prompts: D 5.00 · DL 4.58 · L, R, E 4.64 · V, N, O, C 4.73.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions