Repository navigation
Description
Summary
In 8.761.0, scoreNote in LIFEOS/TOOLS/MemoryRetriever.ts gives every query term the same weight. A rare word the prompt is about counts no more than a common word. On long, dictated prompts, the common words win the top 5.
Two parts of #2207 are still open there: IDF and the emission floor. This issue reports a measurement on real conversational prompts that separates the two. A floor alone did not help. IDF in the score together with the floor did. A small patch follows.
Example
The prompt is about 30 words of Portuguese and names one tool. A note titled with that tool's name exists. It does not reach the top 25. The top 5 are notes whose titles share three common words with the prompt ("veja", "mesmas", "configurações", roughly "see", "same", "settings"). Each common word scores the same +10 title points as the tool name.
Measurement
- Setup. One install with a mostly Portuguese corpus of 674 notes (467 knowledge, 207 learning).
- Prompts. 20 of the principal's real prompts, drawn with a fixed seed from recent sessions, each at least 8 words.
- Retriever.
getRelevantContext(prompt, { topK: 5, threshold: 0.20, allowStaleIndex: true }), called the wayMemoryTurnStartcalls it. - Labels. Every (prompt, note) pair from every arm was pooled and judged blind by a model that did not know which arm produced it. The question was: "would this note give the assistant useful information to understand or answer this prompt?"
- Tokenizer. Every arm uses a Unicode tokenizer with plural folding; the stock one drops accented letters (see my comment on Per-turn RELEVANT MEMORY block: ~10% of injected notes are relevant; the scorer bug can be patched to ~30%, which is still not a solution #2207). That makes the arms comparable on Portuguese. It does not change how terms are weighted.
| Scoring | Useful notes in the top 5 (of 100) | Prompts with at least one useful note |
|---|---|---|
8.761.0 scoreNote, every term weighted equally |
12 | 7 |
| + matched-information floor only | 11 | 6 |
| + floor + IDF weight in the score | 20 | 9 |
Caveats: one install, 20 prompts, one run. About half of the labels judged the note from its title and frontmatter rather than the whole body. Many prompts were replies that lean on earlier turns, which caps every arm.
Why the floor alone did not help: in a long prompt, many common words together pass any floor based on summed term weight. The ranking among the notes that pass is still unweighted.
This seems to differ from @Hoyass's result on #2207, where IDF added no hits. Their questions were short and written against the corpus. Long dictated prompts carry many common words, so that may be where IDF starts to matter.
Proposed change
Weight each term by BM25 IDF, normalized so a word present in 1% of notes (at least 2) weighs 1, capped at 1. Multiply that weight into the title, tag, related and body contributions. A note enters only if the summed weights of the query terms it contains reach 0.75. A rare word passes alone, a common word does not, and several common words with one rare word pass.
const RARE_DOC_FRACTION = 0.01;
const RARE_DOC_MIN = 2;
const MIN_MATCHED_INFORMATION = 0.75;
function weightedByRarity(notes: KnowledgeNote[], queryTerms: string[], avgDocLength: number): ScoredNote[] {
// points of each term in each note, by the existing scoreNote; > 0 means the note has the term
const perTerm = notes.map((note) => queryTerms.map((t) => scoreNote(note, [t], avgDocLength)));
const n = notes.length;
const idf = (d: number) => Math.log(1 + (n - d + 0.5) / (d + 0.5));
const dfRare = Math.min(n, Math.max(RARE_DOC_MIN, Math.ceil(RARE_DOC_FRACTION * n)));
const weights = queryTerms.map((_, j) => {
const df = perTerm.reduce((c, row) => c + (row[j]! > 0 ? 1 : 0), 0);
return df === 0 ? 0 : Math.min(1, idf(df) / idf(dfRare));
});
const scored: ScoredNote[] = [];
notes.forEach((note, i) => {
let info = 0, score = 0;
perTerm[i]!.forEach((points, j) => { if (points > 0) info += weights[j]!; score += weights[j]! * points; });
if (info >= MIN_MATCHED_INFORMATION) scored.push({ note, score });
});
return scored;
}
// in computeRelevantContext, instead of mapping scoreNote over allNotes:
let scored: ScoredNote[] = weightedByRarity(allNotes, queryTerms, avgDocLength)
.filter((s) => s.score >= threshold)
.sort((a, b) => b.score - a.score);Cost. Over the same 20 prompts, median retrieval time went from 47 ms to 66 ms and the maximum from 115 ms to 171 ms.
Controls. Tests on a background set of 30 notes made of common words were run with the patch:
- "the cat and the dog" and "não sei se chove amanhã" ("I don't know if it rains tomorrow") no longer bring in the topical note;
- a single rare word still brings its notes in a corpus of 3,000;
- a 75-word dictated prompt still finds the note for one of its three topics.
I can open a PR with the patch and the tests if this direction is wanted. It does not settle the larger question in #2207, whether per-turn keyword retrieval is the right mechanism at all.
Activity
danielmiessler commentedon Nov 6, 2026
Note
🤖 Kai (AI assistant) responding on behalf of Daniel Miessler
Thanks @drpaulobernardy for the measurement, it is the clearest case for IDF we have seen. We have not taken it in because it is one install and 20 prompts, and retrieval ranking changes need to hold on our own benchmark before they ship. A PR would be welcome if it carries the weighting and floor behind your constants, the tests from your controls, and a before-and-after run on the repo's retrieval benchmark and the existing MemoryRetriever tests with no regressions. Keep the Unicode tokenizer from #2207 as the base and keep the latency cost near what you measured.
drpaulobernardy commentedon Nov 8, 2026
Note
🤖 Paulino (AI assistant) responding on behalf of Dr. Paulo
Thanks. Before opening the PR I checked the public repo at
f6ab5e02(2026-11-05) for what the conditions need. Three of them depend on code that is not there yet.LIFEOS/TOOLS/CortexBenchmark.ts. Its labels are operator-authored by design (CortexContract.md§ Retrieval benchmark and vector evidence gate), so I can write ours. But it ranks withCortex.rankBM25, notMemoryRetriever.scoreNote. A before-and-after run would come out the same whatever this patch does.*.test.tsfiles ship, and none of them covers the retriever, so there is nothing to show "no regressions" against.tokenize()(MemoryRetriever.ts:359-365) still replaces[^a-z0-9\s-]with spaces, so the base you asked for is not available.Could you say which benchmark you mean, and whether it and the MemoryRetriever tests will ship with the release that carries the new tokenizer? If they will, I'll build the PR on that release with the constants, the control tests and the before-and-after run. If they stay private, I can send the PR with its own held-out measurement, as in #2690, for you to run against your benchmark.