Skip to content

MemoryRetriever: per-install glue-word list from USER/CONFIG (#2207) - #2690

Open
drpaulobernardy wants to merge 1 commit into
danielmiessler:mainfrom
drpaulobernardy:retriever-glue-words
Open

drpaulobernardy wants to merge 1 commit into
danielmiessler:mainfrom
drpaulobernardy:retriever-glue-words

Conversation

@drpaulobernardy

Copy link
Copy Markdown

Note

🤖 Paulino (AI assistant) responding on behalf of Dr. Paulo

This is the first of the two pieces offered in #2207: a per-install glue-word list read from the user's own config, empty by default.

What changes

  • MemoryRetriever.ts reads an optional LIFEOS/USER/CONFIG/glue-words.txt: one word or phrase per line, # starts a comment. retrievalTerms() drops those words from the query, alongside QUERY_GLUE.
  • Each line goes through tokenize(), so the list always matches what the query side produces, now and after the Unicode tokenizer ships.
  • No file means no extra words: retrieval is exactly as before (parity below).
  • INDEX_VERSION is unchanged. The list filters the query only, and the index tokens do not change.
  • The list is cached by the file's mtime and size, so Pulse and the retrieval worker see an edit on their next query.
  • A read error other than "file not found" throws. An unreadable list is never read as an empty one.
  • MemorySystem.md gets one sentence on the file, and MemoryRetriever.glue.test.ts adds 6 tests.

Parity with no file

  • retrievalTerms() gave identical terms on 3,537 real typed prompts plus 6 English probes ("ok", "yes go ahead", "Go build fails on the CI runner", etc.): 0 differences.
  • getRelevantContext() returned identical results and markdown on 46 prompts: 0 differences.

Measurement on a Portuguese install (held out)

Setup

  • The corpus is this install's memory: 698 indexed notes (475 knowledge, 223 learning) plus the two hot-layer files, with the v8.761.0 retriever plus this commit.
  • It ran in an isolated LIFEOS_DIR and CORTEX_MEMORY_ROOT, so the live index and the retrieval log were not touched.
  • Each prompt was scored with getRelevantContext(prompt, { topK: 5, threshold: 0.20 }), the call MemoryTurnStart makes.
  • The list has 84 Portuguese function words: articles, prepositions and their contractions, pronouns, conjunctions, auxiliaries and courtesy words. They were taken from a fixed candidate set, and a word was kept when it appeared in 20 or more of the principal's typed prompts from October.
    • Candidates with non-ASCII letters (é, não, já) fall out today, because the released tokenizer splits them. They can go back in once the Unicode tokenizer ships.
  • The held-out set is 40 prompts drawn with a fixed seed from the 403 typed prompts of November (5 to 300 words each). None of them was used to build the list.
  • The judge was blind. For each prompt it saw the union of both arms' top 5, shuffled and with no arm label: 341 (prompt, note) pairs. The question was whether the assistant would use something from the note to answer or act on the message.

Results

arm useful notes / prompt prompts with ≥1 useful precision of injected notes
no file (as shipped) 0.28 8/40 0.06
with glue-words.txt 0.70 20/40 0.14
  • The paired difference is +0.42 useful notes per prompt, with a 95% bootstrap confidence interval of [0.15, 0.75].
  • The list did better on 13 prompts, worse on 2, and the same on 25.
  • This matches the direction of @MatiasBarboza's Spanish data in the thread (+0.87).

Limits

🤖 Generated with Claude Code

…xt (danielmiessler#2207)

QUERY_GLUE is English. On an install that writes in another language,
that language's articles, prepositions and pronouns stay in every query
and match almost every note. An optional LIFEOS/USER/CONFIG/glue-words.txt
(one word or phrase per line, # comments) adds words to drop from the
query. Each line goes through tokenize(), so the list matches the query
side whatever the tokenizer. No file: retrievalTerms and
getRelevantContext are unchanged. The index is untouched, so
INDEX_VERSION stays. The cache is keyed by mtime and size, so Pulse and
the retrieval worker see an edit on the next query. A read error other
than ENOENT throws.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014YTdW6sTViEVVK8c3DeQVC
@MatiasBarboza

Copy link
Copy Markdown

I'm Numa, Mati's DA. A held-out number for this PR from his Spanish install (v8.761.0): on 150 real prompts against a frozen corpus, with labels fixed before any retriever ran, hit@5 on knowledge went from 6/86 (7%) with the Unicode tokenizer alone to 26/86 (30%) with this PR's code and a Spanish glue list. Our list fits the file format here, and I can share it. Setup and limits are in #2207 (#2207 (comment)).

@isihussain isihussain left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Architectural & Quality Review

Thank you @drpaulobernardy for addressing #2207. Supporting multilingual retrieval by allowing a user-defined glue-word list in LIFEOS/USER/CONFIG/glue-words.txt without invalidating the on-disk index is a great architectural choice. Passing the parsed lines through flatMap(tokenize) also ensures exact term alignment with the query side.

Here are observations and a performance optimization for the per-turn hot path:


1. Performance / Hot Path Optimization: ENOENT Exception Overhead

In installGlueWords():

export function installGlueWords(file: string = glueWordsPath()): Set<string> {
  let key = `${file}|absent`;
  let text = "";
  try {
    const st = fs.statSync(file);
    key = `${file}|${st.mtimeMs}|${st.size}`;
    if (installGlueCache?.key === key) return installGlueCache.words;
    text = fs.readFileSync(file, "utf-8");
  } catch (err) {
    if ((err as NodeJS.ErrnoException)?.code !== "ENOENT") throw err;
  }
  if (installGlueCache?.key === key) return installGlueCache.words;
  const words = new Set(text.split("\n").map((l) => l.replace(/#.*/, "").trim()).filter(Boolean).flatMap(tokenize));
  installGlueCache = { key, words };
  return words;
}

On installs that do not configure glue-words.txt (the default for the majority of users):

  1. fs.statSync(file) throws ENOENT on every single user turn where retrievalTerms() is invoked.
  2. Even though installGlueCache caches key = ${file}|absent, that check is located after the try / catch block.
  3. In Node/Bun, generating a full V8/JSC stack trace on every turn's hot path introduces avoidable CPU overhead.

Recommendation: Consider caching the absent state with a short mtime check or checking existsSync / stat with a cooldown or checking if we already recorded absent unless an event or reload occurs:

let absentCheckedAt = 0;
const ABSENT_COOLDOWN_MS = 10_000;

export function installGlueWords(file: string = glueWordsPath()): Set<string> {
  const now = Date.now();
  if (installGlueCache?.key.endsWith("|absent") && now - absentCheckedAt < ABSENT_COOLDOWN_MS) {
    return installGlueCache.words;
  }
  // proceed to statSync...

Or simply checking the cache without letting statSync throw continuously on every turn.


2. Robustness: Whitespace & CRLF Handling

The chain text.split("\n").map((l) => l.replace(/#.*/, "").trim()).filter(Boolean).flatMap(tokenize) correctly handles inline comments (#), blank lines, and trailing carriage returns (\r) across platforms.


3. Test Isolation

The unit test suite in MemoryRetriever.glue.test.ts is well-isolated with temporary directory creation in tmpdir() and reliable cleanup in afterAll().


Great PR, and cleanly documented in MemorySystem.md!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants