Repository navigation
MemoryRetriever: opt-in local embedding leg for per-turn retrieval (#2207) - #2728
henrique-franze wants to merge 1 commit into
Conversation
…anielmiessler#2207) Off by default (LIFEOS_MEMORY_EMBEDDING=1), byte-identical when off. Cosine over a local Ollama embedding (loopback only), empty below a cutoff, previous 3 user messages embedded with the prompt. Query embed in the existing retrieval worker under the prompt budget; note vectors in the async MemoryIndexRefresh Stop hook, which also warms a cold model. Falls back to BM25 on any failure. INDEX_VERSION unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Note 🤖 Cida (AI assistant) responding on behalf of Henrique Mateus Franze Added: the same held-out set in English. The original set had no English turns: none of my real turns since Oct 16 was in English. To cover English I translated the same 70 turns, the prompt and each of the 3 previous messages, keeping the same labels.
What the English round shows:
So my honest summary across both languages:
|
|
[feedback] Russian-language install: held-out numbers in support of the opt-in embedding… What happenedRussian-language install: held-out numbers in support of the opt-in embedding leg in this PR. SummaryA data point for this PR from a Russian-language install (v8.761.0, about 1,470 Cortex notes, almost all in Russian). We have run the same idea locally since Nov 10: a local Ollama embedding leg behind a flag, the keyword result kept as the fallback, and byte-identical output with the flag off. The questions were generated by a model from the notes and spot-checked blind by hand. They were split in two halves before any tuning, and the held-out half was computed once. The measure is "the needed note is in the top 5":
On the same held-out half, bge-m3 scored 68 / 83 / 86 / 24% and embeddinggemma 60 / 74 / 81 / 30%, so the gain does not depend on one model. Broad questions stay weak for every model: one vector per note does not gather an answer spread across several notes. We would rather switch to your flag than keep a local patch, so +1 for merging this. Details
Filed with the LifeOS |
isihussain
left a comment
There was a problem hiding this comment.
Architectural & Quality Review
Outstanding engineering in this PR @henrique-franze. Adding an opt-in semantic embedding leg to MemoryRetriever while maintaining strict privacy boundaries and zero overhead when disabled is a model implementation of system enhancement.
Here is an architectural review across security, reliability, and performance:
1. Security & Privacy: Local-Only Loopback Boundary
The security safeguards around embedding generation are thorough:
- Loopback Assertion (
isLoopbackUrl): Refusing any non-loopback server ensures that private knowledge notes and conversation turns never egress to an external network endpoint. - SSRF / Redirect Protection: Setting
redirect: "error"on the Ollama fetch call:is an excellent defensive move that blocks local proxies or misconfigured services from redirecting sensitive requests off-box.redirect: "error", // a loopback server must not forward note text elsewhere
2. Resilience & Failure Isolation: Non-Blocking Fallback
In MemoryRetriever.getRelevantContextAsync:
- Keyword BM25 retrieval is calculated as the dependable baseline.
- If Ollama is unavailable, the model is unpulled (HTTP 404), or embedding times out, the error is caught cleanly and retrieval serves the keyword results with zero interruption to the user's turn.
- Note embeddings are generated out-of-band in the async Stop hook (
MemoryIndexRefresh), keeping expensive multi-document vectorization off the interactive turn path.
3. Binary Cache Performance & Memory Alignment
In loadCache and saveCache:
- Storing vectors in a contiguous binary block (
vecs.<generation>.bin) alongside atomic index swapping avoids JSON parsing overhead for large vector arrays. new Float32Array(buf.buffer.slice(buf.byteOffset, buf.byteOffset + buf.byteLength))correctly handlesArrayBufferalignment and creates clean, unshared memory views.- Vector normalization at generation time ensures that runtime scoring is reduced to an efficient dot product across candidates.
The test suite in MemoryEmbedding.test.ts (287 lines) with synthetic vectors and mock embedders validates cutoffs, dimensions, and fallbacks cleanly. Ready for merge!
This is the second piece offered in #2207: an opt-in embedding leg behind a flag, with a held-out hit@5 measurement. It is written against
mainatf6ab5e0(8.761.0). Off is the default, and off produces the same output as today.What it does. With
LIFEOS_MEMORY_EMBEDDING=1, the per-turn retriever ranks notes by cosine similarity to an embedding of the turn: the prompt plus the previous 3 user messages. The embeddings come from a local Ollama server,bge-m3by default, set up withollama pull bge-m3. When the best cosine is under a cutoff, the turn injects nothing. That gives the thread the "nothing fits" signal that the keyword score could not provide.Where it lives (small seams, so it ports into the private tree):
LIFEOS/TOOLS/MemoryEmbedding.ts(new)getRelevantContextAsync(),refreshEmbeddingIndex()MemoryRetriever.ts, new exports besidegetRelevantContext, which is untouchedgetRelevantContext(...)/nullhooks/lib/retrieval-worker.ts,hooks/lib/recent-user-messages.ts(new)hooks/MemoryTurnStart.hook.ts(+9/−4)hooks/MemoryIndexRefresh.hook.ts, the async Stop hook that already refreshes the BM25 indexMemoryRetriever.tsheader,MemorySystem.md,CortexContract.md§ vector evidence gateDesign choices, each tied to something in this repo:
redirect: "error"), so a local listener cannot forward note text elsewhere.MemoryIndexRefresh, and each batch starts only if its own timeout still fits in 40 s. A first index therefore spreads over a few turns.INDEX_VERSIONunchanged.retriever-index.jsonare not touched.MEMORY/STATE/cache/memory-embeddings/<model>/, keyed by path, mtime and model, with its own version.index.jsonto point at it. A reader always sees an index together with the vectors written with it, even when two sessions' Stop hooks race.MEMORY/LEARNINGnotes are no longer injected per turn: they read like conversation and crowded the top 5 in my Per-turn RELEVANT MEMORY block: ~10% of injected notes are relevant; the scorer bug can be patched to ~30%, which is still not a solution #2207 data. With the leg off nothing changes.Parity with the flag off.
ok,y,go ahead, an empty string and whitespace.getRelevantContextAsyncand the worker both matchgetRelevantContext, with 0 differences.async, and it imports two small modules.Held-out measurement. The protocol is the one in my #2207 comment: hit@5 against known notes, no judge, and the decision rule committed before the run.
{topK: 5, threshold: 0.20, allowStaleIndex: true}on today's corpus of 486 notes.Cost.
bge-m3.indexedNotes()to compare mtimes. That is async and off the prompt path, but it could read only the changed notes.Limits.
bge-m3. Another model needs its own cutoff (LIFEOS_MEMORY_EMBEDDING_CUTOFF).Tests are for the port. The two
*.test.tsfiles sit beside the code, as in #2690, for running here. I know the public payload ships no tests, so move them into the private test tree or drop them as you prefer.bun test LifeOS/install/LIFEOS/TOOLS/MemoryEmbedding.test.ts LifeOS/install/hooks/lib/recent-user-messages.test.tsgives 19 pass and 0 fail.HOMEand no network.127.0.0.1.CORTEX_MEMORY_ROOT,LIFEOS_DIRandCLAUDE_CONFIG_DIR, so a dev shell cannot point them at a live install.bun .github/scripts/leak-scan.ts --stagedreports no blocking findings, andgitleaks git --staged(8.30.1) reports no leaks.Config. Everything is optional except the flag.
LIFEOS_MEMORY_EMBEDDING=1turns the leg on.LIFEOS_MEMORY_EMBEDDING_MODEL, defaultbge-m3.LIFEOS_MEMORY_EMBEDDING_URL, defaulthttp://127.0.0.1:11434, loopback only.LIFEOS_MEMORY_EMBEDDING_CUTOFF, default0.56.LIFEOS_MEMORY_EMBEDDING_KEEP_ALIVE, sent only if set.LIFEOS_MEMORY_EMBEDDING_CACHE.I used env vars, following the
CORTEX_*precedent. If you'd rather have aUSER/CONFIGfile like #2690, or a fixed port instead of a URL, it is one function (embeddingConfig).Happy to trim, rename or split this however it ports best. For example, the 3-message context could be a follow-up if you want the leg alone first.