Skip to content

Freeze memtomem stores written by v0.2.0 and v0.3.0 and read them with the current readers (#271) - #317

Merged
memtomem merged 4 commits into
mainfrom
test/frozen-memory-271
Nov 15, 2026
Merged

memtomem merged 4 commits into
mainfrom
test/frozen-memory-271

Conversation

@memtomem

Copy link
Copy Markdown
Owner

Refs #271. This is the first part of the Memory family: the memtomem stores that v0.2.0 and v0.3.0 (both memtomem 0.6.8) wrote. The issue stays open: the schema 1 stores of v0.1.2 and v0.1.4 (memtomem 0.3.x), the Mem0 and LangGraph stores with memory migration, and the TUI jobs family remain.

What

tests/fixtures/persisted/memory/<release>/store is one memtomem store per release, written by that release's own CLI through its default backend (freeze.py beside it, run against a git worktree of the tag with uv sync --frozen --no-dev, fake codex and kimi on PATH, no embedding provider). Every scenario goes into the same store, one after another:

  • run under route, pipeline, compete, council (two critique rounds) and decompose: private rows and their shared copies with task: and shared-from= tags and every kind tag;
  • a failed run (leaves no row), a reply with Unicode and CRLF, one with ### headings, a prompt long enough to split into two chunks;
  • a finished board run (its board-run: scratch row and the board-result: row), room apply (room-result);
  • last, a board run -s pipeline SIGKILLed during its second stage, so the store is frozen with that row only in the WAL and a live lease on the board.

freeze.py stops unless each scenario added exactly the rows its definition names and left every earlier row exactly as stored, and unless the killed row is in the WAL only. index.json holds the scenario definitions and a sha256 of every file; release-read.json is what the release's own interpreter read back from every row. Output is published whole (a hidden sibling, then a rename).

tests/test_frozen_memory.py (default suite, no engine): the stored rows against the scenario definitions, the day-file names and namespaces HEAD derives, a shared copy naming its private row, the release read tied to the rows, chunks.source_file (and its FTS mirror) as the only column binding a store to where it was written, the killed row absent from the main database file alone, embedding_provider = none.

tests/test_frozen_memory_real.py (SYNCMILL_TEST_MEMTOMEM_REAL=1, now in CI's memory-backend-real job): HEAD's first and second open of a moved copy, reading every row out of the WAL; MemtomemBackend.get / search / count_namespace compared with the release's read, search with each later-phase kind filter, search_multi, status; share_chunk and add_and_share beside released rows (a later day and the release's own day); purge_namespace of a released scratch namespace; a reindex that changes no row; SyncmillMemory.search_context (filters, agent namespace, board-scoped view), search_shared / get_shared / shared_stats; the gateway shared_* tools; syncmill status; each strategy's recall in a HEAD run (the filters sent, the released rows returned, each handed to every agent of that phase); a HEAD run and room round handed a released result; the store found at the project default location through configure_project_memory; a released lease reclaimed and its scratch namespace purged by the board GC. Copies that write, purge or reindex are rebound first with the two statements memtomem itself uses, asserting every other cell unchanged.

Docs: persisted-contracts.md gets a "Memory (memtomem)" row, with what is not covered; Mem0 and LangGraph stay "not yet" in their own row. The real smoke before a memtomem pin bump now includes tests/test_frozen_memory_real.py (CLAUDE.md, CONTRIBUTING.md, README.md).

No src/ change. No current reader misread a released store.

Measured along the way

  • memtomem tries an AND match of every query word and falls back to OR only when that finds nothing. A strategy-test query with a word the released rows lacked left every later phase with the run's own rows; the query now uses words all released strategy results contain.
  • Closing an ordinary sqlite3 connection checkpoints the WAL into the main file, so the killed store is opened first by the engine under test, and the main-file bytes are taken before any connection.
  • v0.1.4 (memtomem 0.3.12) is a layout of its own: per-namespace day files like v0.2.0, but shared copies tagged only shared-from= and a scratch row v0.1.2 lacks. It goes with v0.1.2 in the follow-up.

Verification

  • suite-run CLEAN; ruff check . clean; CI's real-memtomem command run locally: 68 passed.
  • 39 injected regressions each fail the intended test on an assertion (26 from the template, 13 from review rounds), among them: day-file naming, namespace and tag filters dropped in search, search_multi and search_context, shared-from tags, the share link, the gateway's namespace checks, each strategy's kind string, the room round's recall, the project store binding, FTS rows lost beside a fresh share, reads that fail on a moved copy, recalled context dropped by a strategy or by a later pipeline stage. Fixture-side: a checkpointed WAL, a recorded embedding provider, and a release read missing a namespace or a kind each fail.
  • freeze.py regenerated v0.3.0 into a scratch directory after the review fixes: same 13 scenarios and files, and all default tests pass on it.
  • Reviews (agy): implementation r1 NEEDS-FIX (5 Major), r2 NEEDS-FIX (1 Major: the AND cutoff above), an unprimed r3 over the whole branch NEEDS-FIX (the CI line; a memory migration --from memtomem reader test, declined as Memory-2 scope), post-commit SHIP with two doc nits, fixed.

🤖 Generated with Claude Code

memtomem and others added 4 commits November 15, 2026 21:20
…t readers (#271)

First part of the Memory backends family: one release, as the template for
v0.1.2 and v0.2.0.

tests/fixtures/persisted/memory/v0.3.0/store is a memtomem store the v0.3.0
CLI wrote (freeze.py beside it): run under every strategy, a failed run, a
board run, a room apply, and last a board run SIGKILLed during its second
pipeline stage, so the store is as that kill left it, with its newest rows
in the WAL and a live lease on the board. release-read.json records what the
v0.3.0 interpreter read from it; index.json holds the scenario definitions
and a sha256 of every file.

test_frozen_memory.py runs in the default suite with no memtomem engine: the
stored rows against the scenario definitions, the day-file names and
namespaces HEAD derives, and that chunks.source_file is the only column
binding a store to where it was written.

test_frozen_memory_real.py runs with SYNCMILL_TEST_MEMTOMEM_REAL=1 against
real memtomem: opening the store, the adapter's reads and writes beside the
released rows, a scratch purge, a reindex, SyncmillMemory, the gateway,
status, the filters each strategy sends, a run and a room round carrying a
released result, the store found at the project default location, and a
released lease reclaimed and its scratch namespace purged.

Not in this commit: the v0.1.2 and v0.2.0 stores, the persisted-contracts
row, and the CI line that adds the real file to the real-memtomem job.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…271)

Answers three review rounds of the v0.3.0 template.

test_frozen_memory_real.py:
- A HEAD strategy now runs a task whose every word is in the released
  strategy results. memtomem tries an AND match and falls back to OR only
  when that finds nothing, so a word the released rows lacked left every
  later phase with the run's own rows alone. Each recall must now return
  released rows of the kinds it asks for, none of another kind, and every
  agent of that phase must be handed them.
- The first open of a moved, not rebound copy (the SIGKILL's WAL still
  beside it) reads every row and searches every namespace, not only counts.
- A fresh share is checked against the exact set of released shared rows;
  the old set comparison could not fail.
- rebind compares the directory prefix exactly, separator included, instead
  of a LIKE pattern in which a temporary path's `_` is a wildcard.

test_frozen_memory.py:
- The killed run's row is absent from the main database file alone (its
  bytes are taken before any connection opens the copy, since closing one
  checkpoints the WAL).
- The store records embedding_provider none and has no vector table.
- The release read searched every namespace by every kind; the memtomem
  version is compared as numbers.

freeze.py (the v0.3.0 output is unchanged; regenerated into scratch: same
13 scenarios and files, the namespace-hash lock names aside):
- Every earlier row must stay exactly as stored, ids and full source path
  included, after each scenario.
- The killed run's row must be in the WAL only.
- Databases the release wrote are read through a scratch copy, rooms too.
- Output is published whole: a hidden sibling, then one rename after the
  target is checked again; the sibling is removed on any failure, and a
  symlink at the target is refused, not followed.
- A stalled killed run kills its whole process group before reading output.

CI's real-memtomem job now runs test_frozen_memory_real.py.

Not here: the memory migration reader over these stores
(`memory migration --from memtomem`), which the plan puts in Memory-2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ly (#271)

tests/fixtures/persisted/memory/v0.2.0 is written by the v0.2.0 CLI with the
same freeze.py as v0.3.0. Both releases lock memtomem 0.6.8, and every one of
the 13 scenarios left the same rows as at v0.3.0, ids aside. The default and
real-engine tests now run over both releases.

persisted-contracts.md gets a "Memory (memtomem)" row (documents, readers and
what is not covered); Mem0 and LangGraph stay "not yet" in a row of their own.
The real smoke required before a memtomem pin bump now includes
tests/test_frozen_memory_real.py (CLAUDE.md, CONTRIBUTING.md, README.md).

Not here: the schema 1 stores of v0.1.2 to v0.1.4 (memtomem 0.3.x). Measured:
v0.1.2 writes one mixed day file with untagged rows; v0.1.4 writes
per-namespace day files whose shared copies carry only shared-from=, plus a
scratch row v0.1.2 lacks; v0.1.3 is unmeasured. They follow in their own PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ock (#271)

The doc row and the test docstring called the job by a name it does not
have. The README command wrapped inside an inline code span, so copying it
from the raw file split it into two shell lines; it is now a fenced block
with continuations, run as written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant