Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -124,5 +124,6 @@ jobs:
- run: uv sync --locked --group dev --managed-python
- name: Verify real SQLite extension loading
run: uv run python -c "import sqlite3, sqlite_vec; db = sqlite3.connect(':memory:'); db.enable_load_extension(True); sqlite_vec.load(db); db.close()"
# Temporary stores, real writer/watcher/locks; no provider calls.
- run: SYNCMILL_TEST_MEMTOMEM_REAL=1 uv run pytest -q tests/test_memory_backend_memtomem_integration.py tests/test_memory_gc_process.py
# Temporary stores and copies of release-written ones, real
# writer/watcher/locks; no provider calls.
- run: SYNCMILL_TEST_MEMTOMEM_REAL=1 uv run pytest -q tests/test_memory_backend_memtomem_integration.py tests/test_memory_gc_process.py tests/test_frozen_memory_real.py
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ uv run syncmill status # check the selected memory backend
uv run syncmill serve # read-only MCP gateway over `shared` (stdio; --http/--host/--port)
uv run syncmill memory migration plan --from memtomem --to mem0 # source-retaining snapshot; apply/verify/rollback by id
SYNCMILL_TEST_MEMORY_MIGRATION_REAL=1 uv run pytest -q tests/test_memory_migration_integration.py # required before changing migration adapters
SYNCMILL_TEST_MEMTOMEM_REAL=1 uv run pytest -q tests/test_memory_backend_memtomem_integration.py # required before changing MemtomemBackend's replicated add flow or the memtomem pin
SYNCMILL_TEST_MEMTOMEM_REAL=1 uv run pytest -q tests/test_memory_backend_memtomem_integration.py tests/test_frozen_memory_real.py # required before changing MemtomemBackend's replicated add flow or the memtomem pin (the second file reads release-written stores)
SYNCMILL_TEST_LANGGRAPH_REAL=1 uv run pytest -q tests/test_memory_backend_langgraph_integration.py # required before changing LangGraph SDK pins/API calls
uv add <pkg> # add a dependency (edits pyproject.toml + uv.lock)
uv sync --group dev # install dev deps (pytest)
Expand Down Expand Up @@ -184,7 +184,7 @@ Rules that must hold:
- **Write-path invariants — do not change without the real smoke.** Each write invalidates the search result cache; writes go to namespace-suffixed day files `{date}.{ns}.md`; each entry gets an explicit unique heading (`_entry_title`); payloads open with a **bold** label, never `## ` (heading-led content skips the unique title).
- **Locking — do not change without the real smoke.** A write takes the namespace lock (`_namespace_lock_path`) before selecting its day file, then holds the resolved file's L2 sidecar (`memory_lock_path`) across mix-check → append → index; `purge_namespace` holds the namespace lock plus every owned source's sidecar, removes sources before rows, and raises when a purge cannot be proven complete. `add_and_share` (#117) holds ONE span over both appends: both namespace locks, then both day files' sidecars, from before the clock is read until the last copy is indexed; its copies share one `index_file` and are matched to their source chunks by `shared-from=`, never by index order. **Ordering rule** (on `_namespace_lock_path`; tested): every namespace lock before any sidecar, never a namespace lock while holding a sidecar, and within each level ascending `str(lock_path)`, deduplicated (`_namespace_lock_order`; sidecars by `memory_lock_path`). An unlocked append lets the watcher index the file first and strand the chunk in the wrong namespace.
- **Protected targets.** An unattended append refuses a symlinked day file, a non-`user` tier, a read-only source and an excluded source. `purge_namespace` refuses, before deleting anything, a symlinked source, one in a `read_only_memory_dirs` root, and one in a `project_shared` tier (a `project_local` source may be purged).
- **Pins.** `memtomem>=0.6.8,<0.7` and `mcp>=2.0.0,<3` move together; bump only with `SYNCMILL_TEST_MEMTOMEM_REAL=1 uv run pytest -q tests/test_memory_backend_memtomem_integration.py`.
- **Pins.** `memtomem>=0.6.8,<0.7` and `mcp>=2.0.0,<3` move together; bump only with `SYNCMILL_TEST_MEMTOMEM_REAL=1 uv run pytest -q tests/test_memory_backend_memtomem_integration.py tests/test_frozen_memory_real.py`.

### Interactive collaboration (human drives one agent, syncmill runs the rest)

Expand Down
3 changes: 2 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ Some changes require an opt-in real smoke before they can be reviewed:

```bash
SYNCMILL_TEST_MEMTOMEM_REAL=1 uv run pytest -q \
tests/test_memory_backend_memtomem_integration.py # the memtomem add flow or pin
tests/test_memory_backend_memtomem_integration.py \
tests/test_frozen_memory_real.py # the memtomem add flow or pin
SYNCMILL_TEST_MEMORY_MIGRATION_REAL=1 uv run pytest -q \
tests/test_memory_migration_integration.py # migration adapters
SYNCMILL_TEST_LANGGRAPH_REAL=1 uv run pytest -q \
Expand Down
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -827,8 +827,14 @@ The suite replaces agents and memory with in-memory fakes (see

The [2026-09-08 stabilization roadmap](https://github.com/memtomem/syncmill/blob/main/docs/improvement-roadmap-2026-09-08.ko.md)
reconciles historical handoffs with current source and documents memory GC recovery.
Opt-in real memory tests use temporary stores: `SYNCMILL_TEST_MEMTOMEM_REAL=1 uv run
pytest -q tests/test_memory_backend_memtomem_integration.py tests/test_memory_gc_process.py`.
Opt-in real memory tests use temporary stores and copies of release-written ones:

```bash
SYNCMILL_TEST_MEMTOMEM_REAL=1 uv run pytest -q \
tests/test_memory_backend_memtomem_integration.py \
tests/test_memory_gc_process.py \
tests/test_frozen_memory_real.py
```

## Contributing

Expand Down
3 changes: 2 additions & 1 deletion docs/architecture/persisted-contracts.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,8 @@ A golden over the current writer does not protect these. A reader can stop accep
| Workflows | the workflow database (`workflows.db`): rows with `spec_json`, manifest `schema_version` 3 with `calls_used`, events, and from v0.3.0 the lineage columns and `file`/`revision` origins; per workflow `spec.json`, planner output, node bundles (`result.json`, `winner.patch`, `commit.json` v3) and stream logs; workflow-origin run artifacts, approved or not, with `plan.final_agent` from v0.3.0. No release wrote manifest v1 or v2 (both normalize to v3 before the insert); those readers stay covered by `test_workflow_durable.py::test_marked_v1_migration_preserves_exact_count_and_control_fields` and `test_workflow.py::test_a_workflow_whose_call_accounting_is_ambiguous_is_refused` | `workflow list` / `show` (`--prompts`) / `status` / `logs` / `lineage` / `export` / `cancel` / `prune`, a dead release-written lease recovered by `show` and `status`, `WorkflowStore.events` and `read_node_bundle`; for the final artifact `diff`, `runs show`, `apply` (approved, after a prune, approval healed from the database, refused for a workflow that never completed, `no_changes`), the standalone `recover` skip. A final artifact naming another run, and an unreadable store: `test_cli.py::test_apply_refuses_an_artifact_the_named_workflow_did_not_finalize` and `test_apply_guard_fails_closed_when_the_workflow_store_cannot_be_read`. HEAD execution (`WorkflowCoordinator.run`, what the job worker calls): `run` of released planned, created and revised rows (accounting, stored prompts, dependency output); `resume` of released crashes and failures against the stored deadline (expired: fails before dispatch; before it: the completed dependency's bundle is reused and the unfinished node's attempt refunded, a published final artifact is reused and approved without backfilling `final_agent`, an exhausted node dispatches nothing), refused from a final or unstarted state and while the owner lives, with the released execution history and events kept; `workflow revise` (a v0.2.0 parent gains a lineage); the launch's team peek and state check; `apply --verify` narrowing (v0.2.0: the gate's list only, with a notice; also after a prune); control plans enabled refuse a released workflow planned without one. Written later by the same releases (`freeze.py --part 2b`, `bindings-2b.json`): a node bundle saved just before the crash, reclaimed on resume through `read_node_result` / `read_node_bundle` with its usage added once (`_record_usage`), and re-run when its patch, its commit record or its digest disagrees; control-plan workflows, whose released `control-plan.json` is the control preflight's request, whose binding and validation topology refuse a changed configuration, and whose stored limits and node timeouts bound a run with control plans on, off, or Toolgraph off (`_execution_limits`), plus `show`, `read_control_plan`, `conversations show` and `apply` of a completed one; an interrupted apply of a final artifact, classified by `recover` (completed, not applied, manual recovery; `_bound_receipt`) without touching its approval or the workflow row. Not covered: the TUI's `_read_artifact_result`; no reader revalidates a released execution's `control-preflight.json` | `test_frozen_workflows.py` (v0.2.0, v0.3.0) |
| Sessions and project | isolated-session documents in every state a release leaves (open, checkpointed, closed, and from injected exits creating, closing, recovery_required) and their checkpoint run artifacts; the project marker `.syncmill/project.json` and the `syncmill.toml` that `init` writes | `session isolated-list` (`IsolatedSessionStore.list`: live, source staleness in the stored fingerprint's version), `IsolatedSessionStore.load` path binding (repository, workspace run root, worktree, derived tmux name) through `attach-isolated` (state, liveness, pane cwd), `checkpoint` (a missing worktree) and `close` (each state, a missing workspace), `tui_collaboration.render_session`; for the checkpoint artifact `runs list` / `show`, `diff`, `conversations show` / `list` / `close`, `continue` (refused), `recover` (skipped), `apply` and `validate_applicable`, `discard`, `prune` (pending kept), the TUI `render_artifact` / `_read_artifact_result`; `project_id` with the marker untracked, tracked or symlinked, `load_project_config` of the released config, `initialize_project` falling back to the marker. Not covered: a successful checkpoint or worktree cleanup (a release-written workspace is not relocatable) | `test_frozen_sessions.py` (v0.1.2, v0.2.0, v0.3.0) |
| Board | the board database (`board.db`, with the WAL a killed run leaves): items in every state a release's `board add` / `run` / `update` / `import-review-candidates` leave them (gate evidence passing and failing, the agent's `done` and `review` blocks with handoff and outputs, an agent failure, a killed and a reclaimed claim, a SIGINT and an orchestration crash, manual evidence and sign-off, dependencies including a missing one, imported review candidates), their transitions, decisions including a superseded one, the run ledger, and from v0.2.0 the compaction rollups and counters (partly compacted: a count needs both the live audit and the rollups). No release wrote the lesson columns (`project_id`, `provenance_json`, `source_key`); `test_lessons.py::test_an_old_board_gains_the_lesson_columns` and `::test_a_damaged_provenance_reads_as_not_recorded` cover those readers, and the mixed-project tests there the project filter. Every release writes a claim's ledger row in the claim's transaction, so the ledger backfill's positive branch stays covered by the synthetic tests in `test_board.py`; no release writer produces damaged JSON or a malformed evidence entry either | HEAD's first open (columns and tables added, every released cell kept, a second open a no-op), `Board.get` / `list` / `transitions` (`_row_to_item`, `_load_list`, `_load_evidence`) compared field by field with what the release's own interpreter read from the same database and with the raw cells, `board list` / `show`, `active_decisions` and `_row_to_decision` on every row (retired ones included), `board decisions`, `project_lessons` (empty: no released decision is a lesson), `stats` at fixed clocks including both sides of the aging cutoff, and `board stats` at a fresh and an aged clock; the human path on released rows: `update --state done` counting stored evidence or refused without it, `--sign-off`, `--clear-human-required`, an annotate keeping a released lease, `decide --supersedes` in and out of scope, a re-import skipped, and refused when the stored provenance differs. HEAD operating a released board: `board run` dispatch (FIFO order, skipping human-required items and unmet or missing dependencies, `--goal`, `--id`) with the released handoff, inputs and active decisions in the prompt and the run's outcome added to the released row, an agent's `done` counted only with this run's gate; a released running lease reclaimed exactly at its timeout (the ledger's `reclaimed_at` stamped, on v0.1.2 in the column HEAD added), redispatch counted, `apply_update`, the crash block and `renew_claim` fenced by the released lease; the scratch GC over the released ledger (purged runs never collected, a live lease never, a reclaimed run after one lease timeout of grace, a failed purge scheduled and then cleared in the columns HEAD added); `board compact` of released history (stats unchanged, rollups and counters added to, a cutoff between two claims); the gateway `board_list` / `board_get` / `board_stats`, `dashboard.take_snapshot` and the TUI board tab; a lesson promoted onto a released board leaves the released decisions unbound, and cannot supersede one (scope refused before project). Not covered: a run v0.1.2 reclaimed but never purged (its `reclaimed_at` is NULL, so HEAD collects it with no grace; v0.1.2 had none either, and its own GC purged every such run in the fixtures); `room promote` onto a released board (needs a room; `test_lessons.py` covers the CLI); the memory a released board run wrote (outside `board.db`) | `test_frozen_board.py`, `test_frozen_board_ops.py` (v0.1.2, v0.2.0, v0.3.0) |
| Memory backends | Mem0 and LangGraph stored payloads | the backends' read paths | not yet |
| Memory (memtomem) | the memtomem store a release's CLI writes through its default backend (`mm.db` with the WAL a SIGKILLed board run leaves, the per-namespace day files and their lock sidecars), one store per release holding every scenario: `run` under each strategy (private rows and their `shared` copies with `task:` and `shared-from=` tags and every kind tag), a failed run (no row), Unicode with CRLF, `###` headings, a body split into two chunks, a finished `board run` (its `board-run:` scratch row and the `board-result:` row), `room apply` (`room-result`), and last a `board run` SIGKILLed in its second stage (its scratch row only in the WAL, its lease live on the board). The store is bound to the directory it was written in by `chunks.source_file` and its FTS mirror only; the real-engine tests rebind a copy with the two statements memtomem itself uses and assert every other cell unchanged. `release-read.json` is what the release's own interpreter read from every row | without an engine: the stored rows against the scenario definitions, HEAD's day-file and namespace names, the location binding. With real memtomem (`SYNCMILL_TEST_MEMTOMEM_REAL=1`, CI's `memory-backend-real` job): HEAD's first and second open of a moved copy (rows unchanged, read out of the WAL), `MemtomemBackend.get`, `search` (every namespace) and `count_namespace` compared with the release's read, `search` with each later-phase kind filter, `search_multi`, `status`, `share_chunk` and `add_and_share` beside released rows (a later day and the release's own day), `purge_namespace` of a released scratch namespace, a reindex of the released day files (no row changes), `SyncmillMemory.search_context` (each later-phase filter, the agent namespace, a board-scoped view), `search_shared` / `get_shared` / `shared_stats`, the gateway `shared_*` tools, `syncmill status`, each strategy's recall in a HEAD run (the filters it sends, the released rows they return, each handed to that phase's agents), a HEAD `run` and a room round handed a released result, the store found at the project default location through `configure_project_memory`, and a released lease reclaimed with its scratch namespace purged by the board GC. Not covered: the schema 1 stores of v0.1.2 to v0.1.4 (memtomem 0.3.x, migrated on open; v0.1.2 and v0.1.4 measured as two different layouts, v0.1.3 unmeasured; a follow-up); `memory migration --from memtomem` over these stores (with Mem0 and LangGraph); stores with dense vectors (the frozen stores record `embedding_provider = none`); `sessions` / `session_events` rows (no reader); an external watcher reindexing a released day file; memtomem versions no syncmill release locked | `test_frozen_memory.py`, `test_frozen_memory_real.py` (v0.2.0, v0.3.0) |
| Memory backends (Mem0, LangGraph) | Mem0 and LangGraph stored payloads | the backends' read paths, `memory migration` into memtomem | not yet |
| TUI jobs | job documents, `schema_version` 1 and 2 | `JobStore.list_jobs` | not yet |
| Run envelope | report run directories as a release's `run` and `board run` leave them with recording on: `envelope.json` (completed, no change, route failure, gate passed and failed, a compete team, a file trace, a Phoenix export error, a trace error, breaker and multiple degradations, a board block, the skeleton a SIGKILL leaves, a SIGINT's finalized crash), `result.json`, `trace.otlp.json`, and a trace-only run with no envelope; v0.1.2 wrote no `.syncmill-run-dir` sentinel. Under `<release>/preflight/`, its own reports root: run dirs with a real preflight block from the release's `run` against a fake `toolgraph` answering with the release's vendored goldens (advisory ok, drift, unresolved, unmapped, error, timeout; strict allow, warn, block, timeout, unresolved, unavailable under both policies, two identities, a generation mismatch, an unmapped participant; from v0.2.0 control-plan allow, findings, error, timeout, both modes), the persistence failures reached by making the release's own writer fail one write (v0.1.2 and v0.2.0+ leave different files), and a SIGKILL after a strict preflight; `preflight.json`, `preflight/eval-NNN.json`, `control-plan.json`, `control-preflight.json`. A few producer replies are synthetic and named in its `index.json` | HEAD's `contracts/run-envelope.schema.json` (and the release's own, frozen beside it), `load_envelope` returning the stored document, `report show` and `report list` compared with what the release's own CLI printed, including the refusal for a run without an envelope, the TUI Reports tab and the dashboard REPORT pane; retention ownership (`_is_run_dir`) through `RunReporter.start` and the Supervisor's trace-only sweep: a sentinel or, for v0.1.2, an envelope with syncmill's `kind` naming its own directory makes a run dir removable, and a v0.1.2 trace-only run dir (neither) is never removed; a custom `trace.dir` is never swept; for the preflight runs, the same schema, loader, `report show` / `report list` and UI checks, and retention removing a released dir whole with its nested `preflight/` directory. Not covered: `trace.otlp.json` semantics (tracegraph's reader); `report list` also lists a foreign directory holding any nonempty JSON-object `envelope.json` (observed, unchanged); no HEAD code reads the preflight files back, so their checks are fixture integrity; `report show` prints `Preflight: ok` for a strict run that enforcement blocked (observed, unchanged) | `test_frozen_envelope.py`, `test_frozen_preflight_envelope.py` (v0.1.2, v0.2.0, v0.3.0) |

Expand Down
Loading
Loading