Prove one-session-per-document readiness over loopback serve

CHANGELOG.md

@@ -7,6 +7,9 @@ This project uses Semantic Versioning as **interpreted for MemNet**: package `a.
 
 ## [Unreleased]
 
+### Added
+- **Honesty `c` — one session per document over serve (probe)** — Loopback `memnet-serve` readiness for a product gate that keeps one session per document (TTL 60, save-on-expire, `MEMNET_MAX_SESSIONS=1024`, ~1 800 parts). Extra probes E11–E14, E12 on a fulldoc-with-edges fixture (3000 nodes + 4500 edges at 5000 and 10000), E16 latency, E17 `WHERE CONTAINS`, E18 snapshot `value_bytes` (decoded vs escaped) / tab-CR / `max_fields` vs `line_bytes`, and RSS fixtures. Probe and tests; no engine or cap-default change. No SemVer bump. Wire: [`docs/operations/one-session-per-document.md`](docs/operations/one-session-per-document.md).
+
 ### Changed
 - **Invent only — ClusterRoute vs SliceHandCarry (#191 / #47 cousin)** — `MemNetTwoMoves` outside `MemNetSystem` (`MN-REQ-06.9` + `MN-REQ-06.10` / `MN-VER-06-S08`). ClusterRoute = where the session lives (`MemNetLanMcpFront`; one owner; `pin_map` / `find` SHALL NOT span backends). SliceHandCarry = explicit copy into another session (`export_pin_map` or `session_save` → LAN file copy → dest import/`session_load`; `import_slice` same-serve only). Not a live hop. `import_slice(from_url)` not shipped. tip≠face. `inventOnly=true`; `implemented=false`; no engine code; no SemVer bump. Wire: [`docs/operations/cluster-route-vs-slice-hand-carry.md`](docs/operations/cluster-route-vs-slice-hand-carry.md).
 - **Invent only — LAN MCP front over several serves (#191)** — `MemNetLanMcpFront` outside `MemNetSystem` (`MN-REQ-06.9` / `MN-VER-06-S07`). One MCP catalogue, N LAN `memnet serve` backends; `SessionOwnerRegistry` is owner (explicit pin allowed; silent hash is not sole routing). One owner per session; `pin_map` / `find` SHALL NOT span backends. Cousin of #47 (peer sid handoff), not the same invent. tip≠face. `inventOnly=true`; `implemented=false`; no engine code; no SemVer bump. Wire: [`docs/operations/memnet-lan-mcp-front.md`](docs/operations/memnet-lan-mcp-front.md).

docs/README.md

@@ -67,6 +67,7 @@ Multitask MUST for this product. Index: [`operations/README.md`](operations/READ
 | [`operations/memnet-lan-mcp-front.md`](operations/memnet-lan-mcp-front.md) | Later invent #191: ClusterRoute — one MCP catalogue over N LAN serves (tip≠face; not shipped) |
 | [`operations/cluster-route-vs-slice-hand-carry.md`](operations/cluster-route-vs-slice-hand-carry.md) | Two named moves: ClusterRoute vs SliceHandCarry (#191 / #47 cousin; inventOnly) |
 | [`operations/admin-usage-report.md`](operations/admin-usage-report.md) | Admin-only serve usage JSON for a product-gate admin MCP (opaque alias; not agent MCP) |
+| [`operations/one-session-per-document.md`](operations/one-session-per-document.md) | One MemNet session per document over loopback serve (no MCP front) |
 
 Product skill: [`.cursor/skills/memnet-reference/`](../.cursor/skills/memnet-reference/). SysML trail: MN-REQ-12 → [`sysml-models/outputs/multitask-case-study.md`](../sysml-models/outputs/multitask-case-study.md).
 

docs/operations/README.md

@@ -11,5 +11,6 @@ Agent operating doctrine for this product (not domain recipes).
 | [`memnet-lan-mcp-front.md`](memnet-lan-mcp-front.md) | Later invent #191: ClusterRoute — one MCP catalogue, N LAN serves (tip≠face; not shipped) |
 | [`cluster-route-vs-slice-hand-carry.md`](cluster-route-vs-slice-hand-carry.md) | Two named moves: ClusterRoute vs SliceHandCarry (#191 / #47 cousin; inventOnly) |
 | [`admin-usage-report.md`](admin-usage-report.md) | Admin-only serve usage JSON (opaque alias; not agent MCP; MN-REQ-06.11) |
+| [`one-session-per-document.md`](one-session-per-document.md) | Product gate: one serve session per document over loopback (no MCP); 0.19.18 probe |
 
 Application pattern for `modelbasedPrj-*` / `SysMLEdgePrj-*`: [`../application-notes/system/llm-system-dev-multitask.md`](../application-notes/system/llm-system-dev-multitask.md). Index: [`../README.md`](../README.md).

docs/operations/one-session-per-document.md

@@ -0,0 +1,112 @@
+# One session per document over serve
+
+How a **product gate** keeps **one MemNet session per document** by calling `memnet-serve` on loopback. No MCP front. Synthetic proof: `scripts/probe_doc_gate_readiness.py`. Cap refuse/clip strings stay in [`../cap-contract.md`](../cap-contract.md) (unchanged on 0.19.18).
+
+Never log a session id (`mn_…`).
+
+## Process shape
+
+1. Start serve on loopback. The gate talks the length-prefixed JSON argv envelope, not `memnet-mcp`.
+2. Open one session per document (`session open --map-file` with the tech-docs map, or an equivalent SCHEMA).
+3. Populate with product `mutate` (CREATE / MATCH…SET). Split stdin at `MEMNET_MAX_BATCH_LINES` (default 1000).
+4. Read with `query pin-map` (raise `--max-rows` when the neighbourhood must exceed default \(M=50\)).
+5. Close the session when the document leaves the live set. Opt-in expire snapshot if the graph must survive TTL.
+
+Settings this gate uses:
+
+| Knob | Value |
+|------|--------|
+| TTL | 60 minutes (`MEMNET_SESSION_TTL_MINUTES` or `session open --ttl 60`) |
+| Save on expire | on (`MEMNET_SAVE_ON_EXPIRE=1`) plus `MEMNET_EXPIRE_SNAPSHOT_DIR` |
+| Concurrent sessions | 1024 (`MEMNET_MAX_SESSIONS`) |
+| Document size | about 1 800 part nodes plus a few opaque `USR` text nodes |
+
+Ingest caps (`max_nodes=2000` / `max_edges=2000`) are Path-B ingest, not this mutate path. Session row cap remains 5000 non-LAW rows (**nodes plus edges**). `session load` of a snapshot is **not** bound by ingest budget; it walks leftover `parse_line` + `MemStore.upsert` (`MEMNET_MAX_ROWS`, max sessions, leftover value/line/newline/FIELD_COUNT).
+
+## Request envelope (direct serve)
+
+Length-prefixed UTF-8 JSON on TCP `127.0.0.1` (default port 18765):
+
+```json
+{"args": ["query", "pin-map", "--session", "<id>", "--cue", "SEC_0001"], "stdin": null}
+```
+
+`stdin` is omitted unless the CLI flag `--stdin` needs a body (`mutate --stdin`). Reply is only:
+
+```json
+{"exit_code": 0, "stdout": "…", "stderr": "…"}
+```
+
+There is no MCP `errors` array and no `session_id` field on this envelope. Session id appears only as `@SESSION:` on stdout. Hard refuse is stderr `@ERR: {code}|{message}` (exit 1 or 2). Mutate also prints `ok=N fail=M` on stderr.
+
+Client helper: `memnet.serve.send_command(args, stdin=…, host=…, port=…)`. Wait default is 30 s (`SERVE_CLIENT_TIMEOUT_S`); a 1 800-node mutate batch may need a longer `timeout=` from the gate.
+
+## Snapshot
+
+`session save --file` writes `# memnet-snapshot-v1` via `Path.write_text` (overwrite). MemNet does **not** make that file write-once (no `O_EXCL`, no `chmod`, no immutable flag). The caller or the filesystem can.
+
+Opaque text must stay on one snapshot line. Newlines inside a property survive GQL mutate in RAM, but leftover `@TAG` emit does not escape them, so `session load` raises `@ERR: FIELD_COUNT`. Leftover emit **does** escape `\` and `|` (`join_payload`). A 16 KiB string survives CREATE / SET / `pin_map` in RAM (GQL mutate does not enforce pipe `value_bytes` / `line_bytes` — cap-contract bug 4) but snapshot load of a 16 KiB field refuses `@ERR: limit_exceeded|value_bytes 16384/4096`. Unicode, `|`, and quotes on a **short** single line do round-trip.
+
+Expire: with save-on-expire and a dir, TTL drop writes `{dir}/{sid}.snap` (do not log the name). Next use: `@ERR: session_expired|snap_available`. Restore: `session load --session <id>` (no `--file`).
+
+## ACL
+
+CapsPolicy is off until grant/enable. Who and WorkerWriteScope apply to `pin_map`, `mutate`, and `export pin-map` when ACL is on. `session save` / `load` / `close` do not take `--caller` and do not who-check. Bind is skipped on `memnet serve` (`MEMNET_SERVE_INTERNAL=1`).
+
+## Memory figures
+
+Admin `memnet admin usage-report` (not agent MCP) reports **process** `rss_bytes`. `housekeep stats` is per-session row/edge/orphan counts, not bytes. `session expire-status` is flags only. Measure per-document RSS from `/proc/<serve-pid>/statm` by subtracting before/after populate.
+
+The usage report `sessions.live` count is `registry_count()` and can include expired-but-unswept entries. `session list` purges first. Do not treat usage `live` as the true live set until that is fixed. Not fixed in the 0.19.18 probe.
+
+## Probe
+
+```bash
+source .venv/bin/activate
+python scripts/probe_doc_gate_readiness.py --out /opt/cursor/artifacts/doc-gate-readiness-proof.log
+```
+
+`--quick` shrinks nodes/churn/wait (not the product-gate proof). Extra flags: `--load-nodes` (E11, default 3000), `--fat-nodes` / `--fat-text-nodes` / `--fat-rss-samples` / `--fat-churn` (second RSS fixture), `--fulldoc-nodes` / `--fulldoc-fat` / `--e16-n` (third fixture + latency). E18-only: `--churn 0 --rss-samples 0 --expire-wait 0 --fat-churn 0 --fat-rss-samples 0 --load-nodes 0 --fulldoc-nodes 0 --nodes 40`. Tests: `tests/test_doc_gate_readiness.py` (live subprocess serve; E11 uses 3000 nodes).
+
+## Extra probes (E11–E14)
+
+| Item | What holds on 0.19.18 |
+|------|------------------------|
+| E11 | `session load` of a 3000-node snapshot (mutate batches ≤1000 lines, then save/close/load) is **not** `ingest_budget`. Bound by `MEMNET_MAX_ROWS` (5000) at upsert. Neither batched load nor an ingest exemption is needed at 3000. |
+| E12 | **yes** (revised, fulldoc). Counts **nodes plus edges** on write and `session_load`. 3000 nodes then 2000 edges fill default 5000; next edge `@ERR: limit_exceeded\|rows 5001/5000`. Pi 10000 holds 7500 (`rows=7500` `edges=4500`) and `session load` of that snapshot is **not** `ingest_budget` (loaded 7500). Same 7500 snapshot on 5000: `@ERR: limit_exceeded\|rows 5001/5000`. `pin_map` read is **not** the session cap: hub `M=50` → `## Truncation truncated=true M=50 omitted=2956 reason=max_rows`; hub `M=4000` → `@ERR: response_too_large\|response 9491260 bytes exceeds cap 4194304` (4 MiB serve frame). |
+| E13 | 16 KiB strings with LaTeX / quotes / newline / `\|` / CJK survive GQL CREATE/SET/`pin_map` in RAM. Snapshot save/load does **not** survive byte-for-byte (`FIELD_COUNT` on newlines; `value_bytes 16384/4096` otherwise). Pipe leftover: value 4096, line 32768; GQL mutate skips those (bug 4). Escapes: `\\ \' \" \n \r \t` only. |
+| E14 | List literals store as JSON strings and emit as GQL lists. `'k' IN p.citeKeys` is **not** a product filter (`MATCH (p:USR) WHERE … SET` ignores WHERE and raises `cue_conflict` when \|Q\|>1). Locators are `KEY=VAL` equality on the JSON string; leftover `read list --where` can glob that string. |
+| E17 | **note.** `WHERE n.value CONTAINS` is not a product filter. RETURN → `product_gate`; SET drops WHERE (`cue_conflict` at \|Q\|=1500; unique MATCH still SET on a miss). `STARTS WITH` / `ENDS WITH` / `=~` same. Working: `find`/`pin_map --keyword` (casefold). See paragraph below. |
+| E18 | Snapshot `value_bytes` 4096 is the **decoded** field after `split_payload` (`>` not `>=`); `join_payload` expansion of `\\` / `\|` is not the cap. `line_bytes` 32768 is the raw snapshot line. SCHEMA `max_fields=32`. Tab round-trips; CR/LF do not. Live table below. |
+
+Second RSS fixture: 3000 nodes, no edges, 1500 of them with 2/3/4 KiB text (about 4.4 MiB of UTF-8 payload, not 1 MiB). Measure process RSS the same way as the 1800-part fixture. Short fat churn is on (`--fat-churn`, default 8); 110 cycles of this fixture is not the default.
+
+Third fixture (fulldoc with edges): 3000 nodes (1500 with 2–4 KiB text) plus 4500 edges (`inSection`, `cites`, `refersTo`). Order is `SEC.order`, not an edge. New relation types need mutate `--allow-new-relation`. Default 5000 cannot hold 7500 rows; use 10000 (Pi) or split sessions.
+
+E16 (on the 10000 fulldoc session, this VM `Intel(R) Xeon(R) Processor` 4-core KVM; the face host may be slower). Bar 300 ms p95, n=200, direct serve loopback, warm session.
+
+| Leg | p50 / p95 / max (ms) | Versus 300 ms |
+|-----|----------------------|---------------|
+| (a) atomic SET 2 KiB + delete 1 edge + add 2 | 115.686 / **133.181** / 155.571 | under bar |
+| (b) reverse `pin_map` hub `M=400` | 113.096 / **129.613** / 163.101 | under bar |
+
+**note:** MemNet has **no** native delete-refused-while-referenced check. `DETACH DELETE` of a node with inbound edges exits 0 and leaves dangling edges. The gate must refuse from the reverse lookup (`## Truncation truncated=true M=400 omitted=2604 reason=max_rows` on the hub). Documented `MATCH ()-[r {id}]-() DELETE r` is lowered as a node DROP with an empty id and refuses `@ERR: not_found|DELETE matched no element` (not a referenced-delete check). The probe's working edge DROP is `MATCH (n WHERE true)-[r {id}]->() DELETE r`.
+
+E18: **yes.** Snapshot `value_bytes` 4096 is the **decoded** UTF-8 after `split_payload` (`tag_map.validate_values`: `len(val.encode("utf-8")) > caps.max_value_bytes`, so 4096 passes). `join_payload` expansion of `\\` / `|` is not the cap (4000 `\\` or `|` emit 8000 escaped bytes, snap line ~8021, still loads). `parse_line` measures raw line vs `line_bytes` 32768 first. SCHEMA register vs `max_fields=32`. `emit_record` writes SCHEMA columns only. Loopback CREATE → save → load into a fresh session → `pin_map` cue. Binary search skipped (4000 exact for `\\` and `|`). Proof: `/opt/cursor/artifacts/doc-gate-readiness-e18.log`.
+
+| Case | Wire shape | Save / load | Exact? |
+|------|------------|-------------|--------|
+| E18a 4000 `\\` | `CREATE (:USR {id: 'USR_a4kbs', key: 'e18', value: <blob utf8=4000 chars=4000>, recycle: ''})` | 0 / 0, no `@ERR` | **yes** (escaped 8000, snap line 8021) |
+| E18a 4000 `\|` | `CREATE (:USR {id: 'USR_b4kpp', key: 'e18', value: <blob utf8=4000 chars=4000>, recycle: ''})` | 0 / 0 | **yes** (escaped 8000, snap line 8021) |
+| E18a 4000 `"` | `CREATE (:USR {id: 'USR_c4kdq', …})` | 0 / 0 | **yes** (escaped 4000, snap line 4021) |
+| E18a 4000 `'` | `CREATE (:USR {id: 'USR_d4ksq', …})` | 0 / 0 | **yes** (escaped 4000, snap line 4021) |
+| E18a 4000 CJK | 1333 × U+6D4B (`测`, 3-byte UTF-8) + 1 ASCII X; `USR_e4kcj` | 0 / 0 | **yes** (1334 chars, utf8 4000, snap line 4021) |
+| E18a 4096 (a–e) | same shapes; CJK is 1365 × `测` + 1 X | 0 / 0 all five | **yes** (`\\`/`\|` snap line 8215; quotes/CJK 4119) |
+| E18b tab mid/end | `CREATE (:USR {id: 'USR_tabm', … value: 'ab\\tcd'})` / `'ab\\t'` | 0 / 0 | **yes** (byte-exact) |
+| E18b CR mid/end | `… value: 'ab\\rcd'` / `'ab\\r'` | save 0 / load 1 | **no** — `@ERR: FIELD_COUNT\|Expected 4 fields for USR got 3` (same as newline: `str.splitlines` splits on CR before `newline_in_value`) |
+| E18c SCHEMA 64/128 | `SCHEMA WIDE ; fields=id p000 …` (64 / 128 names) | open 1 | `@ERR: limit_exceeded\|fields 64/32` and `128/32` |
+| E18c SCHEMA 32 | `SCHEMA PRT ; fields=id p00 … p30` | save 0 / load 0 | yes |
+| E18c 64/128 extras on USR | CREATE 60 / 124 keys beyond 4-field SCHEMA | save 0 / load 0 | RAM extras yes; load drops them (`emit_record` SCHEMA columns only) |
+| E18c 8 × 4000 ASCII | `CREATE (:FAT {id: 'FAT_8x4000', p000: <4000 A>, … p007: <4000 A>})` | 0 / 0 | **yes** (snap line 32024 < 32768; all eight fields exact) |
+
+E17: **note.** GQL `WHERE n.value CONTAINS '…'` is **not** a product substring filter. `MATCH … WHERE … RETURN n` → `@ERR: product_gate|agent surface forbids RETURN …`. `MATCH … WHERE … SET` GraphGlot-parses (single or double quotes; GQL escapes `\\ \' \" \n \r \t`; CJK and `$` unescaped) but lowering drops WHERE at SET: `|Q|=1500` → `@ERR: cue_conflict|SET  Q =1500; SHALL NOT pick one root or absorb`; a unique MATCH still SET when CONTAINS would miss. Inline `MATCH (n WHERE n.value CONTAINS '…')` → `@ERR: parse_error|unsupported MATCH shape`. Bare WHERE without SET/RETURN → `@ERR: parse_error|unsupported MATCH continuation`. `STARTS WITH` / `ENDS WITH` / `=~` are the same ignored-WHERE SET path. Working substring: `query find --keyword` / `pin_map --keyword` (casefold across all fields, hard `--limit` / `--max-rows`). leftover `read list --where value=*测例*` works on a small graph; on this fulldoc it is `@ERR: response_too_large|response 4659616 bytes exceeds cap 4194304`. Substitute latency (n=200, `find --limit 50`, warm 10000-row session, this VM): common `测例` (1500 USR hits, 50 returned) p50 **67.282** / p95 **84.020** / max **94.672** ms; rare `Part 1500` (1 hit) p50 **51.183** / p95 **69.090** / max **85.424** ms.