[Fengzdadi]

Which issue or RFC does this PR close?

No issue is closed. The defect was found in a memory-core audit and is described below. Related: RFC 1809 (Atomic Memory) and RFC 1560 (recall sufficiency gate, unchanged here). It builds on #2449, which made related enumeration paged; see Relation to #2449 below.

Rationale for this change

Related-memory recall during Atomic Memory processing queries with the candidate's text. Its lexical channel (used in fts and hybrid related modes) admits a stored memory only when the distinct terms it shares with the candidate reach max(lexical_min_matched_terms, ceil(lexical_coverage × n)), where n is the candidate's distinct term count. The requirement grows with the candidate, not with the stored memory. A long candidate can need more shared terms than a short stored memory has, so that memory is never compared. The reverse direction works.

This was reproduced on 38fb3508f through the public runtime, on SQLite and on a seekDB server using the OceanBase index:

• The stored memory is The billing database is PostgreSQL 15..
• A later 25-term candidate is about upgrading the billing database from PostgreSQL 15 to 16.
• The candidate needs 7 shared terms, but the stored memory can supply only 4.
• Reconciliation never runs, and both memories stay active.

RFC 1809 accepts some misses at the threshold. This one is structural, because it depends only on the relative lengths of the two texts.

What changes are included in this PR?

Rule. Related enumeration stays complete. It admits a memory when the distinct candidate terms it contains either:

• meet the candidate's own lexical threshold (unchanged), or
• number at least lexical_min_matched_terms and cover lexical_coverage of the memory's own indexed words (its searchable text, including the kind).

The requirement is therefore min(candidate threshold, max(lexical_min_matched_terms, ceil(lexical_coverage × memory words))). It never exceeds what a short memory can supply. The one-match exception for one- or two-term queries stays a query rule and does not apply to the stored side. Earlier candidates in the same window use the same rule.

Why the stored memory's indexed words. An exact distinct non-stopword count would need a new projection column, a migration and a backfill on every backend. The word count is derived in SQL from the existing searchable_text. Because it includes stopwords, repeats and the kind, it is never lower than the distinct term count, so the rule never admits more than an exact count would.

Where.

• atomic_memory_related_requirement and atomic_memory_related_requirement_sql in persistence/atomic_memory_index.py own the rule.
  • The SQL form compares the integer shared-term count with words × coverage in double precision. For an integer, this is the same as comparing with the ceiling that Python computes.
  • SQLite, seekDB, OceanBase and the Python window check therefore admit the same rows. A test checks this on 36 memory shapes under five admission floors.
• The lexical rank query of related enumeration (_related_fts_rank_sql, added by perf(atomic-memory): bound related recall candidates in SQL to reduce peak memory and recall cost #2449 and used by fts and hybrid) applies the per-row requirement on both backends:
  • SQLite replaces the constant requirement with the per-row requirement. FTS5 already enumerates every memory sharing a term.
  • On OceanBase / seekDB, the whole-word (LOCATE) path applies the per-row requirement directly.
  • The native path keeps its minimum_should_match block. It adds a second native MATCH block, with minimum_should_match = lexical_min_matched_terms, only when that minimum is below the candidate's threshold. That block computes the word count for each row it matches, then whole-word coverage for rows whose requirement is below the candidate's threshold. For a long candidate, those are nearly all memories sharing that many terms; see Cost.
  • perf(atomic-memory): bound related recall candidates in SQL to reduce peak memory and recall cost #2449's skip, when no memory can meet the threshold, now checks the lowest requirement any memory can have. For a valid admission floor it skips exactly the same queries.
• fts_required_matches is extracted from fts_query_requirements in artifacts/search.py. Its behavior is unchanged.
• Prepare-time search, prepare_context admission (RFC 1560), recovery probes, Topic Memory and Experience are unchanged. Their threshold measures how much of a query a result answers. Related recall compares two memories, which is the only place where the stored side's length matters.

Docs.

• The operate/configuration and workflows/atomic-memory pages (en/zh) describe the rule and the cost of the native block for long candidates.
• The workflow page also drops an incorrect claim that related enumeration is prefiltered by Owner columns. Enumeration covers every active same-Scope memory that meets the threshold, and authorizes each hit.

CI. Three new OceanBase cases join test_atomic_memory_fts_coverage.py. To keep the per-case ceilings enforced by tests/test_oceanbase_ci.py, two budgets rise:

• The long-query step: 8 cases per shard, 40 → 45 min.
• The native FTS step: 8 cases, 30 → 45 min. Its job rises 65 → 80 min.

Release metadata. The persistence digest and rationale in release/upgrade-support.json are refreshed. There are no schema, migration resource or stored projection changes.

Relation to #2449. #2449 ranks the complete related set once, with a lightweight query, and then reads bodies page by page. It kept the admission threshold, so a long candidate still missed a short memory of the same fact. This PR changes only that threshold inside #2449's rank query. Paging, the frozen ranking and body batching are unchanged. A long candidate's ranking can hold more memories, so it can span more pages.

Are there any user-facing changes?

Yes. The lexical channel of related recall changes in both fts and hybrid related modes. That includes auto, which resolves to hybrid when an embedding profile is configured.

• A long candidate is now compared with short memories of the same fact, so it can update or supersede them instead of leaving a contradictory duplicate.
• With embeddings, the effect shows where the vector channel misses.
• Short candidates see almost the same related set as before.

Precision trade-off. Production related requests use the default floor: coverage 0.25 and a minimum of 2 matched terms. Under that floor, any memory of at most 8 indexed words (kind and stopwords included) is related to any long candidate that shares 2 content terms with it.

For example, the 25-term billing candidate above now also relates to The holiday traffic dashboard moved.. That memory indexes 6 words (fact the holiday traffic dashboard moved) and shares holiday and traffic with the candidate.

This is accepted because the old rule already related exactly this pair when the short text was the candidate and the long text was stored: a 4-term candidate needs 2 shared terms. Relatedness no longer depends on which memory arrived first, and the reconcile model still decides whether anything changes.

Cost. Related enumeration stays complete; no candidate limit is added. All numbers are measured on a laptop against master 195be9b77, which includes #2449. Each figure is the median time to drain one complete related enumeration: the rank query plus every page.

Real atomic facts. The LoCoMo observations form one Scope of 2,455 memories. Candidates are 40 held-out observations (short) and 40 concatenations of three held-out observations (long). Latencies are shown as two runs, a / b. The share with ≥ 1 related memory and the reconcile calls per candidate are derived from the related counts, with comparison batches of 20. No model was called.

	Before	After
Long: related memories, mean / max	0.21 / 3	8.57 / 24
Long: share with ≥ 1 related	14.3%	92.9%
Long: reconcile calls per candidate (derived)	0.14	1.00
Long: SQLite	13.0 / 11.9 ms	13.3 / 13.0 ms
Long: seekDB native	1.4 / 0.8 ms	7.7 / 8.2 ms
Long: seekDB whole-word	17.1 / 16.8 ms	23.5 / 22.4 ms
Short: related memories, mean	12.70	12.93
Short: share with ≥ 1 related	92.5%	95.0%
Short: reconcile calls per candidate (derived)	1.20	1.23
Short: SQLite	4.0 / 3.7 ms	3.9 / 4.0 ms
Short: seekDB native	1.6 / 1.2 ms	4.3 / 4.3 ms
Short: seekDB whole-word	8.7 / 8.6 ms	12.5 / 12.5 ms

Synthetic stress Scope. Memories have 6–30 words drawn from a Zipf-distributed 6,000-word vocabulary, with 35% function words. Candidates are 10 long (35–45 words) and 10 short (6–10 words). The vocabulary overlaps far more than real facts do, so related sets are much larger than on LoCoMo. Each cell is one run.

Scope, candidate	Related mean	Reconcile calls per candidate (derived)	SQLite	seekDB native	seekDB whole-word
10k, long	102.5 → 575.5	5.6 → 29.4	151 → 173 ms	13.1 → 86.9 ms	66.9 → 133 ms
10k, short	930.2 → 930.2	47.1 → 47.1	35.7 → 34.6 ms	13.5 → 10.8 ms	26.9 → 33.7 ms
50k, long	302.4 → 2,130.7	15.7 → 107.1	700 → 772 ms	59.4 → 370 ms	297 → 531 ms
50k, short	4,257.2 → 4,257.2	213.2 → 213.2	169 → 185 ms	61.7 → 73.9 ms	157 → 147 ms

On the native path, the second block makes long-candidate enumeration several times slower: about 8 ms instead of 1 ms on LoCoMo, and 370 ms instead of 59 ms at 50k synthetic memories. It stays below the whole-word path (531 ms at 50k), because the MATCH prefilter limits the word count and whole-word coverage to rows that share at least the minimum match count of terms. The whole-word path and SQLite were already doing per-row work for those rows; on them, long candidates take about 1.1× (SQLite) and 1.4–2× (whole-word) as long.

The larger cost is downstream. On the synthetic Scope, the long candidates' related set grows about 6–7×, and so do the derived reconcile calls. On LoCoMo the derived increase is under one call per long candidate.

How was this change tested?

Regression tests.

• tests/e2e/test_atomic_memory_fts_coverage.py:
  • A long candidate finds a short memory of the same fact, in fts and hybrid related modes. The reverse direction still works, unrelated and one-shared-term memories stay out, and search with the long query still admits nothing.
  • SQL and Python agree on the related threshold across 36 memory shapes and five admission floors.
• tests/e2e/test_atomic_memory_unrelated_reconciliation.py: the built-in processor reconciles a long candidate with the earlier short memory, in two Scopes.
• tests/e2e/test_atomic_memory_long_fts_query.py: the below-threshold related fixture adds 300 unshared terms. The old fixture shared 249 of its own 250 terms with the candidate, so it is now legitimately related.

The three behavior tests above (short memory found in fts and hybrid modes, long candidate reconciled) fail on master's src/ at 195be9b77 and pass with this change. They do so on SQLite and on a local seekDB server, through both native MATCH and LOCATE. The SQL/Python agreement test and the below-threshold fixture pass on both by design: they check that the backends agree with Python and that an unrelated memory stays out.

Commands. The test suites ran on the merge of master 195be9b77. A later comment-only edit to the OceanBase index changed the persistence digest; after it, make check, the release gate and the OceanBase CI budget tests were rerun on the final head.

• make check: passes.
• Release gates (tests/test_upgrade_release_gate.py, tests/test_upgrade_publication.py, tests/test_upgrade_schema_equivalence.py, tests/release_upgrades) and tests/test_oceanbase_ci.py: 185 passed and 6 skipped.
  • tests/release_upgrades/test_full_server.py needs the CI upgrade harness (PC_UPGRADE_SOURCE_VERSION). It errors locally the same way on master.
  • persistence_sha256 is recomputed with managed_digest from the final tree, and matches a recomputation from a git archive of the merge commit.
• tests/e2e/ -k atomic_memory plus tests/builtin/persistence/: 350 passed and 222 skipped.
• Unit tests matching atomic_memory, reconcil, related or recall_sufficiency: 551 passed and 7 skipped.
• The new and changed FTS test files together with perf(atomic-memory): bound related recall candidates in SQL to reduce peak memory and recall cost #2449's related pagination and embedding tests: 78 passed and 58 skipped.
• make unit-test was last run before the merge with perf(atomic-memory): bound related recall candidates in SQL to reduce peak memory and recall cost #2449: 8,899 passed and 13 failed. 12 of those also fail on master in this environment (4 in tests/test_ci_installation_roles.py, 8 in tests/evaluation/test_jev_rerank_comparison.py). The 13th was the OceanBase step budget, which the CI change above fixes.

Backends.

• SQLite: the full e2e and persistence suites.
• A local seekDB server ran, through both the native MATCH path and the whole-word (LOCATE) path:
  • every OceanBase-backed case of test_atomic_memory_fts_coverage.py;
  • the related case of test_atomic_memory_long_fts_query.py;
  • perf(atomic-memory): bound related recall candidates in SQL to reduce peak memory and recall cost #2449's pagination scenarios: page coverage, boundaries and tie breaks, hybrid fusion, vector readiness, body batches, and corpus changes in other Scopes.
  • All passed except test_related_page_bodies_are_fetched_in_page_sized_batches on the native path. It fails the same way with master's src/: the test's _RecordingConnection has no run_sync, which _supports_native_fts calls.
• A real OceanBase server was not run locally. A maintainer review ran the focused checks at dde597979, before the merge with perf(atomic-memory): bound related recall candidates in SQL to reduce peak memory and recall cost #2449, on embedded seekDB 1.4.0.post2 and real OceanBase CE 4.4.2.1. CI's OceanBase acceptance jobs run these e2e tests.
• Embedded seekDB (pylibseekdb) was not run locally; CI's seekDB job covers it.

Self-review. These points were reviewed and either addressed or accepted:

• The hybrid related path was untested at first; the regression test now runs both modes.
• The float binding for embedded seekDB is left to CI's seekDB run.
• The extra native MATCH block cost is measured above.
• The word count can overstate stopword-heavy memories. That only makes admission stricter than an exact distinct count.
• The native union mixes SCORE() values from two queries. That affects only the order of a complete enumeration.
• Two independent reviews of the diff found no correctness defect. They led to the cost, precision and validation notes in this description.
• The merge with perf(atomic-memory): bound related recall candidates in SQL to reduce peak memory and recall cost #2449 was reviewed again, in-session and by an independent review, and no defect was found. It led to a more precise comment on the native block.

AI usage statement

Claude Code (Claude Opus 5.5) reproduced the defect and wrote the change, tests, docs and this description. It also ran the benchmarks and checks, and reviewed the diff in two independent passes before this PR was opened and again after merging #2449.

[Copilot]

🔵 Needs a closer look

The backend-specific MATCH/UNION queries and floating-point bindings require OceanBase and embedded seekDB acceptance evidence that this review could not establish.

No open issues 🎉

What changed in this PR

This PR fixes Atomic Memory related recall so long candidates can find short stored memories using a threshold that also considers the stored memory’s indexed words. Ordinary search and prepare-time admission remain unchanged.

Changes:

• Apply the shared relatedness rule across SQLite, OceanBase/seekDB, and earlier candidates in the same processing window.
• Add regression coverage for lexical and hybrid recall, backend threshold consistency, and reconciliation.
• Update bilingual documentation, persistence review metadata, and OceanBase CI time budgets.

File	Description
tests/​e2e/​test_atomic_memory_unrelated_reconciliation.py	Tests long-candidate reconciliation with stored memories and window siblings.
tests/​e2e/​test_atomic_memory_long_fts_query.py	Adjusts the below-threshold fixture for the new rule.
tests/​e2e/​test_atomic_memory_fts_coverage.py	Tests short-memory recall and SQL/Python admission consistency.
src/​powercontext/​builtin/​runtime/​atomic_memory_processing.py	Applies the shared threshold to window siblings.
src/​powercontext/​builtin/​persistence/​sqlite/​atomic_memory_index.py	Uses per-memory thresholds for lexical related recall.
src/​powercontext/​builtin/​persistence/​oceanbase/​atomic_memory_index.py	Updates native and whole-word related enumeration.
src/​powercontext/​builtin/​persistence/​atomic_memory_index.py	Defines shared Python and SQL relatedness thresholds.
src/​powercontext/​builtin/​artifacts/​search.py	Extracts the existing query-threshold calculation.
release/​upgrade-support.json	Refreshes the persistence digest and compatibility rationale.
docs/​zh/​docs/​workflows/​atomic-memory.md	Documents relatedness and corrects owner-prefilter wording in Chinese.
docs/​zh/​docs/​operate/​configuration.md	Explains the rule and native-query cost in Chinese.
docs/​en/​docs/​workflows/​atomic-memory.md	Documents relatedness and corrects owner-prefilter wording in English.
docs/​en/​docs/​operate/​configuration.md	Explains the rule and native-query cost in English.
.github/​workflows/​e2e-harness.yml	Raises OceanBase time budgets for the added cases.

_{Review effort: Balanced}

[hidb4ai]

Reviewed dde5979; I found no blocking correctness defect in the changed recall rule. The focused checks passed on SQLite, embedded seekDB 1.4.0.post2, and real OceanBase CE 4.4.2.1, including native MATCH and the long-query fallback.

Merge-readiness note: merging this head with current master (2d47927) produces a content conflict in release/upgrade-support.json. Please resolve it and recompute the persistence digest from the resolved tree before merging. The current head's digest validates, but it does not cover the combined tree. The unit/e2e CI checks are also still pending.

[Fengzdadi]

@hidb4ai Thanks for the review, and for running it on seekDB and OceanBase. I merged current master into this branch (merge commit 1f9be31, no rebase). The only conflict was the persistence digest in release/upgrade-support.json, from #2597. I kept master's rationale, appended this PR's sentences unchanged, and recomputed persistence_sha256 from the resolved tree with managed_digest. Recomputing it from a git archive of the merge commit gives the same value. Apart from that file, the PR's diff against master is the same as at dde5979. make check, the upgrade release gate and the FTS e2e files pass on the merge commit.

[hireview001]

• CI failed: CI result (failure).
• CI failed: desktop-validation / linux (failure).
• CI failed: cindy-package (failure).
• CI failed: dsh-package (failure).
• CI failed: Main (failure). Failed steps: dsh-package / Test DSH package; cindy-package / Build and test the standalone Cindy archive; desktop-validation / linux / Run cargo clippy --locked --manifest-path src-tauri/Cargo.toml --all-targets -- -D warnings.

[Fengzdadi]

@hidb4ai #2449 merged after your review and conflicted with this branch, so I merged current master (195be9b) again, as 6bef39c, without a rebase.

#2449 changed how related memories are enumerated: one frozen lightweight ranking, then bodies page by page. It kept the admission threshold, so a long candidate still missed a short memory of the same fact. This PR now changes only the two _related_fts_rank_sql methods that #2449 added:

• SQLite compares the shared-term count with the per-memory requirement.
• On OceanBase, the whole-word fallback uses the per-memory requirement. The native path keeps perf(atomic-memory): bound related recall candidates in SQL to reduce peak memory and recall cost #2449's MATCH block and adds the second block for memories short enough to qualify below the candidate's threshold.
• perf(atomic-memory): bound related recall candidates in SQL to reduce peak memory and recall cost #2449's skip for an unreachable threshold now checks the lowest requirement. For any valid admission floor the two checks are equivalent.

Paging, the frozen ranking and body batching are unchanged.

The workflow docs keep #2449's paging description. They still drop the claim that related enumeration is prefiltered by Owner, because the related filter carries only tags and kind.

I recomputed persistence_sha256 from the merged tree and checked it against a git archive of the merge commit.

The three behavior tests fail on master's src/ and pass here, on SQLite and on a local seekDB server through both native MATCH and LOCATE. #2449's pagination scenarios also pass on that server through both paths, with one exception: test_related_page_bodies_are_fetched_in_page_sized_batches fails on the native path. It fails the same way on master, because the test's _RecordingConnection has no run_sync, which _supports_native_fts calls. I didn't run a real OceanBase server locally.

[hireview001]

• Resolve merge conflicts with the target branch.