feat(memory): persist versioned Source evidence and neutral Atomic Memory projection by MaoMengww · Pull Request #2004 · oceanbase/powercontext

MaoMengww · 2026-10-14T17:47:54Z

Which issue or RFC does this PR close?

Closes #1975.

Implements the Source authority and verification prerequisite in RFC #1652, while preserving the Source access and lineage_only semantics in RFC #1458.

Rationale for this change

Source generation eligibility does not describe the authority or verification of its material. Atomic Memory needs explicit, versioned evidence declarations that remain inspectable after registration changes, restarts, and projection rebuilds.

This change records those facts without introducing trust scores, ranking bonuses, or additional write authority.

What changes are included in this PR?

Add MemoryEvidenceDeclaration and exact Source material bindings through the existing Source Definition registry.
Accept declarations only from trusted native adapters or deployment-owned configuration for exact remote Definition manifest identities.
Retain historical Definition manifests, Source declaration bindings, and evidence snapshots keyed to exact Atomic Memory revisions.
Add an authorized Python evidence-read API and reuse the existing projection rebuild path to verify retained historical material.
Resolve missing, invalid, or unverifiable declaration records to neutral untrusted / unknown values. Missing or corrupt authoritative Source material and database failures remain errors.
Add the additive p0005 evidence schema migration for SQLite and MySQL-compatible backends, without backfilling historical trust.
Add compact SQLite/OceanBase acceptance scenarios covering revision history, write rollback, registration drift, cross-Scope isolation, missing-snapshot neutrality, synthetic model trust claims, and unchanged recall consumers.

Are there any user-facing changes?

This adds optional Source Definition evidence declarations and a Python API for inspecting an exact Atomic Memory revision's evidence.

It does not change Atomic Memory bodies or content hashes, Artifact revision identities, HTTP/OpenAPI search schemas, public search scores, or citation identities. It introduces no quality ranking or new generation eligibility rules.

Persistence gains additive evidence tables. Existing Sources and Memory revisions without accepted declarations remain neutral; current configuration does not retroactively attest them.

How was this change tested?

make check: passed, including lock consistency, pre-commit checks, generated guidance checks, and type checking.
make contract-test: 52 passed.
make opendal-test: 8 passed; connector lint, formatting, and type checks passed.
Full suite with doctests and the required DSH configuration runtime, using Python 3.12.13 / SQLite 3.50.4:
- 5294 passed, 385 skipped, 0 failures.
- Command: uv run --frozen --python 3.12 python -m pytest -n 6 --doctest-modules --require-dsh-config-runtime
Focused Source, persistence, migration, lifecycle, and recall regressions: 128 passed.
Real OceanBase CE 4.4.2.1 evidence acceptance: 3 passed, 0 skipped.
Wheel and source distribution builds passed; the wheel contains the new evidence migration resources.

The initial full run under the local Python 3.11 / SQLite 3.37.2 environment exposed an existing Topic Memory vector-prefilter compatibility failure. A standalone SQLite reproducer isolated the cause: sqlite-vec's rowid IN KNN prefilter requires SQLite 3.38 or newer. The original test and full suite passed with the newer SQLite runtime, without changing code, assertions, or skip conditions.

The complete GitHub Actions Python-version matrix has not been run locally.

AI usage statement

AI assistance was used through Oh My Pi with gpt-6.1sol for implementation, test design and consolidation, code inspection, and validation analysis.

Implement issue01 of the Source evidence declarations specification. Bind trusted native and remote declarations to exact Source material and retain per-revision Atomic Memory snapshots in the existing transaction. Preserve neutral fallback, exact identities, scope authorization and public retrieval contracts. Add versioned p0004 migrations and SQLite persistence regressions. Refs oceanbase#1975. Full suite: 5255 passed, 431 skipped, 7 failed; introduced observability adapter failures fixed and its file passes 18 tests. Topic Memory vector failure also reproduces at the unchanged baseline.

Cover persisted declarations, scope isolation, historical neutrality, synthetic model trust claims, and unchanged recall consumers on SQLite and OceanBase.

Preserve the published p0004 artifact handle revision and introduce evidence as p0005. Retain both schema intents, recompute exact SQLite fingerprints, and update MySQL migration head expectations.

hidb4ai

Please address the two native Source evidence issues below before merging. Both reproduce through public SQLite flows and break evidence inspection and projection rebuild after successful Memory generation.

hidb4ai

Native Source evidence inspection and projection rebuild pass the input-alias and model-default scenarios. Remote observation ingestion still fails for serialization aliases; please address the inline P2 before merging.

hidb4ai

The three reported reproduction scenarios now pass. One remote Source regression remains when aliases apply to inherited envelope fields; please address the inline P2 before merging.

Retain native Source identity independently of adapter aliases and mutable description defaults, preserve legacy read/replay behavior, and keep serialized business fields intact. Capture evidence at the shared Atomic Memory body-commit boundary, including restoration revisions, without retroactively attesting state-only writes. Refs oceanbase#1975.

Check native serialization compatibility at acceptance without blocking valid captures or Memory writes. Keep previously accepted incompatible declarations neutral while preserving exact material and digest checks. Reject legacy identities that cannot be decoded exactly instead of supplying a default Definition version.

Preserve Source evidence verification and upstream revision metadata initialization in Atomic Memory projection rebuild.

Reject native serialized material that omits an identity field before journal allocation or persistence, rather than accepting Sources that later fail evidence capture. Preserve strict retained identity checks.

Merge 90355b8 while retaining the Source evidence contract. Append evidence revisions after v1_4_1_005, preserve existing upstream revision resources and checksums, and derive the combined schema fingerprints for both source chains and vector profiles. Unify the automatically merged Atomic Memory service so description handling and transactional evidence capture share one implementation. Preserve both upgrade review narratives and bind the merged persistence inputs.

…idence startup A Source field whose default is null can reach the payload as null, so a renamed nested shape below it never reaches the material. Traversing such a field for schema coverage replaced the manifest a deployment already accepted: its fingerprint changed and re-registration failed with SourceConflictError even though the payload still carried null. Only required fields, and models that forbid extras, now decide between the validation and serialization schemas. Populating a null-defaulted field with a renamed key requires the nested model to serialize by alias, which keeps the validation keys authoritative and the fingerprint stable. Cover the upgrade path: the nullable nested alias keeps the accepted manifest, its null payload registers across two sessions, a populated field-name payload is rejected, and a nested model serializing by alias carries populated values under that same manifest. Document the rule in both locales. The MySQL-family table fast path returned before the registered additive evidence migration, so an unmanaged OceanBase or seekDB database created the evidence tables without their p0005 marker and every runtime open failed readiness with "Atomic Memory evidence schema version is unsupported". Run the migration before the dialect branch and leave evidence tables to their registered DDL, as the SQLite path already does. The upgrade digest is recomputed over the final persistence inputs.

…gins Upstream master adds the Source plugin framework with Server revision v1_4_1_006 (pc_source_write_plugins), which conflicts with the evidence revisions appended by this branch. - Keep both chains on one linear history: v1_4_1_006 now precedes v1_5_0_001/v1_5_0_002, and v1_5_0_001 rebases onto v1_4_1_006. - Register both revision sets in the server and 1.4.0 source manifests, including domain steps, readiness, affected objects and MySQL layouts. - Recompute the SQLite fingerprints for v1_5_0_001/v1_5_0_002 because their layouts now include pc_source_write_plugins; the earlier fingerprints remain byte-identical. - Recompose the reviewed persistence digest and keep both rationale paragraphs in release/upgrade-support.json. - Update revision-chain expectations in tests and docs; prune helpers now take an explicit previous-head revision.

Upstream master adds Artifact plugin storage as Server revision v1_5_0_001 (registration columns, family declarations, search state and text projections, index coverage and changes, vector-space and vector-index registrations) on top of v1_4_1_006, which conflicts with the evidence revisions this branch had appended at v1_5_0_001/v1_5_0_002. Keep one linear history and treat the upstream registration as authoritative: - Artifact storage keeps v1_5_0_001; retained Source evidence and its schema marker move to v1_5_0_002, the disposable exact-revision evidence projection to v1_5_0_003. - Rename the matching schema resources, SQLite/MySQL version wrappers and MySQL layout fragments, and rewire down_revision so the chain is v1_4_1_006 -> v1_5_0_001 -> v1_5_0_002 -> v1_5_0_003. - Compose both revision sets in the server and 1.4.0 source manifests: revision resources, domain steps, readiness, affected objects, MySQL state layouts and wrappers. - Recompute the SQLite fingerprints for the new revisions from a real full-chain migration; earlier fingerprints stay byte-identical. - Head readiness follows the branch convention: base steps plus the step introduced by that revision, so v1_5_0_003 validates artifact_storage, portable_storage and the evidence projection without re-running evidence validation during concurrent startup. - Recompose the reviewed persistence digest and keep both rationale paragraphs in release/upgrade-support.json. - Merge the schema helpers so unmanaged MySQL-family creation still applies the registered evidence migration before other tables while keeping upstream's artifact search and vector-space provisioning, and keep both sides' source/test additions. - Update revision-chain expectations, source batch tests and docs in both locales for the three-revision 1.5.0 tail. Validated with make test (8440 passed, 799 skipped) and make check.

Upstream master (0a98473) consolidates CI selection and reduces pull request runner demand: it rewrites master.yml and the per-suite workflows, adds .github/ci-paths.toml scopes plus ci_changes/ci_installation_roles/ ci_queue_report scripts with their tests, and records a new paragraph in the reviewed persistence rationale. - release/upgrade-support.json: keep this branch's evidence revisions and the upstream code-publication-cache paragraph in one rationale. The declared persistence_sha256 stays e7c44d89 because upstream changed no managed persistence input; it was recomputed from the merged tree. - test_mysql_vector_admission.py: keep upstream's documentation that the fixture always has a database, so missing-database admission is inert. - e2e-harness.yml: the rewritten workflow keeps this branch's 900s budget and thirteen-case assertion for the native OceanBase recovery group. Checks: make check (conflict/toml/yaml/json, ruff, ty, integration manifest) and pytest on the gate, CI-contract, admission and code publication suites.

AlexStocks · 2026-11-01T05:02:45Z

@MaoMengww please handle the file confliction.

Merge oceanbase/powercontext master into the source evidence branch. - Take the replay-based SQLite layout builder; it covers evidence revisions. - Keep both evidence and dependency-snapshot reads. - Preserve upstream upgrade recovery observation and branch migration head. - Keep 1.5.0 policy target and refresh the compatibility review digest. - Skip evidence readiness only inside frozen historical conversion; the evidence revision still rebuilds and verifies the projection.

- Narrow the prepared context union before reading status/content so the repository type check accepts the reuse envelope. - Defer evidence DDL during the historical native conversion instead of creating it through the unmanaged bootstrap, which stamped p0005 early and made the registered marker INSERT fail as a duplicate. - Refresh the release review digest.

hidb4ai

LGTM

Merge oceanbase/powercontext master at 63e9665. Retain the Source evidence compatibility declarations and append upstream ownership recovery and bounded preset-key declarations. Recompute the persistence review digest for the combined tree.

Resolve conflicts with ownership migration when enabling enforcement (oceanbase#2304), Source evidence declarations (oceanbase#2004), and the 13 other commits that landed since the previous merge. - `server/authz/keys.py` is new upstream; it lands in `builtin/authz/keys.py` and `builtin/authz/__init__.py` exports `bounded_idempotency_key` and `IDEMPOTENCY_KEY_LIMIT` from there. - `server/processing_security.py` keeps only `build_worker_security`. The upstream change to `WorkerSecurity` (bounded idempotency keys) is applied to `builtin/authz/worker.py`, where this branch moved that class. - `server/app.py`, `server/factory.py`, `server/cli.py`: keep the relocated `builtin.authz` imports and add only the names the new upstream code uses (`bounded_idempotency_key`, `OwnerRecovery`). `_scheduled_access_runners` stays deleted; its only upstream change was the bounded key. - `release/upgrade-support.json`: keep the rationale added upstream for the v1_5_0_002 and v1_5_0_003 revisions, append the sentence about this branch's code-organization changes, and recompute the persistence digest. The generated table and index DDL for all 75 tables is identical to master on the SQLite and MySQL dialects. Follow the moves into code that arrived with the merge: - `server/ownership.py`, `builtin/authz/service.py` (new `static_scope_bootstrap_would_grant`), and the new ownership-migration, artifact-plugin, and access-control tests import access control from `powercontext.builtin.authz`. - The new ownership-migration, Atomic Memory evidence, and Source evidence tests stop passing the removed `scheduler_path` argument. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

PR #2004 exhausted the 25-minute step after passing individual native cases. Spread all 39 nodes over three shards for each supported CI Python version; retain the preliminary checks and baseline stability repetitions. Constraint: Keep the 180-second node, 25-minute step and 60-minute job budgets. Scope-risk: CI scheduling only; no test selection or runtime behavior changes. Tested: pytest tests/test_ci_changes.py tests/test_ci_pipeline.py -q (125 passed); exact 39-node partition; full Python 3.11 shard 0 replay including prelude and baseline repetitions passed in 1041.98 seconds.

* ci(seekdb): distribute native migrations across three shards PR #2004 exhausted the 25-minute step after passing individual native cases. Spread all 39 nodes over three shards for each supported CI Python version; retain the preliminary checks and baseline stability repetitions. Constraint: Keep the 180-second node, 25-minute step and 60-minute job budgets. Scope-risk: CI scheduling only; no test selection or runtime behavior changes. Tested: pytest tests/test_ci_changes.py tests/test_ci_pipeline.py -q (125 passed); exact 39-node partition; full Python 3.11 shard 0 replay including prelude and baseline repetitions passed in 1041.98 seconds. * fix(sqlite): drain failed connection workers Install the aiosqlite 0.22.1 stop waiter before the initial connection, which may fail before SQLAlchemy emits its connect event. Await the worker's final callback before exposing failed opens or cancellation to callers that may immediately close their event loop. Reuse that waiter after pool admission and report thread-start failure without waiting. Constraint: Keep the workaround restricted to aiosqlite 0.22.1 and preserve cancellation. Scope-risk: Connection lifecycle only; no stored schema, migration or release-policy changes. Refresh the managed-input compatibility digest. Tested: Python 3.12 focused regressions 396 passed with thread warnings as errors; Python 3.11 and 3.14 profile/CLI suites 65 passed each; failed-connect regressions red on original code; thread-start hang reproduced before guard; full Linux ty and changed-file hooks passed.

MaoMengww added 5 commits October 15, 2026 01:02

feat(memory): verify historical evidence during projection rebuild

5dfa2ad

fix(memory): enforce evidence schema safety and backend conformance

034e0cc

fix(memory): preserve unrelated database inspection

4d1bf17

test(memory): consolidate source evidence E2E acceptance

cf8068a

Cover persisted declarations, scope isolation, historical neutrality, synthetic model trust claims, and unchanged recall consumers on SQLite and OceanBase.

MaoMengww requested review from AlexStocks, PsiACE and frostming as code owners October 14, 2026 17:47

merge: integrate upstream artifact handles with evidence migration

dd1d128

Preserve the published p0004 artifact handle revision and introduce evidence as p0005. Retain both schema intents, recompute exact SQLite fingerprints, and update MySQL migration head expectations.

hidb4ai requested changes Oct 15, 2026

View reviewed changes

Comment thread src/powercontext/builtin/persistence/sources.py Resolved

Comment thread src/powercontext/builtin/persistence/sources.py Resolved

fix(memory): verify evidence against retained source material

0d70cba

hidb4ai requested changes Oct 15, 2026

View reviewed changes

Comment thread src/powercontext/sources/observations.py Resolved

fix(sources): align remote payload aliases with manifests

3305b6b

MaoMengww requested a review from hidb4ai October 16, 2026 08:20

hidb4ai requested changes Oct 16, 2026

View reviewed changes

Comment thread src/powercontext/sources/observations.py Resolved

fix(sources): preserve envelope keys with alias generators

c9ee82d

MaoMengww requested a review from hidb4ai October 16, 2026 10:00

hireview001 reviewed Oct 17, 2026

View reviewed changes

Comment thread src/powercontext/sources/models.py Resolved

fix(sources): align envelope validation with serialization

dc9b6a1

AlexStocks reviewed Oct 17, 2026

View reviewed changes

Comment thread ...wercontext/builtin/persistence/migrations/resources/versions/p0005_atomic_memory_evidence.py Resolved

AlexStocks mentioned this pull request Oct 17, 2026

feat(dashboard): browse, compare, and roll back artifact revisions #1813

Merged

AlexStocks added the 1.4.1 v1.4.1 release tracking label Oct 17, 2026

hireview001 reviewed Oct 18, 2026

View reviewed changes

Comment thread src/powercontext/sources/models.py Resolved

Comment thread src/powercontext/builtin/artifacts/atomic_memory/service.py Resolved

Comment thread src/powercontext/builtin/persistence/sources.py Resolved

hireview001 reviewed Oct 19, 2026

View reviewed changes

Comment thread src/powercontext/builtin/persistence/migrations/atomic_memory_evidence.py Resolved

Comment thread src/powercontext/builtin/persistence/sources.py Resolved

Comment thread src/powercontext/builtin/persistence/sources.py Resolved

MaoMengww added 3 commits October 19, 2026 12:05

fix(sqlite): preserve startup catalog compatibility

6a836b1

Merge upstream master into source evidence declarations

40d6c80

Preserve Source evidence verification and upstream revision metadata initialization in Atomic Memory projection rebuild.

hireview001 reviewed Oct 19, 2026

View reviewed changes

Comment thread src/powercontext/builtin/persistence/sources.py Resolved

fix(sources): reject omitted identity before capture

2bce54e

Reject native serialized material that omits an identity field before journal allocation or persistence, rather than accepting Sources that later fail evidence capture. Preserve strict retained identity checks.

MaoMengww added 2 commits October 30, 2026 17:14

MaoMengww force-pushed the feature/1975-source-evidence-declarations branch from 7ba77de to 8444f08 Compare October 30, 2026 10:19

MaoMengww added 2 commits October 31, 2026 13:34

MaoMengww force-pushed the feature/1975-source-evidence-declarations branch from ef168be to 9e1a47b Compare October 31, 2026 10:24

MaoMengww requested a review from hidb4ai October 31, 2026 11:37

hidb4ai removed the 1.5.0 v1.5.0 release tracking label Nov 2, 2026

This was referenced Nov 3, 2026

evaluation(context): consume PreparedContext Receipts in opt-in LongMemEval diagnostics #2436

Open

feat(capture): declare bounded projection coverage for Hermes and Gateway Sources #2437

Closed

AlexStocks added the 1.5.1 v1.5.1 release tracking label Nov 3, 2026

AlexStocks mentioned this pull request Nov 3, 2026

Tracking: stage recall reliability, provenance, diversity, and receipt-linked feedback #1978

Closed

MaoMengww added 2 commits November 3, 2026 22:51

MaoMengww force-pushed the feature/1975-source-evidence-declarations branch from 5f66257 to c9cede7 Compare November 4, 2026 01:30

hidb4ai approved these changes Nov 4, 2026

View reviewed changes

AlexStocks merged commit 1190dcb into oceanbase:master Nov 4, 2026
64 of 65 checks passed

jackie-cqz mentioned this pull request Nov 4, 2026

refactor!: restore package layering in src #2244

Merged

dobet mentioned this pull request Nov 4, 2026

[review] Correctness issues in recent master changes: Dream route 500, full scans in Source management, credential lock failures under concurrency, ownership migration conflicts #2456

Open

This was referenced Nov 4, 2026

ci(seekdb): distribute native migrations across three shards #2458

Merged

fix(sqlite): drain failed connection workers #2468

Merged

Navigation Menu

Repository navigation

feat(memory): persist versioned Source evidence and neutral Atomic Memory projection - #2004

feat(memory): persist versioned Source evidence and neutral Atomic Memory projection#2004
AlexStocks merged 42 commits into
oceanbase:masterfrom
MaoMengww:feature/1975-source-evidence-declarations

MaoMengww commented Oct 14, 2026 •

edited

Loading

Uh oh!

hidb4ai left a comment

Uh oh!

Uh oh!

Uh oh!

hidb4ai left a comment

Uh oh!

Uh oh!

hidb4ai left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexStocks commented Nov 1, 2026

Uh oh!

hidb4ai left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

MaoMengww commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue or RFC does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

How was this change tested?

AI usage statement

Uh oh!

hidb4ai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hidb4ai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hidb4ai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexStocks commented Nov 1, 2026

Uh oh!

hidb4ai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MaoMengww commented Oct 14, 2026 •

edited

Loading