Repository navigation
feat(sources): preserve citation position locators - #2296
Conversation
hireview001
left a comment
There was a problem hiding this comment.
- CI failed: Upgrade release gate (failure).
- CI failed: upgrade-acceptance / Verify the selected upgrade matrix (failure).
- CI failed: Recall effort (OceanBase) (failure).
- CI failed: macos (macos-15) (failure).
- CI failed: windows (failure).
- CI failed: macos (macos-15-intel) (failure).
- CI failed: linux (failure).
- CI failed: tests (3.13, unit) (failure).
- CI failed: tests (3.11, unit) (failure).
- CI failed: tests (3.12, unit) (failure).
- Additional CI issues: PR checks.
hidb4ai
left a comment
There was a problem hiding this comment.
The populated legacy-database upgrade is blocked before the locator-table migration can run. The existing unresolved Desktop and OpenDAL findings also remain applicable. Required quality, unit, and upgrade checks need to pass before merge.
hidb4ai
left a comment
There was a problem hiding this comment.
@Tech-Circle-48 Re-reviewed 2d9eb62d. All four findings remain reproducible; I have added current evidence in their existing threads, including the OpenDAL finding currently marked resolved. Please fix the populated-database upgrade, locator decoding, versioned worker registration, and restore coverage, then resolve the merge conflicts and rerun the failing quality/unit/upgrade checks. Approval remains blocked.
hidb4ai
left a comment
There was a problem hiding this comment.
The remaining blocker is the conflicting registered migration revision with current master, confirmed again in the existing thread. Please rebase, give the locator migration its own successor revision, and validate the combined upgrade chain before merge. The prior four runtime/compatibility findings were verified fixed; the latest changes only update migration documentation and revert the unrelated DMG retry.
hireview001
left a comment
There was a problem hiding this comment.
- CI failed: windows (failure).
- CI failed: macos (macos-15) (failure).
- CI failed: macos (macos-15-intel) (failure).
- CI failed: linux (failure).
- CI failed: Desktop validation (failure). Failed steps: linux / Run cargo test --locked --manifest-path src-tauri/Cargo.toml; macos (macos-15-intel) / Run cargo test --locked --manifest-path src-tauri/Cargo.toml; macos (macos-15) / Run cargo test --locked --manifest-path src-tauri/Cargo.toml.
hidb4ai
left a comment
There was a problem hiding this comment.
@Tech-Circle-48 Re-reviewed ba222e35. The migration revision now conflicts with the subsequently merged Source-plugin history on current master, and the regenerated Desktop contract still has no matching qualified profile. I added exact-head evidence to the existing threads. Please rebase onto the current migration chain and qualify the resulting Desktop contract before merging.
hireview001
left a comment
There was a problem hiding this comment.
- Resolve merge conflicts with the target branch.
- CI failed: windows (failure).
- CI failed: macos (macos-15) (failure).
- CI failed: macos (macos-15-intel) (failure).
- CI failed: linux (failure).
- CI failed: Desktop validation (failure). Failed steps: linux / Run cargo test --locked --manifest-path src-tauri/Cargo.toml; macos (macos-15-intel) / Run cargo test --locked --manifest-path src-tauri/Cargo.toml; macos (macos-15) / Run cargo test --locked --manifest-path src-tauri/Cargo.toml.
knqiufan
left a comment
There was a problem hiding this comment.
The two correctness issues below remain on this head. The new migration also makes six existing recovery tests fail because they still expect the previous bundle head; the failure was reproduced locally and in CI. Please update and rerun the recovery checks.
knqiufan
left a comment
There was a problem hiding this comment.
Re-reviewed the current head and found no remaining blocking correctness issue. The previously reported sequence-locator and legacy OpenDAL import cases are addressed. Local SQLite and migration checks passed; the current CI run and native seekDB/OceanBase acceptance remain outstanding.
AlexStocks
left a comment
There was a problem hiding this comment.
本轮核验小结(exact head fe761522)
本轮新提出 3 条,其中 dev-fresh.db 一条是二进制文件、无法行内定位,放在这里说明。
[P1]src/powercontext/sources/models.py:78—Field(exclude_if=...)需要 pydantic >= 2.12,而pyproject.toml仍声明pydantic>=2.10,<3;在 2.11.9 上locator会序列化成"locator": null,本 PR 自带的tests/test_source_locators.py有 2 条用例在该版本下失败。见 #discussion_r4501494433。[P2]src/powercontext/builtin/runtime/atomic_memory_rebuild.py:286—for_update=True不再对pc_artifact_heads发FOR UPDATE。见 #discussion_r4501495097。[P2]根目录新增dev-fresh.db(见下)。
[P2] 根目录新增的 dev-fresh.db 未被任何代码引用,也未被 .gitignore 覆盖
本 PR 在仓库根目录新增二进制文件 dev-fresh.db(192512 字节,27 张表)。在 exact head fe761522 全树检索 dev-fresh / dev_fresh 没有任何命中:测试、脚本、文档、配置里都没有加载它的代码;.gitignore 只覆盖 db.sqlite3 与 db.sqlite3-journal,与该文件名无关。
内容侧已确认不涉及数据泄漏:业务表全部 0 行,只有 pc_artifact_processing_schema 保留 1 条配置行,pc_atomic_memory_evidence_schema = p0005、vec_info = v0.1.9、fts_config version 4,特征是本地开发实例的残留快照。
判断:该文件与本次 locator 契约、迁移修订都无关,一旦进入历史就无法回收(后续删除也不会从对象库移除)。建议从本 PR 中删掉,并在 .gitignore 增加本地开发库规则(例如 dev-*.db),避免再次误提交。
本轮已核验通过的部分
- 迁移链:
v1_5_0_004的down_revision为v1_5_0_003,与 master 当前头部一致,不存在编号冲突;server_resources/manifest.json与server_sources/1.4.0/manifest.json两份副本均已登记revision_resources/sqlite_fingerprints/affected_objects/domain_readiness/mysql_state_layouts。 - 生成物与契约摘要:
scripts/generate_api.py --check与dify/generate_contract.py --check均退出 0;openapi 摘要18af205a886552b6c2d9f41fb72a52cbf4453ddc70a98a47103212b4c16f2a6f在transport/operations.json、ui/src/generated/operations.ts、connections/compatibility.json三处一致。 - 行为核验(本地按 exact head 运行):
tests/e2e/test_source_citation_locators.py24 passed,其中覆盖SourceSequenceLocator.sequence与引用 Source 的journal_position不一致返回 422、span 超过 200 返回 422;另用探针跑rebuild_atomic_memory_evidence后verify_atomic_memory_current,evidence_projection_differences: 0。
覆盖缺口
- CI 尚未跑完:
tests (3.11/3.12/3.13/3.14 × unit/e2e)共 8 个 job,当前 6 个 success、6 个 pending;Desktop validation经由 Main 编排的workflow_call调用,Main 仍处于 queued,因此desktop/src-tauri/tests/api.rs中新增的两条 Rust 用例在本轮没有可用的 CI 结果。
hidb4ai
left a comment
There was a problem hiding this comment.
Re-reviewed this head. The Pydantic compatibility fix is effective, but required CI preflight still fails on stale nested lockfiles; see the inline finding.
hidb4ai
left a comment
There was a problem hiding this comment.
The current CI failure is reproducible and specific to this branch; see the evaluation evidence finding below.
hidb4ai
left a comment
There was a problem hiding this comment.
The earlier evaluation-report failure is fixed. The remaining CI blocker is a reconciliation test expectation mismatch introduced by the locator behavior.
| source_id=stored.ref.source_id, | ||
| # Journal positions belong to the persisted Source, not the adapter. | ||
| locator=( | ||
| SourceSequenceLocator(sequence=stored.journal_position) |
There was a problem hiding this comment.
[P2] Update reconciliation acceptance expectations for located references
Deriving a journal locator here changes the persisted lineage, but the tests merged from master still compare it with the unlocated capture receipt in tests/e2e/test_atomic_memory_reconciliation_retry.py:346,369,393. This produces all seven failures in the current Python 3.11 E2E job. The unchanged test file passes on base ed8902b2; head aa77ac80 reproduces the seven failures. In a diagnostic copy, adding SourceSequenceLocator(sequence=receipt.sequence) (or later.sequence) to only those three expected references makes the complete file pass, including restart and history checks, without changing runtime code. Update the assertions to verify the exact journal position while retaining the recovery checks. The required upgrade acceptance is currently skipped because this E2E job fails.
There was a problem hiding this comment.
[P2] 复核 head aa77ac80:该根因仍在,影响面比线程正文所述更大
tests (3.11, e2e)(job 128428414052,run 42380498052,event pull_request,head aa77ac80)汇总为 7 failed, 1473 passed, 569 skipped,7 条失败全部落在 tests/e2e/test_atomic_memory_reconciliation_retry.py。该文件在本 head 与 master 逐字节相同(blob 6d1717a0),且不在本 PR 文件集内。
失败之后的连带范围(同一 head 的 check-runs):
- matrix fail-fast 把
tests (3.12, e2e)、tests (3.13, unit)、tests (3.13, e2e)、tests (3.14, unit)、tests (3.14, e2e)变为cancelled; - 随后 30 个 suite job 全部
skipped,其中含desktop-validation、harness-validation、code-seekdb、windows-unit-portability、upgrade-acceptance; Upgrade release gate(job128436321006)的红与本条同源:env 为UPGRADE_SELECTED=true、UPGRADE_RESULT=skipped,test "$UPGRADE_RESULT" = success以退出码 1 结束。本地在 head 上执行python scripts/upgrade_release_gate.py退出 0 并输出Upgrade policy: upgrade-required; source/backend combinations are resolved against the target wheel,说明release/upgrade-support.json的persistence_sha256与 head 的 persistence 源码一致,这一项不是独立缺陷。
断言校准的行内依据:src/powercontext/builtin/runtime/application.py:688-691 的单条 capture 回执已把位置放在 SourceReceipt.sequence,SourceRef 本身不带 locator;因此 tests/e2e/test_atomic_memory_reconciliation_retry.py:346、:369、:393 三处期望值按 SourceSequenceLocator(sequence=<receipt.sequence|later.sequence>) 校准即可,不需要改运行时。
|
@Tech-Circle-48 as you see, there still are some review comments left to be replied and the github ci process still failed. |
hidb4ai
left a comment
There was a problem hiding this comment.
Post-merge follow-up: the earlier reconciliation expectation issue was fixed in #2432; all 11 tests now pass on master 63f021d6. A separate OpenDAL type-check blocker introduced by this PR is still reproducible; see the inline finding.
|
|
||
| TEXT_FILE_SNAPSHOT_SOURCE_NAME = "text-file-snapshot" | ||
| # Older runtimes expose the same frozen schema under TextEvidence alone. | ||
| _TEXT_EVIDENCE_MODEL: type[BaseModel] = getattr(source_models, "TextEvidenceV1", source_models.TextEvidence) |
There was a problem hiding this comment.
[P2] Preserve the concrete text evidence model type
This annotation erases the model's declared fields. The unpinned ty invoked by make opendal-test now reports pydantic-discarded-extra-argument at the constructor on line 160 and exits 1, blocking opendal-package on current master (CI job). With the same ty 0.0.100 and dependencies, base ed8902b2 passes, while this PR's final head aa77ac80 and master 63f021d6 reproduce the diagnostic. The locked ty 0.0.56 passes, so this is a checker-version interaction exposed by the standalone target, not evidence of runtime field loss; the runtime model still preserves all four fields.
Keep the legacy export fallback, but retain the concrete TextEvidence constructor type rather than widening it to type[BaseModel]. In a diagnostic copy, changing only this annotation to type[source_models.TextEvidence] clears the check without changing the v1 schema. The original PR run skipped this downstream job after its E2E failure.
There was a problem hiding this comment.
Verified fixed downstream in #2503 (7338a89f). The projection now uses _TEXT_EVIDENCE_MODEL.model_validate(...), retaining the legacy export fallback and historical v1 manifest/schema. On current master d64381bf, the previously failing ty 0.0.100 check passes, the projected output is unchanged, and the OpenDAL CI job succeeds. This finding no longer applies to current master; #2296 itself remains at its original merged head.
Which issue or RFC does this PR close?
Closes #1869
Rationale for this change
What changes are included in this PR?
Are there any user-facing changes?
How was this change tested?
AI usage statement