Repository navigation
Tracking: stage recall reliability, provenance, diversity, and receipt-linked feedback #1978
Description
Purpose
Track the staged path from deterministic recall reliability to provenance-aware retrieval evaluation and, only after an explainable delivery contract exists, a possible host-confirmed aggregate usage observation.
This is a sequencing and acceptance tracker. It does not authorize a new public API, a durable event ledger, query rewriting, read-path mutation, ranking changes, automatic retention, or automatic deactivation by itself.
Work items and gates
| Stage | Issue / prerequisite | Required outcome before the next gated stage |
|---|---|---|
| 1. Deterministic recall baseline | #1912 and implementation task #1976 | The documented equivalent-query forms have stable FTS behavior through both Memory search and prepare_context; backend/capability coverage and negative cases are recorded. |
| 2. Reconstructible source declaration | #1975 | Each Memory entry version can carry a rebuildable, exact Source materialization/declaration snapshot. Missing history is neutral untrusted/unknown; no ranking or lifecycle behavior changes. |
| 3. Candidate-density evidence | #1977, with #1942 reported separately when applicable | A reproducible Scope-local report shows baseline lineage/near-duplicate concentration, legitimate shared-lineage counterexamples, recall/task effects, latency, bytes, and false-suppression risk. A product reorder is not authorized merely by opening this item. |
| 4. Explainable delivery receipt | RFC/design issue #1356, then a separately approved implementation issue | A bounded, optional Context Receipt identifies the bytes actually delivered, exact selected origins, omission reasons, and relevant preparation outcomes without prompt expansion, raw content retention, or default request persistence. |
| 5. Host-confirmed aggregate usage observation | Create a separate implementation issue only after stages 1, 2, and 4 are complete | The host has a stable receipt identity and explicit acknowledgement/use semantics. The first observation remains Scope-local, asynchronous, bounded, idempotent, privacy-safe, and evaluation-only. |
Hard rules for a future usage-observation proposal
A follow-up proposal for stage 5 must meet all of these conditions:
- Observed fact: distinguish
prepared,delivered,host_acknowledged, and any stronger host-specific use signal. A retrieval hit, a rendered citation, or a model response is not automatically proof of use. - Identity: bind an acknowledgement to the exact receipt identity/output digest and exact selected entry versions; reject stale, malformed, unauthorized, cross-Scope, or replayed acknowledgements idempotently.
- Storage: start with bounded, aggregate, Scope-local observations. Do not persist raw query text, prompt/context body, model response, host transcript, vectors, secrets, absolute paths, or an unbounded per-turn event ledger by default.
- Authority: acknowledgement cannot mutate a Memory body, citation, Artifact Revision, Source, or lifecycle state. It is derived evaluation evidence only.
- No immediate product policy: do not create
use_count,last_used_at, ranking boosts, freshness decay, retention, deletion, or automatic deactivation in the first slice. Any later policy must be a separately calibrated RFC/Issue and must use feat(memory): add versioned Source evidence declarations and neutral lifecycle projection #1975's source protection. - Failure semantics: host non-response, receipt loss, privacy refusal, or observation-sink failure must not alter normal context preparation or Memory write/read behavior. They are not negative utility evidence.
- Evaluation: compare enabled and disabled observation paths through replay/workload evidence. Report acknowledgement coverage separately from usefulness, recall quality, task outcome, bad recall, false retention/suppression risk, latency, byte cost, and storage cost.
Dependency semantics
- Stage 1 is a deterministic reliability prerequisite for evaluating later retrieval changes; it must not be bypassed by an LLM query rewrite.
- Stage 2 is a hard safety prerequisite for using frequency or source class in any future ranking/maintenance decision.
- Stage 3 is evidence gathering only. A favorable result may justify a new implementation proposal, not an implicit production reorder.
- Stage 4 is a hard contract prerequisite for host-confirmed observations. RFC: explainable PreparedContext receipts and progressive disclosure #1356 is design-first; its accepted RFC and a concrete implementation are both required.
- feat(runtime): enable privacy-safe hosted-redacted Memory write-gate egress #1973 is related privacy work for hosted DecisionModel egress, but is independent of this sequence and does not authorize receipt or usage tracking.
Non-goals
- No global
MEMORY.md-style memory, session transcript consolidation, mandatory LLM service, cross-host behavioral surveillance, user-global usage profile, or automatic lifecycle policy. - No replacement for RFC docs: add memory quality,forgetting and lifecycle RFC #1652, feat: strengthen memory governance, provenance, and diagnostics #1698, evaluation(memory): assess Jev for duplicate and conflicting Memory candidates #1942, feat(runtime): enable privacy-safe hosted-redacted Memory write-gate egress #1973, or RFC: explainable PreparedContext receipts and progressive disclosure #1356.
- No claim that a high acknowledgement count makes evidence true, current, authoritative, or suitable for automatic retention.
Definition of done
This tracker closes only after every linked implementation/evaluation task has its stated outcome and any proposed stage-5 implementation has separately passed its design, authorization, privacy, backend-parity, and replay gates. Until then, the normal read path remains read-only.
References
- bug(recall): identical fact is returned for one query form and missed for another — retrieval flips on 1-2 character prefixes (CJK) #1912
- fix(recall): stabilize deterministic FTS recall across equivalent query forms #1976
- feat(memory): add versioned Source evidence declarations and neutral lifecycle projection #1975
- evaluation(recall): measure lineage concentration and near-duplicate occupancy #1977
- evaluation(memory): assess Jev for duplicate and conflicting Memory candidates #1942
- RFC: explainable PreparedContext receipts and progressive disclosure #1356
- feat(runtime): enable privacy-safe hosted-redacted Memory write-gate egress #1973
- RFC docs: add memory quality,forgetting and lifecycle RFC #1652
- RFC docs(rfc): propose decision policy governance #1770
- RFC docs(rfc): define long-horizon workloads #1229
Activity
AlexStocks commentedon Nov 3, 2026
Gate status audit (2026-11-03, against master
729dc4c5)Stage-by-stage verification of this tracker against the current code:
builtin/artifacts/search.py,builtin/artifacts/topic_memory/search.py,builtin/persistence/lexical_projection.py) with coverage intests/builtin/artifacts/memory/test_fusion.py. Note: PR #1969 was closed unmerged — equivalent code landed via other commits.CHANGES_REQUESTEDreview.builtin/runtime/receipt.py,receipt_builder.py,receipt_capture.py;include_receiptwired through runtime, client, and generated HTTP models).Caveats worth recording
untrusted/unknowndeclarations). The data is still valid, but the caveat should be recorded; a re-run after feat(memory): persist versioned Source evidence and neutral Atomic Memory projection #2004 merges would strengthen the report. The "evaluation(memory): assess Jev for duplicate and conflicting Memory candidates #1942 reported separately" item in the stage-3 row also has no visible deliverable yet.Recommended next steps
AlexStocks commentedon Nov 12, 2026
Final gate audit (2026-11-12, against
origin/mastera8f923343c843f14539eedb91bcab921f4d33145)This supersedes the 2026-11-03 audit above. One gate changed: stage 2 landed, which was the only blocker named in that audit.
builtin/artifacts/search.py,builtin/artifacts/topic_memory/search.py,builtin/persistence/lexical_projection.py), with coverage intests/builtin/artifacts/memory/test_fusion.py. PR #1969 was closed unmerged; equivalent code landed through other commits.1190dcbb), issue #1975 closed 2026-11-04T06:25:56Z (completed). Contract insources/evidence.py:MemoryEvidenceDeclaration(authority/verification/declaration_version, defaulting tountrusted/unknown) andSourceEvidencecarryingsource_ref,materialization,material_digest,definition_name/definition_version,manifest_fingerprint,declaration. Neutral projection is applied inbuiltin/runtime/atomic_memory_rebuild.py:255. Acceptance:tests/e2e/test_atomic_memory_source_evidence.py. The priorCHANGES_REQUESTEDreview is resolved by the merge; no open PR touches these paths.include_receipt(defaultfalse, strict) is wired end to end:builtin/runtime/models.py:338,builtin/runtime/application.py:1117/:1195,builtin/runtime/prepared_context.py:347/:417,builtin/runtime/prepared_code.py:185/:192,client/client.py:1446, generatedhttp/_generated/models.py:6003, andopenapi/powercontext.yaml:12185.ReceiptEvidencelives inbuiltin/runtime/receipt_builder.py:141.Caveats carried forward (not blockers)
untrusted/unknownthroughout. The reported lineage-concentration numbers stay valid as an observation of that condition; re-running after feat(memory): persist versioned Source evidence and neutral Atomic Memory projection #2004 would strengthen them. This is optional follow-up, not an unmet gate.Definition of done
Every linked implementation/evaluation task reached its stated outcome (#1976, #1975, #1977, #2235 closed as completed; #1356 accepted as RFC). No stage-5 implementation has been proposed, so the clause requiring any proposed stage-5 implementation to pass its gates does not apply. The normal read path stayed read-only throughout; no ranking, retention, or lifecycle policy was introduced by any stage.
Closing as completed.