Skip to content

docs(rfc): propose scope-based knowledge bases - #2570

Open
Zxf-xufeng wants to merge 10 commits into
masterfrom
codex/rag-knowledge-base-rfc
Open

Zxf-xufeng wants to merge 10 commits into
masterfrom
codex/rag-knowledge-base-rfc

Conversation

@Zxf-xufeng

@Zxf-xufeng Zxf-xufeng commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Which issue or RFC does this PR close?

Related to #2440, #2441, #2442, #2443, and #2444. This design-only PR does not close implementation issues. Framework contracts are defined in #2549 and #2579, with implementation in #2616. Source collection in #2617 is a future integration, not a first-phase requirement.

Rationale for this change

Provide document knowledge bases through one single-family business plugin: one ordinary Scope is one knowledge base, and each document Artifact represents a whole file. Reuse framework Source consumption, synchronous user changes, and native lifecycle semantics while keeping immutable chunks and search indexes in plugin storage.

What changes are included in this PR?

Only the English and Chinese RFCs under docs/{en,zh}/rfcs/2570-knowledge-base.md:

  • A knowledge-base plugin declares the document family, original-file and processing-request Source definitions, APIs, rendering, and search. Knowledge bases are Scopes rather than another Artifact family.
  • Two plugin domain tables: fixed-purpose resources (Scope configuration, logical-file identities, and accepted requests/observations) and immutable chunks. No independent jobs table, lease system, cleanup queue, or new core schema changes for this plugin.
  • Local OpenDAL snapshots, Markdown parsing, RapidOCR/ONNX text-image OCR, structural/recursive chunking, original provenance, and pinned request bases/configuration.
  • Background import/reprocessing uses framework generate, Supervisor, and consumption cursors. Deterministic input failures can be recorded and skipped; infrastructure failures roll back the window and use framework retry. Status observations do not replace committed core results.
  • Chunk edits/removals are synchronous generic Artifact PUT operations. They prepare new chunks/indexes, then commit a new document revision and core-generated lineage_only evidence with revision preconditions.
  • Document soft deletion uses native forget: deprecated and excluded from normal retrieval, without a new content revision or plugin deletion evidence. Originals, bodies, and authorized history remain available. No Source deletion, in-flight cancellation, knowledge-base cascade, or implicit restoration through reimport.
  • Immutable results and indexes become effective only through a committed manifest. Failed publication preserves old searchable content. Exact chunk APIs and plugin retrieval supplement generic Artifact search and Prepared Context.
  • Dedicated plugin storage on SQLite, SeekDB, and OceanBase follows the framework RFC; implementation gaps remain dependencies. Models use independent plugin configuration and fingerprints, with no automatic Server-model inheritance or online model switch.
  • Scope creation retains server.admin; plugin configuration retries after Scope creation, migrations use upgrade(ctx), and first-phase deployments may restart rather than require hot reload.

Are there any user-facing changes?

Documentation only. No runtime, dependency, OpenAPI, migration, or generated-site files change. The proposed first phase includes all three database backends, synchronous manual chunk editing, and native document soft deletion. It does not promise that forgetting a document revokes historical access or deletes its Source. Framework proposals are not treated as already implemented capabilities.

How was this change tested?

  • git diff --check and git diff --cached --check.
  • Cached installed trailing-whitespace, end-of-file, and merge-conflict hooks on both RFCs.
  • YAML front matter, MDX compilation without unintended executable expressions, local links, matching locale sections/table names/workflow, and eight identical JSON examples per locale.
  • Checked the detailed Yuque design and framework-consumption companion against the RFC; checked removal of obsolete multi-base IDs, independent queue contracts, Source-delete operations, and online model-switch operations.
  • UV_OFFLINE=1 uv run --locked prek run -a could not start because Pillow 12.3.0 was absent from the local cache. Full hooks/type checks and the full website build are not verified. These documentation checks do not establish runtime, OCR, concurrency, or real database support.

AI usage statement

OpenAI Codex inspected the framework contracts, drafted the English RFC and Chinese translation from the requester's confirmed decisions, and ran focused documentation checks.

@AlexStocks AlexStocks added the 1.6.0 v1.6.0 release tracking label Nov 10, 2026

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found in this RFC at bb57856. This approval covers the bilingual design documents only; the proposed tables and APIs, real OCR, and three-backend acceptance remain implementation work.

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Zxf-xufeng Zxf-xufeng changed the title docs(rfc): propose rag document knowledge bases docs(rfc): propose knowledge-base business plugin Nov 11, 2026

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking design issues found at 71e842c. This approval covers the bilingual RFC only; the three plugin domain tables and plugin APIs remain proposed implementation work. Current CI is incomplete because preflight was cancelled, so this approval does not clear the CI gate or establish runtime, OCR, or three-backend acceptance.

@Zxf-xufeng Zxf-xufeng changed the title docs(rfc): propose knowledge-base business plugin docs(rfc): propose scope-based knowledge bases Nov 13, 2026

@hireview001 hireview001 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.6.0 v1.6.0 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants