From 368f990f7be4cf8feaaaadd01660a74eb4beb64c Mon Sep 17 00:00:00 2001 From: jasper-yl Date: Sun, 8 Nov 2026 10:45:10 -0800 Subject: [PATCH 01/12] docs(rfc): propose plugin interfaces Replace the Source and Artifact plugin frameworks with a few interfaces that PC Server calls through the Supervisor. Core keeps only Sources, Artifacts, collection targets, and consumption progress; plugins no longer have managed lifecycle state. Co-Authored-By: Claude Opus 5.5 --- docs/en/rfcs/0000-plugin-interfaces.md | 846 +++++++++++++++++++++++++ docs/zh/rfcs/0000-plugin-interfaces.md | 748 ++++++++++++++++++++++ 2 files changed, 1594 insertions(+) create mode 100644 docs/en/rfcs/0000-plugin-interfaces.md create mode 100644 docs/zh/rfcs/0000-plugin-interfaces.md diff --git a/docs/en/rfcs/0000-plugin-interfaces.md b/docs/en/rfcs/0000-plugin-interfaces.md new file mode 100644 index 0000000000..98f926f469 --- /dev/null +++ b/docs/en/rfcs/0000-plugin-interfaces.md @@ -0,0 +1,846 @@ +--- +title: Plugin interfaces +description: Define Source and Artifact plugins through a few interfaces that PC Server calls through the Supervisor when needed; core manages only the two core data kinds, Sources and Artifacts, and their consumption progress, and no longer manages plugin lifecycle. +--- + +- Proposal Name: `plugin_interfaces` +- Start Date: 2026-11-08 +- RFC PR: [oceanbase/powercontext#0000](https://github.com/oceanbase/powercontext/pull/0000) + +# Summary + +This proposal redefines PowerContext's third-party plugin model: each plugin kind implements a few fixed interfaces, +and PC Server calls them when needed. Plugins are callees that only turn inputs into results; they take no part in +scheduling, state management, or persistence protocols. + +- **Source plugins** provide Source definitions (reusing the existing `SourceDefinition` / `SourceProjection`) and may + implement `collect`. For writes, Server calls the definitions to validate, transform, and save. Collection works on + **collection targets**, such as "one account on X"; hosts can add, disable, and re-enable targets at runtime without a + restart. The Supervisor dispatches a Worker on schedule to call `collect`, and core commits the returned Sources and + each target's checkpoint in one transaction. +- **Artifact plugins** implement `generate`. The Supervisor dispatches a Worker on a configured schedule, when new + Sources arrive, or on a manual request; the Worker reads that Scope's unconsumed Source window and passes it to + `generate` together with a **writer** (`ArtifactWriter`). The plugin does slow work such as model calls outside the + writer, then performs Artifact writes inside an `async with writer.transaction()` block: core opens a transaction when + the block starts, writes in the block act directly on core data and immediately return real revisions, and when the + block ends normally core commits those writes, the consumption cursor, and the completion acknowledgment together. An + exception in the block rolls everything back. Plugins can neither read nor change the cursor. +- Core manages only two kinds of core data, `pc_sources` and `pc_artifacts` (with heads, lineage, merge records, + lifecycle, handles, and tags), and their consumption progress. Plugins may keep their own indexes and storage, which + they manage and keep consistent themselves. Applications read and search plugin families through the existing public + Artifact interfaces, and the read path runs no plugin code. +- `collect` and `generate` run in Worker child processes managed by the Supervisor, reusing its existing process + isolation, timeouts, concurrency budgets, leadership lease, and failure backoff. Users deploy only one Server. +- Plugins load from a directory at Server startup; code or manifest changes require a restart. Server stores no plugin + enabled state, registration generation, or management phase, and offers no enable / disable / reload / unload + management interface. + +This proposal replaces the plugin contracts and management mechanisms of the +[Source plugin framework RFC #2012](2012-source-plugin-system.md) and the +[Artifact plugin framework RFC #2175](2175-artifact-plugin-system.md). Neither implementation has shipped as a stable +interface, so this proposal keeps no compatibility: old plugin manifests, management interfaces, the management +service, and plugin-management tables are removed, and custom Artifact data written by old Artifact plugins is not +migrated. + +This document describes the target design, not availability in mainline or a released version. Interface names and +code examples illustrate the contract; final syntax ships with the implementation. + +# Motivation + +Both existing plugin frameworks make PowerContext carry substantial lifecycle management for third-party plugins: + +| Framework | Work Server carries for plugins | +| --- | --- | +| Artifact plugins (#2175) | `desired_state`, `phase`, `binding_generation`, and `last_error` in `pc_artifact_processing_binding_states`; seven management states and their transitions; `enable-all / reload / enable / disable / unload` management interfaces with `expected_generation` concurrency checks; registration-generation fencing of old Worker commits; content schemas and index definitions frozen in `pc_artifact_family_schemas`, with changes returning 409; framework-owned building, rebuilding, and availability of custom declarative indexes | +| Source plugins (#2012) | A separate plugin management service with its own SQLite (instances, capabilities, targets, desired state, submission records); one Runtime process per plugin instance; a poll / reply channel and capability-catalog synchronization between management and Server; the `source load` deployment command; MCP tools to enable and disable instances | + +Everyone pays for these mechanisms: + +- **Plugin authors** must understand bindings, generations, schema-freezing rules, index declarations, and the + management channel before they can write a plugin. +- **Deployers** run several processes, maintain two databases, and use management APIs before a plugin can run. +- **Maintainers** keep state machines, concurrency fencing, schema and index governance, and recovery paths for + third-party code; that code and its test surface already exceed the plugins' business logic. + +A plugin needs to be called at the right time, receive the input it needs, and hand results back to core. Scheduling, +isolation, consumption progress, and transactional commits are capabilities core already has; the Supervisor provides +them to built-in Artifacts today. Plugin models such as Dify's show that a host defining interfaces and invocation +timing, with plugins only implementing interfaces, is enough for a large ecosystem of third-party extensions. + +Goals: + +1. Plugin authors implement a few interfaces without learning scheduling, cursor, and commit protocols. +2. Plugin code runs in isolated Worker processes; plugin failures do not affect Server or other plugins. +3. Plugin results go into `pc_sources` / `pc_artifacts` and commit atomically with consumption progress, so existing + read interfaces work. +4. Core manages only these two kinds of core data and their consumption progress; content structure, indexes, and other + storage belong to plugins. +5. Plugin writes to core data go only through a core-provided writer that covers every generic Artifact write operation + and reuses core's existing validation rules. +6. Artifact generation can run on a configured schedule or be triggered by new Sources or manual requests; collection + targets can be added and removed at runtime. +7. Users deploy only one Server; plugins load when Server starts. + +# Guide-level explanation + +## 1. Core idea + +```mermaid +flowchart LR + Host["Application / Agent / host"] -->|"HTTP / MCP"| Server + subgraph Server["PC Server"] + API["Public interfaces
authentication and Scope authorization"] + Sup["Supervisor
scheduling / triggers / budgets / backoff"] + Core[("core data
pc_sources / pc_artifacts
targets / cursors / checkpoints")] + end + subgraph Worker["Worker child process (per call)"] + Exec["Plugin executor"] + Plugin["Plugin code
collect() / generate()"] + Tx["writer.transaction()
constrained write transaction"] + Exec -->|"targets and checkpoints / Source window"| Plugin + Plugin -->|"put / merge / set_lifecycle ..."| Tx + end + API -->|"read, search"| Core + API -->|"write, add targets"| Core + API -->|"manual trigger"| Sup + Sup -->|"dispatch (plugin, Scope)"| Exec + Tx -->|"on exit: commit writes, cursor, acknowledgment together"| Core + Plugin -.->|"own indexes / storage"| Own[("Plugin-owned storage")] + Plugin -.->|"external systems / models"| Ext["X, ticket systems, LLMs, ..."] +``` + +| Responsibility | Owner | +| --- | --- | +| Authentication, Scope authorization, public HTTP / MCP interfaces, collection target management | Server | +| When to call: scheduling, trigger coalescing, Worker budgets, timeouts, backoff, leadership lease | Supervisor (existing) | +| Reading inputs, providing a constrained write transaction, committing core data / cursors / checkpoints / acknowledgments in one transaction | Plugin executor in the Worker (core) | +| Reading external data, identifying and normalizing collection targets, Source definitions and Projection computation | Source plugin | +| Artifact content structure, generation algorithm, plugin-owned indexes and storage and their consistency | Artifact plugin | +| Deploying plugin code, dependencies, and plugin-owned storage | Deployer | + +Plugin code does one thing: given input, produce results or raise. Raising is failure; core commits no core data and +retries the same input later. + +## 2. Plugin directory and manifest + +Server loads plugins from `POWERCONTEXT_SERVER_PLUGINS_DIR`. Each direct child directory is one plugin containing +`plugin.yml` and code: + +```text +/opt/powercontext/plugins/ +├── x-posts/ +│ ├── plugin.yml +│ ├── definition.py +│ └── plugin.py +└── incident-report/ + ├── plugin.yml + └── plugin.py +``` + +```yaml +name: incident-report # Plugin name, unique in the deployment, matches the directory name +kind: artifact # source or artifact +entry: ./plugin.py:create # Factory function; path relative to the plugin directory +scopes: [team-a] # Optional; omitted means every Scope with matching input +config: # Business configuration validated by the plugin's Config model + title_prefix: Incident report + index_url: ${REPORT_INDEX_URL} # ${NAME} resolves from the environment inside the Worker +``` + +The manifest describes only **what the plugin is and how it runs**, not whether it is enabled. If the directory exists +and loads successfully, the plugin is available; to stop it, move it out of the directory and restart Server. The same +code can be deployed several times through multiple plugin directories with different `name` and `config` values. + +## 3. Source plugin example + +A Source plugin exports Source definitions. With definitions alone, hosts can submit complete content through the +existing Source write interface (the write use of MCP `pc_source_add`); Server calls the definition's `resolve` and +each Projection, then saves to `pc_sources`. + +To collect continuously from an external system, the plugin also implements `collect`. Collection works per +**collection target**: each target is one external resource the plugin can collect, such as one account on X. This +plugin collects new posts from each target account every 5 minutes: + +```yaml +name: x-posts +kind: source +entry: ./plugin.py:create +scopes: [team-a] +collect: + schedule: {interval_seconds: 300} + max_workers: 1 + timeout_seconds: 120 + max_items: 200 + targets: # Optional; initial targets imported on first start + - {scope_id: team-a, input: {account: "@oceanbase"}} +config: + bearer_token: ${X_BEARER_TOKEN} +``` + +```python +from pydantic import BaseModel + +from powercontext.plugins import ( + CollectedItem, CollectRequest, CollectResult, PluginContext, TargetIdentity, +) + +from .definition import PostCapture, XPostDefinition + +definitions = (XPostDefinition(),) + + +class Config(BaseModel): + bearer_token: str + + +class TargetInput(BaseModel): # Input a host submits when adding a target + account: str + + +def identify_target(value: TargetInput) -> TargetIdentity: + account = value.account.strip().removeprefix("@").lower() + if not account: + raise ValueError("account must not be blank") + return TargetIdentity(key=f"account:{account}", input={"account": account}) + + +class XCollector: + def __init__(self, client: XClient) -> None: + self.client = client + + async def collect(self, request: CollectRequest) -> CollectResult: + result = CollectResult() + for target in request.targets: + since = (target.checkpoint or {}).get("since_id") + page = await self.client.user_posts(target.input["account"], since_id=since) + result.add(target, [CollectedItem(definition="x-post", value=PostCapture(**p)) for p in page.posts], + checkpoint={"since_id": page.newest_id or since}) + return result + + +async def create(ctx: PluginContext, config: Config) -> XCollector: + client = await ctx.enter_async_context(XClient(config.bearer_token)) + return XCollector(client) +``` + +A host adds a target at runtime without a restart: + +```text +pc_source_add(scope_id="team-a", capability="x-posts", input={"account": "@PowerContextAI"}) +``` + +Server validates the input with `TargetInput`, calls `identify_target` for a stable target key and normalized input, +saves the target, and immediately requests a collection. Adding the same account again yields the same key and reuses +the existing target; targets can be disabled and re-enabled with progress retained. + +Each target's checkpoint is read-only input; the plugin proposes new checkpoints through its return value. Core commits +all Sources returned by one call and each target's new checkpoint in one transaction. If `collect` raises or times out, +nothing is committed and the next call receives the old checkpoints. + +## 4. Artifact plugin example + +This plugin generates one report per ticket Source. It first looks up a similar existing report in its own vector +index, revises that report on a match and creates a new one otherwise, and writes the report into its own vector index. + +```yaml +name: incident-report +kind: artifact +entry: ./plugin.py:create +scopes: [team-a] +generate: + family: incident-report + inputs: + source_types: [example-ticket] + projection: powercontext.text-evidence@1 + max_sources: 50 + schedule: {interval_seconds: 300} # Periodic check + on_sources: true # Trigger when new Sources arrive + manual: true # Allow manual triggers + max_workers: 1 + timeout_seconds: 300 # Limit for the whole call + transaction_timeout_seconds: 30 # Limit for the write transaction block +config: + title_prefix: Incident report + index_url: ${REPORT_INDEX_URL} +``` + +```python +from pydantic import BaseModel + +from powercontext.plugins import ArtifactWriter, GenerateRequest, PluginContext +from powercontext.sources import TextEvidence + + +class Config(BaseModel): + title_prefix: str = "Incident report" + index_url: str + + +class IncidentReports: + def __init__(self, config: Config, index: VectorIndex, model: Summarizer) -> None: + self.config, self.index, self.model = config, index, model + + async def generate(self, request: GenerateRequest, writer: ArtifactWriter) -> None: + # 1. Slow work stays outside the write transaction: model calls, vector computation + planned = [] + for source in request.sources: + evidence = TextEvidence.model_validate(source.projection) + summary = await self.model.summarize(evidence.content) + vector = await self.model.embed(summary) + planned.append((source, summary, vector)) + + # 2. Short write transaction: reads in the block see what the block has written + async with writer.transaction() as tx: + for source, summary, vector in planned: + match = await self.index.nearest(request.scope_id, vector, threshold=0.92) + current = await tx.get(match.artifact_id) if match else None + artifact_id = current.ref.artifact_id if current else writer.stable_id( + "report", source.ref.source_type, source.ref.source_id) + ref = await tx.put( + artifact_id, + content={"title": self.config.title_prefix, "summary": summary}, + search_text=summary, + source_refs=[source.ref], + ) + # With the real artifact_id and revision in hand, upsert into the plugin's own index by ID + await self.index.upsert(request.scope_id, ref.artifact_id, ref.revision, vector) + # The block ended normally: core commits the Artifact writes, the cursor advance, and the acknowledgment + + +async def create(ctx: PluginContext, config: Config) -> IncidentReports: + index = await ctx.enter_async_context(VectorIndex.connect(config.index_url)) + return IncidentReports(config, index, Summarizer()) +``` + +When the block raises, core rolls back all Artifact writes in the block, the cursor stays put, and the same window is +retried later. The plugin's own index is not part of core's transaction: the plugin upserts by stable ID so a retry +overwrites entries left by the previous attempt, and reads from its own index defer to core's current revision (see +reference-level section 1.6). + +Applications access plugin families through the existing public Artifact interfaces, which read `pc_artifacts` +directly and run no plugin code: + +| Purpose | Interface | +| --- | --- | +| Browse | `GET /v1/scopes/{scope_id}/artifacts/{family}` | +| Current content | `GET /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | +| Exact revision / revision history | `GET .../{artifact_id}/revisions/{revision}`, `GET .../{artifact_id}/revisions` | +| Tags | `GET .../{artifact_id}/tags` | +| Text search | `POST /v1/scopes/{scope_id}/artifacts/{family}/search` | +| Trigger generation manually | `POST /v1/scopes/{scope_id}/artifacts/{family}/generate` | + +## 5. Scheduling and triggers + +Plugin scheduling and triggers belong to the existing Artifact processing Supervisor. Each plugin registers at startup +as one Supervisor binding named `plugin:`; the scheduling key remains `(binding, Scope)`. + +Artifact generation supports three call origins in any combination; at least one must be configured: + +| Configuration | When a Worker is dispatched | `request.reason` | +| --- | --- | --- | +| `schedule: {interval_seconds: N}` or `schedule: {cron: "0 2 * * *", timezone: Asia/Shanghai}` | When due, for Scopes with pending work | `schedule` | +| `on_sources: true` | The Scope's Source journal is ahead of the plugin's cursor | `sources` | +| `manual: true` | The manual trigger interface is called | `manual` | + +- For plugins with `inputs`, every call carries that Scope's unconsumed Source window; when the window is empty, + scheduled and Source-triggered calls skip the plugin and acknowledge completion directly. +- Plugins without `inputs` (for example, a daily cleanup of existing Artifacts) consume no Sources, are called when + due, and must configure `scopes` explicitly. +- Manual triggers always call the plugin and may carry `params`, which the plugin interprets. A manual request is + persisted on acceptance and returns `202 Accepted`; it still runs after a Server restart. + +Source plugin `collect` supports `schedule` and `manual` +(`POST /v1/scopes/{scope_id}/sources/plugins/{name}/collect`); adding or re-enabling a target also requests a collection. + +At most one Worker runs at a time for the same plugin and Scope. The Supervisor coalesces triggers that arrive during a +call into one call after it finishes. + +## 6. Operations + +- Startup: Server checks plugin directories one by one. A plugin that fails its check (invalid manifest, import + failure, missing required exports, family or Source type conflicting with a built-in or another plugin) affects only + itself; Server and other plugins start normally. +- Inspection: `GET /v1/plugins` (requires `server.admin`) lists each plugin's check result, kind, family or Source + definitions, schedule configuration, and per-Scope latest call outcome and next retry time. Collection targets and + their latest collection outcomes are shown by `pc_source_status` and the target interfaces. +- Changes: after changing code or the manifest, restart Server. Server does not watch files or reload plugins at + runtime. Collection targets are business data; adding or removing them needs no restart. +- Removal: move the plugin out of the directory and restart. Saved Sources, Artifacts, collection targets, cursors, and + checkpoints are retained and remain readable; redeploying a plugin with the same name continues from where it stopped. + The deployer keeps or cleans up plugin-owned storage. + +# Reference-level explanation + +## 1. Interface contract + +The public interfaces live in a new `powercontext.plugins` module, replacing the plugin-author-facing parts of +`powercontext.artifact_plugins` and `powercontext.source_management`. Interfaces are expressed as `typing.Protocol` +and Pydantic models. + +### 1.1 Entry module, factory, and context + +The entry module exports: + +| Export | Required | Notes | +| --- | --- | --- | +| Factory function (named by `entry`) | Yes | `async def create(ctx, config) -> plugin instance` | +| `Config` | No | Pydantic model validating the manifest's `config`; without it, `config` must be empty | +| `definitions` | Source plugins | Sequence of `SourceDefinition` | +| `TargetInput`, `identify_target` | Source plugins with `collect` | Collection target input model and identification function | + +```python +class PluginContext: + name: str # Plugin name from the manifest + scope_id: str # Scope of this call + plugin_dir: Path # Plugin directory (read-only by convention) + logger: logging.Logger # Logger bound to the plugin name + + async def enter_async_context(self, cm: AbstractAsyncContextManager[T]) -> T: ... +``` + +- The factory runs once per Worker call; instances are not reused across calls. Resources that need closing (including + connections to plugin-owned storage) register through `ctx.enter_async_context` and are released by the executor when + the call ends. +- `${NAME}` environment references resolve inside the Worker before `Config` validation, so credentials never enter + Supervisor dispatch parameters or logs. +- Plugins get no connection to core's database and no write interface for cursors or checkpoints. There are only two + paths for writing core data: the return value of a Source plugin's `collect`, and an Artifact plugin's + `ArtifactWriter`. + +### 1.2 Source plugins + +```python +class TargetIdentity(BaseModel): + key: str # Stable target key; always the same for the same external resource + input: dict[str, JsonValue] # Normalized input keeping every parameter that affects collection + +class CollectTarget(BaseModel): + key: str + input: dict[str, JsonValue] + checkpoint: JsonValue | None # Last successfully committed checkpoint for this target; None the first time; read-only + +class CollectRequest(BaseModel): + scope_id: str + reason: Literal["schedule", "manual", "target_added"] + targets: Sequence[CollectTarget] # Enabled targets to collect in this call + max_items: int + params: JsonValue | None = None # Manual triggers only + +class CollectedItem(BaseModel): + definition: str # name of one definition in definitions + value: BaseModel # Instance of that definition's input_class + +class CollectResult(BaseModel): + targets: dict[str, TargetResult] = {} # Keyed by target key + def add(self, target: CollectTarget, items: Sequence[CollectedItem], *, checkpoint: JsonValue | None) -> None: ... + +class TargetResult(BaseModel): + items: Sequence[CollectedItem] + checkpoint: JsonValue | None + has_more: bool = False + +class SourceCollector(Protocol): + async def collect(self, request: CollectRequest, /) -> CollectResult: ... +``` + +- Plugins that provide `definitions` without a `collect` configuration support write only. The factory runs only for + collect calls. +- Source types and Projections in `definitions` register into the existing Source definition manifests at startup; + conflicts with built-ins or other plugins fail the check. +- **write** needs no plugin method. Server validates input with `definition.input_class`, calls `resolve` to obtain the + Source, computes each Projection, and saves, with the same semantics as existing built-in Source writes. +- Each **collect** call covers one Scope and carries at most `max_targets` (manifest setting, default 50) enabled + targets; with more targets, calls rotate by target key, and targets reporting `has_more` go first in the next call. + The executor applies the same validation, `resolve`, and Projections as write to every returned Source, then saves all + Sources, writes each target's checkpoint and latest collection time, and acknowledges the call in one transaction. If + any entry fails, nothing is committed. A total entry count above `max_items` or a target key absent from the request + is a failure, not truncated. +- Targets absent from the result keep their checkpoints unchanged. +- Source identity is determined by `resolve`. When repeated collection of the same external state yields the same + identity, existing deduplication rules apply and no duplicate Source is created. + +### 1.3 Collection targets + +Collection targets are business data stored by core; like Sources, they are not plugin lifecycle state: + +| Operation | Interface | Notes | +| --- | --- | --- | +| Add | MCP `pc_source_add` (collect use), `POST /v1/scopes/{scope_id}/sources/plugins/{name}/targets` | Validate input with `TargetInput` and call `identify_target` for the key and normalized input. The same key with the same normalized input returns the existing target; the same key with different input returns a conflict. A new target is saved enabled and a collection is requested | +| Inspect | MCP `pc_source_status`, `GET /v1/scopes/{scope_id}/sources/plugins/{name}/targets` | Target key, normalized input, enabled state, latest successful collection time, latest failure category | +| Disable / re-enable | MCP `pc_source_target_set_enabled`, `PATCH .../targets/{key}` | Disabled targets leave collect requests while their checkpoints and collected Sources are retained; re-enabled targets continue from the old checkpoint | + +- Adding and changing targets requires Source write permission on the Scope, and the Scope must be within the plugin's + `scopes`. +- `identify_target` must be pure: it only validates and normalizes, without contacting external systems or checking + external authorization. Whether an external resource exists or is readable is determined by the plugin during + collect and expressed as an exception or empty result. Like write, it runs synchronously in the Server process under + a timeout. +- The manifest's `collect.targets` are imported once only when their keys do not exist. Afterward core's data is + authoritative; changing the manifest neither overwrites nor deletes existing targets. + +### 1.4 Artifact plugins, the writer, and the write transaction + +```python +class ArtifactGenerator(Protocol): + async def generate(self, request: GenerateRequest, writer: ArtifactWriter, /) -> None: ... + +class GenerateRequest(BaseModel): + scope_id: str + reason: Literal["schedule", "sources", "manual"] + sources: Sequence[GenerationSource] # This window; empty when inputs is not configured + params: JsonValue | None = None # Manual triggers only + +class GenerationSource(BaseModel): + ref: SourceRef # Exact Source reference + projection: JsonValue # Content of the Projection named by inputs.projection + +class ArtifactWriter(ArtifactReads): + family: str # Fixed to the manifest's generate.family + + def stable_id(self, *parts: str) -> str: ... + def transaction(self) -> AbstractAsyncContextManager[ArtifactTransaction]: ... + +class ArtifactTransaction(ArtifactReads): + ... # Reads (see 1.5) plus write operations (see the table below) +``` + +`generate` runs in two parts: + +- **Outside the transaction**: the plugin reads committed data through `writer` (each read uses its own short read-only + transaction) and does slow work such as model calls and vector computation. +- **The write transaction block**: `async with writer.transaction() as tx:`. When the block starts, core opens a write + transaction and rechecks the Supervisor lease and invocation intent. Writes in the block execute directly in that + transaction and immediately return a real `ArtifactRef` (with revision), and reads in the block see what the block has + written. When the block ends normally, core CAS-advances the cursor, acknowledges the call, and commits, all in the same + transaction. If the block raises, exceeds `transaction_timeout_seconds`, or conflicts at commit, the transaction rolls + back entirely and the call fails. + +Rules: + +- Each call opens at most one write transaction block; opening a second raises. The cursor and call acknowledgment + commit only with this one transaction, so a window's core data is never partially committed. +- Returning normally without opening a write transaction block means the window needs no output; the executor commits + the cursor advance and call acknowledgment on its own. +- After the write transaction block commits, exceptions raised by later code in `generate` no longer affect the + committed results; the executor records them as post-commit errors and does not retry the window. Plugins should make + the write transaction block the last step of `generate`. +- The write transaction block's time limit defaults to 30 seconds and counts toward the whole call's `timeout_seconds`. + The block should contain only reads, write operations, and writes to plugin-owned storage, not model calls. + +**Write operations**: the write transaction covers every generic Artifact write operation. Each reuses core's existing +implementation and validation rules and raises immediately on a violation: + +| Operation | Reused core implementation | Semantics | +| --- | --- | --- | +| `put(artifact_id, *, content, search_text=None, source_refs=(), artifact_refs=())` | `ArtifactRepository.create` / `revise` | Creates revision 1 when absent; otherwise appends a revision against the current head. When content and `search_text` equal the current revision, it is a no-op returning the current reference | +| `revert(artifact_id, *, to_revision, reason)` | The rollback path of `revise` | Appends a new revision with a historical revision's content, recording `restored_from_revision` and the reason | +| `set_lifecycle(artifact_id, state, *, replacement_artifact_id=None)` | `ArtifactGovernanceRepository.transition` | active ↔ deprecated (optionally naming a replacement), any state → retired (irreversible). Changes the head's governance state without appending a revision; frozen merge inputs cannot change individually | +| `merge(inputs, *, into, content, search_text=None, source_refs=())` | The shared merge method ([RFC #1859](1859_artifact_merge.md)) | Takes the current revisions of `inputs`, creates `into`, records the merge relation, and freezes the inputs. `into` must be a new ID | +| `preview_undo_merge(artifact_id)` / `undo_merge(artifact_id, *, preview=None)` | Preview and restoration of the shared merge method | Undoes the merge that produced the Artifact, restoring inputs and retiring the result; when executing a preview, stale previews are rejected if related state changed | +| `bind_handle(handle_key, ref)` | Artifact handle binding | Binds a stable handle to a given revision | +| `set_tags(artifact_id, tags)` | Artifact tag replacement | Replaces the Artifact's tags entirely | + +**Write constraints**: + +- Only Artifacts in this Scope and in the manifest's `generate.family` can be touched; plugins cannot name another + family or Scope. +- `source_refs` may reference only Sources in this window; `artifact_refs`, `to_revision`, and `bind_handle` targets must + be revisions that actually exist in this family. +- Operation count, single-content size, and total bytes per write transaction block are bounded; exceeding a limit + raises and rolls back, without truncation or partial commit. +- Core neither validates the business structure of `content` nor stores or freezes a schema. The plugin owns content + structure evolution and may carry a version field inside `content`. + +### 1.5 Reads + +`ArtifactWriter` (outside the transaction, committed data only) and `ArtifactTransaction` (inside the transaction, also +seeing what the block has written) offer the same read methods, scoped to this Scope and this family: + +```python +class ArtifactReads(Protocol): + async def get(self, artifact_id: str, *, revision: int | None = None) -> ArtifactSnapshot | None: ... + async def revisions(self, artifact_id: str, *, cursor: str | None = None, limit: int = 50) -> RevisionPage: ... + async def list(self, *, states: Collection[LifecycleState] = ("active",), tags: Collection[str] = (), + cursor: str | None = None, limit: int = 50) -> ArtifactPage: ... + async def search(self, query: str, *, states: Collection[LifecycleState] = ("active",), + limit: int = 10) -> Sequence[ArtifactSnapshot]: ... + async def by_source(self, ref: SourceRef, *, cursor: str | None = None, limit: int = 50) -> ArtifactRefPage: ... + async def lineage(self, ref: ArtifactRef) -> ArtifactLineage: ... + async def governance(self, artifact_id: str) -> ArtifactGovernance: ... # Lifecycle, replacement, merge ownership + async def merge_record(self, artifact_id: str) -> ArtifactMergeRecord | None: ... + async def tags(self, artifact_id: str) -> ArtifactTagSet: ... + async def resolve_handle(self, handle_key: str) -> ArtifactRef | None: ... +``` + +- `get` reads the current content or a given revision; `revisions` lists the full history. +- `list` and `search` can filter by lifecycle state, including deprecated, retired, and frozen merge inputs, to find + "data that already existed". `search` is core's text search (see section 5); inside the write transaction it finds + content the block has written but not yet committed. +- `by_source` finds Artifacts whose lineage references a Source, so plugins can locate Artifacts to revise after a + Source changes. +- Snapshots returned by reads carry exact revisions and can be used directly as `artifact_refs` or `to_revision` in + write operations. +- When a plugin needs search core does not provide, such as vector similarity, it uses its own index and then defers to + core's current revision through `get`. + +### 1.6 Consistency and idempotency + +**Core data**: a window's Artifact writes, cursor advance, and call acknowledgment commit in one transaction. A failure +leaves core data completely unchanged, so retries start clean; after success the cursor has passed the window, so the +same window is never delivered again. Core data is committed exactly once per window. Operations in the write +transaction act directly on the database, and conflicts (for example, a head changed concurrently after being read) are +reported immediately at the operation or at commit, rolling back everything for the Supervisor to reschedule. Plugin +families expose no public write interfaces in the first version, and calls are single-flight per plugin and Scope, so +such conflicts occur only in abnormal cases. + +**Reprocessing the same business object**: manual reruns, scheduled cleanups without `inputs`, and similar scenarios +process objects already processed. Plugins derive Artifact IDs from business identity with `writer.stable_id(...)`; +`put` is a no-op for identical content, `set_lifecycle` is a no-op when the target state equals the current one, and +`merge` is a no-op when all inputs are already merged into the same result, so no duplicate Artifacts or meaningless +revisions appear. + +**Plugin-owned storage**: plugin indexes and other storage are a different database from core and are outside core's +transaction. Core does not guarantee consistency between them; the plugin does. Recommended practice: + +1. Write plugin-owned storage inside the write transaction block, after obtaining the `ArtifactRef`, upserting by + `artifact_id` (with revision when needed). A failed write to plugin-owned storage then rolls back the core writes too. +2. If the write transaction fails to commit, plugin-owned storage may keep entries from that attempt. The retry + processes the same window, derives the same stable IDs, and overwrites them. +3. Check results read from plugin-owned storage with `get`, confirming the Artifact exists in core with a matching + revision; ignore mismatched entries, or periodically rebuild from core data. + +`generate` itself may run more than once due to retries, so model calls and external requests issued by the plugin are +not exactly once. + +## 2. Plugin executor and consumption cursor + +The plugin executor is core code in the Worker that implements the Supervisor Worker entry and replaces the old +`run_artifact_plugin_worker`. One Artifact plugin call proceeds as follows: + +```text +Worker receives (plugin:, scope_id, request_generation) + -> Short transaction: ScopeInvocation.start; load the cursor for (scope_id, plugin:) from pc_source_cursors + -> read the journal high watermark and fix the window (after, through], through ≤ after + max_sources + -> read the window with list_window_for_generation, filter by inputs.source_types, compute inputs.projection + -> Load the plugin, call the factory, create the writer, call generate (under timeout_seconds) + -> Reads outside the transaction: separate short read-only transactions + -> Write transaction block starts: open a transaction; guard (lease and invocation intent) + Writes in the block execute in that transaction + Block ends: CAS the cursor to through; acknowledge; commit + -> generate returns without opening a write transaction block: a short transaction performs the cursor CAS and acknowledgment +``` + +- Sources in the window whose type does not match are filtered out, but the cursor still advances past the whole window; + a matching Source missing the required Projection fails the call rather than being skipped. +- The cursor advances only in the executor's commit transaction; plugins never receive its value and have no write + interface for it. The cursor CAS ensures a window commits only once even if leadership changes or duplicate dispatch + happens between reading the window and committing. +- `on_sources` reuses the existing `SourceProcessingPendingProvider`: when the journal high watermark exceeds the + cursor, the Scope is marked pending and the Supervisor dispatches within budget. The check uses persisted data, so a + Server restart loses no pending input. +- Manual triggers and target additions reuse existing invocation intents + (`ArtifactProcessingIntentRepository.request`): the acceptance transaction advances the requested generation, + multiple requests coalesce, and requests arriving during a run are handled by a later call. +- Without `scopes`, automatic scheduling applies to every Scope whose journal is ahead of the cursor; with `scopes`, + both automatic scheduling and manual triggers accept only listed Scopes. + +Collect calls have the same structure: the input becomes the Scope's enabled targets and their checkpoints, and the +commit carries Sources and each target's new checkpoint. Collect needs no write transaction block; the executor commits +from the return value at once. + +## 3. Loading and lifecycle + +```text +Server start + -> Scan direct children of POWERCONTEXT_SERVER_PLUGINS_DIR (load nothing when unset) + -> For each directory: parse and validate the manifest (name matches directory, kind matches configuration sections) + -> In a time-limited check child process, import the entry and confirm the factory, Config, and other required + exports match the kind, without calling the factory + -> Source plugins: import definitions, TargetInput, and identify_target in the Server process and register Source types + -> Import initial collection targets from the manifest that do not exist yet + -> Register the family access profile; register a Supervisor binding when collect / generate is configured + -> Any step fails: record the error, mark the plugin failed, continue with the next +Server stop + -> Stop dispatch and reclaim Workers through the existing Supervisor shutdown +``` + +A plugin has only two outcomes, `loaded` and `failed`, held in process memory. Server does not persist plugin enabled +state, registration generations, or configuration snapshots; each start rechecks the directory as it is then. Without +hot reload, no registration generation is needed to fence old Workers; the existing Supervisor lease fence already +rejects stale commits after leadership changes. + +Workers import plugin code on every call. If a plugin directory changes while Server runs, newly dispatched Workers may +see the new code; deployers should restart Server after changes, and the first version does not detect this case. + +## 4. Execution constraints + +- **Isolation**: `collect` / `generate` run in Worker child processes started by the Supervisor; crashes, memory leaks, + and blocking do not affect Server. Child processes isolate ordinary failures but are not a sandbox for untrusted code: + plugins run with the Worker's operating-system permissions and can access its files, network, and environment. +- **Timeouts and budgets**: `timeout_seconds` (default 300) is the total Worker time limit, + `transaction_timeout_seconds` (default 30) the write transaction block limit, and `max_workers` (default 1) the + plugin's concurrency limit across Scopes. These map to existing Supervisor binding parameters or are enforced by the + executor. Input bytes, write operation count, and output bytes per call are bounded; exceeding a limit fails the call. +- **Lock impact of the write transaction**: while the write transaction block is open, it locks the heads and cursor row + it touches; on SQLite, a write transaction blocks all other writes to the database. The block must therefore be short, + and exceeding its limit rolls it back. +- **Failure and backoff**: a failed call commits no core data, and that plugin and Scope enter the Supervisor's existing + backoff without affecting other Scopes or plugins. Core never skips input automatically after repeated failures; + persistent failures require fixing the plugin or data and restarting. A target that keeps failing during collect + blocks the other targets in the same Scope; disable it first. +- **Multi-instance deployments**: with several OceanBase Server instances, dispatch happens only on the instance holding + the Supervisor leadership lease; writes, target management, and reads may run on any instance. Plugin-owned storage + must be reachable from instances that run Workers. SQLite deployments have one Server process. + +## 5. Public reads and search + +Core handles all public reads of plugin families without running plugin code: + +- list / get / revisions / tags reuse the existing generic Artifact read path. Plugin family `content` is returned as + JSON without a schema from `pc_artifact_family_schemas`. Frozen merge inputs and deprecated and retired Artifacts are + presented under the shared merge and lifecycle rules. +- search is a generic text search core adds for plugin families; like the existing search interface, it accepts only + text queries. On write, `search_text` passes through existing lexical analysis into `pc_artifact_heads.searchable_text`, + and search matches on that column, returning only revisions in the requested Scope that the caller may read. Artifacts + without `search_text` are not searchable. The writer's `search` uses the same implementation. The implementation PR + specifies the search method on SQLite and OceanBase and validates it at scale. +- Review, Recall, and Prepared Context are not available for plugin families in the first version, nor are the public + create, revise, merge, lifecycle, handle, and tag write interfaces; writes to plugin families come only from the + plugin. Those interfaces return an explicit "unsupported" error, and `GET /v1/capabilities` reflects the operations + actually supported. + +## 6. Persistence + +| Data | Handling | +| --- | --- | +| `pc_sources`, Projections, the Source journal, `pc_source_definition_manifests` | Kept; results of Source plugins | +| `pc_artifacts`, `pc_artifact_heads`, `pc_artifact_lineage_*`, merge records, handles, tags | Kept; results of Artifact plugins | +| `pc_source_cursors` | Kept; Artifact plugin consumption cursors keyed by `(scope_id, plugin:)` | +| `pc_connector_checkpoints` | Kept; per-target checkpoints keyed by `(scope_id, target binding ID)` | +| `pc_source_collect_targets` | Added; collection targets: Scope, plugin name, target key, normalized input, enabled state, creation and update times, latest successful collection time, latest failure category | +| Supervisor invocation intent, pending, and lease tables | Kept; shared by plugin and built-in bindings | +| Columns added for plugins to `pc_artifact_processing_binding_states` (`plugin_id`, `plugin_family`, `binding_generation`, `desired_state`, `phase`, `last_error`) | Removed | +| `pc_artifact_family_schemas` | Removed | +| Search state and vector index data used by custom-family declarative indexes | Removed; the implementation PR drops the tables after confirming built-in families do not depend on them | +| Artifacts, heads, and lineage written by old Artifact plugins | Deleted, not migrated | +| `pc_source_write_plugins` | Removed; write entries come from the plugin directory at startup | +| The Source plugin management service's SQLite | Removed along with the service | +| Plugin-owned indexes and storage | Not part of core; managed by the plugin and deployer | + +A target's binding ID derives deterministically from the plugin name and target key, using a digest when it exceeds the +column width. Additions and removals happen in one new database migration whose notes list the added and dropped +tables, columns, and data ranges. + +## 7. Interface changes + +| Change | Content | +| --- | --- | +| Removed | `/v1/admin/artifact-plugins` and its subpaths; the Source plugin management service, plugin instance Runtime, and poll / reply channel; the `source load` command; MCP `pc_source_instances`, `pc_source_instance_set_enabled` | +| Removed | Declarative indexes for custom families (`ArtifactSearchDeclaration`, `TextIndex`, and related types), `ArtifactFamily` schema declarations, `ArtifactDraft`, and the old Worker plugin executor | +| Added | `GET /v1/plugins`, `GET /v1/plugins/{name}` (`server.admin`, read-only) | +| Added | `POST /v1/scopes/{scope_id}/artifacts/{family}/generate`, `POST /v1/scopes/{scope_id}/sources/plugins/{name}/collect` (`202 Accepted`) | +| Added | `GET` / `POST /v1/scopes/{scope_id}/sources/plugins/{name}/targets`, `PATCH .../targets/{key}` | +| Changed | Generic Artifact read and search interfaces support plugin families; `GET /v1/capabilities` lists plugin families | +| Changed | The write use of MCP `pc_source_add` uses plugin definitions and its collect use adds collection targets; `pc_source_capabilities` lists plugin Source types and target inputs; `pc_source_status` reports targets and their latest collection outcomes; `pc_source_target_set_enabled` acts on targets stored by core | +| Changed | Environment variable `POWERCONTEXT_SERVER_PLUGINS_DIR` replaces `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` and the Source plugin directory settings | + +`openapi/powercontext.yaml` is updated per this table and `src/powercontext/http/_generated/` regenerated. + +## 8. Observability + +Plugin calls reuse the Supervisor and Worker's existing operation vocabulary, execution identities, and outcome fields, +and scheduled calls create no request ID. New fields: + +| Field | Meaning | +| --- | --- | +| `plugin_name` | Plugin name from the manifest | +| `plugin_method` | `collect` or `generate` | +| `trigger` | `schedule`, `sources`, `manual`, `target_added` | +| `operation_count` | Number of write operations (Artifact) or Sources (collect) committed by the call | +| `target_count` | Number of targets carried by a collect call | +| `transaction_ms` | Time from the start of the write transaction block to commit or rollback, in milliseconds | + +Metric labels may use `plugin_name` and `plugin_method`, never Scope IDs or target keys. Logs go through `log_safely` +and exclude Source content, Projections, Artifact content, target inputs, `params`, `config` values, and environment +variable values; plugin exceptions record only their type and known error category, never the exception message. New +fields are registered in the allowlists per the developer guide and documented in both locales' observability +documentation. + +# Drawbacks + +- **Some plugin code runs in the Server process**: write must return synchronously, so Source definitions' `resolve`, + Projections, and `identify_target` do not go through a Worker. They should be pure data transformations, but blocking + or exceptions still affect request handling. Server applies timeouts to these calls and converts exceptions into + request errors. +- **The write transaction holds locks**: the write transaction block holds database locks, and on SQLite it blocks other + writes. A plugin doing slow work inside the block slows writes across Server until the block times out and rolls back. +- **Plugin-owned storage is not guaranteed consistent with core data**: they are not in one transaction, so a failed + commit may leave entries in plugin-owned storage; plugins overwrite them by stable ID and verify against core. +- **Every call starts a Worker and loads the plugin**: child startup and factory initialization have fixed costs, + unsuitable for high-frequency calls every few seconds; plugins cannot reuse connections or caches across calls. +- **Not a security sandbox**: Workers isolate ordinary failures but do not restrict plugin access to files, network, and + credentials, nor provide hard memory or CPU quotas. The writer constrains plugin writes to core data; it cannot stop + plugin code from reaching other resources available to the Worker. +- **Compatible dependencies required**: all plugins share one Python environment with Server; deployers resolve + dependency conflicts. +- **Content structure is not governed by core**: core does not validate `content` schemas; plugins and applications + keep the interpretation of historical revisions compatible themselves. +- **Limited core search**: core provides only generic text search over `search_text`; vector and other search requires + plugin-built indexes. +- **Collection runs with Server**: collection executes in Server Workers, so Server must reach external data sources. +- **Code changes require restarts**: without hot reload, Server is briefly unavailable during plugin updates. + +# Rationale and alternatives + +| Approach | Tradeoff | +| --- | --- | +| Keep the current design and only simplify its interfaces | Limited improvement for plugin authors; the lifecycle state machine, management service, and schema freezing still cost maintenance | +| Run plugins in the Server process | Simplest deployment and calls, but plugin failures can bring Server down; the Supervisor already provides isolation and scheduling, so there is no reason for another mechanism | +| Plugins fully manage their own storage and cursors | Core read interfaces would not work directly, every plugin would reimplement window reads and idempotency, and cursors and results would be hard to commit atomically | +| Plugins write item by item through a context interface, each committed separately, then return success or failure | Writes and cursor advance would be in separate transactions; failures would leave partial results and retries would write again | +| Plugins return drafts, or a writer stages operations in memory and commits them at the end | Commits atomically, but plugins cannot obtain real revisions before commit, making it hard to write their own indexes alongside; merge, merge undo, lifecycle, and other operations would need core's semantics reimplemented in memory; search cannot see staged content | +| Wrap all of `generate` in one transaction | Model calls may take minutes, and holding locks that long blocks other writes, blocking the whole database on SQLite; plugin-owned storage is another database that one transaction could not cover anyway | +| A short write transaction block inside `generate` (this proposal) | Slow work stays outside the block; inside, core's write operations and validation are reused directly, real revisions are returned, and reads see the block's writes; the block commits with the cursor and acknowledgment and rolls back on exceptions. The cost is holding locks inside the block, bounded by a time limit | +| Resident plugin processes (like Dify's plugin daemon) | Would reuse connections and support other languages, but needs process management, serialization, and cancellation protocols. Interfaces are request / response models, so execution can change later without changing plugin code | +| Support hot reload | Reintroduces load states, registration-generation fencing, and reclamation of old calls, conflicting with the goals | + +"The Supervisor dispatches Workers, plugins write directly inside a short write transaction block, and core commits it +with the cursor" was chosen because plugin authors need only a factory and one method; isolation, scheduling, triggers, +backoff, and leadership all reuse the Supervisor already in production; write operations reuse core's existing +implementation instead of a second set of semantics; Sources and Artifacts stay in core's shared storage, so existing +read interfaces, lineage, merge records, and access control apply directly; and core manages only the two kinds of core +data and their progress, leaving the rest to plugins. + +# Prior art + +- **Dify plugins**: interfaces are defined per plugin type (tools, models, agent strategies, extension endpoints, and + others). A plugin declares its type and parameters in a manifest and implements the matching methods; Dify calls them + when a workflow or application needs them. Plugins run in a separate plugin daemon. This proposal keeps the "host + defines interfaces and invocation timing" approach and reuses PowerContext's existing Supervisor Workers for isolation. +- **Airbyte connector protocol**: a Source connector's `read` receives the previous state and emits records and new + state, which the platform persists. Collection checkpoints here use the same division of responsibility, stored per + target. +- **Transaction context managers**: interfaces such as SQLAlchemy's `async with engine.begin()` open a transaction when + the block starts, commit when it ends normally, and roll back on exceptions. `writer.transaction()` follows this usage + and adds the cursor and call acknowledgment to the commit. +- **Existing PowerContext foundations**: the [Artifact processing Supervisor](1515_artifact_processing_supervisor.md)'s + scheduling, invocation intents, Workers, and lease fence; the [shared merge method](1859_artifact_merge.md); + `ArtifactRepository`, `ArtifactGovernanceRepository`, and the handle and tag write implementations; + `SourceDefinition` / `SourceProjection`; the Source journal, `pc_source_cursors`, and `list_window_for_generation`; + `pc_connector_checkpoints`; and the `identify_target` identification rule from the old Source plugin design. + +# Unresolved questions + +| Question | Notes | +| --- | --- | +| Plugin-defined search | The existing search interface accepts only text queries, so core provides generic text search in the first version. Whether plugins may later provide implementations for the public search interface requires deciding which process runs search code | +| Write transaction block limits and defaults | Whether 30 seconds is the right default; whether SQLite and OceanBase use different defaults; per-block operation and byte limits | +| Shared model client | Whether `PluginContext` should expose Server's configured model client so LLM plugins need not bring their own credentials, with usage counted in existing model usage statistics | +| Deleting collection targets | The first version supports only disabling and re-enabling. Whether to fully delete a target and its checkpoint, and how to treat Sources already collected | +| Plugin families in Recall / Prepared Context | Requires candidate content and budget constraints; suggested as a separate proposal | + +# Future possibilities + +- Resident plugin processes that reuse connections and initialization while keeping interfaces unchanged. +- More plugin kinds, such as providing candidates to Recall or text fragments to Prepared Context. +- Plugin packaging and installation tools, such as installing from a Git URL or wheel into the plugin directory. diff --git a/docs/zh/rfcs/0000-plugin-interfaces.md b/docs/zh/rfcs/0000-plugin-interfaces.md new file mode 100644 index 0000000000..adea4fb5c1 --- /dev/null +++ b/docs/zh/rfcs/0000-plugin-interfaces.md @@ -0,0 +1,748 @@ +--- +title: 插件接口 +description: 用少量接口定义 Source 和 Artifact 插件，由 PC Server 通过 Supervisor 在需要时调用；core 只管理 Source 与 Artifact 两类核心数据及其消费进度，不再托管插件生命周期。 +--- + +- Proposal Name: `plugin_interfaces` +- Start Date: 2026-11-08 +- RFC PR: [oceanbase/powercontext#0000](https://github.com/oceanbase/powercontext/pull/0000) + +# 概要 + +本提案重新定义 PowerContext 的第三方插件模型：每类插件只实现几个固定接口，PC Server 在需要时调用。插件是被调用方， +只负责“把输入变成结果”，不参与调度、状态管理和持久化协议。 + +- **Source 插件**提供 Source 定义（沿用现有 `SourceDefinition` / `SourceProjection`），可选实现 `collect`。写入时 Server + 调用定义完成校验、转换和保存。采集按**采集目标**进行，例如“X 上的某个账号”；宿主可以在运行中添加、停用和恢复目标， + 不需要重启。Supervisor 按调度派发 Worker 调用 `collect`，core 把返回的 Source 和各目标的 checkpoint 在同一事务中提交。 +- **Artifact 插件**实现 `generate`。Supervisor 按配置的调度、新 Source 到达或手动请求派发 Worker，Worker 读取该 Scope + 尚未消费的 Source 窗口，连同一个**写入器**（`ArtifactWriter`）交给 `generate`。插件在写入器之外完成模型调用等耗时工作， + 然后在 `async with writer.transaction()` 块中执行 Artifact 写操作：块开始时 core 开启事务，块内的写操作直接作用于 + core 数据并立即返回真实的 revision，块正常结束时 core 把这些写入、消费游标和完成确认一起提交，块内抛出异常则整体回滚。 + 插件读不到也改不了游标。 +- core 只管理两类核心数据：`pc_sources` 和 `pc_artifacts`（含 heads、lineage、合并记录、生命周期、handle 和 tags）， + 以及它们的消费进度。插件可以有自己的索引和存储，由插件自己管理和保证一致性。应用通过现有的 Artifact 公开接口读取 + 和检索插件 family，读取路径不执行插件代码。 +- `collect` 和 `generate` 在 Supervisor 管理的 Worker 子进程中执行，沿用现有的进程隔离、超时、并发预算、领导租约和 + 失败退避。用户只需部署一个 Server。 +- 插件在 Server 启动时按目录加载，代码或清单变更需要重启。Server 不保存插件的启用状态、注册代次或管理阶段，也不提供 + enable / disable / reload / unload 管理接口。 + +本提案取代 [Source 插件框架 RFC #2012](2012-source-plugin-system.md) 和 [Artifact 插件框架 RFC #2175](2175-artifact-plugin-system.md) +的插件契约与管理机制。两者的实现尚未作为稳定接口发布，本提案不保留兼容：旧的插件清单、管理接口、管理服务和插件管理相关的 +表直接移除，旧 Artifact 插件写入的自定义 Artifact 数据不迁移。 + +本文描述目标设计，不代表主线或发布版本已经可用。接口名称和代码示例用于说明契约，最终语法以实现为准。 + +# 动机 + +现有两套插件框架都让 PowerContext 替第三方插件承担了大量生命周期管理： + +| 框架 | Server 为插件承担的工作 | +| --- | --- | +| Artifact 插件（#2175） | 在 `pc_artifact_processing_binding_states` 中维护 `desired_state`、`phase`、`binding_generation`、`last_error`；七种管理状态及其转换；`enable-all / reload / enable / disable / unload` 管理接口与 `expected_generation` 并发校验；用注册代次屏蔽旧 Worker 提交；在 `pc_artifact_family_schemas` 中固化内容 schema 和索引定义，变更返回 409；框架负责自定义声明式索引的构建、重建和可用状态 | +| Source 插件（#2012） | 独立的插件管理服务及其 SQLite（实例、capability、target、期望状态、提交记录）；每个插件实例一个 Runtime 进程；管理服务与 Server 之间的 poll / reply 通道和能力目录同步；`source load` 部署命令；实例启停 MCP 工具 | + +这些机制的代价由所有人承担： + +- **插件作者**需要理解 binding、generation、schema 固化规则、索引声明和管理通道等框架概念，才能写出一个插件。 +- **部署者**需要运行多个进程、维护两个数据库，并通过管理 API 让插件进入可运行状态。 +- **维护者**需要为第三方代码维护状态机、并发屏蔽、schema 和索引治理以及恢复路径，这部分代码的规模和测试面已经超过 + 插件业务本身。 + +插件真正需要的是：在合适的时机被调用，拿到需要的输入，把结果交回 core。调度、隔离、消费进度和事务提交是 core +本来就有的能力，Supervisor 已经为内置 Artifact 提供了这些能力。Dify 等系统的插件模型也说明，宿主定义接口和调用时机、 +插件只实现接口，足以支撑大量第三方扩展。 + +本提案的目标： + +1. 插件作者只需实现少量接口，不需要了解调度、游标和提交协议。 +2. 插件代码在隔离的 Worker 进程中执行，插件故障不影响 Server 和其他插件。 +3. 插件结果进入 `pc_sources` / `pc_artifacts`，与消费进度原子提交，现有读取接口直接可用。 +4. core 只管理这两类核心数据及其消费进度；内容结构、索引和其他存储交给插件自己。 +5. 插件对核心数据的写入只经过 core 提供的写入器，覆盖 Artifact 的全部通用写操作，并沿用 core 现有的校验规则。 +6. Artifact 生成既可以按配置的调度执行，也可以由新 Source 或手动请求触发；采集目标可以在运行中增减。 +7. 用户只需部署一个 Server；插件随 Server 启动加载。 + +# 指南级说明 + +## 1. 核心思路 + +```mermaid +flowchart LR + Host["应用 / Agent / 宿主"] -->|"HTTP / MCP"| Server + subgraph Server["PC Server"] + API["公开接口
认证与 Scope 授权"] + Sup["Supervisor
调度 / 触发 / 预算 / 退避"] + Core[("core 数据
pc_sources / pc_artifacts
采集目标 / 游标 / checkpoint")] + end + subgraph Worker["Worker 子进程（每次调用）"] + Exec["插件执行器"] + Plugin["插件代码
collect() / generate()"] + Tx["writer.transaction()
受约束的写事务"] + Exec -->|"目标与 checkpoint / Source 窗口"| Plugin + Plugin -->|"put / merge / set_lifecycle ..."| Tx + end + API -->|"读取、检索"| Core + API -->|"write、添加目标"| Core + API -->|"手动触发"| Sup + Sup -->|"派发 (插件, Scope)"| Exec + Tx -->|"退出时一并提交写入、游标与确认"| Core + Plugin -.->|"自有索引 / 存储"| Own[("插件自己的存储")] + Plugin -.->|"外部系统 / 模型"| Ext["X、工单系统、LLM……"] +``` + +| 职责 | 归属 | +| --- | --- | +| 认证、Scope 授权、公开 HTTP / MCP 接口、采集目标的增删改查 | Server | +| 何时调用：调度、触发合并、Worker 预算、超时、退避、领导租约 | Supervisor（现有） | +| 读取输入、提供受约束的写事务、在同一事务中提交核心数据 / 游标 / checkpoint / 完成确认 | Worker 中的插件执行器（core） | +| 外部数据读取、采集目标的识别与规范化、Source 定义与 Projection 计算 | Source 插件 | +| Artifact 内容结构、生成算法、插件自有的索引与存储及其一致性 | Artifact 插件 | +| 插件代码、依赖和插件自有存储的部署 | 部署者 | + +插件代码只做一件事：给定输入，产出结果或抛出异常。抛出异常即失败，core 不提交任何核心数据，下次重试同一输入。 + +## 2. 插件目录与清单 + +Server 从 `POWERCONTEXT_SERVER_PLUGINS_DIR` 加载插件。每个直接子目录是一个插件，包含 `plugin.yml` 和代码： + +```text +/opt/powercontext/plugins/ +├── x-posts/ +│ ├── plugin.yml +│ ├── definition.py +│ └── plugin.py +└── incident-report/ + ├── plugin.yml + └── plugin.py +``` + +```yaml +name: incident-report # 插件名，部署内唯一，与目录名一致 +kind: artifact # source 或 artifact +entry: ./plugin.py:create # 工厂函数，路径相对插件目录 +scopes: [team-a] # 可选；省略时处理所有存在匹配输入的 Scope +config: # 业务配置，由插件的 Config 模型校验 + title_prefix: 工单报告 + index_url: ${REPORT_INDEX_URL} # ${NAME} 在 Worker 中从环境变量解析 +``` + +清单只描述**插件是什么、怎样运行**，不包含启用状态。目录存在且加载成功，插件就可用；需要停用时移出目录并重启 Server。 +同一份代码可以通过多个插件目录、不同的 `name` 和 `config` 部署多次。 + +## 3. Source 插件示例 + +Source 插件导出 Source 定义。只提供定义时，宿主可以通过现有的 Source 写入接口（MCP `pc_source_add` 的 write 用法） +提交完整内容，Server 调用定义的 `resolve` 和各 Projection，然后保存到 `pc_sources`。 + +需要从外部系统持续采集时，插件再实现 `collect`。采集以**采集目标**为单位：每个目标是插件能采集的一项外部资源，例如 +X 上的一个账号。下面的插件每 5 分钟采集各目标账号的新帖子： + +```yaml +name: x-posts +kind: source +entry: ./plugin.py:create +scopes: [team-a] +collect: + schedule: {interval_seconds: 300} + max_workers: 1 + timeout_seconds: 120 + max_items: 200 + targets: # 可选；首次启动时导入的初始目标 + - {scope_id: team-a, input: {account: "@oceanbase"}} +config: + bearer_token: ${X_BEARER_TOKEN} +``` + +```python +from pydantic import BaseModel + +from powercontext.plugins import ( + CollectedItem, CollectRequest, CollectResult, PluginContext, TargetIdentity, +) + +from .definition import PostCapture, XPostDefinition + +definitions = (XPostDefinition(),) + + +class Config(BaseModel): + bearer_token: str + + +class TargetInput(BaseModel): # 宿主添加目标时提交的输入 + account: str + + +def identify_target(value: TargetInput) -> TargetIdentity: + account = value.account.strip().removeprefix("@").lower() + if not account: + raise ValueError("account must not be blank") + return TargetIdentity(key=f"account:{account}", input={"account": account}) + + +class XCollector: + def __init__(self, client: XClient) -> None: + self.client = client + + async def collect(self, request: CollectRequest) -> CollectResult: + result = CollectResult() + for target in request.targets: + since = (target.checkpoint or {}).get("since_id") + page = await self.client.user_posts(target.input["account"], since_id=since) + result.add(target, [CollectedItem(definition="x-post", value=PostCapture(**p)) for p in page.posts], + checkpoint={"since_id": page.newest_id or since}) + return result + + +async def create(ctx: PluginContext, config: Config) -> XCollector: + client = await ctx.enter_async_context(XClient(config.bearer_token)) + return XCollector(client) +``` + +宿主在运行中添加一个目标，无需重启： + +```text +pc_source_add(scope_id="team-a", capability="x-posts", input={"account": "@PowerContextAI"}) +``` + +Server 用 `TargetInput` 校验输入，调用 `identify_target` 得到稳定的目标键和规范化输入，保存目标后立即请求一次采集。 +重复添加同一账号得到同一个键，复用已有目标；目标可以停用和恢复，进度保留。 + +每个目标的 checkpoint 是只读输入，插件通过返回值提出新的 checkpoint。core 把本次返回的所有 Source 与各目标的新 +checkpoint 在同一事务中提交；`collect` 抛出异常或超时则不提交任何内容，下一次调用仍拿到旧 checkpoint。 + +## 4. Artifact 插件示例 + +下面的插件为每条工单 Source 生成一份报告：先用自己的向量索引查找相似的已有报告，命中则修订那份报告，否则新建； +同时把报告写进自己的向量索引。 + +```yaml +name: incident-report +kind: artifact +entry: ./plugin.py:create +scopes: [team-a] +generate: + family: incident-report + inputs: + source_types: [example-ticket] + projection: powercontext.text-evidence@1 + max_sources: 50 + schedule: {interval_seconds: 300} # 定时检查 + on_sources: true # 新 Source 到达后触发 + manual: true # 允许手动触发 + max_workers: 1 + timeout_seconds: 300 # 整次调用的时限 + transaction_timeout_seconds: 30 # 写事务块的时限 +config: + title_prefix: 工单报告 + index_url: ${REPORT_INDEX_URL} +``` + +```python +from pydantic import BaseModel + +from powercontext.plugins import ArtifactWriter, GenerateRequest, PluginContext +from powercontext.sources import TextEvidence + + +class Config(BaseModel): + title_prefix: str = "工单报告" + index_url: str + + +class IncidentReports: + def __init__(self, config: Config, index: VectorIndex, model: Summarizer) -> None: + self.config, self.index, self.model = config, index, model + + async def generate(self, request: GenerateRequest, writer: ArtifactWriter) -> None: + # 1. 耗时工作放在写事务之外：模型调用、向量计算 + planned = [] + for source in request.sources: + evidence = TextEvidence.model_validate(source.projection) + summary = await self.model.summarize(evidence.content) + vector = await self.model.embed(summary) + planned.append((source, summary, vector)) + + # 2. 短写事务：块内读取能看到本块已写入的内容 + async with writer.transaction() as tx: + for source, summary, vector in planned: + match = await self.index.nearest(request.scope_id, vector, threshold=0.92) + current = await tx.get(match.artifact_id) if match else None + artifact_id = current.ref.artifact_id if current else writer.stable_id( + "report", source.ref.source_type, source.ref.source_id) + ref = await tx.put( + artifact_id, + content={"title": self.config.title_prefix, "summary": summary}, + search_text=summary, + source_refs=[source.ref], + ) + # 拿到真实的 artifact_id 与 revision 后写入自有索引，按 ID upsert + await self.index.upsert(request.scope_id, ref.artifact_id, ref.revision, vector) + # 块正常结束：core 提交 Artifact 写入、游标推进与完成确认 + + +async def create(ctx: PluginContext, config: Config) -> IncidentReports: + index = await ctx.enter_async_context(VectorIndex.connect(config.index_url)) + return IncidentReports(config, index, Summarizer()) +``` + +块内抛出异常时，core 回滚块内全部 Artifact 写入，游标不动，下次重试同一窗口。插件自有索引不在 core 的事务中： +插件按稳定 ID upsert，重试时会覆盖上一次留下的条目；读取自有索引时以 core 的当前 revision 为准（见参考级说明 1.6）。 + +应用通过现有的 Artifact 公开接口访问插件 family，这些接口直接读取 `pc_artifacts`，不执行插件代码： + +| 用途 | 接口 | +| --- | --- | +| 浏览 | `GET /v1/scopes/{scope_id}/artifacts/{family}` | +| 读取当前内容 | `GET /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | +| 读取指定 revision / revision 历史 | `GET .../{artifact_id}/revisions/{revision}`、`GET .../{artifact_id}/revisions` | +| 读取 tags | `GET .../{artifact_id}/tags` | +| 文本检索 | `POST /v1/scopes/{scope_id}/artifacts/{family}/search` | +| 手动触发生成 | `POST /v1/scopes/{scope_id}/artifacts/{family}/generate` | + +## 5. 调度与触发 + +插件的调度和触发都交给现有的 Artifact processing Supervisor。每个插件在启动时注册为一个 Supervisor binding， +binding 名为 `plugin:`，调度键仍是 `(binding, Scope)`。 + +Artifact 生成支持三种调用来源，可以任意组合，至少配置一种： + +| 配置 | 何时派发 Worker | `request.reason` | +| --- | --- | --- | +| `schedule: {interval_seconds: N}` 或 `schedule: {cron: "0 2 * * *", timezone: Asia/Shanghai}` | 到期时，对有待处理工作的 Scope | `schedule` | +| `on_sources: true` | Scope 的 Source journal 超过该插件的游标 | `sources` | +| `manual: true` | 调用手动触发接口 | `manual` | + +- 配置了 `inputs` 的插件，每次调用都附带该 Scope 尚未消费的 Source 窗口；窗口为空时，定时和 Source 触发不调用插件， + 直接确认完成。 +- 未配置 `inputs` 的插件（例如每天整理一次已有 Artifact）不消费 Source，到期即调用；必须显式配置 `scopes`。 +- 手动触发总是调用插件，可以附带 `params`，由插件自行解释。手动请求在接受时持久化，返回 `202 Accepted`， + Server 重启后仍会执行。 + +Source 插件的 `collect` 支持 `schedule` 和 `manual`（`POST /v1/scopes/{scope_id}/sources/plugins/{name}/collect`）， +添加或恢复目标时也会请求一次采集。 + +同一插件、同一 Scope 同时最多运行一个 Worker。运行期间到达的触发由 Supervisor 合并成结束后的一次调用。 + +## 6. 运维 + +- 启动：Server 逐个检查插件目录。某个插件检查失败（清单无效、导入失败、缺少必需导出、family 或 Source 类型与内置或其他 + 插件冲突）只影响该插件，Server 和其他插件正常启动。 +- 查看：`GET /v1/plugins`（需要 `server.admin`）列出每个插件的检查结果、类型、family 或 Source 定义、调度配置， + 以及每个 Scope 的最近一次调用结果和下次重试时间。采集目标及其最近采集结果通过 `pc_source_status` 和目标接口查看。 +- 变更：修改代码或清单后重启 Server。Server 不监视文件，也不在运行中重新加载插件。采集目标是业务数据，增减不需要重启。 +- 停用：从插件目录移走后重启。已保存的 Source、Artifact、采集目标、游标和 checkpoint 保留并可继续读取；重新部署 + 同名插件时从原位置继续。插件自有存储由部署者保留或清理。 + +# 参考级说明 + +## 1. 接口契约 + +公开接口位于新模块 `powercontext.plugins`，替换现有的 `powercontext.artifact_plugins` 和 `powercontext.source_management` +中面向插件作者的部分。接口以 `typing.Protocol` 和 Pydantic 模型表达。 + +### 1.1 入口模块、工厂与上下文 + +入口模块导出： + +| 导出 | 必需 | 说明 | +| --- | --- | --- | +| 工厂函数（`entry` 指定） | 是 | `async def create(ctx, config) -> 插件实例` | +| `Config` | 否 | Pydantic 模型，校验清单中的 `config`；未导出时 `config` 必须为空 | +| `definitions` | Source 插件必需 | `SourceDefinition` 序列 | +| `TargetInput`、`identify_target` | 配置了 `collect` 的 Source 插件必需 | 采集目标的输入模型与识别函数 | + +```python +class PluginContext: + name: str # 清单中的插件名 + scope_id: str # 本次调用的 Scope + plugin_dir: Path # 插件目录（只读约定） + logger: logging.Logger # 已绑定插件名的 logger + + async def enter_async_context(self, cm: AbstractAsyncContextManager[T]) -> T: ... +``` + +- 工厂在每次 Worker 调用中执行一次，实例不跨调用复用。需要关闭的资源（包括插件自有存储的连接）通过 + `ctx.enter_async_context` 注册，调用结束时由执行器释放。 +- `${NAME}` 环境变量引用在 Worker 中解析后再用 `Config` 校验，凭据不进入 Supervisor 的派发参数和日志。 +- 插件得不到 core 数据库的连接，也没有游标或 checkpoint 的写入接口。对核心数据的写入只有两条路径：Source 插件的 + `collect` 返回值，和 Artifact 插件的 `ArtifactWriter`。 + +### 1.2 Source 插件 + +```python +class TargetIdentity(BaseModel): + key: str # 稳定的目标键，同一外部资源始终相同 + input: dict[str, JsonValue] # 规范化后的输入，保留所有影响采集的参数 + +class CollectTarget(BaseModel): + key: str + input: dict[str, JsonValue] + checkpoint: JsonValue | None # 该目标上次成功提交的 checkpoint；首次为 None；只读 + +class CollectRequest(BaseModel): + scope_id: str + reason: Literal["schedule", "manual", "target_added"] + targets: Sequence[CollectTarget] # 本次要采集的已启用目标 + max_items: int + params: JsonValue | None = None # 仅手动触发 + +class CollectedItem(BaseModel): + definition: str # definitions 中某个定义的 name + value: BaseModel # 该定义 input_class 的实例 + +class CollectResult(BaseModel): + targets: dict[str, TargetResult] = {} # 键为目标键 + def add(self, target: CollectTarget, items: Sequence[CollectedItem], *, checkpoint: JsonValue | None) -> None: ... + +class TargetResult(BaseModel): + items: Sequence[CollectedItem] + checkpoint: JsonValue | None + has_more: bool = False + +class SourceCollector(Protocol): + async def collect(self, request: CollectRequest, /) -> CollectResult: ... +``` + +- 只提供 `definitions`、不配置 `collect` 的插件只支持 write。工厂只在 collect 调用时执行。 +- `definitions` 中的 Source 类型与 Projection 在启动时注册到现有的 Source 定义清单，与内置类型和其他插件冲突时检查失败。 +- **write** 不需要插件方法。Server 按 `definition.input_class` 校验输入，调用 `resolve` 得到 Source，计算各 Projection 后 + 保存，语义与现有内置 Source 写入一致。 +- **collect** 每次调用对应一个 Scope，最多携带 `max_targets`（清单配置，默认 50）个已启用目标；目标较多时按目标键轮转， + `has_more` 的目标优先进入下一次调用。执行器对返回的每条 Source 执行与 write 相同的校验、`resolve` 和 Projection， + 然后在一个事务中保存全部 Source、写入各目标的 checkpoint 和最近采集时间，并确认本次调用。任一条失败则整次不提交。 + 条目总数超过 `max_items`、返回了请求中不存在的目标键，都视为失败，不截断。 +- 结果中没有出现的目标，checkpoint 保持不变。 +- Source 身份由 `resolve` 决定。重复采集同一外部状态得到相同身份时，沿用现有去重规则，不产生重复 Source。 + +### 1.3 采集目标 + +采集目标是 core 保存的业务数据，与 Source 一样不属于插件生命周期状态： + +| 操作 | 接口 | 说明 | +| --- | --- | --- | +| 添加 | MCP `pc_source_add`（collect 用法）、`POST /v1/scopes/{scope_id}/sources/plugins/{name}/targets` | 用 `TargetInput` 校验输入，调用 `identify_target` 得到键和规范化输入。同键且规范化输入相同则返回已有目标；同键但输入不同返回冲突。新目标保存为启用状态并请求一次采集 | +| 查看 | MCP `pc_source_status`、`GET /v1/scopes/{scope_id}/sources/plugins/{name}/targets` | 目标键、规范化输入、启用状态、最近成功采集时间、最近失败分类 | +| 停用 / 恢复 | MCP `pc_source_target_set_enabled`、`PATCH .../targets/{key}` | 停用后不再进入 collect 请求，checkpoint 和已采集的 Source 保留；恢复后从原 checkpoint 继续 | + +- 添加和变更需要调用者对该 Scope 有 Source 写权限，并且 Scope 在插件的 `scopes` 范围内。 +- `identify_target` 必须是纯函数：只做校验和规范化，不访问外部系统、不检查外部授权。外部资源是否存在、是否可读， + 在 collect 时由插件判断并以异常或空结果表达。它与 write 一样在 Server 进程中同步执行，有超时限制。 +- 清单中的 `collect.targets` 只在对应键不存在时导入一次。之后以 core 中的数据为准，修改清单不会覆盖或删除已有目标。 + +### 1.4 Artifact 插件、写入器与写事务 + +```python +class ArtifactGenerator(Protocol): + async def generate(self, request: GenerateRequest, writer: ArtifactWriter, /) -> None: ... + +class GenerateRequest(BaseModel): + scope_id: str + reason: Literal["schedule", "sources", "manual"] + sources: Sequence[GenerationSource] # 本次窗口；未配置 inputs 时为空 + params: JsonValue | None = None # 仅手动触发 + +class GenerationSource(BaseModel): + ref: SourceRef # 精确 Source 引用 + projection: JsonValue # 清单 inputs.projection 指定的 Projection 内容 + +class ArtifactWriter(ArtifactReads): + family: str # 固定为清单中的 generate.family + + def stable_id(self, *parts: str) -> str: ... + def transaction(self) -> AbstractAsyncContextManager[ArtifactTransaction]: ... + +class ArtifactTransaction(ArtifactReads): + ... # 读取（见 1.5）加写操作（见下表） +``` + +`generate` 的执行分两部分： + +- **事务外**：插件通过 `writer` 读取已提交的数据（每次读取使用独立的只读短事务），完成模型调用、向量计算等耗时工作。 +- **写事务块**：`async with writer.transaction() as tx:`。块开始时，core 开启写事务并重新校验 Supervisor 租约和调用意图； + 块内的写操作直接在该事务中执行，立即返回真实的 `ArtifactRef`（含 revision），块内读取能看到本块已写入的内容； + 块正常结束时，core 在同一事务中以 CAS 推进游标、确认本次调用并提交；块内抛出异常、超过 `transaction_timeout_seconds` + 或提交冲突，事务整体回滚，本次调用失败。 + +规则： + +- 每次调用最多开启一个写事务块；第二次开启抛出异常。游标和调用确认只随这一个事务提交，因此一个窗口的核心数据不会 + 被部分提交。 +- 插件没有开启写事务块就正常返回，表示本窗口无需产出，执行器单独提交游标推进和调用确认。 +- 写事务块提交后，`generate` 中后续代码抛出的异常不再影响已提交的结果，执行器记录为提交后错误，不重试该窗口。 + 插件应把写事务块作为 `generate` 的最后一步。 +- 写事务块的时限默认 30 秒，计入整次调用的 `timeout_seconds`。块内只应做读取、写操作和插件自有存储的写入， + 不应调用模型。 + +**写操作**：写事务覆盖 Artifact 的全部通用写操作，每个操作复用 core 现有的实现与校验规则，违反规则时立即抛出异常： + +| 操作 | 复用的 core 实现 | 语义 | +| --- | --- | --- | +| `put(artifact_id, *, content, search_text=None, source_refs=(), artifact_refs=())` | `ArtifactRepository.create` / `revise` | 不存在则新建 revision 1；已存在则按当前 head 追加 revision。内容与 `search_text` 和当前 revision 相同时为 no-op，返回当前引用 | +| `revert(artifact_id, *, to_revision, reason)` | `revise` 的回滚路径 | 以历史 revision 的内容追加新 revision，记录 `restored_from_revision` 与原因 | +| `set_lifecycle(artifact_id, state, *, replacement_artifact_id=None)` | `ArtifactGovernanceRepository.transition` | active ↔ deprecated（可指定替代者），任意状态 → retired（不可逆）。修改 head 的治理状态，不追加 revision；被合并冻结的输入不能单独变更 | +| `merge(inputs, *, into, content, search_text=None, source_refs=())` | 公共合并方法（[RFC #1859](1859_artifact_merge.md)） | 以 `inputs` 的当前 revision 为输入，新建 `into`，记录合并关系并冻结输入。`into` 必须是新 ID | +| `preview_undo_merge(artifact_id)` / `undo_merge(artifact_id, *, preview=None)` | 公共合并方法的预览与恢复 | 撤销产生该 Artifact 的合并，恢复输入并退役结果；按预览执行时，相关状态变化则拒绝过期预览 | +| `bind_handle(handle_key, ref)` | Artifact handle 绑定 | 把稳定 handle 绑定到指定 revision | +| `set_tags(artifact_id, tags)` | Artifact tags 替换 | 整体替换该 Artifact 的 tags | + +**写入约束**： + +- 只能操作本 Scope、清单 `generate.family` 中的 Artifact，插件不能指定其他 family 或 Scope。 +- `source_refs` 只能引用本次窗口中的 Source；`artifact_refs`、`to_revision` 和 `bind_handle` 的目标只能是本 family + 中实际存在的 revision。 +- 每个写事务块的操作数、单份内容大小和总字节数有上限，超出即抛出异常并回滚，不截断、不部分提交。 +- core 不校验 `content` 的业务结构，也不保存或固化 schema。内容结构的演进由插件负责，插件可以在 `content` 中自带版本字段。 + +### 1.5 读取 + +`ArtifactWriter`（事务外，只读已提交数据）和 `ArtifactTransaction`（事务内，同时看到本块已写入的内容）提供相同的读取方法， +范围固定为本 Scope、本 family： + +```python +class ArtifactReads(Protocol): + async def get(self, artifact_id: str, *, revision: int | None = None) -> ArtifactSnapshot | None: ... + async def revisions(self, artifact_id: str, *, cursor: str | None = None, limit: int = 50) -> RevisionPage: ... + async def list(self, *, states: Collection[LifecycleState] = ("active",), tags: Collection[str] = (), + cursor: str | None = None, limit: int = 50) -> ArtifactPage: ... + async def search(self, query: str, *, states: Collection[LifecycleState] = ("active",), + limit: int = 10) -> Sequence[ArtifactSnapshot]: ... + async def by_source(self, ref: SourceRef, *, cursor: str | None = None, limit: int = 50) -> ArtifactRefPage: ... + async def lineage(self, ref: ArtifactRef) -> ArtifactLineage: ... + async def governance(self, artifact_id: str) -> ArtifactGovernance: ... # 生命周期、替代者、合并归属 + async def merge_record(self, artifact_id: str) -> ArtifactMergeRecord | None: ... + async def tags(self, artifact_id: str) -> ArtifactTagSet: ... + async def resolve_handle(self, handle_key: str) -> ArtifactRef | None: ... +``` + +- `get` 可以读当前内容或指定 revision；`revisions` 列出完整历史。 +- `list` 和 `search` 可以按生命周期状态过滤，包括 deprecated、retired 和被合并冻结的 Artifact，用于查找“之前已经存在的 + 数据”。`search` 是 core 提供的文本检索（见第 5 节），在写事务中能检索到本块已写入、尚未提交的内容。 +- `by_source` 反查 lineage 引用了某条 Source 的 Artifact，用于 Source 更新后找到需要修订的 Artifact。 +- 读取返回的快照都带精确 revision，可以直接作为写操作的 `artifact_refs` 或 `to_revision`。 +- 插件需要向量相似度等 core 不提供的检索时，使用自己的索引，再用 `get` 以 core 中的当前 revision 为准。 + +### 1.6 一致性与幂等 + +**核心数据**：每个窗口的 Artifact 写操作、游标推进和调用确认在一个事务中提交。失败时核心数据完全不变，重试从干净状态 +开始；成功后游标已越过该窗口，同一窗口不会再次投递。core 数据对每个窗口恰好提交一次。写事务内的操作直接作用于数据库， +冲突（例如 head 在读取后被并发修改）在操作时或提交时立即报告，整体回滚后由 Supervisor 重新调度。插件 family 第一版不开放 +公开写接口，同一插件同一 Scope 又是单飞执行，因此这类冲突只在异常情况下出现。 + +**重复处理同一业务对象**：手动重跑、未配置 `inputs` 的定时整理等场景会再次处理已经处理过的对象。插件用 +`writer.stable_id(...)` 从业务身份派生 Artifact ID，`put` 在内容相同时为 no-op，`set_lifecycle` 目标状态与当前相同时 +为 no-op，`merge` 在输入已全部合并到同一结果时为 no-op，从而不会产生重复 Artifact 或无意义的 revision。 + +**插件自有存储**：插件的索引和其他存储与 core 是不同的数据库，不在 core 的事务中，core 不保证两者一致，由插件负责。 +推荐做法： + +1. 在写事务块内、拿到 `ArtifactRef` 之后写自有存储，按 `artifact_id`（必要时加 revision）upsert。这样自有存储的写入 + 失败会让核心写入一起回滚。 +2. 写事务提交失败时，自有存储可能留下本次的条目。重试会处理同一窗口、得到同一个稳定 ID，覆盖这些条目。 +3. 从自有存储读出的结果，用 `get` 校验 core 中是否存在、revision 是否一致，忽略不一致的条目，或定期按 core 数据重建。 + +`generate` 本身可能因失败重试而执行多次，模型调用和插件发出的外部请求不是恰好一次。 + +## 2. 插件执行器与消费游标 + +插件执行器是 Worker 中的 core 代码，实现 Supervisor 的 Worker 入口，替代旧的 `run_artifact_plugin_worker`。 +它对 Artifact 插件的一次调用如下： + +```text +Worker 收到 (plugin:, scope_id, request_generation) + -> 短事务：ScopeInvocation.start；读取 pc_source_cursors 中 (scope_id, plugin:) 的游标 + -> 读取 journal 高水位，确定窗口 (after, through]，through ≤ after + max_sources + -> list_window_for_generation 读取窗口，按 inputs.source_types 过滤，计算 inputs.projection + -> 加载插件、调用工厂、创建写入器、调用 generate（受 timeout_seconds 约束） + -> 事务外读取：独立只读短事务 + -> 写事务块开始：开启事务；guard（租约与调用意图） + 块内写操作在该事务中执行 + 块结束：游标 CAS 到 through；acknowledge；提交 + -> generate 返回且未开启写事务块：短事务完成游标 CAS 与 acknowledge +``` + +- 窗口内类型不匹配的 Source 被过滤，但游标仍推进过整个窗口；匹配类型的 Source 缺少所需 Projection 时调用失败，不跳过。 +- 游标只在执行器的提交事务中推进，插件得不到游标值，也没有写入接口。游标 CAS 保证在读取窗口与提交之间即使发生 + 领导权变更或重复派发，同一窗口也只提交一次。 +- `on_sources` 复用现有的 `SourceProcessingPendingProvider`：journal 高水位大于游标时标记待处理，由 Supervisor 按预算派发。 + 判断基于持久化数据，Server 重启后不会丢失待处理输入。 +- 手动触发和添加采集目标复用现有的调用意图（`ArtifactProcessingIntentRepository.request`）：在接受事务中提升 + requested generation，多次请求合并，运行期间的新请求由后续调用处理。 +- 未配置 `scopes` 时，自动调度对所有 journal 超过游标的 Scope 生效；配置了 `scopes` 时，自动调度和手动触发都只接受 + 名单内的 Scope。 + +collect 调用结构相同：输入换成该 Scope 的已启用目标及其 checkpoint，提交内容换成 Source 与各目标的新 checkpoint。 +collect 不需要写事务块，由执行器根据返回值一次提交。 + +## 3. 加载与生命周期 + +```text +Server 启动 + -> 扫描 POWERCONTEXT_SERVER_PLUGINS_DIR 的直接子目录（未设置则不加载插件） + -> 对每个目录：解析并校验清单（name 与目录名一致、kind 与配置段匹配） + -> 在有时限的检查子进程中导入入口，确认工厂、Config 和其他必需导出符合 kind，不调用工厂 + -> Source 插件：在 Server 进程导入 definitions、TargetInput 与 identify_target，注册 Source 类型 + -> 导入清单中尚不存在的初始采集目标 + -> 注册 family 的访问 profile；有 collect / generate 配置时注册 Supervisor binding + -> 任一步失败：记录错误，该插件标记为 failed，继续下一个 +Server 停止 + -> 按现有 Supervisor 关闭流程停止派发并回收 Worker +``` + +插件只有 `loaded` 和 `failed` 两种结果，只存在于进程内存中。Server 不持久化插件的启用状态、注册代次或配置快照； +每次启动按当时的目录内容重新检查。没有热加载，因此不需要注册代次来屏蔽旧 Worker，现有的 Supervisor 租约 fence +已经足以屏蔽领导权变更后的旧提交。 + +Worker 每次调用重新导入插件代码。插件目录在 Server 运行期间被修改时，新派发的 Worker 可能读到新代码；部署者应在 +修改后重启 Server，第一版不检测这种情况。 + +## 4. 执行约束 + +- **隔离**：`collect` / `generate` 在 Supervisor 启动的 Worker 子进程中执行，崩溃、内存泄漏和阻塞不影响 Server。 + 子进程隔离普通故障，不是不可信代码的沙箱：插件以 Worker 的操作系统权限运行，可以访问其文件、网络和环境变量。 +- **超时与预算**：`timeout_seconds`（默认 300）是 Worker 总时限，`transaction_timeout_seconds`（默认 30）是写事务块时限， + `max_workers`（默认 1）是该插件跨 Scope 的并发上限。前两者和并发上限映射到 Supervisor binding 的现有参数或由执行器强制。 + 单次调用的输入字节、写操作数和输出字节有上限，超出即失败。 +- **写事务的锁影响**：写事务块持有期间会锁住涉及的 head 和游标行；在 SQLite 上，写事务会阻塞整个数据库的其他写入。 + 因此写事务块必须短，时限超出即回滚。 +- **失败与退避**：调用失败时不提交任何核心数据，该插件该 Scope 进入 Supervisor 现有的退避，不影响其他 Scope 和插件。 + core 不会因为重复失败自动跳过输入；持续失败需要修复插件或数据后重启。collect 中某个目标持续失败会阻塞同一 Scope + 的其他目标，可以先停用该目标。 +- **多实例部署**：OceanBase 多 Server 实例时，派发只发生在持有 Supervisor 领导租约的实例上；write、目标管理和读取 + 可以在任意实例执行。插件自有存储需要能被执行 Worker 的实例访问。SQLite 部署只有一个 Server 进程。 + +## 5. 公开读取与检索 + +插件 family 的公开读取完全由 core 处理，不执行插件代码： + +- list / get / revisions / tags 复用现有的通用 Artifact 读取路径。插件 family 的 `content` 按原样 JSON 返回，不需要 + `pc_artifact_family_schemas` 中的 schema。被合并冻结、deprecated 和 retired 的 Artifact 按公共合并与生命周期规则展示。 +- search 是 core 为插件 family 新增的通用文本检索，与现有检索接口一样只接受文本 query：写入时把 `search_text` + 经现有词法分析写入 `pc_artifact_heads.searchable_text`，检索时在该列上匹配，只返回请求 Scope 中调用者有权读取的 + revision。未提供 `search_text` 的 Artifact 不参与检索。写入器的 `search` 使用同一实现。实现 PR 需要给出 SQLite 与 + OceanBase 上的检索方式和规模验证。 +- Review、Recall 和 Prepared Context 第一版不对插件 family 开放；公开的创建、修订、合并、生命周期、handle 和 tags 写接口 + 也不对插件 family 开放，插件 family 的写入只来自插件。对应接口返回明确的“不支持”错误，`GET /v1/capabilities` + 反映实际支持的操作。 + +## 6. 持久化 + +| 数据 | 处理 | +| --- | --- | +| `pc_sources`、Projection、Source journal、`pc_source_definition_manifests` | 保留；Source 插件的结果 | +| `pc_artifacts`、`pc_artifact_heads`、`pc_artifact_lineage_*`、合并记录、handle、tags | 保留；Artifact 插件的结果 | +| `pc_source_cursors` | 保留；Artifact 插件的消费游标，键为 `(scope_id, plugin:)` | +| `pc_connector_checkpoints` | 保留；各采集目标的 checkpoint，键为 `(scope_id, 目标的 binding ID)` | +| `pc_source_collect_targets` | 新增；采集目标：Scope、插件名、目标键、规范化输入、启用状态、创建与更新时间、最近成功采集时间、最近失败分类 | +| Supervisor 的调用意图、pending、租约表 | 保留；插件 binding 与内置 binding 共用 | +| `pc_artifact_processing_binding_states` 中为插件新增的列（`plugin_id`、`plugin_family`、`binding_generation`、`desired_state`、`phase`、`last_error`） | 移除 | +| `pc_artifact_family_schemas` | 移除 | +| 自定义 family 声明式索引使用的搜索状态与向量索引数据 | 移除；实现 PR 确认内置 family 不依赖后删除对应表 | +| 旧 Artifact 插件写入的 Artifact、head、lineage | 删除，不迁移 | +| `pc_source_write_plugins` | 移除；write 入口改由插件目录在启动时提供 | +| Source 插件管理服务的 SQLite | 随管理服务一起移除 | +| 插件自有的索引与存储 | 不属于 core，由插件和部署者管理 | + +目标的 binding ID 由插件名和目标键确定性派生，长度超过列宽时使用其摘要。新增与移除通过一次新的数据库迁移完成， +迁移说明中列出新增、删除的表、列和数据范围。 + +## 7. 接口变更 + +| 变更 | 内容 | +| --- | --- | +| 移除 | `/v1/admin/artifact-plugins` 及其子路径；Source 插件管理服务、插件实例 Runtime、poll / reply 通道；`source load` 命令；MCP `pc_source_instances`、`pc_source_instance_set_enabled` | +| 移除 | 自定义 family 的声明式索引（`ArtifactSearchDeclaration`、`TextIndex` 等）、`ArtifactFamily` 的 schema 声明、`ArtifactDraft` 和旧的 Worker 插件执行器 | +| 新增 | `GET /v1/plugins`、`GET /v1/plugins/{name}`（`server.admin`，只读） | +| 新增 | `POST /v1/scopes/{scope_id}/artifacts/{family}/generate`、`POST /v1/scopes/{scope_id}/sources/plugins/{name}/collect`（`202 Accepted`） | +| 新增 | `GET` / `POST /v1/scopes/{scope_id}/sources/plugins/{name}/targets`、`PATCH .../targets/{key}` | +| 调整 | 通用 Artifact 读取与检索接口支持插件 family；`GET /v1/capabilities` 列出插件 family | +| 调整 | MCP `pc_source_add` 的 write 用法使用插件定义，collect 用法添加采集目标；`pc_source_capabilities` 列出插件 Source 类型与目标输入；`pc_source_status` 报告目标及其最近采集结果；`pc_source_target_set_enabled` 改为作用于 core 保存的目标 | +| 调整 | 环境变量 `POWERCONTEXT_SERVER_PLUGINS_DIR` 取代 `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` 和 Source 插件目录配置 | + +`openapi/powercontext.yaml` 按上表更新后重新生成 `src/powercontext/http/_generated/`。 + +## 8. 可观测性 + +插件调用复用 Supervisor 与 Worker 现有的操作词汇、执行身份和结果字段，不为定时调用生成 request ID。新增字段： + +| 字段 | 含义 | +| --- | --- | +| `plugin_name` | 清单中的插件名 | +| `plugin_method` | `collect` 或 `generate` | +| `trigger` | `schedule`、`sources`、`manual`、`target_added` | +| `operation_count` | 本次提交的写操作数（Artifact）或 Source 条数（collect） | +| `target_count` | 本次 collect 携带的目标数 | +| `transaction_ms` | 写事务块从开始到提交或回滚的耗时，毫秒 | + +指标标签可以使用 `plugin_name` 和 `plugin_method`，不包含 Scope ID 或目标键。日志经 `log_safely` 输出，不包含 Source +内容、Projection、Artifact 内容、目标输入、`params`、`config` 值和环境变量值；插件异常只记录类型和已知错误分类， +不记录异常消息。新增字段按开发指南登记到允许列表，并在中英文可观测性文档中说明。 + +# 缺点 + +- **部分插件代码在 Server 进程中执行**：write 需要同步返回，Source 定义的 `resolve`、Projection 和 `identify_target` + 不经过 Worker。它们应当是纯数据转换，但阻塞或异常仍会影响请求处理。Server 对这些调用设置超时并把异常转换为请求错误。 +- **写事务持锁**：写事务块期间持有数据库锁，SQLite 上会阻塞其他写入。插件若在块内执行慢操作，会拖慢整个 Server 的写入， + 直到块超时回滚。 +- **插件自有存储与核心数据不保证一致**：两者不在同一事务中，提交失败可能在自有存储中留下条目，插件需要按稳定 ID 覆盖 + 并以 core 为准校验。 +- **每次调用都要启动 Worker 并加载插件**：子进程启动和工厂初始化有固定开销，不适合秒级高频调用；插件不能在调用之间 + 复用连接或缓存。 +- **不是安全沙箱**：Worker 隔离普通故障，不限制插件的文件、网络和凭据访问，也不提供硬性的内存或 CPU 配额。 + 写入器约束的是插件对核心数据的写入，不能阻止插件代码访问 Worker 可达的其他资源。 +- **依赖必须兼容**：所有插件与 Server 共用一个 Python 环境，依赖冲突需要部署者解决。 +- **内容结构不受 core 治理**：core 不校验 `content` 的 schema，历史 revision 的解释需要插件和应用自行保持兼容。 +- **core 检索能力有限**：core 只提供基于 `search_text` 的通用文本检索；向量等检索需要插件自建索引。 +- **采集与 Server 部署在一起**：采集在 Server 的 Worker 中执行，需要 Server 能访问外部数据源。 +- **代码变更需要重启**：没有热加载，插件更新期间 Server 会短暂不可用。 + +# 理由与备选 + +| 方案 | 取舍 | +| --- | --- | +| 保留现有设计，只简化接口 | 插件作者体验改善有限，生命周期状态机、管理服务和 schema 固化的维护成本仍在 | +| 插件在 Server 进程内执行 | 部署和调用最简单，但插件故障会拖垮 Server；Supervisor 已经提供隔离和调度，没有理由另起一套 | +| 插件完全自管存储和游标 | core 读取接口无法直接使用，各插件需要重复实现窗口读取与幂等，游标与结果也难以原子提交 | +| 插件通过上下文接口逐条写入、各自提交，最后返回成功与否 | 写入与游标推进分属不同事务，失败时会留下部分结果，重试会重复写入 | +| 插件返回草稿，或由写入器在内存中暂存操作，最后统一提交 | 可以原子提交，但插件在提交前拿不到真实的 revision，难以同步写入自有索引；合并、撤销合并、生命周期等操作需要在内存中重新实现一遍 core 的语义；检索也看不到暂存内容 | +| 用一个事务包住整个 `generate` | 模型调用可能持续数分钟，长时间持锁会阻塞其他写入，SQLite 上会阻塞整个数据库；插件自有存储是另一个数据库，一个事务本来也覆盖不了 | +| `generate` 内的短写事务块（本提案） | 耗时工作在块外；块内直接复用 core 的写操作和校验，拿到真实 revision，读取能看到本块写入；块结束时与游标、确认一起提交，异常即回滚。代价是块内持锁，需要时限约束 | +| 常驻插件进程（类似 Dify 的插件守护进程） | 可以复用连接、支持多语言，但需要进程管理、序列化协议和取消协议。接口以请求 / 响应模型定义，后续可以替换执行方式而不改插件代码 | +| 支持热加载 | 需要重新引入加载状态、注册代次屏蔽和旧调用回收，与目标冲突 | + +选择“Supervisor 派发 Worker、插件在短写事务块中直接写入、core 与游标一并提交”的原因：插件作者只需写一个工厂和一个方法； +隔离、调度、触发、退避和领导权都复用已经在生产中使用的 Supervisor；写操作直接复用 core 现有实现，不需要另一套语义； +Source 与 Artifact 留在 core 的统一存储中，现有读取接口、lineage、合并记录和访问控制直接适用；core 只管理两类核心数据 +及其进度，其余交给插件。 + +# 先例 + +- **Dify 插件**：按插件类型（工具、模型、Agent 策略、扩展端点等）定义接口，插件用清单声明类型与参数并实现对应方法， + Dify 在工作流或应用需要时调用。插件运行在独立的插件守护进程中。本提案沿用“宿主定义接口与调用时机”的思路，执行 + 隔离复用 PowerContext 已有的 Supervisor Worker。 +- **Airbyte 连接器协议**：Source 连接器的 `read` 接收上次的 state 并输出记录和新 state，由平台持久化 state。 + 本提案的采集 checkpoint 采用相同分工，并按目标分别保存。 +- **事务上下文管理器**：SQLAlchemy 的 `async with engine.begin()` 等接口在块开始时开启事务、正常结束时提交、异常时回滚。 + `writer.transaction()` 沿用这一使用方式，并在提交时附带游标与调用确认。 +- **PowerContext 现有基础**：[Artifact processing Supervisor](1515_artifact_processing_supervisor.md) 的调度、调用意图、 + Worker 与租约 fence；[公共合并方法](1859_artifact_merge.md)；`ArtifactRepository`、`ArtifactGovernanceRepository`、 + handle 与 tags 的写入实现；`SourceDefinition` / `SourceProjection`；Source journal、`pc_source_cursors` 和 + `list_window_for_generation`；`pc_connector_checkpoints`；旧 Source 插件设计中的 `identify_target` 目标识别规则。 + +# 未决问题 + +| 问题 | 说明 | +| --- | --- | +| 插件自定义检索 | 现有检索接口只接受文本 query，第一版由 core 提供通用文本检索。后续是否允许插件为公开检索接口提供实现，需要确定检索代码在哪个进程执行 | +| 写事务块的时限与限额默认值 | 默认 30 秒是否合适；SQLite 与 OceanBase 是否采用不同默认值；单块操作数与字节上限 | +| 共享模型客户端 | 是否在 `PluginContext` 中提供 Server 已配置的模型客户端，让 LLM 插件不必自带凭据，并计入现有模型用量统计 | +| 采集目标的删除 | 第一版只支持停用和恢复。是否需要彻底删除目标及其 checkpoint，以及删除后已采集 Source 的处理 | +| 插件 family 接入 Recall / Prepared Context | 需要定义候选内容和预算约束，建议另行提案 | + +# 后续可能 + +- 常驻插件进程，在保持接口不变的前提下复用连接和初始化结果。 +- 更多插件类型，例如为 Recall 提供候选内容、为 Prepared Context 提供文本片段。 +- 插件打包与安装工具，例如从 Git 地址或 wheel 安装到插件目录。 From 647236acb49f80be46aee055d88a96f2ebf54f10 Mon Sep 17 00:00:00 2001 From: jasper-yl Date: Sun, 8 Nov 2026 10:45:37 -0800 Subject: [PATCH 02/12] docs(rfc): number plugin interfaces RFC as 2549 Co-Authored-By: Claude Opus 5.5 --- .../rfcs/{0000-plugin-interfaces.md => 2549-plugin-interfaces.md} | 0 .../rfcs/{0000-plugin-interfaces.md => 2549-plugin-interfaces.md} | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename docs/en/rfcs/{0000-plugin-interfaces.md => 2549-plugin-interfaces.md} (100%) rename docs/zh/rfcs/{0000-plugin-interfaces.md => 2549-plugin-interfaces.md} (100%) diff --git a/docs/en/rfcs/0000-plugin-interfaces.md b/docs/en/rfcs/2549-plugin-interfaces.md similarity index 100% rename from docs/en/rfcs/0000-plugin-interfaces.md rename to docs/en/rfcs/2549-plugin-interfaces.md diff --git a/docs/zh/rfcs/0000-plugin-interfaces.md b/docs/zh/rfcs/2549-plugin-interfaces.md similarity index 100% rename from docs/zh/rfcs/0000-plugin-interfaces.md rename to docs/zh/rfcs/2549-plugin-interfaces.md From a5428109f37e46b36adfd236afb6742d92af5e37 Mon Sep 17 00:00:00 2001 From: jasper-yl Date: Sun, 8 Nov 2026 11:03:43 -0800 Subject: [PATCH 03/12] docs(rfc): link plugin interfaces RFC to PR 2549 Co-Authored-By: Claude Opus 5.5 --- docs/en/rfcs/2549-plugin-interfaces.md | 2 +- docs/zh/rfcs/2549-plugin-interfaces.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/rfcs/2549-plugin-interfaces.md b/docs/en/rfcs/2549-plugin-interfaces.md index 98f926f469..3e6326e20c 100644 --- a/docs/en/rfcs/2549-plugin-interfaces.md +++ b/docs/en/rfcs/2549-plugin-interfaces.md @@ -5,7 +5,7 @@ description: Define Source and Artifact plugins through a few interfaces that PC - Proposal Name: `plugin_interfaces` - Start Date: 2026-11-08 -- RFC PR: [oceanbase/powercontext#0000](https://github.com/oceanbase/powercontext/pull/0000) +- RFC PR: [oceanbase/powercontext#2549](https://github.com/oceanbase/powercontext/pull/2549) # Summary diff --git a/docs/zh/rfcs/2549-plugin-interfaces.md b/docs/zh/rfcs/2549-plugin-interfaces.md index adea4fb5c1..d4c2c24137 100644 --- a/docs/zh/rfcs/2549-plugin-interfaces.md +++ b/docs/zh/rfcs/2549-plugin-interfaces.md @@ -5,7 +5,7 @@ description: 用少量接口定义 Source 和 Artifact 插件，由 PC Server - Proposal Name: `plugin_interfaces` - Start Date: 2026-11-08 -- RFC PR: [oceanbase/powercontext#0000](https://github.com/oceanbase/powercontext/pull/0000) +- RFC PR: [oceanbase/powercontext#2549](https://github.com/oceanbase/powercontext/pull/2549) # 概要 From 1dd59e6a2a1be4183cc0c23c9d2123ca53eedb8f Mon Sep 17 00:00:00 2001 From: jasper-yl Date: Mon, 9 Nov 2026 18:55:02 -0800 Subject: [PATCH 04/12] docs(rfc): declare plugins in code and reuse the Source and Artifact APIs Co-Authored-By: Claude Opus 5.5 --- docs/en/rfcs/2549-plugin-interfaces.md | 1312 +++++++++++++----------- docs/zh/rfcs/2549-plugin-interfaces.md | 816 +++++++++------ 2 files changed, 1217 insertions(+), 911 deletions(-) diff --git a/docs/en/rfcs/2549-plugin-interfaces.md b/docs/en/rfcs/2549-plugin-interfaces.md index 3e6326e20c..fce15316a2 100644 --- a/docs/en/rfcs/2549-plugin-interfaces.md +++ b/docs/en/rfcs/2549-plugin-interfaces.md @@ -1,6 +1,6 @@ --- title: Plugin interfaces -description: Define Source and Artifact plugins through a few interfaces that PC Server calls through the Supervisor when needed; core manages only the two core data kinds, Sources and Artifacts, and their consumption progress, and no longer manages plugin lifecycle. +description: Source and Artifact plugins each implement a few interfaces that PC Server calls through the Supervisor when needed; callers use only the existing Source and Artifact APIs, and core manages only the two core data kinds and their consumption progress, without managing plugin lifecycle. --- - Proposal Name: `plugin_interfaces` @@ -9,191 +9,223 @@ description: Define Source and Artifact plugins through a few interfaces that PC # Summary -This proposal redefines PowerContext's third-party plugin model: each plugin kind implements a few fixed interfaces, -and PC Server calls them when needed. Plugins are callees that only turn inputs into results; they take no part in -scheduling, state management, or persistence protocols. - -- **Source plugins** provide Source definitions (reusing the existing `SourceDefinition` / `SourceProjection`) and may - implement `collect`. For writes, Server calls the definitions to validate, transform, and save. Collection works on - **collection targets**, such as "one account on X"; hosts can add, disable, and re-enable targets at runtime without a - restart. The Supervisor dispatches a Worker on schedule to call `collect`, and core commits the returned Sources and - each target's checkpoint in one transaction. -- **Artifact plugins** implement `generate`. The Supervisor dispatches a Worker on a configured schedule, when new - Sources arrive, or on a manual request; the Worker reads that Scope's unconsumed Source window and passes it to - `generate` together with a **writer** (`ArtifactWriter`). The plugin does slow work such as model calls outside the - writer, then performs Artifact writes inside an `async with writer.transaction()` block: core opens a transaction when - the block starts, writes in the block act directly on core data and immediately return real revisions, and when the - block ends normally core commits those writes, the consumption cursor, and the completion acknowledgment together. An - exception in the block rolls everything back. Plugins can neither read nor change the cursor. -- Core manages only two kinds of core data, `pc_sources` and `pc_artifacts` (with heads, lineage, merge records, - lifecycle, handles, and tags), and their consumption progress. Plugins may keep their own indexes and storage, which - they manage and keep consistent themselves. Applications read and search plugin families through the existing public - Artifact interfaces, and the read path runs no plugin code. -- `collect` and `generate` run in Worker child processes managed by the Supervisor, reusing its existing process - isolation, timeouts, concurrency budgets, leadership lease, and failure backoff. Users deploy only one Server. -- Plugins load from a directory at Server startup; code or manifest changes require a restart. Server stores no plugin - enabled state, registration generation, or management phase, and offers no enable / disable / reload / unload - management interface. +This proposal redefines PowerContext's third-party plugin model: plugins implement only a few fixed interfaces, and +PC Server calls them when it needs them. Callers never call plugins directly; they use the existing Source and Artifact +APIs exactly as they do for built-in capabilities. + +- **Two plugin kinds**: + - **Source plugins** provide Source definitions, and implement `identify_target` and `collect` when they need to + collect continuously. + - **Artifact plugins** implement `generate`, which turns unconsumed Sources into Artifacts. +- **Plugin shape**: a plugin is a directory containing `plugin.py`, which assigns a declaration object (`SourcePlugin` + or `ArtifactPlugin`) to `plugin`. The declaration lives in code; there is no separate manifest file. The plugin reads + deployment-specific settings from environment variables. +- **How callers use plugins**: + - Writing Sources and adding collection targets use the existing Source APIs. + - Reading, searching, and changing Artifacts use the existing Artifact APIs. + - User operations such as forget, restore, and edit are executed by core directly, without the plugin. +- **Execution**: `collect` and `generate` run in Worker child processes managed by the Supervisor, reusing its existing + isolation, timeouts, concurrency budgets, leader lease, and failure backoff. +- **Writes**: an Artifact plugin performs its writes inside an `async with writer.transaction()` block. When the block + ends, core commits the writes, the consumption cursor, and the completion acknowledgment together; an exception rolls + everything back. Plugins can neither read nor change the cursor. +- **What core owns**: only `pc_sources`, `pc_artifacts` (including heads, lineage, merge records, lifecycle, handles, and + tags), and their consumption progress. Plugins may have their own indexes and storage and manage them themselves. +- **Lifecycle**: plugins load from a directory at Server startup, and changes require a restart. Server stores no plugin + enabled state, registration generation, or management phase. This proposal replaces the plugin contracts and management mechanisms of the [Source plugin framework RFC #2012](2012-source-plugin-system.md) and the -[Artifact plugin framework RFC #2175](2175-artifact-plugin-system.md). Neither implementation has shipped as a stable -interface, so this proposal keeps no compatibility: old plugin manifests, management interfaces, the management -service, and plugin-management tables are removed, and custom Artifact data written by old Artifact plugins is not -migrated. +[Artifact plugin framework RFC #2175](2175-artifact-plugin-system.md). Neither implementation has been released as a +stable interface, so this proposal keeps no compatibility: the old plugin manifests, management APIs, management service, +and plugin-management tables are removed, and custom Artifact data written by old Artifact plugins is not migrated. -This document describes the target design, not availability in mainline or a released version. Interface names and -code examples illustrate the contract; final syntax ships with the implementation. +This document describes the target design; it does not mean the mainline or a release already provides it. Interface +names and code examples illustrate the contract; the implementation determines the final syntax. # Motivation -Both existing plugin frameworks make PowerContext carry substantial lifecycle management for third-party plugins: +Both existing plugin frameworks make PowerContext carry a large amount of lifecycle management for third-party plugins: | Framework | Work Server carries for plugins | | --- | --- | -| Artifact plugins (#2175) | `desired_state`, `phase`, `binding_generation`, and `last_error` in `pc_artifact_processing_binding_states`; seven management states and their transitions; `enable-all / reload / enable / disable / unload` management interfaces with `expected_generation` concurrency checks; registration-generation fencing of old Worker commits; content schemas and index definitions frozen in `pc_artifact_family_schemas`, with changes returning 409; framework-owned building, rebuilding, and availability of custom declarative indexes | -| Source plugins (#2012) | A separate plugin management service with its own SQLite (instances, capabilities, targets, desired state, submission records); one Runtime process per plugin instance; a poll / reply channel and capability-catalog synchronization between management and Server; the `source load` deployment command; MCP tools to enable and disable instances | +| Artifact plugins (#2175) | `desired_state`, `phase`, `binding_generation`, and `last_error` in `pc_artifact_processing_binding_states`; seven management states and their transitions; the `enable-all / reload / enable / disable / unload` management APIs with `expected_generation` concurrency checks; registration generations that fence old Worker commits; content schemas and index definitions pinned in `pc_artifact_family_schemas`, with 409 on change; the framework builds, rebuilds, and tracks availability of custom declarative indexes | +| Source plugins (#2012) | A separate plugin management service with its own SQLite (instances, capabilities, targets, desired state, commit records); one Runtime process per plugin instance; poll / reply channels and capability catalog sync between the service and Server; the `source load` deployment command; MCP tools for starting and stopping instances | Everyone pays for these mechanisms: -- **Plugin authors** must understand bindings, generations, schema-freezing rules, index declarations, and the - management channel before they can write a plugin. -- **Deployers** run several processes, maintain two databases, and use management APIs before a plugin can run. -- **Maintainers** keep state machines, concurrency fencing, schema and index governance, and recovery paths for - third-party code; that code and its test surface already exceed the plugins' business logic. +- **Plugin authors** must understand framework concepts such as bindings, generations, schema pinning rules, index + declarations, and management channels before they can write a plugin. +- **Deployers** must run several processes, maintain two databases, write manifest files, and use management APIs to get + plugins into a runnable state. +- **Callers** must tell built-in capabilities from plugin capabilities and use different APIs for them. +- **Maintainers** must maintain state machines, concurrency fencing, schema and index governance, and recovery paths for + third-party code; this code and its test surface are already larger than the plugins' own business logic. -A plugin needs to be called at the right time, receive the input it needs, and hand results back to core. Scheduling, -isolation, consumption progress, and transactional commits are capabilities core already has; the Supervisor provides -them to built-in Artifacts today. Plugin models such as Dify's show that a host defining interfaces and invocation -timing, with plugins only implementing interfaces, is enough for a large ecosystem of third-party extensions. +What a plugin actually needs is to be called at the right time, receive the input it needs, and hand its results back to +core. Scheduling, isolation, consumption progress, and transactional commits are capabilities core already has; the +Supervisor already provides them for built-in Artifacts. Plugin models such as Dify's also show that letting the host +define the interfaces and when to call them, while plugins only implement the interfaces, is enough to support a large +number of third-party extensions. Goals: -1. Plugin authors implement a few interfaces without learning scheduling, cursor, and commit protocols. -2. Plugin code runs in isolated Worker processes; plugin failures do not affect Server or other plugins. -3. Plugin results go into `pc_sources` / `pc_artifacts` and commit atomically with consumption progress, so existing - read interfaces work. -4. Core manages only these two kinds of core data and their consumption progress; content structure, indexes, and other - storage belong to plugins. -5. Plugin writes to core data go only through a core-provided writer that covers every generic Artifact write operation - and reuses core's existing validation rules. -6. Artifact generation can run on a configured schedule or be triggered by new Sources or manual requests; collection - targets can be added and removed at runtime. -7. Users deploy only one Server; plugins load when Server starts. +1. A plugin author writes only `plugin.py`: one declaration object and a few methods, without learning the internals of + plugin management. +2. Callers are unaware of plugins: the existing Source and Artifact APIs apply equally to data plugins produce. +3. Plugin code runs in isolated Worker processes, so a plugin failure does not affect Server or other plugins. +4. Plugin results go into `pc_sources` / `pc_artifacts` and commit atomically with consumption progress. +5. Core manages only these two core data kinds and their consumption progress; content structure, indexes, and other + storage belong to the plugin. +6. Plugin writes to core data go only through a writer core provides, which covers every generic Artifact write + operation and keeps core's existing validation rules. +7. Artifact generation can run on a schedule or be triggered by new Sources; collection targets can be added and removed + at runtime. +8. Users deploy a single Server; plugins load when Server starts. # Guide-level explanation -## 1. Core idea +## 1. Overall design ```mermaid flowchart LR - Host["Application / Agent / host"] -->|"HTTP / MCP"| Server - subgraph Server["PC Server"] - API["Public interfaces
authentication and Scope authorization"] - Sup["Supervisor
scheduling / triggers / budgets / backoff"] - Core[("core data
pc_sources / pc_artifacts
targets / cursors / checkpoints")] + Caller["Application / Agent / Host"] + subgraph Server["PC Server (core)"] + SourceAPI["Existing Source APIs
write Sources, add collection targets"] + ArtifactAPI["Existing Artifact APIs
read, search, change"] + Sup["Supervisor
schedules / triggers / budgets / backoff"] + Core[("Core data
pc_sources / pc_artifacts
collection targets / checkpoints / cursors")] end - subgraph Worker["Worker child process (per call)"] - Exec["Plugin executor"] - Plugin["Plugin code
collect() / generate()"] - Tx["writer.transaction()
constrained write transaction"] - Exec -->|"targets and checkpoints / Source window"| Plugin - Plugin -->|"put / merge / set_lifecycle ..."| Tx + subgraph SW["Worker: Source plugin"] + Collect["collect(request)"] end - API -->|"read, search"| Core - API -->|"write, add targets"| Core - API -->|"manual trigger"| Sup - Sup -->|"dispatch (plugin, Scope)"| Exec - Tx -->|"on exit: commit writes, cursor, acknowledgment together"| Core - Plugin -.->|"own indexes / storage"| Own[("Plugin-owned storage")] - Plugin -.->|"external systems / models"| Ext["X, ticket systems, LLMs, ..."] + subgraph AW["Worker: Artifact plugin"] + Generate["generate(request, writer)"] + Tx["writer.transaction()"] + Generate --> Tx + end + Caller --> SourceAPI + Caller --> ArtifactAPI + SourceAPI -->|"resolve / Projections (plugin definitions)"| Core + ArtifactAPI <--> Core + Sup -->|"targets and checkpoints"| Collect + Collect -->|"Sources and new checkpoints, one commit"| Core + Core -->|"new Sources trigger"| Sup + Sup -->|"unconsumed Source window"| Generate + Tx -->|"Artifact writes, cursor, acknowledgment, one commit"| Core + Collect -.-> Ext["External systems"] + Generate -.-> Own[("Plugin-owned storage / models")] ``` +The diagram has two independent paths, and one Worker invocation runs only one plugin: + +- **Collection path**: the Supervisor dispatches a Source plugin's Worker on schedule with the Scope's enabled collection + targets and their checkpoints. The plugin reads from the external system and returns Sources; core validates them and + saves the Sources and the new checkpoints in one transaction. +- **Generation path**: when new Sources enter the journal, Artifact plugins that consume them are marked as pending. The + Supervisor dispatches a Worker with the Scope's unconsumed Source window and a writer; the plugin writes Artifacts in a + write transaction block, and core commits them together with the cursor and the acknowledgment. + +Callers deal only with core's existing APIs: + +| What the caller does | API | How the plugin takes part | +| --- | --- | --- | +| Submit a Source | Existing Source write API | Core calls the plugin's `resolve` and Projections to validate and convert it | +| Add, view, or disable a collection target | Existing Source management APIs | Core calls the plugin's `identify_target` to normalize the input | +| Browse, read, or search Artifacts | Existing Artifact read and search APIs | Not involved | +| Edit, tag, merge, forget, restore, undo a merge | Existing Artifact write API, tag APIs, and the generic merge APIs | Not involved; the plugin sees the new state in its next invocation | + +Responsibilities: + | Responsibility | Owner | | --- | --- | -| Authentication, Scope authorization, public HTTP / MCP interfaces, collection target management | Server | -| When to call: scheduling, trigger coalescing, Worker budgets, timeouts, backoff, leadership lease | Supervisor (existing) | -| Reading inputs, providing a constrained write transaction, committing core data / cursors / checkpoints / acknowledgments in one transaction | Plugin executor in the Worker (core) | -| Reading external data, identifying and normalizing collection targets, Source definitions and Projection computation | Source plugin | +| Authentication, Scope authorization, public HTTP / MCP APIs, storing collection targets and checkpoints, user changes to Artifacts | Core | +| When to call: schedules, trigger coalescing, Worker budgets, timeouts, backoff, leader lease | Core (the existing Supervisor) | +| Reading inputs, providing the constrained write transaction, committing core data / cursor / checkpoints / acknowledgment in one transaction | Core (the plugin executor in the Worker) | +| Source definitions and Projection computation, identifying and normalizing collection targets, reading external data | Source plugin | | Artifact content structure, generation algorithm, plugin-owned indexes and storage and their consistency | Artifact plugin | -| Deploying plugin code, dependencies, and plugin-owned storage | Deployer | +| Deploying plugin code, dependencies, environment variables, and plugin-owned storage | Deployer | -Plugin code does one thing: given input, produce results or raise. Raising is failure; core commits no core data and -retries the same input later. +Plugin code does one thing: given an input, produce a result or raise. Raising means failure; core commits no core data +and retries the same input next time. -## 2. Plugin directory and manifest +## 2. Plugin shape -Server loads plugins from `POWERCONTEXT_SERVER_PLUGINS_DIR`. Each direct child directory is one plugin containing -`plugin.yml` and code: +Server loads plugins from `POWERCONTEXT_SERVER_PLUGINS_DIR`. Each direct subdirectory is one plugin, and the directory name +is the plugin name. The directory must contain `plugin.py` and may contain other modules: ```text /opt/powercontext/plugins/ ├── x-posts/ -│ ├── plugin.yml -│ ├── definition.py -│ └── plugin.py +│ ├── plugin.py +│ └── definition.py └── incident-report/ - ├── plugin.yml └── plugin.py ``` -```yaml -name: incident-report # Plugin name, unique in the deployment, matches the directory name -kind: artifact # source or artifact -entry: ./plugin.py:create # Factory function; path relative to the plugin directory -scopes: [team-a] # Optional; omitted means every Scope with matching input -config: # Business configuration validated by the plugin's Config model - title_prefix: Incident report - index_url: ${REPORT_INDEX_URL} # ${NAME} resolves from the environment inside the Worker -``` +`plugin.py` assigns a declaration object to the module-level variable `plugin`: -The manifest describes only **what the plugin is and how it runs**, not whether it is enabled. If the directory exists -and loads successfully, the plugin is available; to stop it, move it out of the directory and restart Server. The same -code can be deployed several times through multiple plugin directories with different `name` and `config` values. - -## 3. Source plugin example - -A Source plugin exports Source definitions. With definitions alone, hosts can submit complete content through the -existing Source write interface (the write use of MCP `pc_source_add`); Server calls the definition's `resolve` and -each Projection, then saves to `pc_sources`. - -To collect continuously from an external system, the plugin also implements `collect`. Collection works per -**collection target**: each target is one external resource the plugin can collect, such as one account on X. This -plugin collects new posts from each target account every 5 minutes: - -```yaml -name: x-posts -kind: source -entry: ./plugin.py:create -scopes: [team-a] -collect: - schedule: {interval_seconds: 300} - max_workers: 1 - timeout_seconds: 120 - max_items: 200 - targets: # Optional; initial targets imported on first start - - {scope_id: team-a, input: {account: "@oceanbase"}} -config: - bearer_token: ${X_BEARER_TOKEN} +```python +plugin = ArtifactPlugin(family="incident-report", ..., create=create) # or SourcePlugin(...) ``` +The declaration answers what core needs to know before it runs any plugin code: what the plugin provides (Source types or +an Artifact family), what it consumes, when to call it, and its resource limits. It is an ordinary Python object validated +by Pydantic; there is no YAML or other manifest file. + +- **Deployment settings**: values that vary by deployment, such as credentials, external service addresses, and model + names, are read by the plugin from environment variables in `create`. Server neither stores, passes, nor logs them. +- **Enabling and disabling**: if the directory exists and passes the check, the plugin is available; to disable it, move + it out of the directory and restart Server. +- **Check**: at startup, Server imports `plugin.py` in a time-limited child process and validates the declaration without + calling `create`. A failed check affects only that plugin. + +## 3. Source plugins + +### 3.1 Interfaces to implement + +| Interface | Required | Description | +| --- | --- | --- | +| `definitions` | Yes | A sequence of `SourceDefinition`, the existing Source definition: `input_class`, `resolve`, and its Projections | +| `target_input` | For collection | A Pydantic model validating the input a caller submits when adding a target | +| `identify_target(value) -> TargetIdentity` | For collection | A pure function that normalizes the input into a stable target key and normalized input | +| `create(ctx) -> SourceCollector` | For collection | A factory run once per invocation in the Worker, returning an object that implements `collect` | +| `collect(request) -> CollectResult` | For collection | Reads external data per target and checkpoint, returning Sources and new checkpoints | + +A plugin that provides only `definitions` only extends Source types: callers submit content through the existing Source +write APIs, and core uses the definitions to validate, convert, and save it. + +### 3.2 Declaration + +| Field | Default | Description | +| --- | --- | --- | +| `definitions` | Required | See the table above | +| `collect` | `None` | Collection settings; empty means the plugin only provides definitions | +| `collect.schedule` | Required | `Interval(seconds=...)` or `Cron(expression=..., timezone=...)` | +| `collect.max_targets` | 50 | Maximum targets per invocation | +| `collect.max_items` | 200 | Maximum Sources returned per invocation | +| `collect.timeout_seconds` | 120 | Total Worker time limit | +| `collect.max_workers` | 1 | Concurrency limit across Scopes | +| `target_input`, `identify_target`, `create` | Required for collection | See the table above | +| `scopes` | `None` | Scopes the plugin may serve; empty means every Scope | + +### 3.3 Example + +The plugin below collects new posts from each target account on X every 5 minutes: + ```python +import os + from pydantic import BaseModel from powercontext.plugins import ( - CollectedItem, CollectRequest, CollectResult, PluginContext, TargetIdentity, + Collect, CollectedItem, CollectRequest, CollectResult, Interval, PluginContext, SourcePlugin, TargetIdentity, ) -from .definition import PostCapture, XPostDefinition - -definitions = (XPostDefinition(),) +from definition import PostCapture, XPostDefinition -class Config(BaseModel): - bearer_token: str - - -class TargetInput(BaseModel): # Input a host submits when adding a target +class TargetInput(BaseModel): account: str @@ -218,267 +250,308 @@ class XCollector: return result -async def create(ctx: PluginContext, config: Config) -> XCollector: - client = await ctx.enter_async_context(XClient(config.bearer_token)) +async def create(ctx: PluginContext) -> XCollector: + client = await ctx.enter_async_context(XClient(os.environ["X_BEARER_TOKEN"])) return XCollector(client) + + +plugin = SourcePlugin( + definitions=(XPostDefinition(),), + collect=Collect(schedule=Interval(seconds=300), timeout_seconds=120), + target_input=TargetInput, + identify_target=identify_target, + create=create, +) ``` -A host adds a target at runtime without a restart: +### 3.4 How callers use it + +Callers use the existing Source management APIs and do not need to know a plugin provides the capability: ```text pc_source_add(scope_id="team-a", capability="x-posts", input={"account": "@PowerContextAI"}) +pc_source_status(scope_id="team-a", target_id="...") +pc_source_target_set_enabled(scope_id="team-a", target_id="...", enabled=false) ``` -Server validates the input with `TargetInput`, calls `identify_target` for a stable target key and normalized input, -saves the target, and immediately requests a collection. Adding the same account again yields the same key and reuses -the existing target; targets can be disabled and re-enabled with progress retained. - -Each target's checkpoint is read-only input; the plugin proposes new checkpoints through its return value. Core commits -all Sources returned by one call and each target's new checkpoint in one transaction. If `collect` raises or times out, -nothing is committed and the next call receives the old checkpoints. - -## 4. Artifact plugin example - -This plugin generates one report per ticket Source. It first looks up a similar existing report in its own vector -index, revises that report on a match and creates a new one otherwise, and writes the report into its own vector index. - -```yaml -name: incident-report -kind: artifact -entry: ./plugin.py:create -scopes: [team-a] -generate: - family: incident-report - inputs: - source_types: [example-ticket] - projection: powercontext.text-evidence@1 - max_sources: 50 - schedule: {interval_seconds: 300} # Periodic check - on_sources: true # Trigger when new Sources arrive - manual: true # Allow manual triggers - max_workers: 1 - timeout_seconds: 300 # Limit for the whole call - transaction_timeout_seconds: 30 # Limit for the write transaction block -config: - title_prefix: Incident report - index_url: ${REPORT_INDEX_URL} -``` +- **Adding**: core validates the input with `target_input` and calls `identify_target` to get the key and normalized + input. The same account yields the same key and reuses the existing target. A new target is saved as enabled and one + collection is requested immediately, with no restart. +- **Checkpoints**: each target's checkpoint is a read-only input, and the plugin proposes a new checkpoint through its + return value. Core commits all returned Sources and every target's new checkpoint in one transaction; if `collect` + raises or times out, nothing is committed and the next invocation still receives the old checkpoint. -```python -from pydantic import BaseModel +## 4. Artifact plugins -from powercontext.plugins import ArtifactWriter, GenerateRequest, PluginContext -from powercontext.sources import TextEvidence +### 4.1 Interfaces to implement + +| Interface | Required | Description | +| --- | --- | --- | +| `create(ctx) -> ArtifactGenerator` | Yes | A factory run once per invocation in the Worker | +| `generate(request, writer) -> None` | Yes | Reads `request.sources` (the unconsumed Source window) and writes Artifacts in the write transaction block | +The plugin only generates. Core executes user changes to Artifacts directly (see 4.4), so the plugin implements no command +handling. -class Config(BaseModel): - title_prefix: str = "Incident report" - index_url: str +### 4.2 Declaration + +| Field | Default | Description | +| --- | --- | --- | +| `family` | Required | The Artifact family the plugin owns; must not duplicate a built-in family or another plugin's | +| `inputs` | `None` | `Inputs(source_types=..., projection=..., max_sources=50)`: the Source types consumed, the Projection each Source is delivered in, and the maximum Sources per window; empty means the plugin consumes no Sources | +| `on_sources` | `False` | Trigger after new Sources arrive; requires `inputs` | +| `schedule` | `None` | `Interval(seconds=...)` or `Cron(expression=..., timezone=...)`; declare at least one of this and `on_sources` | +| `scopes` | `None` | Scopes the plugin may serve; required for plugins that consume no Sources | +| `search_fields` | `None` | Top-level content fields indexed for full-text search; empty means every top-level string field | +| `timeout_seconds` | 300 | Time limit for the whole invocation | +| `transaction_timeout_seconds` | 30 | Time limit for the write transaction block, at most `timeout_seconds` | +| `max_workers` | 1 | Concurrency limit across Scopes | +| `create` | Required | See the table above | + +### 4.3 Example + +The plugin below generates one report per ticket Source: it first uses its own vector index to find a similar existing +report and revises it on a match, otherwise it creates a new one; it also writes the report into its own vector index. + +```python +import os + +from powercontext.plugins import ArtifactPlugin, ArtifactWriter, GenerateRequest, Inputs, PluginContext +from powercontext.sources import TextEvidence class IncidentReports: - def __init__(self, config: Config, index: VectorIndex, model: Summarizer) -> None: - self.config, self.index, self.model = config, index, model + def __init__(self, index: VectorIndex, model: Summarizer) -> None: + self.index, self.model = index, model async def generate(self, request: GenerateRequest, writer: ArtifactWriter) -> None: - # 1. Slow work stays outside the write transaction: model calls, vector computation + # 1. Slow work outside the write transaction: model calls, vector computation planned = [] for source in request.sources: evidence = TextEvidence.model_validate(source.projection) summary = await self.model.summarize(evidence.content) - vector = await self.model.embed(summary) - planned.append((source, summary, vector)) + planned.append((source, summary, await self.model.embed(summary))) - # 2. Short write transaction: reads in the block see what the block has written + # 2. A short write transaction: reads inside the block see what the block has written async with writer.transaction() as tx: for source, summary, vector in planned: match = await self.index.nearest(request.scope_id, vector, threshold=0.92) current = await tx.get(match.artifact_id) if match else None artifact_id = current.ref.artifact_id if current else writer.stable_id( "report", source.ref.source_type, source.ref.source_id) - ref = await tx.put( - artifact_id, - content={"title": self.config.title_prefix, "summary": summary}, - search_text=summary, - source_refs=[source.ref], - ) - # With the real artifact_id and revision in hand, upsert into the plugin's own index by ID - await self.index.upsert(request.scope_id, ref.artifact_id, ref.revision, vector) + written = await tx.put(artifact_id, content={"title": "Incident report", "summary": summary}, + source_refs=[source.ref]) + # Write the owned index once the real artifact_id and revision are known; upsert by ID + await self.index.upsert(request.scope_id, written.ref.artifact_id, written.ref.revision, vector) # The block ended normally: core commits the Artifact writes, the cursor advance, and the acknowledgment -async def create(ctx: PluginContext, config: Config) -> IncidentReports: - index = await ctx.enter_async_context(VectorIndex.connect(config.index_url)) - return IncidentReports(config, index, Summarizer()) +async def create(ctx: PluginContext) -> IncidentReports: + index = await ctx.enter_async_context(VectorIndex.connect(os.environ["REPORT_INDEX_URL"])) + return IncidentReports(index, Summarizer()) + + +plugin = ArtifactPlugin( + family="incident-report", + inputs=Inputs(source_types=("example-ticket",), projection="powercontext.text-evidence@1"), + on_sources=True, + search_fields=("title", "summary"), + create=create, +) ``` -When the block raises, core rolls back all Artifact writes in the block, the cursor stays put, and the same window is -retried later. The plugin's own index is not part of core's transaction: the plugin upserts by stable ID so a retry -overwrites entries left by the previous attempt, and reads from its own index defer to core's current revision (see -reference-level section 1.6). +If the block raises, core rolls back every Artifact write in it, the cursor stays put, and the same window is retried. +The plugin's own index is not part of core's transaction: the plugin upserts by stable ID, so a retry overwrites entries +left by the previous attempt, and reads from the owned index defer to core's current data (see reference 3.5). -Applications access plugin families through the existing public Artifact interfaces, which read `pc_artifacts` -directly and run no plugin code: +### 4.4 How callers use it -| Purpose | Interface | +Callers access plugin families through the existing Artifact APIs. Read APIs read `pc_artifacts` directly, and write APIs +are executed by core with the same core implementation plugin writes use; neither runs plugin code: + +| Purpose | API | | --- | --- | | Browse | `GET /v1/scopes/{scope_id}/artifacts/{family}` | -| Current content | `GET /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | -| Exact revision / revision history | `GET .../{artifact_id}/revisions/{revision}`, `GET .../{artifact_id}/revisions` | -| Tags | `GET .../{artifact_id}/tags` | +| Read the current content, a given revision, revision history, tags | `GET .../{artifact_id}`, `GET .../{artifact_id}/revisions/{revision}`, `GET .../{artifact_id}/revisions`, `GET .../{artifact_id}/tags` | | Text search | `POST /v1/scopes/{scope_id}/artifacts/{family}/search` | -| Trigger generation manually | `POST /v1/scopes/{scope_id}/artifacts/{family}/generate` | +| Edit (submit complete content; `If-Match` checks the current revision) | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | +| Change tags | `PUT .../{artifact_id}/tags`; query by tag with `POST /v1/scopes/{scope_id}/artifact-tags/query` | +| Read the governance state (the `state_version` needed to forget or merge) | `GET .../{artifact_id}/governance` | +| Merge | `POST /v1/scopes/{scope_id}/artifacts/{family}/merges` | +| Forget | `POST .../{artifact_id}/forget` | +| Restore, restore an earlier revision, undo a merge | `POST .../{artifact_id}/restorations` | +| Add knowledge | Write a Source; the plugin processes it in its next invocation | + +Changes return their result synchronously. The plugin reads the changed state through its writer in its next invocation. ## 5. Scheduling and triggers -Plugin scheduling and triggers belong to the existing Artifact processing Supervisor. Each plugin registers at startup -as one Supervisor binding named `plugin:`; the scheduling key remains `(binding, Scope)`. +Scheduling and triggering are delegated to the existing Artifact processing Supervisor. At startup each plugin registers +as one Supervisor binding named `plugin:`, and the scheduling key is `(binding, Scope)`. -Artifact generation supports three call origins in any combination; at least one must be configured: +Artifact plugins: -| Configuration | When a Worker is dispatched | `request.reason` | +| Declaration | When a Worker is dispatched | `request.reason` | | --- | --- | --- | -| `schedule: {interval_seconds: N}` or `schedule: {cron: "0 2 * * *", timezone: Asia/Shanghai}` | When due, for Scopes with pending work | `schedule` | -| `on_sources: true` | The Scope's Source journal is ahead of the plugin's cursor | `sources` | -| `manual: true` | The manual trigger interface is called | `manual` | +| `schedule=Interval(...)` or `schedule=Cron(...)` | When due, for Scopes with pending work | `sources` when there are new Sources, otherwise `schedule` | +| `on_sources=True` | When the Scope's Source journal passes the plugin's cursor, within about 2 seconds | `sources` | -- For plugins with `inputs`, every call carries that Scope's unconsumed Source window; when the window is empty, - scheduled and Source-triggered calls skip the plugin and acknowledge completion directly. -- Plugins without `inputs` (for example, a daily cleanup of existing Artifacts) consume no Sources, are called when - due, and must configure `scopes` explicitly. -- Manual triggers always call the plugin and may carry `params`, which the plugin interprets. A manual request is - persisted on acceptance and returns `202 Accepted`; it still runs after a Server restart. +- A plugin with `inputs` receives the Scope's unconsumed Source window in every invocation; when the window has no matching + Source, the plugin is not called and the cursor is advanced and acknowledged directly. +- A plugin without `inputs` (for example, one that tidies existing Artifacts once a day) consumes no Sources and is called + once for each Scope in `scopes` when due; the first version supports only `Interval`. +- `Cron` is allowed only for plugins with `inputs` and cannot be combined with `on_sources`: each Supervisor binding + accepts a single interval or cron policy. -Source plugin `collect` supports `schedule` and `manual` -(`POST /v1/scopes/{scope_id}/sources/plugins/{name}/collect`); adding or re-enabling a target also requests a collection. +A Source plugin's `collect` is called on `collect.schedule`; adding or resuming a target also requests one collection. -At most one Worker runs at a time for the same plugin and Scope. The Supervisor coalesces triggers that arrive during a -call into one call after it finishes. +At most one Worker runs per plugin per Scope. Triggers that arrive during a run are coalesced by the Supervisor into one +invocation after it finishes. ## 6. Operations -- Startup: Server checks plugin directories one by one. A plugin that fails its check (invalid manifest, import - failure, missing required exports, family or Source type conflicting with a built-in or another plugin) affects only - itself; Server and other plugins start normally. -- Inspection: `GET /v1/plugins` (requires `server.admin`) lists each plugin's check result, kind, family or Source - definitions, schedule configuration, and per-Scope latest call outcome and next retry time. Collection targets and - their latest collection outcomes are shown by `pc_source_status` and the target interfaces. -- Changes: after changing code or the manifest, restart Server. Server does not watch files or reload plugins at - runtime. Collection targets are business data; adding or removing them needs no restart. -- Removal: move the plugin out of the directory and restart. Saved Sources, Artifacts, collection targets, cursors, and - checkpoints are retained and remain readable; redeploying a plugin with the same name continues from where it stopped. - The deployer keeps or cleans up plugin-owned storage. +- **Startup**: Server checks each plugin directory in turn. A failed check (no `plugin.py`, import failure, an invalid + declaration, or a family or Source type that conflicts with a built-in or another plugin) affects only that plugin; + Server and the other plugins start normally. +- **Inspection** (all require `server.admin`, are read-only, and come from the startup check or the database, so every + Server role returns the same answer): + - `GET /v1/plugins`, `GET /v1/plugins/{name}`: each plugin's check result, kind, family or Source types, consumed Source + types, triggers, and Scope range. + - `GET /v1/plugins/{name}/scopes`: for each Scope, whether work is pending, the journal position consumed through, and + the current journal position. + - Reasons for failed calls are diagnosed through logs and metrics; the API does not store them. +- **Changes**: restart Server after changing code. Server does not watch files or reload plugins at runtime. Collection + targets are business data, so adding or removing them needs no restart. +- **Disabling**: move the plugin out of the directory and restart. Saved Sources, Artifacts, collection targets, cursors, + and checkpoints remain readable; redeploying a plugin with the same name continues where it left off. The deployer keeps + or cleans up plugin-owned storage. # Reference-level explanation -## 1. Interface contract +## 1. Shared contract -The public interfaces live in a new `powercontext.plugins` module, replacing the plugin-author-facing parts of -`powercontext.artifact_plugins` and `powercontext.source_management`. Interfaces are expressed as `typing.Protocol` -and Pydantic models. +The public interfaces live in a new module, `powercontext.plugins`, replacing the plugin-author-facing parts of the existing +`powercontext.artifact_plugins` and `powercontext.source_management`. Interfaces are expressed as `typing.Protocol` and +Pydantic models. -### 1.1 Entry module, factory, and context +### 1.1 Declarations and loading -The entry module exports: +```python +class Interval(BaseModel): + seconds: float # > 0 + +class Cron(BaseModel): + expression: str + timezone: str = "Asia/Shanghai" + +class ArtifactPlugin(BaseModel): # fields in guide 4.2 + family: str + inputs: Inputs | None = None + ... + create: Callable[[PluginContext], Awaitable[ArtifactGenerator]] + +class SourcePlugin(BaseModel): # fields in guide 3.2 + definitions: Sequence[SourceDefinition] + collect: Collect | None = None + ... +``` -| Export | Required | Notes | -| --- | --- | --- | -| Factory function (named by `entry`) | Yes | `async def create(ctx, config) -> plugin instance` | -| `Config` | No | Pydantic model validating the manifest's `config`; without it, `config` must be empty | -| `definitions` | Source plugins | Sequence of `SourceDefinition` | -| `TargetInput`, `identify_target` | Source plugins with `collect` | Collection target input model and identification function | +- The plugin name comes from the directory name and must match `^[a-z][a-z0-9-]{0,63}$`. +- A declaration is validated when constructed, including schedule combinations, the dependency between `inputs` and + `scopes`, time-limit relations, and family naming. `create` must be an async function. +- The parts of a declaration other than code objects such as `create`, `identify_target`, and `definitions` are + serializable. Server obtains them from the check child process to register bindings and answer operational queries. + A Worker re-imports `plugin.py` on every invocation; if the declaration differs from the one checked at startup, the call + fails and asks for a Server restart. + +### 1.2 Context ```python class PluginContext: - name: str # Plugin name from the manifest - scope_id: str # Scope of this call - plugin_dir: Path # Plugin directory (read-only by convention) - logger: logging.Logger # Logger bound to the plugin name + name: str # plugin name + scope_id: str # Scope of this invocation + plugin_dir: Path # plugin directory (read-only by convention) + logger: logging.Logger # logger bound to the plugin name async def enter_async_context(self, cm: AbstractAsyncContextManager[T]) -> T: ... ``` -- The factory runs once per Worker call; instances are not reused across calls. Resources that need closing (including - connections to plugin-owned storage) register through `ctx.enter_async_context` and are released by the executor when - the call ends. -- `${NAME}` environment references resolve inside the Worker before `Config` validation, so credentials never enter - Supervisor dispatch parameters or logs. -- Plugins get no connection to core's database and no write interface for cursors or checkpoints. There are only two - paths for writing core data: the return value of a Source plugin's `collect`, and an Artifact plugin's - `ArtifactWriter`. +- The factory runs once per Worker invocation, and instances are not reused across invocations. Resources that need + closing (including connections to plugin-owned storage) are registered through `ctx.enter_async_context` and released + by the executor when the invocation ends. +- Plugins get no connection to core's database and no interface for writing cursors or checkpoints. Plugins write core + data through exactly two paths: a Source plugin's `collect` return value and an Artifact plugin's `ArtifactWriter`. -### 1.2 Source plugins +## 2. Source plugin contract ```python class TargetIdentity(BaseModel): - key: str # Stable target key; always the same for the same external resource - input: dict[str, JsonValue] # Normalized input keeping every parameter that affects collection + key: str # stable target key, always the same for the same external resource + input: dict[str, JsonValue] # normalized input keeping every parameter that affects collection class CollectTarget(BaseModel): key: str input: dict[str, JsonValue] - checkpoint: JsonValue | None # Last successfully committed checkpoint for this target; None the first time; read-only + checkpoint: JsonValue | None # the target's last committed checkpoint; None the first time; read-only class CollectRequest(BaseModel): scope_id: str - reason: Literal["schedule", "manual", "target_added"] - targets: Sequence[CollectTarget] # Enabled targets to collect in this call + reason: Literal["schedule", "target_added"] + targets: Sequence[CollectTarget] # enabled targets to collect in this invocation max_items: int - params: JsonValue | None = None # Manual triggers only class CollectedItem(BaseModel): - definition: str # name of one definition in definitions - value: BaseModel # Instance of that definition's input_class - -class CollectResult(BaseModel): - targets: dict[str, TargetResult] = {} # Keyed by target key - def add(self, target: CollectTarget, items: Sequence[CollectedItem], *, checkpoint: JsonValue | None) -> None: ... + definition: str # name of one of the definitions + value: BaseModel # an instance of that definition's input_class class TargetResult(BaseModel): items: Sequence[CollectedItem] checkpoint: JsonValue | None has_more: bool = False +class CollectResult(BaseModel): + targets: dict[str, TargetResult] = {} # keyed by target key + def add(self, target: CollectTarget, items: Sequence[CollectedItem], *, checkpoint: JsonValue | None) -> None: ... + class SourceCollector(Protocol): async def collect(self, request: CollectRequest, /) -> CollectResult: ... ``` -- Plugins that provide `definitions` without a `collect` configuration support write only. The factory runs only for - collect calls. -- Source types and Projections in `definitions` register into the existing Source definition manifests at startup; - conflicts with built-ins or other plugins fail the check. -- **write** needs no plugin method. Server validates input with `definition.input_class`, calls `resolve` to obtain the - Source, computes each Projection, and saves, with the same semantics as existing built-in Source writes. -- Each **collect** call covers one Scope and carries at most `max_targets` (manifest setting, default 50) enabled - targets; with more targets, calls rotate by target key, and targets reporting `has_more` go first in the next call. - The executor applies the same validation, `resolve`, and Projections as write to every returned Source, then saves all - Sources, writes each target's checkpoint and latest collection time, and acknowledges the call in one transaction. If - any entry fails, nothing is committed. A total entry count above `max_items` or a target key absent from the request - is a failure, not truncated. -- Targets absent from the result keep their checkpoints unchanged. -- Source identity is determined by `resolve`. When repeated collection of the same external state yields the same +- **Definition registration**: the Source types and Projections in `definitions` are registered in the existing Source + definition registry at startup; a conflict with a built-in type or another plugin fails the check. +- **Writes**: callers submit content through the existing Source write APIs (`POST /v1/scopes/{scope_id}/sources`, or the + write usage of MCP `pc_source_add`). Core validates the input with `definition.input_class`, calls `resolve` to obtain + the Source, computes each Projection, and saves it, with the same semantics as built-in Source writes. This step runs + synchronously in the Server process with a timeout. +- **collect**: each invocation covers one Scope and carries at most `max_targets` enabled targets; when there are more, + targets rotate by key, and targets that returned `has_more` go first in the next invocation. The executor applies the + same validation, `resolve`, and Projections as writes to every returned Source, then in one transaction saves all + Sources, writes each target's checkpoint and last collection time, and acknowledges the invocation. If any item fails, + nothing is committed. Exceeding `max_items` in total, or returning a target key absent from the request, is a failure; + nothing is truncated. Targets absent from the result keep their checkpoint. +- **Source identity**: determined by `resolve`. When repeated collection of the same external state yields the same identity, existing deduplication rules apply and no duplicate Source is created. -### 1.3 Collection targets +**Collection targets** are business data stored by core and, like Sources, are not plugin lifecycle state. Management reuses +the existing Source management APIs: -Collection targets are business data stored by core; like Sources, they are not plugin lifecycle state: - -| Operation | Interface | Notes | +| Operation | API | Description | | --- | --- | --- | -| Add | MCP `pc_source_add` (collect use), `POST /v1/scopes/{scope_id}/sources/plugins/{name}/targets` | Validate input with `TargetInput` and call `identify_target` for the key and normalized input. The same key with the same normalized input returns the existing target; the same key with different input returns a conflict. A new target is saved enabled and a collection is requested | -| Inspect | MCP `pc_source_status`, `GET /v1/scopes/{scope_id}/sources/plugins/{name}/targets` | Target key, normalized input, enabled state, latest successful collection time, latest failure category | -| Disable / re-enable | MCP `pc_source_target_set_enabled`, `PATCH .../targets/{key}` | Disabled targets leave collect requests while their checkpoints and collected Sources are retained; re-enabled targets continue from the old checkpoint | +| Add | MCP / HTTP `pc_source_add` (`capability` is the plugin name) | Validates the input with `target_input` and calls `identify_target` to get the key and normalized input. The same key with the same normalized input returns the existing target; the same key with a different input is a conflict. A new target is saved as enabled and one collection is requested | +| View | MCP / HTTP `pc_source_status` | Target key, normalized input, enabled state, last successful collection time, last failure category | +| Disable / resume | MCP / HTTP `pc_source_target_set_enabled` | A disabled target is left out of collect requests, and its checkpoint and collected Sources are kept; resuming continues from the saved checkpoint | -- Adding and changing targets requires Source write permission on the Scope, and the Scope must be within the plugin's +- Adding and changing targets require Source write permission on the Scope, and the Scope must be within the plugin's `scopes`. -- `identify_target` must be pure: it only validates and normalizes, without contacting external systems or checking - external authorization. Whether an external resource exists or is readable is determined by the plugin during - collect and expressed as an exception or empty result. Like write, it runs synchronously in the Server process under +- `identify_target` must be a pure function: it only validates and normalizes, without contacting external systems or + checking external authorization. Whether an external resource exists or is readable is decided by the plugin during + collect and expressed as an exception or an empty result. Like writes, it runs synchronously in the Server process with a timeout. -- The manifest's `collect.targets` are imported once only when their keys do not exist. Afterward core's data is - authoritative; changing the manifest neither overwrites nor deletes existing targets. +- `pc_source_capabilities` lists the Source types and target inputs plugins provide. + +## 3. Artifact plugin contract -### 1.4 Artifact plugins, the writer, and the write transaction +### 3.1 Request ```python class ArtifactGenerator(Protocol): @@ -486,361 +559,446 @@ class ArtifactGenerator(Protocol): class GenerateRequest(BaseModel): scope_id: str - reason: Literal["schedule", "sources", "manual"] - sources: Sequence[GenerationSource] # This window; empty when inputs is not configured - params: JsonValue | None = None # Manual triggers only + reason: Literal["schedule", "sources"] + sources: Sequence[GenerationSource] # this window; empty when inputs is not declared class GenerationSource(BaseModel): - ref: SourceRef # Exact Source reference - projection: JsonValue # Content of the Projection named by inputs.projection + ref: SourceRef # exact Source reference + position: int # the Source's journal position, for processing in collection order + projection: JsonValue # the Projection named by inputs.projection +``` -class ArtifactWriter(ArtifactReads): - family: str # Fixed to the manifest's generate.family +### 3.2 Writer and write transaction +```python +class ArtifactWriter(ArtifactReads): + family: str # fixed to the declared family def stable_id(self, *parts: str) -> str: ... def transaction(self) -> AbstractAsyncContextManager[ArtifactTransaction]: ... class ArtifactTransaction(ArtifactReads): - ... # Reads (see 1.5) plus write operations (see the table below) + ... # reads (see 3.4) plus write operations (see the table below) ``` `generate` runs in two parts: - **Outside the transaction**: the plugin reads committed data through `writer` (each read uses its own short read-only transaction) and does slow work such as model calls and vector computation. -- **The write transaction block**: `async with writer.transaction() as tx:`. When the block starts, core opens a write - transaction and rechecks the Supervisor lease and invocation intent. Writes in the block execute directly in that - transaction and immediately return a real `ArtifactRef` (with revision), and reads in the block see what the block has - written. When the block ends normally, core CAS-advances the cursor, acknowledges the call, and commits, all in the same - transaction. If the block raises, exceeds `transaction_timeout_seconds`, or conflicts at commit, the transaction rolls - back entirely and the call fails. +- **The write transaction block**: `async with writer.transaction() as tx:`. + - When the block starts, core opens a write transaction and re-checks the Supervisor lease and the invocation intent. + - Writes inside the block execute directly in that transaction and immediately return the written `ArtifactSnapshot` + (with the exact revision); reads inside the block see what the block has written. + - When the block ends normally, core advances the cursor by CAS, acknowledges the invocation, and commits, all in the + same transaction. + - If the block raises, exceeds `transaction_timeout_seconds`, or hits a commit conflict, the whole transaction rolls back + and the invocation fails. Rules: -- Each call opens at most one write transaction block; opening a second raises. The cursor and call acknowledgment - commit only with this one transaction, so a window's core data is never partially committed. -- Returning normally without opening a write transaction block means the window needs no output; the executor commits - the cursor advance and call acknowledgment on its own. -- After the write transaction block commits, exceptions raised by later code in `generate` no longer affect the - committed results; the executor records them as post-commit errors and does not retry the window. Plugins should make - the write transaction block the last step of `generate`. -- The write transaction block's time limit defaults to 30 seconds and counts toward the whole call's `timeout_seconds`. - The block should contain only reads, write operations, and writes to plugin-owned storage, not model calls. +- Each invocation opens at most one write transaction block; opening a second one raises. The cursor and the + acknowledgment commit only with this transaction, so a window's core data is never partially committed. +- If the plugin returns normally without opening a write transaction block, the window produces nothing, and the executor + commits the cursor advance and the acknowledgment on its own. +- Once the write transaction block has committed, exceptions raised by later code in `generate` no longer affect the + committed result; the executor records them as post-commit errors and does not retry the window. Plugins should make the + write transaction block the last step of `generate`. +- The block's time limit counts toward the invocation's `timeout_seconds`. Inside the block, plugins should only read, + write, and write their own storage, never call a model. -**Write operations**: the write transaction covers every generic Artifact write operation. Each reuses core's existing -implementation and validation rules and raises immediately on a violation: +### 3.3 Write operations -| Operation | Reused core implementation | Semantics | +The write transaction covers every generic Artifact write operation. Each reuses core's existing implementation and +validation and raises immediately when a rule is violated: + +| Operation | Core implementation reused | Semantics | | --- | --- | --- | -| `put(artifact_id, *, content, search_text=None, source_refs=(), artifact_refs=())` | `ArtifactRepository.create` / `revise` | Creates revision 1 when absent; otherwise appends a revision against the current head. When content and `search_text` equal the current revision, it is a no-op returning the current reference | -| `revert(artifact_id, *, to_revision, reason)` | The rollback path of `revise` | Appends a new revision with a historical revision's content, recording `restored_from_revision` and the reason | -| `set_lifecycle(artifact_id, state, *, replacement_artifact_id=None)` | `ArtifactGovernanceRepository.transition` | active ↔ deprecated (optionally naming a replacement), any state → retired (irreversible). Changes the head's governance state without appending a revision; frozen merge inputs cannot change individually | -| `merge(inputs, *, into, content, search_text=None, source_refs=())` | The shared merge method ([RFC #1859](1859_artifact_merge.md)) | Takes the current revisions of `inputs`, creates `into`, records the merge relation, and freezes the inputs. `into` must be a new ID | -| `preview_undo_merge(artifact_id)` / `undo_merge(artifact_id, *, preview=None)` | Preview and restoration of the shared merge method | Undoes the merge that produced the Artifact, restoring inputs and retiring the result; when executing a preview, stale previews are rejected if related state changed | -| `bind_handle(handle_key, ref)` | Artifact handle binding | Binds a stable handle to a given revision | -| `set_tags(artifact_id, tags)` | Artifact tag replacement | Replaces the Artifact's tags entirely | - -**Write constraints**: - -- Only Artifacts in this Scope and in the manifest's `generate.family` can be touched; plugins cannot name another - family or Scope. -- `source_refs` may reference only Sources in this window; `artifact_refs`, `to_revision`, and `bind_handle` targets must - be revisions that actually exist in this family. -- Operation count, single-content size, and total bytes per write transaction block are bounded; exceeding a limit - raises and rolls back, without truncation or partial commit. -- Core neither validates the business structure of `content` nor stores or freezes a schema. The plugin owns content - structure evolution and may carry a version field inside `content`. - -### 1.5 Reads +| `put(artifact_id, *, content, source_refs=(), artifact_refs=(), expected_revision=None)` | The change path of the shared merge methods | Creates revision 1 if absent; otherwise appends a revision on the current head. With `expected_revision`, a different current revision is a conflict, enabling optimistic concurrency for reading outside the transaction and writing inside it. Identical content to the current revision is a no-op that returns the current snapshot | +| `revert(artifact_id, *, to_revision)` | The shared restoration method (selected revision) | Appends a new revision with a historical revision's content and records `restored_from_revision`; a forgotten or merge-frozen target is restored as well | +| `set_lifecycle(artifact_id, state, *, replacement_artifact_id=None)` | The shared forget and restore methods; `ArtifactGovernanceRepository.transition` for a replacement or retirement | `deprecated` forgets, `active` restores, and `retired` is irreversible. Restoring uses shared restoration: it records a new revision with the same content, and an input frozen by a merge undoes that merge as one group. Forgetting and retiring append no revision | +| `merge(inputs, *, into, content, source_refs=(), artifact_refs=())` | The shared merge method ([RFC #1859](1859_artifact_merge.md)) | Takes the current revisions of `inputs`, creates `into`, records the merge, and freezes the inputs. `into` must be a new ID; when the same inputs are merged again after an undo, the plugin should include the input revisions in `stable_id` to avoid colliding with the retired result | +| `preview_undo_merge(artifact_id)` / `undo_merge(artifact_id)` | Preview and restoration of the shared merge methods | Undoes the merge that produced the Artifact, restoring its inputs and retiring the result, and returns exact references to what was restored and retired | +| `bind_handle(handle_key, ref)` | Artifact handle binding | Binds a stable handle to the given revision | +| `set_tags(artifact_id, tags)` | Artifact tag replacement | Replaces the Artifact's tags as a whole | + +Write constraints: + +- Writes apply only to Artifacts of the declared `family` in this Scope; plugins cannot name another family or Scope. +- `source_refs` may only reference Sources in this invocation's window; `artifact_refs`, `to_revision`, and the target of + `bind_handle` must be revisions that exist in this family. +- Core extracts search text from the content according to the declared `search_fields`; plugins do not supply it + separately. The same content yields the same search text whether a plugin or a user wrote it. +- Each write transaction block has limits on operations (1000), single content size (256 KiB), and total bytes (8 MiB); + exceeding one raises and rolls back, with no truncation or partial commit. +- Core does not validate the business structure of `content`, nor store or pin a schema. The plugin owns how its content + structure evolves and may keep a version field in `content`. + +### 3.4 Reads `ArtifactWriter` (outside the transaction, committed data only) and `ArtifactTransaction` (inside the transaction, also seeing what the block has written) offer the same read methods, scoped to this Scope and this family: ```python +class ArtifactSnapshot(BaseModel): + ref: ArtifactRef # exact revision + content: dict[str, JsonValue] + lineage: ArtifactLineage # directly referenced Sources and Artifacts + lifecycle_state: LifecycleState # active / deprecated / retired + merged_into_id: str | None # the merge result when frozen by a merge + replacement_artifact_id: str | None + governance_generation: int # governance state version, to detect changes after a read + class ArtifactReads(Protocol): async def get(self, artifact_id: str, *, revision: int | None = None) -> ArtifactSnapshot | None: ... - async def revisions(self, artifact_id: str, *, cursor: str | None = None, limit: int = 50) -> RevisionPage: ... + async def revisions(self, artifact_id: str, *, cursor: str | None = None, limit: int = 50) -> ArtifactPage: ... async def list(self, *, states: Collection[LifecycleState] = ("active",), tags: Collection[str] = (), cursor: str | None = None, limit: int = 50) -> ArtifactPage: ... - async def search(self, query: str, *, states: Collection[LifecycleState] = ("active",), - limit: int = 10) -> Sequence[ArtifactSnapshot]: ... - async def by_source(self, ref: SourceRef, *, cursor: str | None = None, limit: int = 50) -> ArtifactRefPage: ... - async def lineage(self, ref: ArtifactRef) -> ArtifactLineage: ... - async def governance(self, artifact_id: str) -> ArtifactGovernance: ... # Lifecycle, replacement, merge ownership - async def merge_record(self, artifact_id: str) -> ArtifactMergeRecord | None: ... - async def tags(self, artifact_id: str) -> ArtifactTagSet: ... + async def search(self, query: str, *, limit: int = 10) -> Sequence[ArtifactSnapshot]: ... + async def by_source(self, ref: SourceRef) -> Sequence[ArtifactSnapshot]: ... + async def merge_inputs(self, artifact_id: str) -> Sequence[ArtifactRef]: ... + async def tags(self, artifact_id: str) -> Sequence[str]: ... async def resolve_handle(self, handle_key: str) -> ArtifactRef | None: ... ``` -- `get` reads the current content or a given revision; `revisions` lists the full history. -- `list` and `search` can filter by lifecycle state, including deprecated, retired, and frozen merge inputs, to find - "data that already existed". `search` is core's text search (see section 5); inside the write transaction it finds - content the block has written but not yet committed. -- `by_source` finds Artifacts whose lineage references a Source, so plugins can locate Artifacts to revise after a - Source changes. -- Snapshots returned by reads carry exact revisions and can be used directly as `artifact_refs` or `to_revision` in - write operations. -- When a plugin needs search core does not provide, such as vector similarity, it uses its own index and then defers to - core's current revision through `get`. - -### 1.6 Consistency and idempotency - -**Core data**: a window's Artifact writes, cursor advance, and call acknowledgment commit in one transaction. A failure -leaves core data completely unchanged, so retries start clean; after success the cursor has passed the window, so the -same window is never delivered again. Core data is committed exactly once per window. Operations in the write -transaction act directly on the database, and conflicts (for example, a head changed concurrently after being read) are -reported immediately at the operation or at commit, rolling back everything for the Supervisor to reschedule. Plugin -families expose no public write interfaces in the first version, and calls are single-flight per plugin and Scope, so -such conflicts occur only in abnormal cases. - -**Reprocessing the same business object**: manual reruns, scheduled cleanups without `inputs`, and similar scenarios -process objects already processed. Plugins derive Artifact IDs from business identity with `writer.stable_id(...)`; -`put` is a no-op for identical content, `set_lifecycle` is a no-op when the target state equals the current one, and -`merge` is a no-op when all inputs are already merged into the same result, so no duplicate Artifacts or meaningless -revisions appear. - -**Plugin-owned storage**: plugin indexes and other storage are a different database from core and are outside core's -transaction. Core does not guarantee consistency between them; the plugin does. Recommended practice: - -1. Write plugin-owned storage inside the write transaction block, after obtaining the `ArtifactRef`, upserting by - `artifact_id` (with revision when needed). A failed write to plugin-owned storage then rolls back the core writes too. -2. If the write transaction fails to commit, plugin-owned storage may keep entries from that attempt. The retry - processes the same window, derives the same stable IDs, and overwrites them. -3. Check results read from plugin-owned storage with `get`, confirming the Artifact exists in core with a matching - revision; ignore mismatched entries, or periodically rebuild from core data. - -`generate` itself may run more than once due to retries, so model calls and external requests issued by the plugin are -not exactly once. - -## 2. Plugin executor and consumption cursor - -The plugin executor is core code in the Worker that implements the Supervisor Worker entry and replaces the old -`run_artifact_plugin_worker`. One Artifact plugin call proceeds as follows: +- A snapshot carries the exact revision, lineage, and current governance state together, so plugins need not read lineage + and lifecycle separately. +- `get` reads the current content or a given revision; `revisions` lists the full history; `merge_inputs` reads the exact + inputs of a merge result. +- `list` filters by lifecycle state and tags, including deprecated, retired, and merge-frozen Artifacts, for finding data + that already existed. +- `search` is core's text search (see section 6) and covers only active current revisions; inside the write transaction it + also finds content the block has written but not yet committed. +- `by_source` finds the Artifacts whose lineage references a given Source, for locating Artifacts to revise after a Source + changes. +- Every returned snapshot carries an exact revision and can be used directly as `artifact_refs` or `to_revision` in writes. + +### 3.5 Consistency and idempotency + +- **Core data**: each window's Artifact writes, cursor advance, and invocation acknowledgment commit in one transaction. On + failure core data is unchanged and the retry starts clean; on success the cursor has passed the window, so the same + window is never delivered again. Conflicts (for example, a head changed by a user after it was read) are reported when + the operation runs or at commit; the whole transaction rolls back, the Supervisor reschedules, and the next invocation + reads the changed state. +- **Processing the same business object again**: scenarios such as scheduled tidying without Source inputs process objects + that were already processed. Plugins derive Artifact IDs from business identity with `writer.stable_id(...)`; `put` is a + no-op for identical content, `set_lifecycle` is a no-op when the target state equals the current one, and `merge` is a + no-op when every input is already merged into the same result. No duplicate Artifacts or meaningless revisions result. +- **Plugin-owned storage**: the plugin's indexes and other storage are different databases outside core's transaction. + Core does not guarantee their consistency with core data; the plugin does. Recommended practice: + 1. Write owned storage inside the write transaction block after receiving the `ArtifactSnapshot`, upserting by + `artifact_id`. A failed write to owned storage then rolls back the core writes too. + 2. If the write transaction fails to commit, owned storage may keep entries from that attempt. The retry processes the + same window, derives the same stable IDs, and overwrites them. + 3. Treat the owned index as a source of candidates: read each hit's current snapshot from core with `get`, and use that + snapshot's content and state. Entries left by rollbacks and user changes then cannot make the plugin read stale + content. Plugins may also rebuild their index from core data periodically. +- `generate` itself may run several times because of retries; model calls and external requests a plugin makes are not + exactly-once. + +## 4. User changes to plugin Artifacts + +Users change plugin Artifacts through core's generic Artifact APIs. Server executes each change synchronously in a short +transaction, without calling the plugin or going through the Supervisor. + +**Edits and tags** + +| API | Rules | +| --- | --- | +| `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | The body is `{"content": {...}}`, stored as is, with search text taken from the declared `search_fields`; `If-Match` that differs from the current revision returns 412; the Artifact must exist. As with built-in edits, the submitted content is also stored as an internal Content Source (`lineage_only`) recorded in lineage as the new revision's direct source; such Sources never enter a plugin's input window | +| `GET` / `PUT .../{artifact_id}/tags`, `POST /v1/scopes/{scope_id}/artifact-tags/query` | The same as for built-in families; tags create no content revision | + +**Merge, forget, and restoration** + +These come from the shared merge engine (`ArtifactMergeService`, [RFC #1859](1859_artifact_merge.md)), which was designed +to be family-agnostic: each family supplies an adapter that builds merged content and maintains its own index, and the +engine does version checks, cascading restoration, lineage records, and permission checks. The runtime finds a family's +engine instance through a registry (`ArtifactMergeApplication`). Experience and Atomic Memory are the registered built-in +families today; when Server loads a plugin, it creates an engine instance for the plugin family (the plugin adapter plus +the shared Artifact access policy) and registers it. + +A set of family-agnostic APIs sits on the registry and works for every registered family; unregistered families return +422: + +| API | Rules | +| --- | --- | +| `GET .../{artifact_id}/governance` | Returns the current revision, lifecycle state, `state_version`, and merge target | +| `POST /v1/scopes/{scope_id}/artifacts/{family}/merges` | Takes at least two current Artifacts as `artifact_id`, `revision`, and `state_version`, plus the result's complete content; creates the result Artifact and freezes every input. Any changed input returns 409 | +| `POST .../{artifact_id}/forget` | Takes `revision` and `state_version`; marks the Artifact deprecated with no replacement, appends no revision, and removes it from search | +| `POST .../{artifact_id}/restorations` | `operation` is `restore` (default; `revision` restores earlier content) or `undo_merge`; restores and retires as one group | + +- **Permissions**: checked by the engine, the same as for built-in families. Changing an existing Artifact needs write + permission on it (`artifact.write`); creating a merge result needs the Scope contribute permission, and the caller owns + the result. The writer in plugin invocations uses the same engine class and adapter with different permissions: a + plugin has full write access to its family's Artifacts, including results of user merges. +- **No preview**: Atomic Memory's dedicated APIs can preview a restoration's effect and then execute it with a signed + token. The generic APIs offer no preview and therefore need no signing key; restoration and undo merge act on the state + at execution time, while merge and forget are checked by `revision` and `state_version`. +- **Merge content**: merge content for a plugin family is opaque JSON; core checks only that it is a JSON object and takes + search text from `search_fields`. Built-in families validate it against their own content models. +- **Concurrency**: user changes and plugin invocations can overlap. A plugin that reads outside the transaction and writes + with `expected_revision` inside it detects the conflict and rolls back; its next invocation decides again from the + changed state, so a user change is never silently overwritten by the plugin. +- **No notification**: as for built-in families, user changes are recorded only in lineage and governance state and do not + notify the plugin. The plugin reads the new state through its writer in its next invocation. +- **Creation**: the public APIs do not create plugin Artifacts directly, except merge results. A caller who wants to add + knowledge writes a Source. + +## 5. Plugin executor and consumption cursor + +The plugin executor is core code in the Worker that implements the Supervisor's Worker entry point, replacing the old +`run_artifact_plugin_worker`. One Artifact plugin invocation looks like this: ```text Worker receives (plugin:, scope_id, request_generation) - -> Short transaction: ScopeInvocation.start; load the cursor for (scope_id, plugin:) from pc_source_cursors - -> read the journal high watermark and fix the window (after, through], through ≤ after + max_sources - -> read the window with list_window_for_generation, filter by inputs.source_types, compute inputs.projection - -> Load the plugin, call the factory, create the writer, call generate (under timeout_seconds) - -> Reads outside the transaction: separate short read-only transactions - -> Write transaction block starts: open a transaction; guard (lease and invocation intent) - Writes in the block execute in that transaction - Block ends: CAS the cursor to through; acknowledge; commit - -> generate returns without opening a write transaction block: a short transaction performs the cursor CAS and acknowledgment + -> short transaction: ScopeInvocation.start; read the cursor for (scope_id, plugin:) in pc_source_cursors + -> read the journal high watermark and choose the window (after, through], through ≤ after + max_sources + -> list_window_for_generation reads the window, filters by inputs.source_types, computes inputs.projection + -> empty window with inputs declared: a short transaction does the cursor CAS and acknowledge; the plugin is not loaded + -> import plugin.py, compare the declaration, call create, build the writer, call generate (bounded by timeout_seconds) + -> reads outside the transaction: separate short read-only transactions + -> the write transaction block starts: open a transaction; guard (lease and invocation intent) + writes inside the block run in that transaction + the block ends: cursor CAS to through; acknowledge; commit + -> generate returns without opening a write transaction block: a short transaction does the cursor CAS and acknowledge ``` - Sources in the window whose type does not match are filtered out, but the cursor still advances past the whole window; - a matching Source missing the required Projection fails the call rather than being skipped. -- The cursor advances only in the executor's commit transaction; plugins never receive its value and have no write - interface for it. The cursor CAS ensures a window commits only once even if leadership changes or duplicate dispatch - happens between reading the window and committing. -- `on_sources` reuses the existing `SourceProcessingPendingProvider`: when the journal high watermark exceeds the - cursor, the Scope is marked pending and the Supervisor dispatches within budget. The check uses persisted data, so a - Server restart loses no pending input. -- Manual triggers and target additions reuse existing invocation intents - (`ArtifactProcessingIntentRepository.request`): the acceptance transaction advances the requested generation, - multiple requests coalesce, and requests arriving during a run are handled by a later call. -- Without `scopes`, automatic scheduling applies to every Scope whose journal is ahead of the cursor; with `scopes`, - both automatic scheduling and manual triggers accept only listed Scopes. - -Collect calls have the same structure: the input becomes the Scope's enabled targets and their checkpoints, and the -commit carries Sources and each target's new checkpoint. Collect needs no write transaction block; the executor commits -from the return value at once. - -## 3. Loading and lifecycle + a matching Source that lacks the required Projection fails the invocation instead of being skipped. +- The cursor advances only in the executor's commit transaction; plugins never see its value and have no way to write it. + The cursor CAS ensures that even if leadership changes or the invocation is dispatched twice between reading the window + and committing, the same window commits only once. +- Pending work is discovered in two ways: a Source write in the Server process marks the plugin bindings that watch the + Scope as pending, and at startup the existing `SourceProcessingPendingProvider` fills in Sources written by other + processes from the gap between the journal high watermark and the cursor. The Supervisor admits pending Scopes only when + a binding's interval or cron is due, so `on_sources` maps to an admission interval of about 2 seconds. These decisions + rest on persisted data, so pending input is not lost when Server restarts. +- Without `scopes`, automatic scheduling covers every Scope whose journal is past the cursor; with `scopes`, only Scopes + on the list are accepted. + +A collect invocation has the same structure: its input is the Scope's enabled targets and their checkpoints, and its +commit is the Sources and every target's new checkpoint. Collect needs no write transaction block; the executor commits +the return value at once. Adding a collection target requests one collection through the existing invocation intent +(`ArtifactProcessingIntentRepository.request`). + +## 6. Public reads and search + +Core serves all public reads of plugin families without running plugin code: + +- list / get / revisions / tags reuse the existing generic Artifact read paths. Plugin family `content` is returned as JSON + as is, without a schema in `pc_artifact_family_schemas`. Merge-frozen, deprecated, and retired Artifacts are presented + under the shared merge and lifecycle rules. +- List items carry only metadata; `title` and `summary` are empty for plugin families, and content is read with get. +- Search is core's generic text search for plugin families and, like the existing search APIs, accepts only text queries: + - On write, core takes the text of the `search_fields` from the content, analyzes it with the existing lexical analysis, + writes it to `pc_artifact_heads.searchable_text`, and keeps it with the revision, so restoring an old revision restores + its search text with its content. + - On search, core matches that column and returns only active current revisions the caller may read in the requested + Scope. Artifacts without searchable text do not take part. + - The writer's `search` uses the same implementation. + - On SQLite, plugin families share `pc_artifact_fts` with Experience and Skill: rebuilding that helper table backfills + every head with search text, and the readiness check compares only built-in family rows. The implementation PR must + present how search works on OceanBase and validate it at scale. +- Review, Recall, and Prepared Context are not open to plugin families in the first version; those APIs return an explicit + "not supported" error, and `GET /v1/capabilities` lists plugin families. + +## 7. Loading and lifecycle ```text -Server start - -> Scan direct children of POWERCONTEXT_SERVER_PLUGINS_DIR (load nothing when unset) - -> For each directory: parse and validate the manifest (name matches directory, kind matches configuration sections) - -> In a time-limited check child process, import the entry and confirm the factory, Config, and other required - exports match the kind, without calling the factory - -> Source plugins: import definitions, TargetInput, and identify_target in the Server process and register Source types - -> Import initial collection targets from the manifest that do not exist yet - -> Register the family access profile; register a Supervisor binding when collect / generate is configured - -> Any step fails: record the error, mark the plugin failed, continue with the next -Server stop - -> Stop dispatch and reclaim Workers through the existing Supervisor shutdown +Server starts + -> scan the direct subdirectories of POWERCONTEXT_SERVER_PLUGINS_DIR (no plugins load if it is unset) + -> for each directory: import plugin.py in a time-limited check child process and validate the declaration, without calling create + -> Source plugins: import definitions, target_input, and identify_target in the Server process and register Source types + -> register the family for reads and search; register a Supervisor binding when there is a schedule or trigger + -> any step fails: record the error, mark the plugin failed, and continue with the next one +Server stops + -> stop dispatching and reclaim Workers through the existing Supervisor shutdown ``` -A plugin has only two outcomes, `loaded` and `failed`, held in process memory. Server does not persist plugin enabled -state, registration generations, or configuration snapshots; each start rechecks the directory as it is then. Without -hot reload, no registration generation is needed to fence old Workers; the existing Supervisor lease fence already -rejects stale commits after leadership changes. - -Workers import plugin code on every call. If a plugin directory changes while Server runs, newly dispatched Workers may -see the new code; deployers should restart Server after changes, and the first version does not detect this case. - -## 4. Execution constraints - -- **Isolation**: `collect` / `generate` run in Worker child processes started by the Supervisor; crashes, memory leaks, - and blocking do not affect Server. Child processes isolate ordinary failures but are not a sandbox for untrusted code: - plugins run with the Worker's operating-system permissions and can access its files, network, and environment. -- **Timeouts and budgets**: `timeout_seconds` (default 300) is the total Worker time limit, - `transaction_timeout_seconds` (default 30) the write transaction block limit, and `max_workers` (default 1) the - plugin's concurrency limit across Scopes. These map to existing Supervisor binding parameters or are enforced by the - executor. Input bytes, write operation count, and output bytes per call are bounded; exceeding a limit fails the call. -- **Lock impact of the write transaction**: while the write transaction block is open, it locks the heads and cursor row - it touches; on SQLite, a write transaction blocks all other writes to the database. The block must therefore be short, - and exceeding its limit rolls it back. -- **Failure and backoff**: a failed call commits no core data, and that plugin and Scope enter the Supervisor's existing - backoff without affecting other Scopes or plugins. Core never skips input automatically after repeated failures; - persistent failures require fixing the plugin or data and restarting. A target that keeps failing during collect - blocks the other targets in the same Scope; disable it first. -- **Multi-instance deployments**: with several OceanBase Server instances, dispatch happens only on the instance holding - the Supervisor leadership lease; writes, target management, and reads may run on any instance. Plugin-owned storage - must be reachable from instances that run Workers. SQLite deployments have one Server process. - -## 5. Public reads and search - -Core handles all public reads of plugin families without running plugin code: - -- list / get / revisions / tags reuse the existing generic Artifact read path. Plugin family `content` is returned as - JSON without a schema from `pc_artifact_family_schemas`. Frozen merge inputs and deprecated and retired Artifacts are - presented under the shared merge and lifecycle rules. -- search is a generic text search core adds for plugin families; like the existing search interface, it accepts only - text queries. On write, `search_text` passes through existing lexical analysis into `pc_artifact_heads.searchable_text`, - and search matches on that column, returning only revisions in the requested Scope that the caller may read. Artifacts - without `search_text` are not searchable. The writer's `search` uses the same implementation. The implementation PR - specifies the search method on SQLite and OceanBase and validates it at scale. -- Review, Recall, and Prepared Context are not available for plugin families in the first version, nor are the public - create, revise, merge, lifecycle, handle, and tag write interfaces; writes to plugin families come only from the - plugin. Those interfaces return an explicit "unsupported" error, and `GET /v1/capabilities` reflects the operations - actually supported. - -## 6. Persistence +- A plugin has only two outcomes, `loaded` and `failed`, kept only in process memory. Server persists no plugin enabled + state, registration generation, or configuration snapshot; each startup re-checks the directory as it is. +- With no hot reload, no registration generation is needed to fence old Workers; the existing Supervisor lease fence is + enough to fence stale commits after a leadership change. +- A Worker re-imports plugin code and compares the declaration on every invocation. If the plugin directory is modified + while Server runs and the declaration changes, the call fails and asks for a restart; if only implementation code changes, + newly dispatched Workers use the new code. Deployers should restart Server after a change. +- Server instances in the read-only API role also check plugins and register families and Source types for reads, search, + and user changes, but do not dispatch Workers. + +## 8. Execution constraints + +- **Isolation**: `collect` / `generate` run in Worker child processes started by the Supervisor, so crashes, memory leaks, + and blocking do not affect Server. Child processes isolate ordinary failures; they are not a sandbox for untrusted code: + a plugin runs with the Worker's operating-system privileges and can access its files, network, and environment + variables. +- **Timeouts and budgets**: `timeout_seconds` is the total Worker limit, `transaction_timeout_seconds` is the write + transaction block limit, and `max_workers` is the plugin's concurrency limit across Scopes; each maps to an existing + Supervisor binding parameter or is enforced by the executor. +- **Lock impact of the write transaction**: while the block is open it locks the heads and cursor rows involved; on SQLite a + write transaction blocks all other writes to the database. The block must therefore be short, and exceeding its limit + rolls it back. +- **Database access**: Workers open the database without creating tables; Server prepares the schema. The existing + database readiness check verifies that every pending invocation intent has a persistently registered binding; bindings + prefixed with `plugin:` are determined by the plugin directory at startup, so the check admits them, and intents left + behind by a removed plugin are no longer processed. +- **Failure and backoff**: a failed call commits no core data, and that plugin in that Scope enters the Supervisor's + existing backoff without affecting other Scopes or plugins. Core never skips input automatically because of repeated + failures; persistent failures require fixing the plugin or the data and restarting. A target that keeps failing in + collect blocks the other targets in the same Scope; disable it first. +- **Multi-instance deployments**: with several Server instances on OceanBase, dispatch happens only on the instance holding + the Supervisor leader lease; Source writes, target management, reads, and user changes can run on any instance. Plugin-owned + storage must be reachable from the instance that runs Workers. A SQLite deployment has a single Server process. + +## 9. Persistence | Data | Handling | | --- | --- | -| `pc_sources`, Projections, the Source journal, `pc_source_definition_manifests` | Kept; results of Source plugins | -| `pc_artifacts`, `pc_artifact_heads`, `pc_artifact_lineage_*`, merge records, handles, tags | Kept; results of Artifact plugins | -| `pc_source_cursors` | Kept; Artifact plugin consumption cursors keyed by `(scope_id, plugin:)` | -| `pc_connector_checkpoints` | Kept; per-target checkpoints keyed by `(scope_id, target binding ID)` | -| `pc_source_collect_targets` | Added; collection targets: Scope, plugin name, target key, normalized input, enabled state, creation and update times, latest successful collection time, latest failure category | -| Supervisor invocation intent, pending, and lease tables | Kept; shared by plugin and built-in bindings | -| Columns added for plugins to `pc_artifact_processing_binding_states` (`plugin_id`, `plugin_family`, `binding_generation`, `desired_state`, `phase`, `last_error`) | Removed | +| `pc_sources`, Projections, the Source journal, `pc_source_definition_manifests` | Kept; Source plugin results | +| `pc_artifacts`, `pc_artifact_heads`, `pc_artifact_lineage_*`, merge records, handles, tags | Kept; Artifact plugin results | +| `pc_source_cursors` | Kept; Artifact plugin consumption cursors, keyed by `(scope_id, plugin:)` | +| `pc_connector_checkpoints` | Kept; each collection target's checkpoint, keyed by `(scope_id, the target's binding ID)` | +| `pc_source_collect_targets` | Added; collection targets: Scope, plugin name, target key, normalized input, enabled state, creation and update times, last successful collection time, last failure category | +| Supervisor invocation intent, pending, and lease tables | Kept; plugin bindings share them with built-in bindings, and they back `GET /v1/plugins/{name}/scopes` | +| Plugin columns added to `pc_artifact_processing_binding_states` (`plugin_id`, `plugin_family`, `binding_generation`, `desired_state`, `phase`, `last_error`) | Removed | | `pc_artifact_family_schemas` | Removed | -| Search state and vector index data used by custom-family declarative indexes | Removed; the implementation PR drops the tables after confirming built-in families do not depend on them | +| Search state and vector index data for custom family declarative indexes | Removed; the implementation PR confirms built-in families do not depend on them before dropping the tables | | Artifacts, heads, and lineage written by old Artifact plugins | Deleted, not migrated | -| `pc_source_write_plugins` | Removed; write entries come from the plugin directory at startup | -| The Source plugin management service's SQLite | Removed along with the service | -| Plugin-owned indexes and storage | Not part of core; managed by the plugin and deployer | +| `pc_source_write_plugins` | Removed; Source definitions come from the plugin directory at startup | +| The Source plugin management service's SQLite | Removed with the management service | +| Plugin-owned indexes and storage | Not part of core; managed by the plugin and the deployer | -A target's binding ID derives deterministically from the plugin name and target key, using a digest when it exceeds the -column width. Additions and removals happen in one new database migration whose notes list the added and dropped -tables, columns, and data ranges. +A target's binding ID is derived deterministically from the plugin name and the target key, using a digest when it exceeds +the column width. Additions and removals ship as one new database migration whose notes list the tables, columns, and data +ranges added and removed. -## 7. Interface changes +## 10. API changes | Change | Content | | --- | --- | -| Removed | `/v1/admin/artifact-plugins` and its subpaths; the Source plugin management service, plugin instance Runtime, and poll / reply channel; the `source load` command; MCP `pc_source_instances`, `pc_source_instance_set_enabled` | -| Removed | Declarative indexes for custom families (`ArtifactSearchDeclaration`, `TextIndex`, and related types), `ArtifactFamily` schema declarations, `ArtifactDraft`, and the old Worker plugin executor | -| Added | `GET /v1/plugins`, `GET /v1/plugins/{name}` (`server.admin`, read-only) | -| Added | `POST /v1/scopes/{scope_id}/artifacts/{family}/generate`, `POST /v1/scopes/{scope_id}/sources/plugins/{name}/collect` (`202 Accepted`) | -| Added | `GET` / `POST /v1/scopes/{scope_id}/sources/plugins/{name}/targets`, `PATCH .../targets/{key}` | -| Changed | Generic Artifact read and search interfaces support plugin families; `GET /v1/capabilities` lists plugin families | -| Changed | The write use of MCP `pc_source_add` uses plugin definitions and its collect use adds collection targets; `pc_source_capabilities` lists plugin Source types and target inputs; `pc_source_status` reports targets and their latest collection outcomes; `pc_source_target_set_enabled` acts on targets stored by core | -| Changed | Environment variable `POWERCONTEXT_SERVER_PLUGINS_DIR` replaces `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` and the Source plugin directory settings | +| Removed | `/v1/admin/artifact-plugins` and its subpaths; the Source plugin management service, plugin instance Runtimes, and poll / reply channels; the `source load` command; MCP `pc_source_instances` and `pc_source_instance_set_enabled`; the `plugin.yml` and `powercontext.yml` manifests | +| Removed | Custom family declarative indexes (`ArtifactSearchDeclaration`, `TextIndex`, and so on), `ArtifactFamily` schema declarations, `ArtifactDraft`, and the old Worker plugin executor | +| Added | `powercontext.plugins`: `SourcePlugin`, `ArtifactPlugin`, `Inputs`, `Interval`, `Cron`, `PluginContext`, and the writer and read interfaces | +| Added | `GET /v1/plugins`, `GET /v1/plugins/{name}`, `GET /v1/plugins/{name}/scopes` (`server.admin`, read-only) | +| Added | `GET .../{artifact_id}/governance`, `POST /v1/scopes/{scope_id}/artifacts/{family}/merges`, `POST .../{artifact_id}/forget`, `POST .../{artifact_id}/restorations`: generic APIs on the shared merge engine registry that work for every registered family (Experience and Atomic Memory among built-ins, plus loaded plugin families). Atomic Memory's dedicated APIs are unchanged | +| Changed | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` accepts plugin families with a `{"content": {...}}` body | +| Changed | The tag APIs accept plugin families: `TaggableArtifactFamily` changes from an enum to a string in the `ArtifactReadFamily` format, and Server accepts built-in families or loaded plugin families. The wire format is unchanged; generated clients lose the enum type | +| Changed | The generic Artifact read and search APIs support plugin families; `GET /v1/capabilities` lists plugin families | +| Changed | The write usage of `pc_source_add` uses plugin definitions, and its collect usage adds collection targets; `pc_source_capabilities` lists plugin Source types and target inputs; `pc_source_status` reports targets and their latest collection result; `pc_source_target_set_enabled` acts on targets stored by core | +| Changed | The environment variable `POWERCONTEXT_SERVER_PLUGINS_DIR` replaces `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` and the Source plugin directory setting | -`openapi/powercontext.yaml` is updated per this table and `src/powercontext/http/_generated/` regenerated. +After `openapi/powercontext.yaml` is updated per the table, `src/powercontext/http/_generated/` and each integration's +operation table are regenerated. -## 8. Observability +## 11. Observability -Plugin calls reuse the Supervisor and Worker's existing operation vocabulary, execution identities, and outcome fields, -and scheduled calls create no request ID. New fields: +Plugin calls reuse the Supervisor's and Worker's existing operation vocabulary, execution identities, and outcome fields, +and no request ID is generated for scheduled calls. New fields: | Field | Meaning | | --- | --- | -| `plugin_name` | Plugin name from the manifest | +| `plugin_name` | Plugin name | | `plugin_method` | `collect` or `generate` | -| `trigger` | `schedule`, `sources`, `manual`, `target_added` | -| `operation_count` | Number of write operations (Artifact) or Sources (collect) committed by the call | -| `target_count` | Number of targets carried by a collect call | +| `trigger` | `schedule`, `sources`, `target_added` | +| `operation_count` | Write operations committed (Artifact) or Sources committed (collect) in this invocation | +| `target_count` | Targets carried by this collect invocation | | `transaction_ms` | Time from the start of the write transaction block to commit or rollback, in milliseconds | -Metric labels may use `plugin_name` and `plugin_method`, never Scope IDs or target keys. Logs go through `log_safely` -and exclude Source content, Projections, Artifact content, target inputs, `params`, `config` values, and environment -variable values; plugin exceptions record only their type and known error category, never the exception message. New -fields are registered in the allowlists per the developer guide and documented in both locales' observability -documentation. +Metric labels may use `plugin_name` and `plugin_method`, never Scope IDs or target keys. Logs go through `log_safely` and +exclude Source content, Projections, Artifact content, target inputs, and environment variable values; plugin exceptions +are logged only by type and known error category, never by message. New fields are registered in the allowlists per the +developer guide and documented in both locales' observability docs. Call outcomes go only to logs and metrics and are not +persisted, so the operational APIs report no "last call result". # Drawbacks -- **Some plugin code runs in the Server process**: write must return synchronously, so Source definitions' `resolve`, - Projections, and `identify_target` do not go through a Worker. They should be pure data transformations, but blocking - or exceptions still affect request handling. Server applies timeouts to these calls and converts exceptions into - request errors. -- **The write transaction holds locks**: the write transaction block holds database locks, and on SQLite it blocks other - writes. A plugin doing slow work inside the block slows writes across Server until the block times out and rolls back. -- **Plugin-owned storage is not guaranteed consistent with core data**: they are not in one transaction, so a failed - commit may leave entries in plugin-owned storage; plugins overwrite them by stable ID and verify against core. -- **Every call starts a Worker and loads the plugin**: child startup and factory initialization have fixed costs, - unsuitable for high-frequency calls every few seconds; plugins cannot reuse connections or caches across calls. -- **Not a security sandbox**: Workers isolate ordinary failures but do not restrict plugin access to files, network, and - credentials, nor provide hard memory or CPU quotas. The writer constrains plugin writes to core data; it cannot stop - plugin code from reaching other resources available to the Worker. -- **Compatible dependencies required**: all plugins share one Python environment with Server; deployers resolve +- **Some plugin code runs in the Server process**: Source writes must return synchronously, so a Source definition's + `resolve`, Projections, and `identify_target` bypass the Worker. They should be pure data transformations, but blocking + or exceptions still affect request handling. Server bounds these calls with timeouts and turns exceptions into request + errors. +- **The write transaction holds locks**: the block holds database locks, which on SQLite block other writes. A plugin that + does slow work in the block slows every write in Server until the block times out and rolls back. +- **Plugin-owned storage is not guaranteed consistent with core data**: they are not in the same transaction, so a failed + commit can leave entries in owned storage, and user changes do not notify the plugin. Plugins must overwrite by stable ID + and defer to core's current data. +- **Every call starts a Worker and loads the plugin**: process startup and factory initialization have a fixed cost that + rules out second-level high-frequency calls, and plugins cannot reuse connections or caches across calls. +- **Not a security sandbox**: Workers isolate ordinary failures but do not restrict a plugin's access to files, network, or + credentials, nor impose hard memory or CPU quotas. +- **Dependencies must be compatible**: every plugin shares one Python environment with Server, and deployers resolve dependency conflicts. -- **Content structure is not governed by core**: core does not validate `content` schemas; plugins and applications - keep the interpretation of historical revisions compatible themselves. -- **Limited core search**: core provides only generic text search over `search_text`; vector and other search requires - plugin-built indexes. -- **Collection runs with Server**: collection executes in Server Workers, so Server must reach external data sources. -- **Code changes require restarts**: without hot reload, Server is briefly unavailable during plugin updates. +- **Deploying the same code twice takes extra care**: the family is part of the declaration, so copying the same code into + two directories fails the check with a family conflict. A plugin that needs several deployments should read identifiers + such as its family from environment variables. +- **Content structure is not governed by core**: core does not validate a `content` schema, so plugins and applications + must keep historical revisions interpretable; content submitted with `PUT` is interpreted only when the plugin reads it + later. +- **Core search is limited**: core provides only generic text search over declared fields; vector and other search requires + a plugin-owned index. +- **Collection runs with Server**: collection runs in Server's Workers, so Server needs access to the external data sources. +- **Code changes require a restart**: without hot reload, Server is briefly unavailable while plugins update. # Rationale and alternatives -| Approach | Tradeoff | +| Option | Trade-off | | --- | --- | -| Keep the current design and only simplify its interfaces | Limited improvement for plugin authors; the lifecycle state machine, management service, and schema freezing still cost maintenance | -| Run plugins in the Server process | Simplest deployment and calls, but plugin failures can bring Server down; the Supervisor already provides isolation and scheduling, so there is no reason for another mechanism | -| Plugins fully manage their own storage and cursors | Core read interfaces would not work directly, every plugin would reimplement window reads and idempotency, and cursors and results would be hard to commit atomically | -| Plugins write item by item through a context interface, each committed separately, then return success or failure | Writes and cursor advance would be in separate transactions; failures would leave partial results and retries would write again | -| Plugins return drafts, or a writer stages operations in memory and commits them at the end | Commits atomically, but plugins cannot obtain real revisions before commit, making it hard to write their own indexes alongside; merge, merge undo, lifecycle, and other operations would need core's semantics reimplemented in memory; search cannot see staged content | -| Wrap all of `generate` in one transaction | Model calls may take minutes, and holding locks that long blocks other writes, blocking the whole database on SQLite; plugin-owned storage is another database that one transaction could not cover anyway | -| A short write transaction block inside `generate` (this proposal) | Slow work stays outside the block; inside, core's write operations and validation are reused directly, real revisions are returned, and reads see the block's writes; the block commits with the cursor and acknowledgment and rolls back on exceptions. The cost is holding locks inside the block, bounded by a time limit | -| Resident plugin processes (like Dify's plugin daemon) | Would reuse connections and support other languages, but needs process management, serialization, and cancellation protocols. Interfaces are request / response models, so execution can change later without changing plugin code | -| Support hot reload | Reintroduces load states, registration-generation fencing, and reclamation of old calls, conflicting with the goals | - -"The Supervisor dispatches Workers, plugins write directly inside a short write transaction block, and core commits it -with the cursor" was chosen because plugin authors need only a factory and one method; isolation, scheduling, triggers, -backoff, and leadership all reuse the Supervisor already in production; write operations reuse core's existing -implementation instead of a second set of semantics; Sources and Artifacts stay in core's shared storage, so existing -read interfaces, lineage, merge records, and access control apply directly; and core manages only the two kinds of core -data and their progress, leaving the rest to plugins. +| Keep the current design and only simplify the interfaces | Limited improvement for plugin authors; the cost of maintaining lifecycle state machines, the management service, and schema pinning remains | +| Describe plugins in a manifest file (YAML) | Plugins must maintain a manifest alongside code and keep their fields in sync, and variable substitution, config models, and export conventions in the manifest are another set of rules. Declarations in code are validated by the same Pydantic model, and plugin authors write only Python | +| Run plugins in the Server process | Simplest to deploy and call, but a plugin failure can take Server down; the Supervisor already provides isolation and scheduling, so there is no reason to build another | +| Plugins manage their own storage and cursors entirely | Core's read APIs could not be used directly, every plugin would reimplement window reads and idempotency, and cursors would be hard to commit atomically with results | +| Plugins return drafts, or the writer stages operations in memory and commits at the end | Atomic, but plugins cannot see real revisions before the commit and struggle to write their own index; merge, undo merge, and lifecycle semantics would need reimplementing in memory; search would not see staged content | +| Wrap the whole `generate` in one transaction | Model calls can take minutes, and holding locks that long blocks other writes, the whole database on SQLite; plugin-owned storage is another database that one transaction cannot cover anyway | +| A short write transaction block inside `generate` (this proposal) | Slow work stays outside the block; inside it, core's write operations and validation are reused directly, real revisions are returned, and reads see the block's writes; at the end it commits with the cursor and acknowledgment, and an exception rolls back. The cost is holding locks inside the block, bounded by a time limit | +| Run user commands through the plugin (queue manual requests the plugin handles in its next invocation) | Plugins must parse commands and callers can only wait asynchronously; a command that always fails blocks later generation in the Scope; and the request queue must be persisted. Forget, restore, and edit are generic operations core already has, and running them synchronously in core is simpler | +| Plugin-specific trigger and management APIs | Callers would have to tell plugins from built-ins; the existing Source and Artifact APIs already cover the operations callers need | +| Generic merge APIs for plugin families only | Needs a separate family allowlist in the APIs. The registry already decides which families can be changed, and a family without an adapter cannot register; built-in permission checks live in the engine, so both entry points apply the same checks | +| Resident plugin processes (like Dify's plugin daemon) | Could reuse connections and support other languages, but needs process management, a serialization protocol, and a cancellation protocol. The interfaces are request / response models, so the execution mechanism can change later without changing plugin code | +| Hot reload | Would reintroduce load states, registration generation fencing, and reclaiming old calls, which contradicts the goals | + +Reasons for this choice: + +- A plugin author writes only `plugin.py`: one declaration, one factory, and one or two methods. +- Callers use only the existing Source and Artifact APIs and are unaware of plugins. +- Isolation, scheduling, triggers, backoff, and leadership all reuse the Supervisor already in production. +- Writes reuse core's existing implementation, so plugin writes and user changes follow the same semantics. +- Sources and Artifacts stay in core's unified storage, so the existing read APIs, lineage, merge records, and access + control apply directly. +- Core manages only the two core data kinds and their progress and leaves everything else to plugins. # Prior art -- **Dify plugins**: interfaces are defined per plugin type (tools, models, agent strategies, extension endpoints, and - others). A plugin declares its type and parameters in a manifest and implements the matching methods; Dify calls them - when a workflow or application needs them. Plugins run in a separate plugin daemon. This proposal keeps the "host - defines interfaces and invocation timing" approach and reuses PowerContext's existing Supervisor Workers for isolation. -- **Airbyte connector protocol**: a Source connector's `read` receives the previous state and emits records and new - state, which the platform persists. Collection checkpoints here use the same division of responsibility, stored per - target. -- **Transaction context managers**: interfaces such as SQLAlchemy's `async with engine.begin()` open a transaction when - the block starts, commit when it ends normally, and roll back on exceptions. `writer.transaction()` follows this usage - and adds the cursor and call acknowledgment to the commit. -- **Existing PowerContext foundations**: the [Artifact processing Supervisor](1515_artifact_processing_supervisor.md)'s - scheduling, invocation intents, Workers, and lease fence; the [shared merge method](1859_artifact_merge.md); - `ArtifactRepository`, `ArtifactGovernanceRepository`, and the handle and tag write implementations; - `SourceDefinition` / `SourceProjection`; the Source journal, `pc_source_cursors`, and `list_window_for_generation`; - `pc_connector_checkpoints`; and the `identify_target` identification rule from the old Source plugin design. +- **Dify plugins**: interfaces are defined per plugin type, plugins implement the matching methods, and Dify calls them + when a workflow or application needs them; plugins run in a separate plugin daemon. This proposal keeps the idea that the + host defines the interfaces and when to call them, and reuses PowerContext's existing Supervisor Workers for isolation. +- **The Airbyte connector protocol**: a Source connector's `read` receives the previous state and emits records and a new + state, which the platform persists. The collection checkpoints in this proposal split the work the same way and keep a + checkpoint per target. +- **pytest plugins and setuptools entry points**: plugins declare what they provide as Python objects that the host reads + after importing them, with no separate manifest file. +- **Transaction context managers**: interfaces such as SQLAlchemy's `async with engine.begin()` open a transaction when the + block starts, commit when it ends normally, and roll back on an exception. `writer.transaction()` follows the same usage + and also commits the cursor and acknowledgment. +- **Existing PowerContext foundations**: scheduling, invocation intents, Workers, and lease fences of the + [Artifact processing Supervisor](1515_artifact_processing_supervisor.md); the [shared merge methods](1859_artifact_merge.md); + the write implementations of `ArtifactRepository`, `ArtifactGovernanceRepository`, handles, and tags; the generic Artifact + read and write APIs; `SourceDefinition` / `SourceProjection`; the Source journal, `pc_source_cursors`, and + `list_window_for_generation`; `pc_connector_checkpoints`; and the Source management APIs with the `identify_target` + target identification rule. # Unresolved questions | Question | Notes | | --- | --- | -| Plugin-defined search | The existing search interface accepts only text queries, so core provides generic text search in the first version. Whether plugins may later provide implementations for the public search interface requires deciding which process runs search code | -| Write transaction block limits and defaults | Whether 30 seconds is the right default; whether SQLite and OceanBase use different defaults; per-block operation and byte limits | -| Shared model client | Whether `PluginContext` should expose Server's configured model client so LLM plugins need not bring their own credentials, with usage counted in existing model usage statistics | -| Deleting collection targets | The first version supports only disabling and re-enabling. Whether to fully delete a target and its checkpoint, and how to treat Sources already collected | -| Plugin families in Recall / Prepared Context | Requires candidate content and budget constraints; suggested as a separate proposal | +| Plugin-defined search | The existing search APIs accept only text queries, so the first version provides generic text search in core. Whether plugins may later implement the public search API depends on deciding which process runs search code | +| Default limits for the write transaction block | Whether 30 seconds is right; whether SQLite and OceanBase need different defaults; per-block operation and byte limits | +| A shared model client | Whether `PluginContext` should offer the model client Server already configures, so LLM plugins need not bring credentials and their usage counts toward the existing model usage statistics | +| Deleting collection targets | The first version supports only disabling and resuming. Whether targets and their checkpoints should be deletable, and what happens to Sources already collected | +| Plugin families in Recall / Prepared Context | Requires defining candidate content and budget constraints; a separate proposal is recommended | +| Shrinking the window | Built-in Atomic Memory shrinks the next Source window when the model times out. Plugins cannot affect the window size, so an oversized window keeps timing out. Whether plugins may request a smaller window for the Scope with an agreed exception | +| Windows that keep failing | A Source window that always fails blocks later invocations in the Scope. Whether to offer an operation that skips it or moves it to a dead letter, and how to record skipped input | +| Consuming every Source type | `inputs.source_types` must list each type; built-in Atomic Memory consumes every Source usable for generation. Whether to support `"*"` for all types | +| Built-in-only capabilities | Scope Prompt management, the write gate with decision observations, and model usage statistics serve only built-in families today, and plugins cannot use them. Whether and how to open them to plugins | +| Atomic Memory's dedicated APIs | With the generic merge APIs, Atomic Memory merge, forget, and restoration have a dedicated and a generic entry point that behave the same. Whether to keep the dedicated APIs or make them aliases of the generic ones | # Future possibilities -- Resident plugin processes that reuse connections and initialization while keeping interfaces unchanged. -- More plugin kinds, such as providing candidates to Recall or text fragments to Prepared Context. -- Plugin packaging and installation tools, such as installing from a Git URL or wheel into the plugin directory. +- Resident plugin processes that reuse connections and initialization while keeping the interfaces unchanged. +- More plugin types, such as candidate content for Recall or text fragments for Prepared Context. +- Plugin packaging and installation tools, such as installing into the plugin directory from a Git URL or a wheel. diff --git a/docs/zh/rfcs/2549-plugin-interfaces.md b/docs/zh/rfcs/2549-plugin-interfaces.md index d4c2c24137..a9e23f456e 100644 --- a/docs/zh/rfcs/2549-plugin-interfaces.md +++ b/docs/zh/rfcs/2549-plugin-interfaces.md @@ -1,6 +1,6 @@ --- title: 插件接口 -description: 用少量接口定义 Source 和 Artifact 插件，由 PC Server 通过 Supervisor 在需要时调用；core 只管理 Source 与 Artifact 两类核心数据及其消费进度，不再托管插件生命周期。 +description: Source 和 Artifact 插件各自实现少量接口，由 PC Server 通过 Supervisor 在需要时调用；调用方只使用现有的 Source 与 Artifact 接口，core 只管理两类核心数据及其消费进度，不托管插件生命周期。 --- - Proposal Name: `plugin_interfaces` @@ -9,24 +9,25 @@ description: 用少量接口定义 Source 和 Artifact 插件，由 PC Server # 概要 -本提案重新定义 PowerContext 的第三方插件模型：每类插件只实现几个固定接口，PC Server 在需要时调用。插件是被调用方， -只负责“把输入变成结果”，不参与调度、状态管理和持久化协议。 - -- **Source 插件**提供 Source 定义（沿用现有 `SourceDefinition` / `SourceProjection`），可选实现 `collect`。写入时 Server - 调用定义完成校验、转换和保存。采集按**采集目标**进行，例如“X 上的某个账号”；宿主可以在运行中添加、停用和恢复目标， - 不需要重启。Supervisor 按调度派发 Worker 调用 `collect`，core 把返回的 Source 和各目标的 checkpoint 在同一事务中提交。 -- **Artifact 插件**实现 `generate`。Supervisor 按配置的调度、新 Source 到达或手动请求派发 Worker，Worker 读取该 Scope - 尚未消费的 Source 窗口，连同一个**写入器**（`ArtifactWriter`）交给 `generate`。插件在写入器之外完成模型调用等耗时工作， - 然后在 `async with writer.transaction()` 块中执行 Artifact 写操作：块开始时 core 开启事务，块内的写操作直接作用于 - core 数据并立即返回真实的 revision，块正常结束时 core 把这些写入、消费游标和完成确认一起提交，块内抛出异常则整体回滚。 - 插件读不到也改不了游标。 -- core 只管理两类核心数据：`pc_sources` 和 `pc_artifacts`（含 heads、lineage、合并记录、生命周期、handle 和 tags）， - 以及它们的消费进度。插件可以有自己的索引和存储，由插件自己管理和保证一致性。应用通过现有的 Artifact 公开接口读取 - 和检索插件 family，读取路径不执行插件代码。 -- `collect` 和 `generate` 在 Supervisor 管理的 Worker 子进程中执行，沿用现有的进程隔离、超时、并发预算、领导租约和 - 失败退避。用户只需部署一个 Server。 -- 插件在 Server 启动时按目录加载，代码或清单变更需要重启。Server 不保存插件的启用状态、注册代次或管理阶段，也不提供 - enable / disable / reload / unload 管理接口。 +本提案重新定义 PowerContext 的第三方插件模型：插件只实现几个固定接口，PC Server 在需要时调用；调用方不直接调用插件， +而是像使用内置能力一样使用现有的 Source 和 Artifact 接口。 + +- **两类插件**： + - **Source 插件**提供 Source 定义，需要持续采集时再实现 `identify_target` 和 `collect`。 + - **Artifact 插件**实现 `generate`，把尚未消费的 Source 加工成 Artifact。 +- **插件形态**：一个插件是一个目录，包含 `plugin.py`，其中把一个声明对象（`SourcePlugin` 或 `ArtifactPlugin`）赋给 + `plugin`。声明写在代码里，没有单独的清单文件；部署相关的设置由插件从环境变量读取。 +- **调用方式**： + - 写入 Source、添加采集目标，使用现有的 Source 接口。 + - 读取、检索和修改 Artifact，使用现有的 Artifact 接口。 + - 遗忘、恢复、编辑等用户操作由 core 直接执行，不经过插件。 +- **执行**：`collect` 和 `generate` 在 Supervisor 管理的 Worker 子进程中执行，沿用现有的隔离、超时、并发预算、 + 领导租约和失败退避。 +- **写入**：Artifact 插件在 `async with writer.transaction()` 块中执行写操作，块结束时 core 把写入、消费游标和 + 完成确认一起提交，异常则整体回滚。插件读不到也改不了游标。 +- **core 的职责**：只管理 `pc_sources`、`pc_artifacts`（含 heads、lineage、合并记录、生命周期、handle 和 tags）及其 + 消费进度。插件可以有自己的索引和存储，由插件自己管理。 +- **生命周期**：插件在 Server 启动时按目录加载，变更需要重启。Server 不保存插件的启用状态、注册代次或管理阶段。 本提案取代 [Source 插件框架 RFC #2012](2012-source-plugin-system.md) 和 [Artifact 插件框架 RFC #2175](2175-artifact-plugin-system.md) 的插件契约与管理机制。两者的实现尚未作为稳定接口发布，本提案不保留兼容：旧的插件清单、管理接口、管理服务和插件管理相关的 @@ -46,7 +47,8 @@ description: 用少量接口定义 Source 和 Artifact 插件，由 PC Server 这些机制的代价由所有人承担： - **插件作者**需要理解 binding、generation、schema 固化规则、索引声明和管理通道等框架概念，才能写出一个插件。 -- **部署者**需要运行多个进程、维护两个数据库，并通过管理 API 让插件进入可运行状态。 +- **部署者**需要运行多个进程、维护两个数据库、编写清单文件，并通过管理 API 让插件进入可运行状态。 +- **调用方**需要区分内置能力和插件能力，使用不同的接口。 - **维护者**需要为第三方代码维护状态机、并发屏蔽、schema 和索引治理以及恢复路径，这部分代码的规模和测试面已经超过 插件业务本身。 @@ -56,122 +58,152 @@ description: 用少量接口定义 Source 和 Artifact 插件，由 PC Server 本提案的目标： -1. 插件作者只需实现少量接口，不需要了解调度、游标和提交协议。 -2. 插件代码在隔离的 Worker 进程中执行，插件故障不影响 Server 和其他插件。 -3. 插件结果进入 `pc_sources` / `pc_artifacts`，与消费进度原子提交，现有读取接口直接可用。 -4. core 只管理这两类核心数据及其消费进度；内容结构、索引和其他存储交给插件自己。 -5. 插件对核心数据的写入只经过 core 提供的写入器，覆盖 Artifact 的全部通用写操作，并沿用 core 现有的校验规则。 -6. Artifact 生成既可以按配置的调度执行，也可以由新 Source 或手动请求触发；采集目标可以在运行中增减。 -7. 用户只需部署一个 Server；插件随 Server 启动加载。 +1. 插件作者只需写一个 `plugin.py`：一个声明对象和几个方法，不需要理解插件管理的内部机制。 +2. 调用方不感知插件：Source 和 Artifact 的现有接口对插件产生的数据同样适用。 +3. 插件代码在隔离的 Worker 进程中执行，插件故障不影响 Server 和其他插件。 +4. 插件结果进入 `pc_sources` / `pc_artifacts`，与消费进度原子提交。 +5. core 只管理这两类核心数据及其消费进度；内容结构、索引和其他存储交给插件自己。 +6. 插件对核心数据的写入只经过 core 提供的写入器，覆盖 Artifact 的全部通用写操作，并沿用 core 现有的校验规则。 +7. Artifact 生成可以按调度执行，也可以由新 Source 触发；采集目标可以在运行中增减。 +8. 用户只需部署一个 Server；插件随 Server 启动加载。 # 指南级说明 -## 1. 核心思路 +## 1. 总体设计 ```mermaid flowchart LR - Host["应用 / Agent / 宿主"] -->|"HTTP / MCP"| Server - subgraph Server["PC Server"] - API["公开接口
认证与 Scope 授权"] + Caller["应用 / Agent / 宿主"] + subgraph Server["PC Server（core）"] + SourceAPI["现有 Source 接口
写入 Source、添加采集目标"] + ArtifactAPI["现有 Artifact 接口
读取、检索、修改"] Sup["Supervisor
调度 / 触发 / 预算 / 退避"] - Core[("core 数据
pc_sources / pc_artifacts
采集目标 / 游标 / checkpoint")] + Core[("core 数据
pc_sources / pc_artifacts
采集目标 / checkpoint / 游标")] end - subgraph Worker["Worker 子进程（每次调用）"] - Exec["插件执行器"] - Plugin["插件代码
collect() / generate()"] - Tx["writer.transaction()
受约束的写事务"] - Exec -->|"目标与 checkpoint / Source 窗口"| Plugin - Plugin -->|"put / merge / set_lifecycle ..."| Tx + subgraph SW["Worker：Source 插件"] + Collect["collect(request)"] end - API -->|"读取、检索"| Core - API -->|"write、添加目标"| Core - API -->|"手动触发"| Sup - Sup -->|"派发 (插件, Scope)"| Exec - Tx -->|"退出时一并提交写入、游标与确认"| Core - Plugin -.->|"自有索引 / 存储"| Own[("插件自己的存储")] - Plugin -.->|"外部系统 / 模型"| Ext["X、工单系统、LLM……"] + subgraph AW["Worker：Artifact 插件"] + Generate["generate(request, writer)"] + Tx["writer.transaction()"] + Generate --> Tx + end + Caller --> SourceAPI + Caller --> ArtifactAPI + SourceAPI -->|"resolve / Projection（插件定义）"| Core + ArtifactAPI <--> Core + Sup -->|"目标与 checkpoint"| Collect + Collect -->|"Source 与新 checkpoint，一次提交"| Core + Core -->|"新 Source 触发"| Sup + Sup -->|"未消费的 Source 窗口"| Generate + Tx -->|"Artifact 写入、游标、确认，一次提交"| Core + Collect -.-> Ext["外部系统"] + Generate -.-> Own[("插件自有存储 / 模型")] ``` +图中有两条独立的路径，一次 Worker 调用只执行其中一个插件： + +- **采集路径**：Supervisor 按调度派发 Source 插件的 Worker，传入该 Scope 已启用的采集目标及其 checkpoint；插件从外部 + 系统读取数据并返回 Source；core 校验后把 Source 和新 checkpoint 在一个事务中保存。 +- **生成路径**：新 Source 进入 journal 后，关注这些 Source 的 Artifact 插件被标记为待处理；Supervisor 派发 Worker，传入 + 该 Scope 尚未消费的 Source 窗口和写入器；插件在写事务块中写入 Artifact，core 连同游标和确认一起提交。 + +调用方只和 core 的现有接口打交道： + +| 调用方要做的事 | 使用的接口 | 插件参与的方式 | +| --- | --- | --- | +| 提交一条 Source | 现有 Source 写入接口 | core 调用插件定义的 `resolve` 和 Projection 完成校验与转换 | +| 添加、查看、停用采集目标 | 现有 Source 管理接口 | core 调用插件的 `identify_target` 规范化输入 | +| 浏览、读取、检索 Artifact | 现有 Artifact 读取与检索接口 | 不参与 | +| 编辑、打 tag、合并、遗忘、恢复、撤销合并 | 现有 Artifact 写接口、tags 接口与通用合并接口 | 不参与；插件在下一次调用时看到新状态 | + +职责划分： + | 职责 | 归属 | | --- | --- | -| 认证、Scope 授权、公开 HTTP / MCP 接口、采集目标的增删改查 | Server | -| 何时调用：调度、触发合并、Worker 预算、超时、退避、领导租约 | Supervisor（现有） | -| 读取输入、提供受约束的写事务、在同一事务中提交核心数据 / 游标 / checkpoint / 完成确认 | Worker 中的插件执行器（core） | -| 外部数据读取、采集目标的识别与规范化、Source 定义与 Projection 计算 | Source 插件 | +| 认证、Scope 授权、公开 HTTP / MCP 接口、采集目标与 checkpoint 的保存、用户对 Artifact 的修改 | core | +| 何时调用：调度、触发合并、Worker 预算、超时、退避、领导租约 | core（现有 Supervisor） | +| 读取输入、提供受约束的写事务、在同一事务中提交核心数据 / 游标 / checkpoint / 完成确认 | core（Worker 中的插件执行器） | +| Source 定义与 Projection 计算、采集目标的识别与规范化、外部数据读取 | Source 插件 | | Artifact 内容结构、生成算法、插件自有的索引与存储及其一致性 | Artifact 插件 | -| 插件代码、依赖和插件自有存储的部署 | 部署者 | +| 插件代码、依赖、环境变量和插件自有存储的部署 | 部署者 | 插件代码只做一件事：给定输入，产出结果或抛出异常。抛出异常即失败，core 不提交任何核心数据，下次重试同一输入。 -## 2. 插件目录与清单 +## 2. 插件的形态 -Server 从 `POWERCONTEXT_SERVER_PLUGINS_DIR` 加载插件。每个直接子目录是一个插件，包含 `plugin.yml` 和代码： +Server 从 `POWERCONTEXT_SERVER_PLUGINS_DIR` 加载插件。每个直接子目录是一个插件，目录名就是插件名；目录中必须有 +`plugin.py`，可以有其他模块： ```text /opt/powercontext/plugins/ ├── x-posts/ -│ ├── plugin.yml -│ ├── definition.py -│ └── plugin.py +│ ├── plugin.py +│ └── definition.py └── incident-report/ - ├── plugin.yml └── plugin.py ``` -```yaml -name: incident-report # 插件名，部署内唯一，与目录名一致 -kind: artifact # source 或 artifact -entry: ./plugin.py:create # 工厂函数，路径相对插件目录 -scopes: [team-a] # 可选；省略时处理所有存在匹配输入的 Scope -config: # 业务配置，由插件的 Config 模型校验 - title_prefix: 工单报告 - index_url: ${REPORT_INDEX_URL} # ${NAME} 在 Worker 中从环境变量解析 -``` +`plugin.py` 把一个声明对象赋给模块级变量 `plugin`： -清单只描述**插件是什么、怎样运行**，不包含启用状态。目录存在且加载成功，插件就可用；需要停用时移出目录并重启 Server。 -同一份代码可以通过多个插件目录、不同的 `name` 和 `config` 部署多次。 - -## 3. Source 插件示例 - -Source 插件导出 Source 定义。只提供定义时，宿主可以通过现有的 Source 写入接口（MCP `pc_source_add` 的 write 用法） -提交完整内容，Server 调用定义的 `resolve` 和各 Projection，然后保存到 `pc_sources`。 - -需要从外部系统持续采集时，插件再实现 `collect`。采集以**采集目标**为单位：每个目标是插件能采集的一项外部资源，例如 -X 上的一个账号。下面的插件每 5 分钟采集各目标账号的新帖子： - -```yaml -name: x-posts -kind: source -entry: ./plugin.py:create -scopes: [team-a] -collect: - schedule: {interval_seconds: 300} - max_workers: 1 - timeout_seconds: 120 - max_items: 200 - targets: # 可选；首次启动时导入的初始目标 - - {scope_id: team-a, input: {account: "@oceanbase"}} -config: - bearer_token: ${X_BEARER_TOKEN} +```python +plugin = ArtifactPlugin(family="incident-report", ..., create=create) # 或 SourcePlugin(...) ``` +声明对象回答 core 在运行插件代码之前需要知道的问题：插件提供什么（Source 类型或 Artifact family）、消费什么、何时调用、 +资源上限多少。它是普通的 Python 对象，由 Pydantic 校验；没有 YAML 或其他清单文件。 + +- **部署设置**：凭据、外部服务地址、模型名等随部署变化的值，由插件在 `create` 中从环境变量读取。Server 不保存、 + 不传递也不记录这些值。 +- **启用与停用**：目录存在且检查通过，插件就可用；需要停用时移出目录并重启 Server。 +- **检查**：Server 启动时在有时限的子进程中导入 `plugin.py`，校验声明，不调用 `create`。检查失败只影响该插件。 + +## 3. Source 插件 + +### 3.1 要实现的接口 + +| 接口 | 必需 | 说明 | +| --- | --- | --- | +| `definitions` | 是 | `SourceDefinition` 序列，沿用现有的 Source 定义：`input_class`、`resolve` 和各 Projection | +| `target_input` | 采集时必需 | Pydantic 模型，校验调用方添加目标时提交的输入 | +| `identify_target(value) -> TargetIdentity` | 采集时必需 | 纯函数，把输入规范化为稳定的目标键和规范化输入 | +| `create(ctx) -> SourceCollector` | 采集时必需 | 工厂，在 Worker 中每次调用执行一次，返回实现了 `collect` 的对象 | +| `collect(request) -> CollectResult` | 采集时必需 | 按目标和 checkpoint 读取外部数据，返回 Source 和新的 checkpoint | + +只提供 `definitions` 的插件只扩展 Source 类型：调用方通过现有的 Source 写入接口提交内容，core 调用定义完成校验、 +转换和保存。 + +### 3.2 声明 + +| 字段 | 默认值 | 说明 | +| --- | --- | --- | +| `definitions` | 必填 | 见上表 | +| `collect` | `None` | 采集配置；为空表示只提供定义 | +| `collect.schedule` | 必填 | `Interval(seconds=...)` 或 `Cron(expression=..., timezone=...)` | +| `collect.max_targets` | 50 | 单次调用携带的目标数上限 | +| `collect.max_items` | 200 | 单次调用返回的 Source 条数上限 | +| `collect.timeout_seconds` | 120 | Worker 总时限 | +| `collect.max_workers` | 1 | 跨 Scope 的并发上限 | +| `target_input`、`identify_target`、`create` | 采集时必填 | 见上表 | +| `scopes` | `None` | 可服务的 Scope 名单；为空表示所有 Scope | + +### 3.3 示例 + +下面的插件每 5 分钟采集各目标账号在 X 上的新帖子： + ```python +import os + from pydantic import BaseModel from powercontext.plugins import ( - CollectedItem, CollectRequest, CollectResult, PluginContext, TargetIdentity, + Collect, CollectedItem, CollectRequest, CollectResult, Interval, PluginContext, SourcePlugin, TargetIdentity, ) -from .definition import PostCapture, XPostDefinition - -definitions = (XPostDefinition(),) - +from definition import PostCapture, XPostDefinition -class Config(BaseModel): - bearer_token: str - -class TargetInput(BaseModel): # 宿主添加目标时提交的输入 +class TargetInput(BaseModel): account: str @@ -196,65 +228,76 @@ class XCollector: return result -async def create(ctx: PluginContext, config: Config) -> XCollector: - client = await ctx.enter_async_context(XClient(config.bearer_token)) +async def create(ctx: PluginContext) -> XCollector: + client = await ctx.enter_async_context(XClient(os.environ["X_BEARER_TOKEN"])) return XCollector(client) + + +plugin = SourcePlugin( + definitions=(XPostDefinition(),), + collect=Collect(schedule=Interval(seconds=300), timeout_seconds=120), + target_input=TargetInput, + identify_target=identify_target, + create=create, +) ``` -宿主在运行中添加一个目标，无需重启： +### 3.4 调用方怎么使用 + +调用方使用现有的 Source 管理接口，不需要知道这是插件提供的能力： ```text pc_source_add(scope_id="team-a", capability="x-posts", input={"account": "@PowerContextAI"}) +pc_source_status(scope_id="team-a", target_id="...") +pc_source_target_set_enabled(scope_id="team-a", target_id="...", enabled=false) ``` -Server 用 `TargetInput` 校验输入，调用 `identify_target` 得到稳定的目标键和规范化输入，保存目标后立即请求一次采集。 -重复添加同一账号得到同一个键，复用已有目标；目标可以停用和恢复，进度保留。 +- **添加**：core 用 `target_input` 校验输入，调用 `identify_target` 得到键和规范化输入。同一账号得到同一个键，复用已有 + 目标。新目标保存为启用状态并立即请求一次采集，不需要重启。 +- **checkpoint**：每个目标的 checkpoint 是只读输入，插件通过返回值提出新的 checkpoint。core 把本次返回的所有 Source + 与各目标的新 checkpoint 在同一事务中提交；`collect` 抛出异常或超时则不提交任何内容，下一次调用仍拿到旧 checkpoint。 + +## 4. Artifact 插件 -每个目标的 checkpoint 是只读输入，插件通过返回值提出新的 checkpoint。core 把本次返回的所有 Source 与各目标的新 -checkpoint 在同一事务中提交；`collect` 抛出异常或超时则不提交任何内容，下一次调用仍拿到旧 checkpoint。 +### 4.1 要实现的接口 + +| 接口 | 必需 | 说明 | +| --- | --- | --- | +| `create(ctx) -> ArtifactGenerator` | 是 | 工厂，在 Worker 中每次调用执行一次 | +| `generate(request, writer) -> None` | 是 | 读取 `request.sources`（未消费的 Source 窗口），在写事务块中写入 Artifact | -## 4. Artifact 插件示例 +插件只负责生成。用户对 Artifact 的修改由 core 直接执行（见 4.4），插件不需要实现命令处理。 + +### 4.2 声明 + +| 字段 | 默认值 | 说明 | +| --- | --- | --- | +| `family` | 必填 | 插件拥有的 Artifact family，不能与内置 family 或其他插件重复 | +| `inputs` | `None` | `Inputs(source_types=..., projection=..., max_sources=50)`：消费的 Source 类型、每条 Source 交付的 Projection、单个窗口的条数上限；为空表示不消费 Source | +| `on_sources` | `False` | 新 Source 到达后触发；需要 `inputs` | +| `schedule` | `None` | `Interval(seconds=...)` 或 `Cron(expression=..., timezone=...)`；与 `on_sources` 至少配置一项 | +| `scopes` | `None` | 可服务的 Scope 名单；不消费 Source 的插件必须填写 | +| `search_fields` | `None` | 参与全文检索的顶层内容字段；为空表示所有顶层字符串字段 | +| `timeout_seconds` | 300 | 整次调用的时限 | +| `transaction_timeout_seconds` | 30 | 写事务块的时限，不超过 `timeout_seconds` | +| `max_workers` | 1 | 跨 Scope 的并发上限 | +| `create` | 必填 | 见上表 | + +### 4.3 示例 下面的插件为每条工单 Source 生成一份报告：先用自己的向量索引查找相似的已有报告，命中则修订那份报告，否则新建； 同时把报告写进自己的向量索引。 -```yaml -name: incident-report -kind: artifact -entry: ./plugin.py:create -scopes: [team-a] -generate: - family: incident-report - inputs: - source_types: [example-ticket] - projection: powercontext.text-evidence@1 - max_sources: 50 - schedule: {interval_seconds: 300} # 定时检查 - on_sources: true # 新 Source 到达后触发 - manual: true # 允许手动触发 - max_workers: 1 - timeout_seconds: 300 # 整次调用的时限 - transaction_timeout_seconds: 30 # 写事务块的时限 -config: - title_prefix: 工单报告 - index_url: ${REPORT_INDEX_URL} -``` - ```python -from pydantic import BaseModel +import os -from powercontext.plugins import ArtifactWriter, GenerateRequest, PluginContext +from powercontext.plugins import ArtifactPlugin, ArtifactWriter, GenerateRequest, Inputs, PluginContext from powercontext.sources import TextEvidence -class Config(BaseModel): - title_prefix: str = "工单报告" - index_url: str - - class IncidentReports: - def __init__(self, config: Config, index: VectorIndex, model: Summarizer) -> None: - self.config, self.index, self.model = config, index, model + def __init__(self, index: VectorIndex, model: Summarizer) -> None: + self.index, self.model = index, model async def generate(self, request: GenerateRequest, writer: ArtifactWriter) -> None: # 1. 耗时工作放在写事务之外：模型调用、向量计算 @@ -262,8 +305,7 @@ class IncidentReports: for source in request.sources: evidence = TextEvidence.model_validate(source.projection) summary = await self.model.summarize(evidence.content) - vector = await self.model.embed(summary) - planned.append((source, summary, vector)) + planned.append((source, summary, await self.model.embed(summary))) # 2. 短写事务：块内读取能看到本块已写入的内容 async with writer.transaction() as tx: @@ -272,91 +314,125 @@ class IncidentReports: current = await tx.get(match.artifact_id) if match else None artifact_id = current.ref.artifact_id if current else writer.stable_id( "report", source.ref.source_type, source.ref.source_id) - ref = await tx.put( - artifact_id, - content={"title": self.config.title_prefix, "summary": summary}, - search_text=summary, - source_refs=[source.ref], - ) + written = await tx.put(artifact_id, content={"title": "工单报告", "summary": summary}, + source_refs=[source.ref]) # 拿到真实的 artifact_id 与 revision 后写入自有索引，按 ID upsert - await self.index.upsert(request.scope_id, ref.artifact_id, ref.revision, vector) + await self.index.upsert(request.scope_id, written.ref.artifact_id, written.ref.revision, vector) # 块正常结束：core 提交 Artifact 写入、游标推进与完成确认 -async def create(ctx: PluginContext, config: Config) -> IncidentReports: - index = await ctx.enter_async_context(VectorIndex.connect(config.index_url)) - return IncidentReports(config, index, Summarizer()) +async def create(ctx: PluginContext) -> IncidentReports: + index = await ctx.enter_async_context(VectorIndex.connect(os.environ["REPORT_INDEX_URL"])) + return IncidentReports(index, Summarizer()) + + +plugin = ArtifactPlugin( + family="incident-report", + inputs=Inputs(source_types=("example-ticket",), projection="powercontext.text-evidence@1"), + on_sources=True, + search_fields=("title", "summary"), + create=create, +) ``` 块内抛出异常时，core 回滚块内全部 Artifact 写入，游标不动，下次重试同一窗口。插件自有索引不在 core 的事务中： -插件按稳定 ID upsert，重试时会覆盖上一次留下的条目；读取自有索引时以 core 的当前 revision 为准（见参考级说明 1.6）。 +插件按稳定 ID upsert，重试时会覆盖上一次留下的条目；读取自有索引时以 core 的当前数据为准（见参考级说明 3.5）。 + +### 4.4 调用方怎么使用 -应用通过现有的 Artifact 公开接口访问插件 family，这些接口直接读取 `pc_artifacts`，不执行插件代码： +调用方使用现有的 Artifact 接口访问插件 family。读取接口直接读 `pc_artifacts`，写接口由 core 执行，与插件写入复用同一套 +core 实现，都不运行插件代码： | 用途 | 接口 | | --- | --- | | 浏览 | `GET /v1/scopes/{scope_id}/artifacts/{family}` | -| 读取当前内容 | `GET /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | -| 读取指定 revision / revision 历史 | `GET .../{artifact_id}/revisions/{revision}`、`GET .../{artifact_id}/revisions` | -| 读取 tags | `GET .../{artifact_id}/tags` | +| 读取当前内容、指定 revision、revision 历史、tags | `GET .../{artifact_id}`、`GET .../{artifact_id}/revisions/{revision}`、`GET .../{artifact_id}/revisions`、`GET .../{artifact_id}/tags` | | 文本检索 | `POST /v1/scopes/{scope_id}/artifacts/{family}/search` | -| 手动触发生成 | `POST /v1/scopes/{scope_id}/artifacts/{family}/generate` | +| 编辑（提交完整内容，`If-Match` 校验当前 revision） | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | +| 修改 tags | `PUT .../{artifact_id}/tags`；按 tag 查询用 `POST /v1/scopes/{scope_id}/artifact-tags/query` | +| 读取治理状态（遗忘、合并前取得 `state_version`） | `GET .../{artifact_id}/governance` | +| 合并 | `POST /v1/scopes/{scope_id}/artifacts/{family}/merges` | +| 遗忘 | `POST .../{artifact_id}/forget` | +| 恢复、恢复到旧 revision、撤销合并 | `POST .../{artifact_id}/restorations` | +| 补充知识 | 写入一条 Source，由插件在下一次调用中处理 | + +修改同步返回结果。插件在下一次调用中通过写入器读到修改后的状态。 ## 5. 调度与触发 插件的调度和触发都交给现有的 Artifact processing Supervisor。每个插件在启动时注册为一个 Supervisor binding， -binding 名为 `plugin:`，调度键仍是 `(binding, Scope)`。 +binding 名为 `plugin:`，调度键是 `(binding, Scope)`。 -Artifact 生成支持三种调用来源，可以任意组合，至少配置一种： +Artifact 插件： -| 配置 | 何时派发 Worker | `request.reason` | +| 声明 | 何时派发 Worker | `request.reason` | | --- | --- | --- | -| `schedule: {interval_seconds: N}` 或 `schedule: {cron: "0 2 * * *", timezone: Asia/Shanghai}` | 到期时，对有待处理工作的 Scope | `schedule` | -| `on_sources: true` | Scope 的 Source journal 超过该插件的游标 | `sources` | -| `manual: true` | 调用手动触发接口 | `manual` | +| `schedule=Interval(...)` 或 `schedule=Cron(...)` | 到期时，对有待处理工作的 Scope | 有新 Source 时为 `sources`，否则为 `schedule` | +| `on_sources=True` | Scope 的 Source journal 超过该插件的游标，约 2 秒内派发 | `sources` | -- 配置了 `inputs` 的插件，每次调用都附带该 Scope 尚未消费的 Source 窗口；窗口为空时，定时和 Source 触发不调用插件， - 直接确认完成。 -- 未配置 `inputs` 的插件（例如每天整理一次已有 Artifact）不消费 Source，到期即调用；必须显式配置 `scopes`。 -- 手动触发总是调用插件，可以附带 `params`，由插件自行解释。手动请求在接受时持久化，返回 `202 Accepted`， - Server 重启后仍会执行。 +- 配置了 `inputs` 的插件，每次调用都附带该 Scope 尚未消费的 Source 窗口；窗口中没有匹配的 Source 时不调用插件， + 直接推进游标并确认。 +- 未配置 `inputs` 的插件（例如每天整理一次已有 Artifact）不消费 Source，到期即对 `scopes` 中的每个 Scope 调用一次； + 第一版只支持 `Interval`。 +- `Cron` 只能用于配置了 `inputs` 的插件，且不能与 `on_sources` 同时使用：Supervisor 的每个 binding 只接受一种 + interval 或 cron 策略。 -Source 插件的 `collect` 支持 `schedule` 和 `manual`（`POST /v1/scopes/{scope_id}/sources/plugins/{name}/collect`）， -添加或恢复目标时也会请求一次采集。 +Source 插件的 `collect` 按 `collect.schedule` 调用；添加或恢复目标时也会请求一次采集。 同一插件、同一 Scope 同时最多运行一个 Worker。运行期间到达的触发由 Supervisor 合并成结束后的一次调用。 ## 6. 运维 -- 启动：Server 逐个检查插件目录。某个插件检查失败（清单无效、导入失败、缺少必需导出、family 或 Source 类型与内置或其他 - 插件冲突）只影响该插件，Server 和其他插件正常启动。 -- 查看：`GET /v1/plugins`（需要 `server.admin`）列出每个插件的检查结果、类型、family 或 Source 定义、调度配置， - 以及每个 Scope 的最近一次调用结果和下次重试时间。采集目标及其最近采集结果通过 `pc_source_status` 和目标接口查看。 -- 变更：修改代码或清单后重启 Server。Server 不监视文件，也不在运行中重新加载插件。采集目标是业务数据，增减不需要重启。 -- 停用：从插件目录移走后重启。已保存的 Source、Artifact、采集目标、游标和 checkpoint 保留并可继续读取；重新部署 +- **启动**：Server 逐个检查插件目录。某个插件检查失败（缺少 `plugin.py`、导入失败、声明无效、family 或 Source 类型 + 与内置或其他插件冲突）只影响该插件，Server 和其他插件正常启动。 +- **查看**（都需要 `server.admin`，只读，数据都来自启动检查或数据库，任何 Server 角色都给出相同结果）： + - `GET /v1/plugins`、`GET /v1/plugins/{name}`：每个插件的检查结果、类型、family 或 Source 类型、消费的 Source 类型、 + 触发方式和 Scope 范围。 + - `GET /v1/plugins/{name}/scopes`：每个 Scope 是否有待处理的工作、已消费到的 journal 位置和当前 journal 位置。 + - 调用失败的原因通过日志和指标排查，不在接口中保存。 +- **变更**：修改代码后重启 Server。Server 不监视文件，也不在运行中重新加载插件。采集目标是业务数据，增减不需要重启。 +- **停用**：从插件目录移走后重启。已保存的 Source、Artifact、采集目标、游标和 checkpoint 保留并可继续读取；重新部署 同名插件时从原位置继续。插件自有存储由部署者保留或清理。 # 参考级说明 -## 1. 接口契约 +## 1. 公共契约 公开接口位于新模块 `powercontext.plugins`，替换现有的 `powercontext.artifact_plugins` 和 `powercontext.source_management` 中面向插件作者的部分。接口以 `typing.Protocol` 和 Pydantic 模型表达。 -### 1.1 入口模块、工厂与上下文 +### 1.1 声明与加载 -入口模块导出： +```python +class Interval(BaseModel): + seconds: float # > 0 + +class Cron(BaseModel): + expression: str + timezone: str = "Asia/Shanghai" + +class ArtifactPlugin(BaseModel): # 字段见指南级说明 4.2 + family: str + inputs: Inputs | None = None + ... + create: Callable[[PluginContext], Awaitable[ArtifactGenerator]] + +class SourcePlugin(BaseModel): # 字段见指南级说明 3.2 + definitions: Sequence[SourceDefinition] + collect: Collect | None = None + ... +``` -| 导出 | 必需 | 说明 | -| --- | --- | --- | -| 工厂函数（`entry` 指定） | 是 | `async def create(ctx, config) -> 插件实例` | -| `Config` | 否 | Pydantic 模型，校验清单中的 `config`；未导出时 `config` 必须为空 | -| `definitions` | Source 插件必需 | `SourceDefinition` 序列 | -| `TargetInput`、`identify_target` | 配置了 `collect` 的 Source 插件必需 | 采集目标的输入模型与识别函数 | +- 插件名取自目录名，必须匹配 `^[a-z][a-z0-9-]{0,63}$`。 +- 声明在构造时校验，包括调度组合、`inputs` 与 `scopes` 的依赖、时限关系和 family 命名。`create` 必须是 async 函数。 +- 声明中除 `create`、`identify_target`、`definitions` 等代码对象外的部分可以序列化，Server 在检查子进程中得到它， + 用于注册 binding 和回答运维查询；Worker 每次调用重新导入 `plugin.py`，声明与启动时不一致则调用失败，提示重启 Server。 + +### 1.2 上下文 ```python class PluginContext: - name: str # 清单中的插件名 + name: str # 插件名 scope_id: str # 本次调用的 Scope plugin_dir: Path # 插件目录（只读约定） logger: logging.Logger # 已绑定插件名的 logger @@ -366,11 +442,10 @@ class PluginContext: - 工厂在每次 Worker 调用中执行一次，实例不跨调用复用。需要关闭的资源（包括插件自有存储的连接）通过 `ctx.enter_async_context` 注册，调用结束时由执行器释放。 -- `${NAME}` 环境变量引用在 Worker 中解析后再用 `Config` 校验，凭据不进入 Supervisor 的派发参数和日志。 -- 插件得不到 core 数据库的连接，也没有游标或 checkpoint 的写入接口。对核心数据的写入只有两条路径：Source 插件的 +- 插件得不到 core 数据库的连接，也没有游标或 checkpoint 的写入接口。插件对核心数据的写入只有两条路径：Source 插件的 `collect` 返回值，和 Artifact 插件的 `ArtifactWriter`。 -### 1.2 Source 插件 +## 2. Source 插件契约 ```python class TargetIdentity(BaseModel): @@ -384,55 +459,54 @@ class CollectTarget(BaseModel): class CollectRequest(BaseModel): scope_id: str - reason: Literal["schedule", "manual", "target_added"] + reason: Literal["schedule", "target_added"] targets: Sequence[CollectTarget] # 本次要采集的已启用目标 max_items: int - params: JsonValue | None = None # 仅手动触发 class CollectedItem(BaseModel): definition: str # definitions 中某个定义的 name value: BaseModel # 该定义 input_class 的实例 -class CollectResult(BaseModel): - targets: dict[str, TargetResult] = {} # 键为目标键 - def add(self, target: CollectTarget, items: Sequence[CollectedItem], *, checkpoint: JsonValue | None) -> None: ... - class TargetResult(BaseModel): items: Sequence[CollectedItem] checkpoint: JsonValue | None has_more: bool = False +class CollectResult(BaseModel): + targets: dict[str, TargetResult] = {} # 键为目标键 + def add(self, target: CollectTarget, items: Sequence[CollectedItem], *, checkpoint: JsonValue | None) -> None: ... + class SourceCollector(Protocol): async def collect(self, request: CollectRequest, /) -> CollectResult: ... ``` -- 只提供 `definitions`、不配置 `collect` 的插件只支持 write。工厂只在 collect 调用时执行。 -- `definitions` 中的 Source 类型与 Projection 在启动时注册到现有的 Source 定义清单，与内置类型和其他插件冲突时检查失败。 -- **write** 不需要插件方法。Server 按 `definition.input_class` 校验输入，调用 `resolve` 得到 Source，计算各 Projection 后 - 保存，语义与现有内置 Source 写入一致。 -- **collect** 每次调用对应一个 Scope，最多携带 `max_targets`（清单配置，默认 50）个已启用目标；目标较多时按目标键轮转， - `has_more` 的目标优先进入下一次调用。执行器对返回的每条 Source 执行与 write 相同的校验、`resolve` 和 Projection， - 然后在一个事务中保存全部 Source、写入各目标的 checkpoint 和最近采集时间，并确认本次调用。任一条失败则整次不提交。 - 条目总数超过 `max_items`、返回了请求中不存在的目标键，都视为失败，不截断。 -- 结果中没有出现的目标，checkpoint 保持不变。 -- Source 身份由 `resolve` 决定。重复采集同一外部状态得到相同身份时，沿用现有去重规则，不产生重复 Source。 - -### 1.3 采集目标 +- **定义注册**：`definitions` 中的 Source 类型与 Projection 在启动时注册到现有的 Source 定义清单，与内置类型和其他插件 + 冲突时检查失败。 +- **写入**：调用方通过现有的 Source 写入接口（`POST /v1/scopes/{scope_id}/sources`、MCP `pc_source_add` 的 write 用法） + 提交内容。core 按 `definition.input_class` 校验输入，调用 `resolve` 得到 Source，计算各 Projection 后保存，语义与内置 + Source 写入一致。这一步在 Server 进程中同步执行，有超时限制。 +- **collect**：每次调用对应一个 Scope，最多携带 `max_targets` 个已启用目标；目标较多时按目标键轮转，`has_more` 的目标 + 优先进入下一次调用。执行器对返回的每条 Source 执行与写入相同的校验、`resolve` 和 Projection，然后在一个事务中保存全部 + Source、写入各目标的 checkpoint 和最近采集时间，并确认本次调用。任一条失败则整次不提交。条目总数超过 `max_items`、 + 返回了请求中不存在的目标键，都视为失败，不截断。结果中没有出现的目标，checkpoint 保持不变。 +- **Source 身份**：由 `resolve` 决定。重复采集同一外部状态得到相同身份时，沿用现有去重规则，不产生重复 Source。 -采集目标是 core 保存的业务数据，与 Source 一样不属于插件生命周期状态： +**采集目标**是 core 保存的业务数据，与 Source 一样不属于插件生命周期状态。管理接口复用现有的 Source 管理接口： | 操作 | 接口 | 说明 | | --- | --- | --- | -| 添加 | MCP `pc_source_add`（collect 用法）、`POST /v1/scopes/{scope_id}/sources/plugins/{name}/targets` | 用 `TargetInput` 校验输入，调用 `identify_target` 得到键和规范化输入。同键且规范化输入相同则返回已有目标；同键但输入不同返回冲突。新目标保存为启用状态并请求一次采集 | -| 查看 | MCP `pc_source_status`、`GET /v1/scopes/{scope_id}/sources/plugins/{name}/targets` | 目标键、规范化输入、启用状态、最近成功采集时间、最近失败分类 | -| 停用 / 恢复 | MCP `pc_source_target_set_enabled`、`PATCH .../targets/{key}` | 停用后不再进入 collect 请求，checkpoint 和已采集的 Source 保留；恢复后从原 checkpoint 继续 | +| 添加 | MCP / HTTP `pc_source_add`（`capability` 为插件名） | 用 `target_input` 校验输入，调用 `identify_target` 得到键和规范化输入。同键且规范化输入相同则返回已有目标；同键但输入不同返回冲突。新目标保存为启用状态并请求一次采集 | +| 查看 | MCP / HTTP `pc_source_status` | 目标键、规范化输入、启用状态、最近成功采集时间、最近失败分类 | +| 停用 / 恢复 | MCP / HTTP `pc_source_target_set_enabled` | 停用后不再进入 collect 请求，checkpoint 和已采集的 Source 保留；恢复后从原 checkpoint 继续 | - 添加和变更需要调用者对该 Scope 有 Source 写权限，并且 Scope 在插件的 `scopes` 范围内。 - `identify_target` 必须是纯函数：只做校验和规范化，不访问外部系统、不检查外部授权。外部资源是否存在、是否可读， - 在 collect 时由插件判断并以异常或空结果表达。它与 write 一样在 Server 进程中同步执行，有超时限制。 -- 清单中的 `collect.targets` 只在对应键不存在时导入一次。之后以 core 中的数据为准，修改清单不会覆盖或删除已有目标。 + 在 collect 时由插件判断并以异常或空结果表达。它与写入一样在 Server 进程中同步执行，有超时限制。 +- `pc_source_capabilities` 列出插件提供的 Source 类型和目标输入。 -### 1.4 Artifact 插件、写入器与写事务 +## 3. Artifact 插件契约 + +### 3.1 请求 ```python class ArtifactGenerator(Protocol): @@ -440,31 +514,35 @@ class ArtifactGenerator(Protocol): class GenerateRequest(BaseModel): scope_id: str - reason: Literal["schedule", "sources", "manual"] + reason: Literal["schedule", "sources"] sources: Sequence[GenerationSource] # 本次窗口；未配置 inputs 时为空 - params: JsonValue | None = None # 仅手动触发 class GenerationSource(BaseModel): ref: SourceRef # 精确 Source 引用 - projection: JsonValue # 清单 inputs.projection 指定的 Projection 内容 + position: int # Source 在 journal 中的位置，用于按采集顺序处理 + projection: JsonValue # inputs.projection 指定的 Projection 内容 +``` -class ArtifactWriter(ArtifactReads): - family: str # 固定为清单中的 generate.family +### 3.2 写入器与写事务 +```python +class ArtifactWriter(ArtifactReads): + family: str # 固定为声明中的 family def stable_id(self, *parts: str) -> str: ... def transaction(self) -> AbstractAsyncContextManager[ArtifactTransaction]: ... class ArtifactTransaction(ArtifactReads): - ... # 读取（见 1.5）加写操作（见下表） + ... # 读取（见 3.4）加写操作（见下表） ``` `generate` 的执行分两部分： - **事务外**：插件通过 `writer` 读取已提交的数据（每次读取使用独立的只读短事务），完成模型调用、向量计算等耗时工作。 -- **写事务块**：`async with writer.transaction() as tx:`。块开始时，core 开启写事务并重新校验 Supervisor 租约和调用意图； - 块内的写操作直接在该事务中执行，立即返回真实的 `ArtifactRef`（含 revision），块内读取能看到本块已写入的内容； - 块正常结束时，core 在同一事务中以 CAS 推进游标、确认本次调用并提交；块内抛出异常、超过 `transaction_timeout_seconds` - 或提交冲突，事务整体回滚，本次调用失败。 +- **写事务块**：`async with writer.transaction() as tx:`。 + - 块开始时，core 开启写事务并重新校验 Supervisor 租约和调用意图。 + - 块内的写操作直接在该事务中执行，立即返回写入后的 `ArtifactSnapshot`（含精确 revision）；块内读取能看到本块已写入的内容。 + - 块正常结束时，core 在同一事务中以 CAS 推进游标、确认本次调用并提交。 + - 块内抛出异常、超过 `transaction_timeout_seconds` 或提交冲突，事务整体回滚，本次调用失败。 规则： @@ -473,79 +551,127 @@ class ArtifactTransaction(ArtifactReads): - 插件没有开启写事务块就正常返回，表示本窗口无需产出，执行器单独提交游标推进和调用确认。 - 写事务块提交后，`generate` 中后续代码抛出的异常不再影响已提交的结果，执行器记录为提交后错误，不重试该窗口。 插件应把写事务块作为 `generate` 的最后一步。 -- 写事务块的时限默认 30 秒，计入整次调用的 `timeout_seconds`。块内只应做读取、写操作和插件自有存储的写入， - 不应调用模型。 +- 写事务块的时限计入整次调用的 `timeout_seconds`。块内只应做读取、写操作和插件自有存储的写入，不应调用模型。 -**写操作**：写事务覆盖 Artifact 的全部通用写操作，每个操作复用 core 现有的实现与校验规则，违反规则时立即抛出异常： +### 3.3 写操作 + +写事务覆盖 Artifact 的全部通用写操作，每个操作复用 core 现有的实现与校验规则，违反规则时立即抛出异常： | 操作 | 复用的 core 实现 | 语义 | | --- | --- | --- | -| `put(artifact_id, *, content, search_text=None, source_refs=(), artifact_refs=())` | `ArtifactRepository.create` / `revise` | 不存在则新建 revision 1；已存在则按当前 head 追加 revision。内容与 `search_text` 和当前 revision 相同时为 no-op，返回当前引用 | -| `revert(artifact_id, *, to_revision, reason)` | `revise` 的回滚路径 | 以历史 revision 的内容追加新 revision，记录 `restored_from_revision` 与原因 | -| `set_lifecycle(artifact_id, state, *, replacement_artifact_id=None)` | `ArtifactGovernanceRepository.transition` | active ↔ deprecated（可指定替代者），任意状态 → retired（不可逆）。修改 head 的治理状态，不追加 revision；被合并冻结的输入不能单独变更 | -| `merge(inputs, *, into, content, search_text=None, source_refs=())` | 公共合并方法（[RFC #1859](1859_artifact_merge.md)） | 以 `inputs` 的当前 revision 为输入，新建 `into`，记录合并关系并冻结输入。`into` 必须是新 ID | -| `preview_undo_merge(artifact_id)` / `undo_merge(artifact_id, *, preview=None)` | 公共合并方法的预览与恢复 | 撤销产生该 Artifact 的合并，恢复输入并退役结果；按预览执行时，相关状态变化则拒绝过期预览 | +| `put(artifact_id, *, content, source_refs=(), artifact_refs=(), expected_revision=None)` | 公共合并方法的修改路径 | 不存在则新建 revision 1；已存在则按当前 head 追加 revision。给出 `expected_revision` 时，当前 revision 不一致即冲突，用于事务外读取、事务内写入的乐观并发。内容与当前 revision 相同时为 no-op，返回当前快照 | +| `revert(artifact_id, *, to_revision)` | 公共恢复方法（选定 revision） | 以历史 revision 的内容追加新 revision，记录 `restored_from_revision`；目标已被遗忘或被合并冻结时一并恢复 | +| `set_lifecycle(artifact_id, state, *, replacement_artifact_id=None)` | 公共遗忘与恢复方法；指定替代者或退役时使用 `ArtifactGovernanceRepository.transition` | `deprecated` 为遗忘，`active` 为恢复，`retired` 不可逆。恢复走共享恢复流程：记录一个内容相同的新 revision，被合并冻结的输入会撤销冻结它的合并，按组恢复。遗忘和退役不追加 revision | +| `merge(inputs, *, into, content, source_refs=(), artifact_refs=())` | 公共合并方法（[RFC #1859](1859_artifact_merge.md)） | 以 `inputs` 的当前 revision 为输入，新建 `into`，记录合并关系并冻结输入。`into` 必须是新 ID；同一组输入在撤销后再次合并时，插件应把输入 revision 纳入 `stable_id`，避免与已退役的结果撞号 | +| `preview_undo_merge(artifact_id)` / `undo_merge(artifact_id)` | 公共合并方法的预览与恢复 | 撤销产生该 Artifact 的合并，恢复输入并退役结果，返回恢复与退役的精确引用 | | `bind_handle(handle_key, ref)` | Artifact handle 绑定 | 把稳定 handle 绑定到指定 revision | | `set_tags(artifact_id, tags)` | Artifact tags 替换 | 整体替换该 Artifact 的 tags | -**写入约束**： +写入约束： -- 只能操作本 Scope、清单 `generate.family` 中的 Artifact，插件不能指定其他 family 或 Scope。 +- 只能操作本 Scope、声明中 `family` 的 Artifact，插件不能指定其他 family 或 Scope。 - `source_refs` 只能引用本次窗口中的 Source；`artifact_refs`、`to_revision` 和 `bind_handle` 的目标只能是本 family 中实际存在的 revision。 -- 每个写事务块的操作数、单份内容大小和总字节数有上限，超出即抛出异常并回滚，不截断、不部分提交。 +- 检索文本由 core 按声明的 `search_fields` 从内容中提取，插件不单独提供；同一份内容无论由插件还是用户写入， + 检索文本都相同。 +- 每个写事务块的操作数（1000）、单份内容大小（256 KiB）和总字节数（8 MiB）有上限，超出即抛出异常并回滚， + 不截断、不部分提交。 - core 不校验 `content` 的业务结构，也不保存或固化 schema。内容结构的演进由插件负责，插件可以在 `content` 中自带版本字段。 -### 1.5 读取 +### 3.4 读取 `ArtifactWriter`（事务外，只读已提交数据）和 `ArtifactTransaction`（事务内，同时看到本块已写入的内容）提供相同的读取方法， 范围固定为本 Scope、本 family： ```python +class ArtifactSnapshot(BaseModel): + ref: ArtifactRef # 精确 revision + content: dict[str, JsonValue] + lineage: ArtifactLineage # 直接引用的 Source 与 Artifact + lifecycle_state: LifecycleState # active / deprecated / retired + merged_into_id: str | None # 被合并冻结时指向合并结果 + replacement_artifact_id: str | None + governance_generation: int # 治理状态版本，用于判断读取后是否被修改 + class ArtifactReads(Protocol): async def get(self, artifact_id: str, *, revision: int | None = None) -> ArtifactSnapshot | None: ... - async def revisions(self, artifact_id: str, *, cursor: str | None = None, limit: int = 50) -> RevisionPage: ... + async def revisions(self, artifact_id: str, *, cursor: str | None = None, limit: int = 50) -> ArtifactPage: ... async def list(self, *, states: Collection[LifecycleState] = ("active",), tags: Collection[str] = (), cursor: str | None = None, limit: int = 50) -> ArtifactPage: ... - async def search(self, query: str, *, states: Collection[LifecycleState] = ("active",), - limit: int = 10) -> Sequence[ArtifactSnapshot]: ... - async def by_source(self, ref: SourceRef, *, cursor: str | None = None, limit: int = 50) -> ArtifactRefPage: ... - async def lineage(self, ref: ArtifactRef) -> ArtifactLineage: ... - async def governance(self, artifact_id: str) -> ArtifactGovernance: ... # 生命周期、替代者、合并归属 - async def merge_record(self, artifact_id: str) -> ArtifactMergeRecord | None: ... - async def tags(self, artifact_id: str) -> ArtifactTagSet: ... + async def search(self, query: str, *, limit: int = 10) -> Sequence[ArtifactSnapshot]: ... + async def by_source(self, ref: SourceRef) -> Sequence[ArtifactSnapshot]: ... + async def merge_inputs(self, artifact_id: str) -> Sequence[ArtifactRef]: ... + async def tags(self, artifact_id: str) -> Sequence[str]: ... async def resolve_handle(self, handle_key: str) -> ArtifactRef | None: ... ``` -- `get` 可以读当前内容或指定 revision；`revisions` 列出完整历史。 -- `list` 和 `search` 可以按生命周期状态过滤，包括 deprecated、retired 和被合并冻结的 Artifact，用于查找“之前已经存在的 - 数据”。`search` 是 core 提供的文本检索（见第 5 节），在写事务中能检索到本块已写入、尚未提交的内容。 +- 快照同时携带精确 revision、lineage 和当前治理状态，插件不需要再分别读取 lineage 和生命周期。 +- `get` 可以读当前内容或指定 revision；`revisions` 列出完整历史；`merge_inputs` 读取合并结果的精确输入。 +- `list` 可以按生命周期状态和 tags 过滤，包括 deprecated、retired 和被合并冻结的 Artifact，用于查找之前已经存在的数据。 +- `search` 是 core 提供的文本检索（见第 6 节），只覆盖 active 的当前 revision；在写事务中能检索到本块已写入、尚未提交的内容。 - `by_source` 反查 lineage 引用了某条 Source 的 Artifact，用于 Source 更新后找到需要修订的 Artifact。 - 读取返回的快照都带精确 revision，可以直接作为写操作的 `artifact_refs` 或 `to_revision`。 -- 插件需要向量相似度等 core 不提供的检索时，使用自己的索引，再用 `get` 以 core 中的当前 revision 为准。 -### 1.6 一致性与幂等 +### 3.5 一致性与幂等 + +- **核心数据**：每个窗口的 Artifact 写操作、游标推进和调用确认在一个事务中提交。失败时核心数据完全不变，重试从干净状态 + 开始；成功后游标已越过该窗口，同一窗口不会再次投递。冲突（例如 head 在读取后被用户修改）在操作时或提交时报告，整体 + 回滚后由 Supervisor 重新调度，下一次调用读到修改后的状态。 +- **重复处理同一业务对象**：不消费 Source 的定时整理等场景会再次处理已经处理过的对象。插件用 `writer.stable_id(...)` 从业务 + 身份派生 Artifact ID；`put` 在内容相同时为 no-op，`set_lifecycle` 目标状态与当前相同时为 no-op，`merge` 在输入已全部 + 合并到同一结果时为 no-op，从而不会产生重复 Artifact 或无意义的 revision。 +- **插件自有存储**：插件的索引和其他存储与 core 是不同的数据库，不在 core 的事务中，core 不保证两者一致，由插件负责。 + 推荐做法： + 1. 在写事务块内、拿到 `ArtifactSnapshot` 之后写自有存储，按 `artifact_id` upsert。这样自有存储的写入失败会让核心写入 + 一起回滚。 + 2. 写事务提交失败时，自有存储可能留下本次的条目。重试会处理同一窗口、得到同一个稳定 ID，覆盖这些条目。 + 3. 把自有索引当作候选来源：命中的 ID 用 `get` 读取 core 中的当前快照，以快照内容和状态为准。这样回滚留下的条目和用户 + 修改都不会让插件读到过期内容。插件也可以定期按 core 数据重建索引。 +- `generate` 本身可能因失败重试而执行多次，模型调用和插件发出的外部请求不是恰好一次。 + +## 4. 用户对插件 Artifact 的修改 + +用户修改通过 core 的通用 Artifact 接口完成，由 Server 在短事务中同步执行，不调用插件，也不进入 Supervisor。 -**核心数据**：每个窗口的 Artifact 写操作、游标推进和调用确认在一个事务中提交。失败时核心数据完全不变，重试从干净状态 -开始；成功后游标已越过该窗口，同一窗口不会再次投递。core 数据对每个窗口恰好提交一次。写事务内的操作直接作用于数据库， -冲突（例如 head 在读取后被并发修改）在操作时或提交时立即报告，整体回滚后由 Supervisor 重新调度。插件 family 第一版不开放 -公开写接口，同一插件同一 Scope 又是单飞执行，因此这类冲突只在异常情况下出现。 +**编辑与 tags** -**重复处理同一业务对象**：手动重跑、未配置 `inputs` 的定时整理等场景会再次处理已经处理过的对象。插件用 -`writer.stable_id(...)` 从业务身份派生 Artifact ID，`put` 在内容相同时为 no-op，`set_lifecycle` 目标状态与当前相同时 -为 no-op，`merge` 在输入已全部合并到同一结果时为 no-op，从而不会产生重复 Artifact 或无意义的 revision。 +| 接口 | 规则 | +| --- | --- | +| `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | 请求体为 `{"content": {...}}`，content 按原样保存，检索文本按声明的 `search_fields` 提取；`If-Match` 与当前 revision 不一致返回 412；Artifact 必须已存在。与内置 family 的编辑一样，提交的内容另存为一条内部 Content Source（`lineage_only`），作为新 revision 的直接来源记入 lineage；这类 Source 不进入任何插件的输入窗口 | +| `GET` / `PUT .../{artifact_id}/tags`、`POST /v1/scopes/{scope_id}/artifact-tags/query` | 与内置 family 相同；tags 不产生新的内容 revision | -**插件自有存储**：插件的索引和其他存储与 core 是不同的数据库，不在 core 的事务中，core 不保证两者一致，由插件负责。 -推荐做法： +**合并、遗忘与恢复** -1. 在写事务块内、拿到 `ArtifactRef` 之后写自有存储，按 `artifact_id`（必要时加 revision）upsert。这样自有存储的写入 - 失败会让核心写入一起回滚。 -2. 写事务提交失败时，自有存储可能留下本次的条目。重试会处理同一窗口、得到同一个稳定 ID，覆盖这些条目。 -3. 从自有存储读出的结果，用 `get` 校验 core 中是否存在、revision 是否一致，忽略不一致的条目，或定期按 core 数据重建。 +这几项能力由共享的合并引擎提供（`ArtifactMergeService`，[RFC #1859](1859_artifact_merge.md)）。引擎本来就按 family 无关的方式 +设计：每个 family 提供一个适配器，负责生成合并后的内容和维护自己的索引；版本校验、级联恢复、lineage 记录和权限检查都在 +引擎中完成。运行时用一个注册表（`ArtifactMergeApplication`）按 family 名找到对应的引擎实例。内置 family 目前登记了 +Experience 和 Atomic Memory；Server 加载插件时，为每个插件 family 创建一个引擎实例（插件适配器 + 共享的 Artifact 访问策略） +并登记进去。 -`generate` 本身可能因失败重试而执行多次，模型调用和插件发出的外部请求不是恰好一次。 +在注册表之上提供一组不区分 family 的接口，对所有登记过的 family 生效；没有登记的 family 返回 422： -## 2. 插件执行器与消费游标 +| 接口 | 规则 | +| --- | --- | +| `GET .../{artifact_id}/governance` | 返回当前 revision、生命周期状态、`state_version` 和合并目标 | +| `POST /v1/scopes/{scope_id}/artifacts/{family}/merges` | 输入为至少两个当前 Artifact 的 `artifact_id`、`revision`、`state_version`，以及结果的完整内容；新建结果 Artifact 并冻结所有输入。任何输入已变化即返回 409 | +| `POST .../{artifact_id}/forget` | 携带 `revision` 和 `state_version`；把 Artifact 标为 deprecated（无替代者），不追加 revision，退出检索 | +| `POST .../{artifact_id}/restorations` | `operation` 为 `restore`（默认，可指定 `revision` 恢复到旧内容）或 `undo_merge`；按组执行级联恢复和退役 | + +- **权限**：由引擎检查，与内置 family 相同：修改已有 Artifact 需要该 Artifact 的写权限（`artifact.write`），新建合并结果 + 需要 Scope 的贡献权限，结果归属于调用者。 + 插件调用中的写入器使用同一个引擎类和适配器，只是权限不同：插件对本 family 的 Artifact 有完整写权限，包括用户合并 + 产生的结果。 +- **不提供预览**：Atomic Memory 的专用接口支持先预览恢复影响、再凭签名 token 执行。通用接口不提供预览，也就不需要签名 + 密钥；恢复和撤销合并按执行时的当前状态进行，合并和遗忘依靠 `revision` 与 `state_version` 校验。 +- **合并内容**：插件 family 的合并内容是不透明 JSON，core 只校验 JSON 结构并按 `search_fields` 提取检索文本；内置 family + 按各自的内容模型校验。 +- **并发**：用户修改与插件调用可能同时发生。插件在事务外读取、事务内用 `expected_revision` 写入时会发现冲突并整体回滚， + 下一次调用基于修改后的状态重新决策；用户修改不会被插件静默覆盖。 +- **不通知插件**：与内置 family 一样，用户修改只记录在 lineage 和治理状态中，不通知插件。插件在下一次调用时通过写入器 + 读到新状态。 +- **新建**：公开接口不为插件 family 直接新建 Artifact（合并结果除外）。调用方要补充知识，就写入 Source。 + +## 5. 插件执行器与消费游标 插件执行器是 Worker 中的 core 代码，实现 Supervisor 的 Worker 入口，替代旧的 `run_artifact_plugin_worker`。 它对 Artifact 插件的一次调用如下： @@ -555,7 +681,8 @@ Worker 收到 (plugin:, scope_id, request_generation) -> 短事务：ScopeInvocation.start；读取 pc_source_cursors 中 (scope_id, plugin:) 的游标 -> 读取 journal 高水位，确定窗口 (after, through]，through ≤ after + max_sources -> list_window_for_generation 读取窗口，按 inputs.source_types 过滤，计算 inputs.projection - -> 加载插件、调用工厂、创建写入器、调用 generate（受 timeout_seconds 约束） + -> 窗口为空且配置了 inputs：短事务完成游标 CAS 与 acknowledge，不加载插件 + -> 导入 plugin.py、核对声明、调用 create、创建写入器、调用 generate（受 timeout_seconds 约束） -> 事务外读取：独立只读短事务 -> 写事务块开始：开启事务；guard（租约与调用意图） 块内写操作在该事务中执行 @@ -566,68 +693,71 @@ Worker 收到 (plugin:, scope_id, request_generation) - 窗口内类型不匹配的 Source 被过滤，但游标仍推进过整个窗口；匹配类型的 Source 缺少所需 Projection 时调用失败，不跳过。 - 游标只在执行器的提交事务中推进，插件得不到游标值，也没有写入接口。游标 CAS 保证在读取窗口与提交之间即使发生 领导权变更或重复派发，同一窗口也只提交一次。 -- `on_sources` 复用现有的 `SourceProcessingPendingProvider`：journal 高水位大于游标时标记待处理，由 Supervisor 按预算派发。 - 判断基于持久化数据，Server 重启后不会丢失待处理输入。 -- 手动触发和添加采集目标复用现有的调用意图（`ArtifactProcessingIntentRepository.request`）：在接受事务中提升 - requested generation，多次请求合并，运行期间的新请求由后续调用处理。 -- 未配置 `scopes` 时，自动调度对所有 journal 超过游标的 Scope 生效；配置了 `scopes` 时，自动调度和手动触发都只接受 - 名单内的 Scope。 +- 待处理工作的发现有两条路径：Server 进程中的 Source 写入会把关注该 Scope 的插件 binding 标记为待处理；启动时 + 现有的 `SourceProcessingPendingProvider` 按 journal 高水位与游标的差值补齐其他进程写入的 Source。Supervisor 只在 + binding 的 interval 或 cron 到期时接纳待处理的 Scope，因此 `on_sources` 映射为约 2 秒的接纳间隔。判断基于持久化数据， + Server 重启后不会丢失待处理输入。 +- 未配置 `scopes` 时，自动调度对所有 journal 超过游标的 Scope 生效；配置了 `scopes` 时只接受名单内的 Scope。 collect 调用结构相同：输入换成该 Scope 的已启用目标及其 checkpoint，提交内容换成 Source 与各目标的新 checkpoint。 -collect 不需要写事务块，由执行器根据返回值一次提交。 +collect 不需要写事务块，由执行器根据返回值一次提交。添加采集目标通过现有的调用意图（`ArtifactProcessingIntentRepository.request`） +请求一次采集。 + +## 6. 公开读取与检索 + +插件 family 的公开读取完全由 core 处理，不执行插件代码： -## 3. 加载与生命周期 +- list / get / revisions / tags 复用现有的通用 Artifact 读取路径。插件 family 的 `content` 按原样 JSON 返回，不需要 + `pc_artifact_family_schemas` 中的 schema。被合并冻结、deprecated 和 retired 的 Artifact 按公共合并与生命周期规则展示。 +- list 返回的条目只含元数据，`title` 和 `summary` 对插件 family 为空；内容需要通过 get 读取。 +- search 是 core 为插件 family 提供的通用文本检索，与现有检索接口一样只接受文本 query： + - 写入时，core 按 `search_fields` 从内容中取出文本，经现有词法分析写入 `pc_artifact_heads.searchable_text`，并随 revision + 保存，恢复旧 revision 时检索文本随内容一起恢复。 + - 检索时在该列上匹配，只返回请求 Scope 中调用者有权读取的 active 当前 revision。没有可检索文本的 Artifact 不参与检索。 + - 写入器的 `search` 使用同一实现。 + - SQLite 上与 Experience、Skill 共用 `pc_artifact_fts`：该辅助表的重建会回填所有带检索文本的 head，就绪校验只比对内置 + family 的行。实现 PR 需要给出 OceanBase 上的检索方式和规模验证。 +- Review、Recall 和 Prepared Context 第一版不对插件 family 开放，对应接口返回明确的“不支持”错误；`GET /v1/capabilities` + 列出插件 family。 + +## 7. 加载与生命周期 ```text Server 启动 -> 扫描 POWERCONTEXT_SERVER_PLUGINS_DIR 的直接子目录（未设置则不加载插件） - -> 对每个目录：解析并校验清单（name 与目录名一致、kind 与配置段匹配） - -> 在有时限的检查子进程中导入入口，确认工厂、Config 和其他必需导出符合 kind，不调用工厂 - -> Source 插件：在 Server 进程导入 definitions、TargetInput 与 identify_target，注册 Source 类型 - -> 导入清单中尚不存在的初始采集目标 - -> 注册 family 的访问 profile；有 collect / generate 配置时注册 Supervisor binding + -> 对每个目录：在有时限的检查子进程中导入 plugin.py，校验声明，不调用 create + -> Source 插件：在 Server 进程导入 definitions、target_input 与 identify_target，注册 Source 类型 + -> 注册 family 的读取与检索；有调度或触发时注册 Supervisor binding -> 任一步失败：记录错误，该插件标记为 failed，继续下一个 Server 停止 -> 按现有 Supervisor 关闭流程停止派发并回收 Worker ``` -插件只有 `loaded` 和 `failed` 两种结果，只存在于进程内存中。Server 不持久化插件的启用状态、注册代次或配置快照； -每次启动按当时的目录内容重新检查。没有热加载，因此不需要注册代次来屏蔽旧 Worker，现有的 Supervisor 租约 fence -已经足以屏蔽领导权变更后的旧提交。 - -Worker 每次调用重新导入插件代码。插件目录在 Server 运行期间被修改时，新派发的 Worker 可能读到新代码；部署者应在 -修改后重启 Server，第一版不检测这种情况。 +- 插件只有 `loaded` 和 `failed` 两种结果，只存在于进程内存中。Server 不持久化插件的启用状态、注册代次或配置快照； + 每次启动按当时的目录内容重新检查。 +- 没有热加载，因此不需要注册代次来屏蔽旧 Worker，现有的 Supervisor 租约 fence 已经足以屏蔽领导权变更后的旧提交。 +- Worker 每次调用重新导入插件代码并核对声明。插件目录在 Server 运行期间被修改、声明随之改变时，调用失败并提示重启； + 只改了实现代码时，新派发的 Worker 会使用新代码。部署者应在修改后重启 Server。 +- 只读 API 角色的 Server 实例也会检查插件并注册 family 和 Source 类型，用于读取、检索和用户修改，但不派发 Worker。 -## 4. 执行约束 +## 8. 执行约束 - **隔离**：`collect` / `generate` 在 Supervisor 启动的 Worker 子进程中执行，崩溃、内存泄漏和阻塞不影响 Server。 子进程隔离普通故障，不是不可信代码的沙箱：插件以 Worker 的操作系统权限运行，可以访问其文件、网络和环境变量。 -- **超时与预算**：`timeout_seconds`（默认 300）是 Worker 总时限，`transaction_timeout_seconds`（默认 30）是写事务块时限， - `max_workers`（默认 1）是该插件跨 Scope 的并发上限。前两者和并发上限映射到 Supervisor binding 的现有参数或由执行器强制。 - 单次调用的输入字节、写操作数和输出字节有上限，超出即失败。 +- **超时与预算**：`timeout_seconds` 是 Worker 总时限，`transaction_timeout_seconds` 是写事务块时限，`max_workers` 是该插件 + 跨 Scope 的并发上限，分别映射到 Supervisor binding 的现有参数或由执行器强制。 - **写事务的锁影响**：写事务块持有期间会锁住涉及的 head 和游标行；在 SQLite 上，写事务会阻塞整个数据库的其他写入。 因此写事务块必须短，时限超出即回滚。 +- **数据库访问**：Worker 打开数据库时不创建表，Schema 由 Server 准备。现有的数据库就绪检查会校验每个待处理调用意图的 + binding 都已持久注册；`plugin:` 前缀的 binding 由插件目录在启动时决定，就绪检查对其放行，插件移除后遗留的调用意图 + 不再被处理。 - **失败与退避**：调用失败时不提交任何核心数据，该插件该 Scope 进入 Supervisor 现有的退避，不影响其他 Scope 和插件。 core 不会因为重复失败自动跳过输入；持续失败需要修复插件或数据后重启。collect 中某个目标持续失败会阻塞同一 Scope 的其他目标，可以先停用该目标。 -- **多实例部署**：OceanBase 多 Server 实例时，派发只发生在持有 Supervisor 领导租约的实例上；write、目标管理和读取 - 可以在任意实例执行。插件自有存储需要能被执行 Worker 的实例访问。SQLite 部署只有一个 Server 进程。 - -## 5. 公开读取与检索 - -插件 family 的公开读取完全由 core 处理，不执行插件代码： +- **多实例部署**：OceanBase 多 Server 实例时，派发只发生在持有 Supervisor 领导租约的实例上；Source 写入、目标管理、 + 读取和用户修改可以在任意实例执行。插件自有存储需要能被执行 Worker 的实例访问。SQLite 部署只有一个 Server 进程。 -- list / get / revisions / tags 复用现有的通用 Artifact 读取路径。插件 family 的 `content` 按原样 JSON 返回，不需要 - `pc_artifact_family_schemas` 中的 schema。被合并冻结、deprecated 和 retired 的 Artifact 按公共合并与生命周期规则展示。 -- search 是 core 为插件 family 新增的通用文本检索，与现有检索接口一样只接受文本 query：写入时把 `search_text` - 经现有词法分析写入 `pc_artifact_heads.searchable_text`，检索时在该列上匹配，只返回请求 Scope 中调用者有权读取的 - revision。未提供 `search_text` 的 Artifact 不参与检索。写入器的 `search` 使用同一实现。实现 PR 需要给出 SQLite 与 - OceanBase 上的检索方式和规模验证。 -- Review、Recall 和 Prepared Context 第一版不对插件 family 开放；公开的创建、修订、合并、生命周期、handle 和 tags 写接口 - 也不对插件 family 开放，插件 family 的写入只来自插件。对应接口返回明确的“不支持”错误，`GET /v1/capabilities` - 反映实际支持的操作。 - -## 6. 持久化 +## 9. 持久化 | 数据 | 处理 | | --- | --- | @@ -636,65 +766,70 @@ Worker 每次调用重新导入插件代码。插件目录在 Server 运行期 | `pc_source_cursors` | 保留；Artifact 插件的消费游标，键为 `(scope_id, plugin:)` | | `pc_connector_checkpoints` | 保留；各采集目标的 checkpoint，键为 `(scope_id, 目标的 binding ID)` | | `pc_source_collect_targets` | 新增；采集目标：Scope、插件名、目标键、规范化输入、启用状态、创建与更新时间、最近成功采集时间、最近失败分类 | -| Supervisor 的调用意图、pending、租约表 | 保留；插件 binding 与内置 binding 共用 | +| Supervisor 的调用意图、pending、租约表 | 保留；插件 binding 与内置 binding 共用，也是 `GET /v1/plugins/{name}/scopes` 的数据来源 | | `pc_artifact_processing_binding_states` 中为插件新增的列（`plugin_id`、`plugin_family`、`binding_generation`、`desired_state`、`phase`、`last_error`） | 移除 | | `pc_artifact_family_schemas` | 移除 | | 自定义 family 声明式索引使用的搜索状态与向量索引数据 | 移除；实现 PR 确认内置 family 不依赖后删除对应表 | | 旧 Artifact 插件写入的 Artifact、head、lineage | 删除，不迁移 | -| `pc_source_write_plugins` | 移除；write 入口改由插件目录在启动时提供 | +| `pc_source_write_plugins` | 移除；Source 定义改由插件目录在启动时提供 | | Source 插件管理服务的 SQLite | 随管理服务一起移除 | | 插件自有的索引与存储 | 不属于 core，由插件和部署者管理 | 目标的 binding ID 由插件名和目标键确定性派生，长度超过列宽时使用其摘要。新增与移除通过一次新的数据库迁移完成， 迁移说明中列出新增、删除的表、列和数据范围。 -## 7. 接口变更 +## 10. 接口变更 | 变更 | 内容 | | --- | --- | -| 移除 | `/v1/admin/artifact-plugins` 及其子路径；Source 插件管理服务、插件实例 Runtime、poll / reply 通道；`source load` 命令；MCP `pc_source_instances`、`pc_source_instance_set_enabled` | +| 移除 | `/v1/admin/artifact-plugins` 及其子路径；Source 插件管理服务、插件实例 Runtime、poll / reply 通道；`source load` 命令；MCP `pc_source_instances`、`pc_source_instance_set_enabled`；`plugin.yml` 与 `powercontext.yml` 清单 | | 移除 | 自定义 family 的声明式索引（`ArtifactSearchDeclaration`、`TextIndex` 等）、`ArtifactFamily` 的 schema 声明、`ArtifactDraft` 和旧的 Worker 插件执行器 | -| 新增 | `GET /v1/plugins`、`GET /v1/plugins/{name}`（`server.admin`，只读） | -| 新增 | `POST /v1/scopes/{scope_id}/artifacts/{family}/generate`、`POST /v1/scopes/{scope_id}/sources/plugins/{name}/collect`（`202 Accepted`） | -| 新增 | `GET` / `POST /v1/scopes/{scope_id}/sources/plugins/{name}/targets`、`PATCH .../targets/{key}` | +| 新增 | `powercontext.plugins`：`SourcePlugin`、`ArtifactPlugin`、`Inputs`、`Interval`、`Cron`、`PluginContext`、写入器与读取接口 | +| 新增 | `GET /v1/plugins`、`GET /v1/plugins/{name}`、`GET /v1/plugins/{name}/scopes`（`server.admin`，只读） | +| 新增 | `GET .../{artifact_id}/governance`、`POST /v1/scopes/{scope_id}/artifacts/{family}/merges`、`POST .../{artifact_id}/forget`、`POST .../{artifact_id}/restorations`：基于共享合并引擎注册表的通用接口，对所有登记过的 family 生效（内置为 Experience 和 Atomic Memory，以及加载的插件 family）。Atomic Memory 的专用接口保持不变 | +| 调整 | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` 接受插件 family，请求体为 `{"content": {...}}` | +| 调整 | tags 接口接受插件 family：`TaggableArtifactFamily` 从枚举改为与 `ArtifactReadFamily` 相同格式的字符串，服务端校验为内置 family 或已加载的插件 family。网络上的数据格式不变，生成的客户端少了这个枚举类型 | | 调整 | 通用 Artifact 读取与检索接口支持插件 family；`GET /v1/capabilities` 列出插件 family | -| 调整 | MCP `pc_source_add` 的 write 用法使用插件定义，collect 用法添加采集目标；`pc_source_capabilities` 列出插件 Source 类型与目标输入；`pc_source_status` 报告目标及其最近采集结果；`pc_source_target_set_enabled` 改为作用于 core 保存的目标 | +| 调整 | `pc_source_add` 的 write 用法使用插件定义，collect 用法添加采集目标；`pc_source_capabilities` 列出插件 Source 类型与目标输入；`pc_source_status` 报告目标及其最近采集结果；`pc_source_target_set_enabled` 作用于 core 保存的目标 | | 调整 | 环境变量 `POWERCONTEXT_SERVER_PLUGINS_DIR` 取代 `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` 和 Source 插件目录配置 | -`openapi/powercontext.yaml` 按上表更新后重新生成 `src/powercontext/http/_generated/`。 +`openapi/powercontext.yaml` 按上表更新后重新生成 `src/powercontext/http/_generated/` 和各集成的操作表。 -## 8. 可观测性 +## 11. 可观测性 插件调用复用 Supervisor 与 Worker 现有的操作词汇、执行身份和结果字段，不为定时调用生成 request ID。新增字段： | 字段 | 含义 | | --- | --- | -| `plugin_name` | 清单中的插件名 | +| `plugin_name` | 插件名 | | `plugin_method` | `collect` 或 `generate` | -| `trigger` | `schedule`、`sources`、`manual`、`target_added` | +| `trigger` | `schedule`、`sources`、`target_added` | | `operation_count` | 本次提交的写操作数（Artifact）或 Source 条数（collect） | | `target_count` | 本次 collect 携带的目标数 | | `transaction_ms` | 写事务块从开始到提交或回滚的耗时，毫秒 | 指标标签可以使用 `plugin_name` 和 `plugin_method`，不包含 Scope ID 或目标键。日志经 `log_safely` 输出，不包含 Source -内容、Projection、Artifact 内容、目标输入、`params`、`config` 值和环境变量值；插件异常只记录类型和已知错误分类， -不记录异常消息。新增字段按开发指南登记到允许列表，并在中英文可观测性文档中说明。 +内容、Projection、Artifact 内容、目标输入和环境变量值；插件异常只记录类型和已知错误分类，不记录异常消息。新增字段按 +开发指南登记到允许列表，并在中英文可观测性文档中说明。调用结果只进入日志和指标，不持久化，运维接口不提供“最近一次 +调用结果”。 # 缺点 -- **部分插件代码在 Server 进程中执行**：write 需要同步返回，Source 定义的 `resolve`、Projection 和 `identify_target` +- **部分插件代码在 Server 进程中执行**：Source 写入需要同步返回，Source 定义的 `resolve`、Projection 和 `identify_target` 不经过 Worker。它们应当是纯数据转换，但阻塞或异常仍会影响请求处理。Server 对这些调用设置超时并把异常转换为请求错误。 - **写事务持锁**：写事务块期间持有数据库锁，SQLite 上会阻塞其他写入。插件若在块内执行慢操作，会拖慢整个 Server 的写入， 直到块超时回滚。 -- **插件自有存储与核心数据不保证一致**：两者不在同一事务中，提交失败可能在自有存储中留下条目，插件需要按稳定 ID 覆盖 - 并以 core 为准校验。 +- **插件自有存储与核心数据不保证一致**：两者不在同一事务中，提交失败可能在自有存储中留下条目；用户修改也不通知插件。 + 插件需要按稳定 ID 覆盖，并以 core 的当前数据为准。 - **每次调用都要启动 Worker 并加载插件**：子进程启动和工厂初始化有固定开销，不适合秒级高频调用；插件不能在调用之间 复用连接或缓存。 - **不是安全沙箱**：Worker 隔离普通故障，不限制插件的文件、网络和凭据访问，也不提供硬性的内存或 CPU 配额。 - 写入器约束的是插件对核心数据的写入，不能阻止插件代码访问 Worker 可达的其他资源。 - **依赖必须兼容**：所有插件与 Server 共用一个 Python 环境，依赖冲突需要部署者解决。 -- **内容结构不受 core 治理**：core 不校验 `content` 的 schema，历史 revision 的解释需要插件和应用自行保持兼容。 -- **core 检索能力有限**：core 只提供基于 `search_text` 的通用文本检索；向量等检索需要插件自建索引。 +- **同一份代码部署多次需要额外处理**：family 写在声明中，同一份代码复制成两个目录会因 family 冲突而检查失败。需要多次 + 部署时，插件应从环境变量读取 family 等标识。 +- **内容结构不受 core 治理**：core 不校验 `content` 的 schema，历史 revision 的解释需要插件和应用自行保持兼容； + 用户通过 `PUT` 提交的内容也只由插件在后续读取时解释。 +- **core 检索能力有限**：core 只提供基于声明字段的通用文本检索；向量等检索需要插件自建索引。 - **采集与 Server 部署在一起**：采集在 Server 的 Worker 中执行，需要 Server 能访问外部数据源。 - **代码变更需要重启**：没有热加载，插件更新期间 Server 会短暂不可用。 @@ -703,33 +838,41 @@ Worker 每次调用重新导入插件代码。插件目录在 Server 运行期 | 方案 | 取舍 | | --- | --- | | 保留现有设计，只简化接口 | 插件作者体验改善有限，生命周期状态机、管理服务和 schema 固化的维护成本仍在 | +| 清单文件（YAML）描述插件 | 插件需要同时维护清单和代码，两者的字段要保持一致；清单中的变量替换、配置模型和导出约定又是一套额外规则。声明写在代码里，由同一个 Pydantic 模型校验，插件作者只写 Python | | 插件在 Server 进程内执行 | 部署和调用最简单，但插件故障会拖垮 Server；Supervisor 已经提供隔离和调度，没有理由另起一套 | | 插件完全自管存储和游标 | core 读取接口无法直接使用，各插件需要重复实现窗口读取与幂等，游标与结果也难以原子提交 | -| 插件通过上下文接口逐条写入、各自提交，最后返回成功与否 | 写入与游标推进分属不同事务，失败时会留下部分结果，重试会重复写入 | | 插件返回草稿，或由写入器在内存中暂存操作，最后统一提交 | 可以原子提交，但插件在提交前拿不到真实的 revision，难以同步写入自有索引；合并、撤销合并、生命周期等操作需要在内存中重新实现一遍 core 的语义；检索也看不到暂存内容 | | 用一个事务包住整个 `generate` | 模型调用可能持续数分钟，长时间持锁会阻塞其他写入，SQLite 上会阻塞整个数据库；插件自有存储是另一个数据库，一个事务本来也覆盖不了 | | `generate` 内的短写事务块（本提案） | 耗时工作在块外；块内直接复用 core 的写操作和校验，拿到真实 revision，读取能看到本块写入；块结束时与游标、确认一起提交，异常即回滚。代价是块内持锁，需要时限约束 | +| 用户命令经插件执行（手动请求排队，插件在下一次调用中处理） | 插件要实现命令解析，调用方只能异步等待；一个始终失败的命令会阻塞该 Scope 的后续生成；还需要持久化请求队列。遗忘、恢复、编辑都是 core 已有的通用操作，由 core 同步执行更简单 | +| 插件专用的触发与管理接口 | 调用方要区分插件与内置能力；现有的 Source 与 Artifact 接口已经覆盖调用方需要的操作 | +| 通用合并接口只对插件 family 开放 | 需要在接口里额外维护一份 family 白名单。能操作哪些 family 本来就由注册表决定，没有适配器的 family 无法登记；内置 family 的权限检查在引擎内，两个入口走同一套检查 | | 常驻插件进程（类似 Dify 的插件守护进程） | 可以复用连接、支持多语言，但需要进程管理、序列化协议和取消协议。接口以请求 / 响应模型定义，后续可以替换执行方式而不改插件代码 | | 支持热加载 | 需要重新引入加载状态、注册代次屏蔽和旧调用回收，与目标冲突 | -选择“Supervisor 派发 Worker、插件在短写事务块中直接写入、core 与游标一并提交”的原因：插件作者只需写一个工厂和一个方法； -隔离、调度、触发、退避和领导权都复用已经在生产中使用的 Supervisor；写操作直接复用 core 现有实现，不需要另一套语义； -Source 与 Artifact 留在 core 的统一存储中，现有读取接口、lineage、合并记录和访问控制直接适用；core 只管理两类核心数据 -及其进度，其余交给插件。 +选择这一方案的原因： + +- 插件作者只需写一个 `plugin.py`：一个声明、一个工厂、一两个方法。 +- 调用方只用现有的 Source 与 Artifact 接口，不感知插件。 +- 隔离、调度、触发、退避和领导权都复用已经在生产中使用的 Supervisor。 +- 写操作直接复用 core 现有实现，插件写入和用户修改遵循同一套语义。 +- Source 与 Artifact 留在 core 的统一存储中，现有读取接口、lineage、合并记录和访问控制直接适用。 +- core 只管理两类核心数据及其进度，其余交给插件。 # 先例 -- **Dify 插件**：按插件类型（工具、模型、Agent 策略、扩展端点等）定义接口，插件用清单声明类型与参数并实现对应方法， - Dify 在工作流或应用需要时调用。插件运行在独立的插件守护进程中。本提案沿用“宿主定义接口与调用时机”的思路，执行 - 隔离复用 PowerContext 已有的 Supervisor Worker。 +- **Dify 插件**：按插件类型定义接口，插件实现对应方法，Dify 在工作流或应用需要时调用；插件运行在独立的插件守护进程中。 + 本提案沿用“宿主定义接口与调用时机”的思路，执行隔离复用 PowerContext 已有的 Supervisor Worker。 - **Airbyte 连接器协议**：Source 连接器的 `read` 接收上次的 state 并输出记录和新 state，由平台持久化 state。 本提案的采集 checkpoint 采用相同分工，并按目标分别保存。 +- **pytest 插件与 setuptools entry points**：插件以 Python 对象声明自己提供的能力，宿主导入后读取，不需要单独的清单文件。 - **事务上下文管理器**：SQLAlchemy 的 `async with engine.begin()` 等接口在块开始时开启事务、正常结束时提交、异常时回滚。 `writer.transaction()` 沿用这一使用方式，并在提交时附带游标与调用确认。 - **PowerContext 现有基础**：[Artifact processing Supervisor](1515_artifact_processing_supervisor.md) 的调度、调用意图、 Worker 与租约 fence；[公共合并方法](1859_artifact_merge.md)；`ArtifactRepository`、`ArtifactGovernanceRepository`、 - handle 与 tags 的写入实现；`SourceDefinition` / `SourceProjection`；Source journal、`pc_source_cursors` 和 - `list_window_for_generation`；`pc_connector_checkpoints`；旧 Source 插件设计中的 `identify_target` 目标识别规则。 + handle 与 tags 的写入实现；通用 Artifact 读写接口；`SourceDefinition` / `SourceProjection`；Source journal、 + `pc_source_cursors` 和 `list_window_for_generation`；`pc_connector_checkpoints`；Source 管理接口与 `identify_target` + 目标识别规则。 # 未决问题 @@ -740,6 +883,11 @@ Source 与 Artifact 留在 core 的统一存储中，现有读取接口、lineag | 共享模型客户端 | 是否在 `PluginContext` 中提供 Server 已配置的模型客户端，让 LLM 插件不必自带凭据，并计入现有模型用量统计 | | 采集目标的删除 | 第一版只支持停用和恢复。是否需要彻底删除目标及其 checkpoint，以及删除后已采集 Source 的处理 | | 插件 family 接入 Recall / Prepared Context | 需要定义候选内容和预算约束，建议另行提案 | +| 窗口大小的回退 | 内置 Atomic Memory 在模型超时时会缩小下一次的 Source 窗口。插件无法影响窗口大小，过大的窗口会持续超时。是否允许插件以约定的异常请求缩小该 Scope 的窗口 | +| 持续失败的窗口 | 始终失败的 Source 窗口会阻塞该 Scope 的后续调用。是否提供跳过或转入死信的运维操作，以及如何记录被跳过的输入 | +| 消费全部 Source 类型 | `inputs.source_types` 需要逐一列出类型；内置 Atomic Memory 消费所有可用于生成的 Source。是否支持 `"*"` 表示全部类型 | +| 内置专属能力 | Scope Prompt 管理、写入闸门与决策观测、模型用量统计目前只服务内置 family，插件无法接入。是否以及如何向插件开放 | +| Atomic Memory 专用接口的去留 | 通用合并接口上线后，Atomic Memory 的合并、遗忘、恢复同时有专用和通用两个入口，行为一致。专用接口是否保留，或改为通用接口的别名 | # 后续可能 From bcadb81eb204b3220166fad8660d22a8d23eb658 Mon Sep 17 00:00:00 2001 From: jasper-yl Date: Tue, 10 Nov 2026 00:29:27 -0800 Subject: [PATCH 05/12] docs(rfc): route user changes through plugin host processes and add edge collection Co-Authored-By: Claude Opus 5.5 --- docs/en/rfcs/2549-plugin-interfaces.md | 589 ++++++++++++++++++------- docs/zh/rfcs/2549-plugin-interfaces.md | 492 +++++++++++++++------ 2 files changed, 794 insertions(+), 287 deletions(-) diff --git a/docs/en/rfcs/2549-plugin-interfaces.md b/docs/en/rfcs/2549-plugin-interfaces.md index fce15316a2..66534c59de 100644 --- a/docs/en/rfcs/2549-plugin-interfaces.md +++ b/docs/en/rfcs/2549-plugin-interfaces.md @@ -1,6 +1,6 @@ --- title: Plugin interfaces -description: Source and Artifact plugins each implement a few interfaces that PC Server calls through the Supervisor when needed; callers use only the existing Source and Artifact APIs, and core manages only the two core data kinds and their consumption progress, without managing plugin lifecycle. +description: Source and Artifact plugins each implement a few interfaces that PC Server calls when needed, with background calls in Supervisor Workers and synchronous calls in plugin host processes that core manages; callers use only the existing Source and Artifact APIs, and core manages only the two core data kinds and their consumption progress, without managing plugin lifecycle. --- - Proposal Name: `plugin_interfaces` @@ -22,23 +22,38 @@ APIs exactly as they do for built-in capabilities. deployment-specific settings from environment variables. - **How callers use plugins**: - Writing Sources and adding collection targets use the existing Source APIs. - - Reading, searching, and changing Artifacts use the existing Artifact APIs. - - User operations such as forget, restore, and edit are executed by core directly, without the plugin. -- **Execution**: `collect` and `generate` run in Worker child processes managed by the Supervisor, reusing its existing - isolation, timeouts, concurrency budgets, leader lease, and failure backoff. + - Reading and searching Artifacts use the existing Artifact APIs. + - Creating, editing, merging, forgetting, and restoring plugin Artifacts also use the existing Artifact APIs. Core + handles authorization, version checks, and provenance, then synchronously calls the matching plugin method; the plugin + validates the content, maintains its own storage, and writes through the writer. An operation the plugin does not + implement returns 422. +- **Execution**: + - `collect` and `generate` are background calls. They run in Worker child processes managed by the Supervisor, + reusing its existing isolation, timeouts, concurrency budgets, leader lease, and failure backoff. + - `resolve` and Projections during Source writes, `identify_target` when adding a collection target, and user changes + to plugin Artifacts are synchronous calls. They run in a long-lived **plugin host process** that core manages: each + Server process starts one host process per plugin on demand, and deployers do not manage it. - **Writes**: an Artifact plugin performs its writes inside an `async with writer.transaction()` block. When the block ends, core commits the writes, the consumption cursor, and the completion acknowledgment together; an exception rolls everything back. Plugins can neither read nor change the cursor. - **What core owns**: only `pc_sources`, `pc_artifacts` (including heads, lineage, merge records, lifecycle, handles, and - tags), and their consumption progress. Plugins may have their own indexes and storage and manage them themselves. -- **Lifecycle**: plugins load from a directory at Server startup, and changes require a restart. Server stores no plugin - enabled state, registration generation, or management phase. + tags), and their consumption progress. Plugins may have their own indexes and storage and manage them themselves. Core + gives each plugin a dedicated storage area on the database backend the deployment uses, and lets plugins use the models + Server has configured. +- **Edge collection**: Server can start in `source-agent` mode, which only runs Source plugin collection and submits + Sources and checkpoints to a remote central Server. This serves scenarios such as embodied AI, where data is collected + on the device. +- **Lifecycle**: plugins load from a directory at Server startup, and changes require a restart. Each Server process + records its active plugins only in memory and persists no plugin enabled state, registration generation, or + management phase. This proposal replaces the plugin contracts and management mechanisms of the [Source plugin framework RFC #2012](2012-source-plugin-system.md) and the -[Artifact plugin framework RFC #2175](2175-artifact-plugin-system.md). Neither implementation has been released as a -stable interface, so this proposal keeps no compatibility: the old plugin manifests, management APIs, management service, -and plugin-management tables are removed, and custom Artifact data written by old Artifact plugins is not migrated. +[Artifact plugin framework RFC #2175](2175-artifact-plugin-system.md). The old Source plugin framework shipped in v1.5, +but no plugins for it have been published, so this proposal keeps no compatibility: the old plugin manifests, management +APIs, management service, and plugin-management tables are removed, and custom Artifact data written by old Artifact +plugins is not migrated. Parts of the old design that work well are kept, including the Source management APIs +(`pc_source_add` and others) and their collection target identity rules, and core's existing remote ingestion APIs. This document describes the target design; it does not mean the mainline or a release already provides it. Interface names and code examples illustrate the contract; the implementation determines the final syntax. @@ -73,15 +88,18 @@ Goals: 1. A plugin author writes only `plugin.py`: one declaration object and a few methods, without learning the internals of plugin management. 2. Callers are unaware of plugins: the existing Source and Artifact APIs apply equally to data plugins produce. -3. Plugin code runs in isolated Worker processes, so a plugin failure does not affect Server or other plugins. +3. Plugin code runs only in child processes core manages (Workers or plugin host processes), so a plugin failure does + not affect Server or other plugins. 4. Plugin results go into `pc_sources` / `pc_artifacts` and commit atomically with consumption progress. 5. Core manages only these two core data kinds and their consumption progress; content structure, indexes, and other storage belong to the plugin. 6. Plugin writes to core data go only through a writer core provides, which covers every generic Artifact write operation and keeps core's existing validation rules. -7. Artifact generation can run on a schedule or be triggered by new Sources; collection targets can be added and removed +7. User changes to plugin Artifacts go through the plugin, so plugin-owned storage is updated together with the change. +8. Artifact generation can run on a schedule or be triggered by new Sources; collection targets can be added and removed at runtime. -8. Users deploy a single Server; plugins load when Server starts. +9. Users deploy a single Server; plugins load when Server starts. When data must be collected on a device, the same + program runs in edge mode. # Guide-level explanation @@ -96,6 +114,9 @@ flowchart LR Sup["Supervisor
schedules / triggers / budgets / backoff"] Core[("Core data
pc_sources / pc_artifacts
collection targets / checkpoints / cursors")] end + subgraph Host["Plugin host process (long-lived, managed by core)"] + Sync["resolve / Projections / identify_target
create_artifact / update_artifact / ..."] + end subgraph SW["Worker: Source plugin"] Collect["collect(request)"] end @@ -106,8 +127,11 @@ flowchart LR end Caller --> SourceAPI Caller --> ArtifactAPI - SourceAPI -->|"resolve / Projections (plugin definitions)"| Core - ArtifactAPI <--> Core + SourceAPI -->|"synchronous call"| Sync + ArtifactAPI -->|"reads"| Core + ArtifactAPI -->|"changes, synchronous call"| Sync + Sync -->|"Source or Artifact writes, one commit"| Core + Sync -.-> Own Sup -->|"targets and checkpoints"| Collect Collect -->|"Sources and new checkpoints, one commit"| Core Core -->|"new Sources trigger"| Sup @@ -117,7 +141,7 @@ flowchart LR Generate -.-> Own[("Plugin-owned storage / models")] ``` -The diagram has two independent paths, and one Worker invocation runs only one plugin: +The diagram has three paths, and one call runs only one plugin: - **Collection path**: the Supervisor dispatches a Source plugin's Worker on schedule with the Scope's enabled collection targets and their checkpoints. The plugin reads from the external system and returns Sources; core validates them and @@ -125,29 +149,35 @@ The diagram has two independent paths, and one Worker invocation runs only one p - **Generation path**: when new Sources enter the journal, Artifact plugins that consume them are marked as pending. The Supervisor dispatches a Worker with the Scope's unconsumed Source window and a writer; the plugin writes Artifacts in a write transaction block, and core commits them together with the cursor and the acknowledgment. +- **Synchronous path**: when a caller writes a Source, adds a collection target, or changes a plugin Artifact, Server + checks authorization and versions and then hands the request to the plugin's host process. The plugin performs the + conversion or write there, and the result is committed in one transaction and returned to the caller. Callers deal only with core's existing APIs: | What the caller does | API | How the plugin takes part | | --- | --- | --- | -| Submit a Source | Existing Source write API | Core calls the plugin's `resolve` and Projections to validate and convert it | -| Add, view, or disable a collection target | Existing Source management APIs | Core calls the plugin's `identify_target` to normalize the input | +| Submit a Source | Existing Source write API | The host process runs the plugin's `resolve` and Projections | +| Add, view, or disable a collection target | Existing Source management APIs | On add, the host process runs the plugin's `identify_target` | | Browse, read, or search Artifacts | Existing Artifact read and search APIs | Not involved | -| Edit, tag, merge, forget, restore, undo a merge | Existing Artifact write API, tag APIs, and the generic merge APIs | Not involved; the plugin sees the new state in its next invocation | +| Create, edit, merge, forget, restore, undo a merge | Existing Artifact create and edit APIs, and the generic merge APIs | The host process runs the matching plugin method; an unimplemented one returns 422 | +| Tag, query by tag | Existing tag APIs | Not involved; tags are core data | Responsibilities: | Responsibility | Owner | | --- | --- | -| Authentication, Scope authorization, public HTTP / MCP APIs, storing collection targets and checkpoints, user changes to Artifacts | Core | +| Authentication, Scope authorization, public HTTP / MCP APIs, storing collection targets and checkpoints, version checks and provenance for user changes | Core | | When to call: schedules, trigger coalescing, Worker budgets, timeouts, backoff, leader lease | Core (the existing Supervisor) | -| Reading inputs, providing the constrained write transaction, committing core data / cursor / checkpoints / acknowledgment in one transaction | Core (the plugin executor in the Worker) | +| Starting, reusing, terminating on timeout, and restarting host processes | Core | +| Reading inputs, providing the constrained write transaction, committing core data / cursor / checkpoints / acknowledgment in one transaction | Core (the plugin executor in the Worker or host process) | +| Allocating dedicated plugin storage, wrapping the models Server has configured | Core | | Source definitions and Projection computation, identifying and normalizing collection targets, reading external data | Source plugin | -| Artifact content structure, generation algorithm, plugin-owned indexes and storage and their consistency | Artifact plugin | -| Deploying plugin code, dependencies, environment variables, and plugin-owned storage | Deployer | +| Artifact content structure, generation algorithm, validating user-submitted content, plugin-owned indexes and storage and their consistency | Artifact plugin | +| Deploying plugin code, dependencies, and environment variables | Deployer | -Plugin code does one thing: given an input, produce a result or raise. Raising means failure; core commits no core data -and retries the same input next time. +Plugin code does one thing: given an input, produce a result or raise. Raising means failure and core commits no core +data; a background call retries the same input next time, and a synchronous call returns the error to the caller. ## 2. Plugin shape @@ -173,8 +203,15 @@ The declaration answers what core needs to know before it runs any plugin code: an Artifact family), what it consumes, when to call it, and its resource limits. It is an ordinary Python object validated by Pydantic; there is no YAML or other manifest file. -- **Deployment settings**: values that vary by deployment, such as credentials, external service addresses, and model - names, are read by the plugin from environment variables in `create`. Server neither stores, passes, nor logs them. +- **Deployment settings**: the plugin's own values that vary by deployment, such as credentials and external service + addresses, are read by the plugin from environment variables in `create`. Server neither stores, passes, nor logs them. +- **Resources core provides**: a plugin that needs a database or models does not configure them separately; it uses + objects core wraps for it through the context (see reference 1.2): + - `ctx.storage`: a dedicated area for the plugin on the database backend the deployment uses. On SQLite it is a + separate file in the data directory; on OceanBase and seekdb it is a separate database. The connection the plugin + gets points only to this area, not to core's tables. + - `ctx.models`: the generation and embedding models Server has configured. The plugin can only call them; it never + sees model addresses or credentials. Calls count toward the existing model usage statistics. - **Enabling and disabling**: if the directory exists and passes the check, the plugin is available; to disable it, move it out of the directory and restart Server. - **Check**: at startup, Server imports `plugin.py` in a time-limited child process and validates the declaration without @@ -193,7 +230,12 @@ by Pydantic; there is no YAML or other manifest file. | `collect(request) -> CollectResult` | For collection | Reads external data per target and checkpoint, returning Sources and new checkpoints | A plugin that provides only `definitions` only extends Source types: callers submit content through the existing Source -write APIs, and core uses the definitions to validate, convert, and save it. +write APIs, and core uses the definitions to validate, convert, and save it. `resolve`, Projections, and +`identify_target` all run in the plugin host process; `collect` runs in a Worker. + +Sources, like all built-in Sources, are append-only and cannot be changed or deleted; to correct content, submit a new +Source. Large content such as files and images goes into the plugin's own storage, and the Source records only +metadata such as its location, digest, and media type. ### 3.2 Declaration @@ -280,6 +322,36 @@ pc_source_target_set_enabled(scope_id="team-a", target_id="...", enabled=false) - **Checkpoints**: each target's checkpoint is a read-only input, and the plugin proposes a new checkpoint through its return value. Core commits all returned Sources and every target's new checkpoint in one transaction; if `collect` raises or times out, nothing is committed and the next invocation still receives the old checkpoint. +- **Format validation**: `pc_source_capabilities` publishes the JSON Schema of `target_input`, so callers know the format + before they submit. On add, core first validates with `target_input` and returns 422 naming the field on failure; a + `ValueError` from `identify_target` also returns 422. Whether the external resource exists or is readable can only be + known during `collect`, and the result shows up as the latest failure in `pc_source_status`. + +### 3.5 Edge collection + +Scenarios such as embodied AI collect data on the device and submit it to a central PowerContext Server. The same +program starts in edge mode: + +```text +powercontext source-agent --server https://memory.example.com +``` + +- The edge loads only the Source plugins in the plugin directory that declare `collect`, and runs the Supervisor to + schedule `collect`. It loads no Artifact plugins and serves no Artifact APIs. +- Collection targets are stored in a local SQLite database on the edge and managed through the edge's local + `pc_source_add`, `pc_source_status`, and `pc_source_target_set_enabled`. The central Server does not see edge targets; + it sees only the submitted Sources and checkpoints. +- Results are submitted to the central Server through core's existing remote ingestion APIs: Sources through + `POST /v1/source-observations`, and checkpoints through `POST /v1/connector-checkpoints/commit`. `resolve` and + Projections run on the edge; the central Server validates against the registered definition manifests and does not + need to load plugin code. +- Disconnects, restarts, and resubmission are idempotent: Sources are submitted first, then the new checkpoint is + committed with the old checkpoint as the expected value. If anything fails midway, the checkpoint does not advance and + the next call collects again from the old checkpoint; resubmitted Sources have the same identity and content, and the + central Server deduplicates them. +- The plugin handles any backlog that builds up while offline. When a submission fails, the call fails and the + checkpoint does not advance. For data the external system cannot replay, such as sensor streams, the plugin first + keeps it in its own local storage and provides it again by checkpoint. ## 4. Artifact plugins @@ -287,11 +359,18 @@ pc_source_target_set_enabled(scope_id="team-a", target_id="...", enabled=false) | Interface | Required | Description | | --- | --- | --- | -| `create(ctx) -> ArtifactGenerator` | Yes | A factory run once per invocation in the Worker | +| `create(ctx) -> ArtifactGenerator` | Yes | A factory; run once per invocation in a Worker, and once per Scope and then reused in the host process | | `generate(request, writer) -> None` | Yes | Reads `request.sources` (the unconsumed Source window) and writes Artifacts in the write transaction block | +| `create_artifact(request, writer) -> None` | No | A user create; core allocates `request.artifact_id` | +| `update_artifact(request, writer) -> None` | No | A user edit of the given Artifact; `request.expected_revision` is the revision the caller saw | +| `merge_artifacts(request, writer) -> None` | No | A user merge; core allocates the result ID | +| `forget_artifact(request, writer) -> None` | No | A user forget | +| `restore_artifact(request, writer) -> None` | No | A user restore, restore to an earlier revision, or undo of a merge | -The plugin only generates. Core executes user changes to Artifacts directly (see 4.4), so the plugin implements no command -handling. +The last five methods handle user changes (see 4.4), and each is optional: for an operation the plugin does not +implement, the matching API returns 422, meaning the family does not support it. In these methods the plugin validates +the content the user submitted, updates its own storage, and performs through the writer the one write operation core +requires. ### 4.2 Declaration @@ -306,17 +385,21 @@ handling. | `timeout_seconds` | 300 | Time limit for the whole invocation | | `transaction_timeout_seconds` | 30 | Time limit for the write transaction block, at most `timeout_seconds` | | `max_workers` | 1 | Concurrency limit across Scopes | +| `change_timeout_seconds` | 30 | Time limit for one user-change call, including its write transaction block | | `create` | Required | See the table above | ### 4.3 Example The plugin below generates one report per ticket Source: it first uses its own vector index to find a similar existing report and revises it on a match, otherwise it creates a new one; it also writes the report into its own vector index. +It also supports user edits of reports; create, merge, forget, and restore are not implemented and return 422 for this +family. The vector index lives in the dedicated storage core allocates, and the models are the ones Server has +configured. ```python -import os - -from powercontext.plugins import ArtifactPlugin, ArtifactWriter, GenerateRequest, Inputs, PluginContext +from powercontext.plugins import ( + ArtifactPlugin, ArtifactWriter, GenerateRequest, Inputs, InvalidContent, PluginContext, UpdateRequest, +) from powercontext.sources import TextEvidence @@ -345,10 +428,19 @@ class IncidentReports: await self.index.upsert(request.scope_id, written.ref.artifact_id, written.ref.revision, vector) # The block ended normally: core commits the Artifact writes, the cursor advance, and the acknowledgment + async def update_artifact(self, request: UpdateRequest, writer: ArtifactWriter) -> None: + # A user edit: validate the content, recompute the vector, then write to core and the owned index + report = Report.model_validate(request.content) # raise InvalidContent on bad input; the API returns 422 + vector = await self.model.embed(report.summary) + async with writer.transaction() as tx: + written = await tx.put(request.artifact_id, content=report.model_dump(), + expected_revision=request.expected_revision) + await self.index.upsert(request.scope_id, written.ref.artifact_id, written.ref.revision, vector) + async def create(ctx: PluginContext) -> IncidentReports: - index = await ctx.enter_async_context(VectorIndex.connect(os.environ["REPORT_INDEX_URL"])) - return IncidentReports(index, Summarizer()) + index = await ctx.enter_async_context(VectorIndex.connect(ctx.storage.engine())) + return IncidentReports(index, Summarizer(ctx.models)) plugin = ArtifactPlugin( @@ -366,28 +458,31 @@ left by the previous attempt, and reads from the owned index defer to core's cur ### 4.4 How callers use it -Callers access plugin families through the existing Artifact APIs. Read APIs read `pc_artifacts` directly, and write APIs -are executed by core with the same core implementation plugin writes use; neither runs plugin code: +Callers access plugin families through the existing Artifact APIs. Read APIs read `pc_artifacts` directly and run no +plugin code; for change APIs, core checks authorization and versions and then synchronously calls the matching plugin +method: -| Purpose | API | -| --- | --- | -| Browse | `GET /v1/scopes/{scope_id}/artifacts/{family}` | -| Read the current content, a given revision, revision history, tags | `GET .../{artifact_id}`, `GET .../{artifact_id}/revisions/{revision}`, `GET .../{artifact_id}/revisions`, `GET .../{artifact_id}/tags` | -| Text search | `POST /v1/scopes/{scope_id}/artifacts/{family}/search` | -| Edit (submit complete content; `If-Match` checks the current revision) | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | -| Change tags | `PUT .../{artifact_id}/tags`; query by tag with `POST /v1/scopes/{scope_id}/artifact-tags/query` | -| Read the governance state (the `state_version` needed to forget or merge) | `GET .../{artifact_id}/governance` | -| Merge | `POST /v1/scopes/{scope_id}/artifacts/{family}/merges` | -| Forget | `POST .../{artifact_id}/forget` | -| Restore, restore an earlier revision, undo a merge | `POST .../{artifact_id}/restorations` | -| Add knowledge | Write a Source; the plugin processes it in its next invocation | - -Changes return their result synchronously. The plugin reads the changed state through its writer in its next invocation. +| Purpose | API | Plugin method | +| --- | --- | --- | +| Browse | `GET /v1/scopes/{scope_id}/artifacts/{family}` | None | +| Read the current content, a given revision, revision history, tags | `GET .../{artifact_id}`, `GET .../{artifact_id}/revisions/{revision}`, `GET .../{artifact_id}/revisions`, `GET .../{artifact_id}/tags` | None | +| Text search | `POST /v1/scopes/{scope_id}/artifacts/{family}/search` | None | +| Create (submit only content; core allocates the ID) | `POST /v1/scopes/{scope_id}/artifacts` with body `{"family": ..., "content": {...}}` | `create_artifact` | +| Edit (submit complete content; `If-Match` checks the current revision) | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | `update_artifact` | +| Read the governance state (the `state_version` needed to forget or merge) | `GET .../{artifact_id}/governance` | None | +| Merge | `POST /v1/scopes/{scope_id}/artifacts/{family}/merges` | `merge_artifacts` | +| Forget | `POST .../{artifact_id}/forget` | `forget_artifact` | +| Restore, restore an earlier revision, undo a merge | `POST .../{artifact_id}/restorations` | `restore_artifact` | +| Change tags | `PUT .../{artifact_id}/tags`; query by tag with `POST /v1/scopes/{scope_id}/artifact-tags/query` | None; handled by core | + +Changes return their result synchronously. A plugin rejecting the content returns 422, a plugin that does not implement +the method also returns 422, and a host process that times out or is unavailable returns 503. ## 5. Scheduling and triggers -Scheduling and triggering are delegated to the existing Artifact processing Supervisor. At startup each plugin registers -as one Supervisor binding named `plugin:`, and the scheduling key is `(binding, Scope)`. +Scheduling and triggering of plugins' background calls (`collect`, `generate`) are delegated to the existing Artifact +processing Supervisor; synchronous calls are triggered by requests and do not go through the Supervisor. At startup each +plugin registers as one Supervisor binding named `plugin:`, and the scheduling key is `(binding, Scope)`. Artifact plugins: @@ -423,8 +518,10 @@ invocation after it finishes. - **Changes**: restart Server after changing code. Server does not watch files or reload plugins at runtime. Collection targets are business data, so adding or removing them needs no restart. - **Disabling**: move the plugin out of the directory and restart. Saved Sources, Artifacts, collection targets, cursors, - and checkpoints remain readable; redeploying a plugin with the same name continues where it left off. The deployer keeps - or cleans up plugin-owned storage. + and checkpoints remain readable; redeploying a plugin with the same name continues where it left off. The plugin's + dedicated storage is not deleted with it; the deployer keeps or cleans it up. +- **Host processes**: core starts, reuses, and restarts them on demand; deployers neither configure nor manage them. + Host process starts, terminations, and restarts appear only in logs and metrics. # Reference-level explanation @@ -472,15 +569,41 @@ class PluginContext: scope_id: str # Scope of this invocation plugin_dir: Path # plugin directory (read-only by convention) logger: logging.Logger # logger bound to the plugin name + storage: PluginStorage # the plugin's dedicated storage + models: PluginModels | None # models Server has configured; None when none are configured async def enter_async_context(self, cm: AbstractAsyncContextManager[T]) -> T: ... + +class PluginStorage(Protocol): + backend: Literal["sqlite", "oceanbase", "seekdb"] + def engine(self) -> AsyncEngine: ... # points only to this plugin's dedicated area + +class PluginModels(Protocol): + embedding_profile: EmbeddingProfile | None # profile ID and vector dimension; None without an embedding model + async def embed(self, texts: Sequence[str], /) -> Sequence[Sequence[float]]: ... + async def generate(self, messages: Sequence[ChatMessage], /, *, max_tokens: int | None = None) -> str: ... ``` -- The factory runs once per Worker invocation, and instances are not reused across invocations. Resources that need - closing (including connections to plugin-owned storage) are registered through `ctx.enter_async_context` and released - by the executor when the invocation ends. +- The factory runs once per Worker invocation, and instances are not reused across invocations; in the host process it + runs once per Scope, and the instance is reused for as long as the host process lives. Resources that need closing + (including connections to plugin-owned storage) are registered through `ctx.enter_async_context` and released by the + executor when the Worker invocation ends or the host process stops. +- **Dedicated storage**: core uses Server's existing database connection settings to allocate an area for the plugin on + the same backend, with no additional database login: + - SQLite: `plugins/.sqlite3` in the data directory. + - OceanBase / seekdb: a database named `powercontext_plugin_` (with `-` replaced by `_`), created by core on + first use. When Server's database login cannot create databases, the deployer creates it in advance. + - The plugin manages its own tables and migrations; core does not read, migrate, or back up this area. It is not part + of core's transactions, and its consistency with core data follows 3.5. + - The plugin and Server use the same database login, so this split prevents a plugin from writing core's tables by + mistake but does not stop a malicious plugin; the plugin directory is treated as trusted code. +- **Models**: `models` wraps Server's inference configuration; the plugin never sees model addresses, headers, or + credentials. `embedding_profile` lets the plugin use the same embedding configuration for indexing and queries. Calls + follow Server's timeout settings and count toward the existing model usage statistics. A plugin may also read its own + model configuration from environment variables and create its own clients. - Plugins get no connection to core's database and no interface for writing cursors or checkpoints. Plugins write core - data through exactly two paths: a Source plugin's `collect` return value and an Artifact plugin's `ArtifactWriter`. + data through exactly three paths: a Source plugin's `collect` return value, the return value of `resolve` in the host + process, and an Artifact plugin's `ArtifactWriter`. ## 2. Source plugin contract @@ -521,8 +644,9 @@ class SourceCollector(Protocol): definition registry at startup; a conflict with a built-in type or another plugin fails the check. - **Writes**: callers submit content through the existing Source write APIs (`POST /v1/scopes/{scope_id}/sources`, or the write usage of MCP `pc_source_add`). Core validates the input with `definition.input_class`, calls `resolve` to obtain - the Source, computes each Projection, and saves it, with the same semantics as built-in Source writes. This step runs - synchronously in the Server process with a timeout. + the Source, computes each Projection, and saves it, with the same semantics as built-in Source writes. Validation, + `resolve`, and Projections run synchronously in the plugin's host process with a timeout. `resolve` must be a pure + function with no side effects such as copying files; to bring external files into plugin storage, use `collect`. - **collect**: each invocation covers one Scope and carries at most `max_targets` enabled targets; when there are more, targets rotate by key, and targets that returned `has_more` go first in the next invocation. The executor applies the same validation, `resolve`, and Projections as writes to every returned Source, then in one transaction saves all @@ -545,9 +669,26 @@ the existing Source management APIs: `scopes`. - `identify_target` must be a pure function: it only validates and normalizes, without contacting external systems or checking external authorization. Whether an external resource exists or is readable is decided by the plugin during - collect and expressed as an exception or an empty result. Like writes, it runs synchronously in the Server process with - a timeout. -- `pc_source_capabilities` lists the Source types and target inputs plugins provide. + collect and expressed as an exception or an empty result. Like writes, it runs synchronously in the host process with a + timeout. +- `pc_source_capabilities` lists the Source types plugins provide and the JSON Schema of their target inputs. + +**Edge mode** (`powercontext source-agent`) keeps the same Source plugin contract and only replaces where results are +committed: + +| Item | Handling in edge mode | +| --- | --- | +| Configuration | `--server` names the central Server; `POWERCONTEXT_SOURCE_AGENT_ID` is the edge's stable identifier; `POWERCONTEXT_SOURCE_AGENT_TOKEN` is a credential with `scope.contribute` on the target Scopes | +| Local data | A SQLite database in the edge's data directory holds collection targets (with the structure of `pc_source_collect_targets`) and Supervisor scheduling state; it holds no Sources | +| Definition manifests | At startup, the declarative manifest of each Source definition is registered with the central Server (`POST /v1/source-definitions/register`, which requires `server.admin`); if the credential lacks that permission, the deployer registers them in advance | +| Each `collect` | Reads each target's checkpoint from the central Server (`/v1/connector-checkpoints/get`), calls the plugin, runs `resolve` and Projections on the edge, submits the Sources one by one (`/v1/source-observations`), and after all succeed commits the new checkpoint with the old one as the expected value (`/v1/connector-checkpoints/commit`) | +| Target binding ID | Derived deterministically from the edge identifier, the plugin name, and the target key, so two edges collecting the same external resource do not overwrite each other's checkpoints | +| Failure | Any failed step fails the call, which enters the Supervisor's backoff; already submitted Sources are kept, the checkpoint does not advance, and the central Server deduplicates the next collection | + +The edge and the central Server do not share a transaction: a Source can be saved while the checkpoint has not advanced, +and Source identity deduplication prevents duplicates. This requires the plugin's `resolve` to give the same identity and +content for the same external state; when the identity matches but the content differs, the central Server returns a +conflict and the call fails. ## 3. Artifact plugin contract @@ -622,9 +763,11 @@ validation and raises immediately when a rule is violated: Write constraints: -- Writes apply only to Artifacts of the declared `family` in this Scope; plugins cannot name another family or Scope. -- `source_refs` may only reference Sources in this invocation's window; `artifact_refs`, `to_revision`, and the target of - `bind_handle` must be revisions that exist in this family. +- Writes apply only to Artifacts of the declared `family` in this Scope; plugins cannot write another family or Scope. +- In `generate`, `source_refs` may only reference Sources in this invocation's window; in user changes they may only + reference Sources already in the target Artifact's current lineage (to keep the original inputs). `artifact_refs` may + reference existing revisions of any family in this Scope; `to_revision` and the target of `bind_handle` must be + revisions that exist in this family. - Core extracts search text from the content according to the declared `search_fields`; plugins do not supply it separately. The same content yields the same search text whether a plugin or a user wrote it. - Each write transaction block has limits on operations (1000), single content size (256 KiB), and total bytes (8 MiB); @@ -635,7 +778,10 @@ Write constraints: ### 3.4 Reads `ArtifactWriter` (outside the transaction, committed data only) and `ArtifactTransaction` (inside the transaction, also -seeing what the block has written) offer the same read methods, scoped to this Scope and this family: +seeing what the block has written) offer the same read methods, scoped by default to this Scope and this family. +`reads(family)` returns a read-only view of another family in the same Scope with the same methods; it can read built-in +families and other plugins' families (content is returned as JSON, as is) but cannot write. This lets one plugin work +from another plugin's results, for example a knowledge base plugin reading the Artifacts a document plugin writes: ```python class ArtifactSnapshot(BaseModel): @@ -657,6 +803,7 @@ class ArtifactReads(Protocol): async def merge_inputs(self, artifact_id: str) -> Sequence[ArtifactRef]: ... async def tags(self, artifact_id: str) -> Sequence[str]: ... async def resolve_handle(self, handle_key: str) -> ArtifactRef | None: ... + def reads(self, family: str) -> ArtifactReads: ... # read-only view of another family in the same Scope ``` - A snapshot carries the exact revision, lineage, and current governance state together, so plugins need not read lineage @@ -665,7 +812,7 @@ class ArtifactReads(Protocol): inputs of a merge result. - `list` filters by lifecycle state and tags, including deprecated, retired, and merge-frozen Artifacts, for finding data that already existed. -- `search` is core's text search (see section 6) and covers only active current revisions; inside the write transaction it +- `search` is core's text search (see section 7) and covers only active current revisions; inside the write transaction it also finds content the block has written but not yet committed. - `by_source` finds the Artifacts whose lineage references a given Source, for locating Artifacts to revise after a Source changes. @@ -682,65 +829,132 @@ class ArtifactReads(Protocol): that were already processed. Plugins derive Artifact IDs from business identity with `writer.stable_id(...)`; `put` is a no-op for identical content, `set_lifecycle` is a no-op when the target state equals the current one, and `merge` is a no-op when every input is already merged into the same result. No duplicate Artifacts or meaningless revisions result. -- **Plugin-owned storage**: the plugin's indexes and other storage are different databases outside core's transaction. +- **Plugin-owned storage**: the plugin's indexes and other storage (including the dedicated storage core allocates) are + outside core's transaction. Core does not guarantee their consistency with core data; the plugin does. Recommended practice: 1. Write owned storage inside the write transaction block after receiving the `ArtifactSnapshot`, upserting by `artifact_id`. A failed write to owned storage then rolls back the core writes too. 2. If the write transaction fails to commit, owned storage may keep entries from that attempt. The retry processes the same window, derives the same stable IDs, and overwrites them. 3. Treat the owned index as a source of candidates: read each hit's current snapshot from core with `get`, and use that - snapshot's content and state. Entries left by rollbacks and user changes then cannot make the plugin read stale - content. Plugins may also rebuild their index from core data periodically. + snapshot's content and state. Entries left by rollbacks then cannot make the plugin read stale content. Plugins may + also rebuild their index from core data periodically. + 4. Store large content (bodies, chunks, file bytes) as immutable records keyed by `(artifact_id, revision)`: write them + to owned storage first, then commit Artifact content that carries their digest in the write transaction block. + Records that were never committed are referenced by no revision, and reads verify them against the digest in the + Artifact content. - `generate` itself may run several times because of retries; model calls and external requests a plugin makes are not exactly-once. ## 4. User changes to plugin Artifacts -Users change plugin Artifacts through core's generic Artifact APIs. Server executes each change synchronously in a short -transaction, without calling the plugin or going through the Supervisor. +Users change plugin Artifacts through core's generic Artifact APIs. After authorization and a version precheck, Server +hands the request to the plugin's host process; the plugin method runs there and writes through the writer, and the +result is committed in a short transaction and returned synchronously. User changes neither go through the Supervisor +nor wait in a queue. -**Edits and tags** +### 4.1 Requests and methods -| API | Rules | -| --- | --- | -| `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | The body is `{"content": {...}}`, stored as is, with search text taken from the declared `search_fields`; `If-Match` that differs from the current revision returns 412; the Artifact must exist. As with built-in edits, the submitted content is also stored as an internal Content Source (`lineage_only`) recorded in lineage as the new revision's direct source; such Sources never enter a plugin's input window | -| `GET` / `PUT .../{artifact_id}/tags`, `POST /v1/scopes/{scope_id}/artifact-tags/query` | The same as for built-in families; tags create no content revision | +```python +class CreateRequest(BaseModel): + scope_id: str + artifact_id: str # allocated by core; the caller does not submit it + content: dict[str, JsonValue] + +class UpdateRequest(BaseModel): + scope_id: str + artifact_id: str + expected_revision: int + content: dict[str, JsonValue] + +class MergeRequest(BaseModel): + scope_id: str + inputs: Sequence[MergeInput] # each is artifact_id, revision, state_version + into: str # result ID allocated by core + content: dict[str, JsonValue] + +class ForgetRequest(BaseModel): + scope_id: str + artifact_id: str + expected_revision: int + expected_state_version: int + +class RestoreRequest(BaseModel): + scope_id: str + artifact_id: str + operation: Literal["restore", "undo_merge"] + revision: int | None = None # for restore, the revision to restore to -**Merge, forget, and restoration** +class InvalidContent(Exception): # the plugin rejects the content; the API returns 422 with the message and field + def __init__(self, message: str, *, field: str | None = None) -> None: ... +``` -These come from the shared merge engine (`ArtifactMergeService`, [RFC #1859](1859_artifact_merge.md)), which was designed -to be family-agnostic: each family supplies an adapter that builds merged content and maintains its own index, and the -engine does version checks, cascading restoration, lineage records, and permission checks. The runtime finds a family's -engine instance through a registry (`ArtifactMergeApplication`). Experience and Atomic Memory are the registered built-in -families today; when Server loads a plugin, it creates an engine instance for the plugin family (the plugin adapter plus -the shared Artifact access policy) and registers it. +| Method | Write operation the plugin must perform | +| --- | --- | +| `create_artifact(request, writer)` | Exactly one `put(request.artifact_id, ...)` on an Artifact that did not exist | +| `update_artifact(request, writer)` | Exactly one `put(request.artifact_id, ..., expected_revision=request.expected_revision)` | +| `merge_artifacts(request, writer)` | Exactly one `merge(input IDs, into=request.into, ...)` with the inputs from the request | +| `forget_artifact(request, writer)` | Exactly one `set_lifecycle(request.artifact_id, "deprecated")` | +| `restore_artifact(request, writer)` | For `restore`, exactly one `revert` or `set_lifecycle(..., "active")`; for `undo_merge`, exactly one `undo_merge(request.artifact_id)` | + +- The write transaction in these calls allows only the matching write operation from the table, plus `set_tags` on the + same Artifact; writing any other Artifact raises immediately. If the method returns normally without performing the + required write, core rolls back and returns 502, meaning the plugin did not complete the contract. +- The plugin may normalize the content (for example, by filling default fields); the written content is what the plugin + submits. +- The plugin should update its own storage inside the write transaction block after receiving the snapshot, as in 3.5. + +### 4.2 What core does + +- **Authorization**: the same as for built-in families. Creating an Artifact or a merge result needs the Scope contribute + permission, and the caller owns the result; changing an existing Artifact needs write permission on it + (`artifact.write`). In these calls the plugin gets only the write scope the request needs. +- **Version checks**: before calling the plugin, core checks `If-Match`, `revision`, and `state_version`, and returns 412 + or 409 without calling the plugin when they differ. At commit it checks again inside the write transaction; if another + write changed the state in the meantime, it rolls back and returns 409. +- **Provenance**: as for built-in families, on create, edit, restore, and undo merge core stores the submitted content as + an internal Content Source in the same transaction (`lineage_only`, with operation `artifact_create`, + `artifact_replace`, `artifact_restore`, or `artifact_undo_merge`) and records it in lineage as the new revision's + direct source. Core writes these Sources, and they never enter a plugin's input window. +- **Merge registry**: the generic merge, forget, and restoration APIs sit on the registry of the shared merge engine + (`ArtifactMergeApplication`, [RFC #1859](1859_artifact_merge.md)) and work for every registered family. Experience and + Atomic Memory are the registered built-in families and use their own adapters; a plugin family is registered at load + and handed to the matching plugin method. Unregistered families return 422. +- **No preview**: Atomic Memory's dedicated APIs can preview a restoration's effect and then execute it with a signed + token. The generic APIs offer no preview and therefore need no signing key; restoration and undo merge act on the state + at execution time, while merge and forget are checked by `revision` and `state_version`. +- **Tags**: `GET` / `PUT .../{artifact_id}/tags` and `POST /v1/scopes/{scope_id}/artifact-tags/query` behave the same for + plugin and built-in families. Core handles them directly, without calling the plugin and without creating a content + revision. -A set of family-agnostic APIs sits on the registry and works for every registered family; unregistered families return -422: +### 4.3 APIs | API | Rules | | --- | --- | -| `GET .../{artifact_id}/governance` | Returns the current revision, lifecycle state, `state_version`, and merge target | -| `POST /v1/scopes/{scope_id}/artifacts/{family}/merges` | Takes at least two current Artifacts as `artifact_id`, `revision`, and `state_version`, plus the result's complete content; creates the result Artifact and freezes every input. Any changed input returns 409 | +| `POST /v1/scopes/{scope_id}/artifacts` | The body gains a plugin branch `{"family": "", "content": {...}}`, where family must not be a built-in family name; core allocates the ID; the response matches built-in creation, including `Location` and `ETag` | +| `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | The body is `{"content": {...}}`; `If-Match` that differs from the current revision returns 412; the Artifact must exist | +| `GET .../{artifact_id}/governance` | Returns the current revision, lifecycle state, `state_version`, and merge target; handled by core | +| `POST /v1/scopes/{scope_id}/artifacts/{family}/merges` | Takes at least two current Artifacts as `artifact_id`, `revision`, and `state_version`, plus the result's complete content; creates the result Artifact and freezes every input | | `POST .../{artifact_id}/forget` | Takes `revision` and `state_version`; marks the Artifact deprecated with no replacement, appends no revision, and removes it from search | | `POST .../{artifact_id}/restorations` | `operation` is `restore` (default; `revision` restores earlier content) or `undo_merge`; restores and retires as one group | -- **Permissions**: checked by the engine, the same as for built-in families. Changing an existing Artifact needs write - permission on it (`artifact.write`); creating a merge result needs the Scope contribute permission, and the caller owns - the result. The writer in plugin invocations uses the same engine class and adapter with different permissions: a - plugin has full write access to its family's Artifacts, including results of user merges. -- **No preview**: Atomic Memory's dedicated APIs can preview a restoration's effect and then execute it with a signed - token. The generic APIs offer no preview and therefore need no signing key; restoration and undo merge act on the state - at execution time, while merge and forget are checked by `revision` and `state_version`. -- **Merge content**: merge content for a plugin family is opaque JSON; core checks only that it is a JSON object and takes - search text from `search_fields`. Built-in families validate it against their own content models. -- **Concurrency**: user changes and plugin invocations can overlap. A plugin that reads outside the transaction and writes - with `expected_revision` inside it detects the conflict and rolls back; its next invocation decides again from the - changed state, so a user change is never silently overwritten by the plugin. -- **No notification**: as for built-in families, user changes are recorded only in lineage and governance state and do not - notify the plugin. The plugin reads the new state through its writer in its next invocation. -- **Creation**: the public APIs do not create plugin Artifacts directly, except merge results. A caller who wants to add - knowledge writes a Source. +| Situation | Response | +| --- | --- | +| The plugin does not implement the method | 422, saying the family does not support the operation | +| The plugin raises `InvalidContent` | 422, with the plugin's message and field | +| A version mismatch | 412 (`If-Match`) or 409 (`revision`, `state_version`) | +| The plugin raises any other exception | 500; only the exception type is logged, and its message is not returned | +| The plugin did not perform the required write | 502 | +| `change_timeout_seconds` is exceeded, or the host process is unavailable or in restart backoff | 503 | + +- The plugin branch of the create request has no discriminator mapping: OpenAPI cannot map arbitrary plugin families, so + generated clients type this branch loosely. This matches the tag APIs, whose family became a string. +- A need that only means removal (for example, taking down a document) uses forget; core provides no physical delete. In + `forget_artifact` the plugin can clean the body and indexes out of its own storage, while the revisions and Sources in + core are kept. +- **Concurrency**: user changes and background `generate` calls can overlap. A `generate` that reads outside the + transaction and writes with `expected_revision` inside it detects the conflict and rolls back; its next invocation + decides again from the changed state, so a user change is never silently overwritten by the plugin. ## 5. Plugin executor and consumption cursor @@ -779,7 +993,42 @@ commit is the Sources and every target's new checkpoint. Collect needs no write the return value at once. Adding a collection target requests one collection through the existing invocation intent (`ArtifactProcessingIntentRepository.request`). -## 6. Public reads and search +## 6. Plugin host process + +Synchronous calls must return within the request's time limit, so they cannot start a Worker each time: a Worker starts a +fresh Python process with `spawn`, and on a 2-core development machine importing core's execution modules alone takes +about 5.5 seconds and 150 MB of memory. Synchronous calls therefore go to a long-lived host process: + +| Call | Where it runs | +| --- | --- | +| Input validation, `resolve`, and Projections for Source writes | Host process | +| `identify_target` when adding a collection target | Host process | +| `create_artifact`, `update_artifact`, `merge_artifacts`, `forget_artifact`, `restore_artifact` | Host process | +| `collect`, `generate` | Supervisor Worker | + +- **Count and startup**: each Server process keeps at most one host process for each plugin that needs synchronous calls, + starts it on the first synchronous call, and reuses it afterward. An Artifact plugin that provides only `generate` + starts no host process. In edge mode, `resolve` and Projections run inside the collection Worker, and no host process + starts. +- **Execution**: the host process imports `plugin.py` once at startup and compares the declaration. As in a Worker, core + code opens the database connection, and the writer runs inside the host process. One host process can serve several + requests concurrently, up to a limit set in the Server configuration. +- **Timeouts and failures**: each call has its own time limit (Source writes and `identify_target` use Server's + synchronous call limit; user changes use `change_timeout_seconds`). On a timeout, crash, or loss of response, core + terminates the process, in-flight requests return 503, and the process restarts after a backoff. During the backoff + the plugin's synchronous calls return 503; background calls are unaffected. +- **Code changes**: a host process does not re-import the plugin while it lives. Restart Server after changing a plugin, + the same rule as for Workers. +- **Isolation**: as with Workers, the host process isolates ordinary failures and is not a sandbox for untrusted code. + +**In-memory record of active plugins**: each Server process records in memory the plugins it has loaded: name, +directory, kind, declaration digest, check result, and the current host process state (not started, running, or in +restart backoff). Routing user requests, starting host processes, and dispatching Workers all use this record. It is not +persisted or synchronized across processes, and is rebuilt from the plugin directory when Server restarts. The +operational APIs return only the check results and declarations, which are the same on every process; host process state +is visible through logs and metrics. + +## 7. Public reads and search Core serves all public reads of plugin families without running plugin code: @@ -800,35 +1049,43 @@ Core serves all public reads of plugin families without running plugin code: - Review, Recall, and Prepared Context are not open to plugin families in the first version; those APIs return an explicit "not supported" error, and `GET /v1/capabilities` lists plugin families. -## 7. Loading and lifecycle +## 8. Loading and lifecycle ```text Server starts -> scan the direct subdirectories of POWERCONTEXT_SERVER_PLUGINS_DIR (no plugins load if it is unset) -> for each directory: import plugin.py in a time-limited check child process and validate the declaration, without calling create - -> Source plugins: import definitions, target_input, and identify_target in the Server process and register Source types - -> register the family for reads and search; register a Supervisor binding when there is a schedule or trigger + -> the check child process returns a serializable declaration: Source types and the JSON Schema of their inputs, + the JSON Schema of target inputs, the family, and which user-change methods are implemented + -> Source plugins: register Source types; plugin code is not imported into the Server process + -> Artifact plugins: register the family for reads and search and in the merge registry; register a Supervisor + binding when there is a schedule or trigger + -> add the plugin to this process's record of active plugins -> any step fails: record the error, mark the plugin failed, and continue with the next one Server stops + -> stop accepting synchronous calls, wait for in-flight calls to finish or time out, then terminate host processes -> stop dispatching and reclaim Workers through the existing Supervisor shutdown ``` - A plugin has only two outcomes, `loaded` and `failed`, kept only in process memory. Server persists no plugin enabled - state, registration generation, or configuration snapshot; each startup re-checks the directory as it is. + state, registration generation, or configuration snapshot; each startup re-checks the directory as it is. Plugin code + is imported only in the check child process, host processes, and Workers, never in the Server process. - With no hot reload, no registration generation is needed to fence old Workers; the existing Supervisor lease fence is enough to fence stale commits after a leadership change. - A Worker re-imports plugin code and compares the declaration on every invocation. If the plugin directory is modified while Server runs and the declaration changes, the call fails and asks for a restart; if only implementation code changes, newly dispatched Workers use the new code. Deployers should restart Server after a change. - Server instances in the read-only API role also check plugins and register families and Source types for reads, search, - and user changes, but do not dispatch Workers. + Source writes, and user changes; they start host processes on demand but do not dispatch Workers. +- Adding or removing plugins at runtime (hot reload) and plugin-defined APIs are left to a separate proposal built on + host processes. -## 8. Execution constraints +## 9. Execution constraints -- **Isolation**: `collect` / `generate` run in Worker child processes started by the Supervisor, so crashes, memory leaks, - and blocking do not affect Server. Child processes isolate ordinary failures; they are not a sandbox for untrusted code: - a plugin runs with the Worker's operating-system privileges and can access its files, network, and environment - variables. +- **Isolation**: `collect` / `generate` run in Worker child processes started by the Supervisor and synchronous calls run in + host processes, so crashes, memory leaks, and blocking do not affect Server. Child processes isolate ordinary failures; + they are not a sandbox for untrusted code: a plugin runs with the child process's operating-system privileges and can + access its files, network, and environment variables. - **Timeouts and budgets**: `timeout_seconds` is the total Worker limit, `transaction_timeout_seconds` is the write transaction block limit, and `max_workers` is the plugin's concurrency limit across Scopes; each maps to an existing Supervisor binding parameter or is enforced by the executor. @@ -844,10 +1101,11 @@ Server stops failures; persistent failures require fixing the plugin or the data and restarting. A target that keeps failing in collect blocks the other targets in the same Scope; disable it first. - **Multi-instance deployments**: with several Server instances on OceanBase, dispatch happens only on the instance holding - the Supervisor leader lease; Source writes, target management, reads, and user changes can run on any instance. Plugin-owned - storage must be reachable from the instance that runs Workers. A SQLite deployment has a single Server process. + the Supervisor leader lease; Source writes, target management, reads, and user changes can run on any instance, each using + its own host processes. Plugin-owned storage must be reachable from every instance; the dedicated storage core allocates + lives on the shared database backend and meets this naturally. A SQLite deployment has a single Server process. -## 9. Persistence +## 10. Persistence | Data | Handling | | --- | --- | @@ -863,22 +1121,26 @@ Server stops | Artifacts, heads, and lineage written by old Artifact plugins | Deleted, not migrated | | `pc_source_write_plugins` | Removed; Source definitions come from the plugin directory at startup | | The Source plugin management service's SQLite | Removed with the management service | -| Plugin-owned indexes and storage | Not part of core; managed by the plugin and the deployer | +| Dedicated plugin storage (`plugins/.sqlite3` or the `powercontext_plugin_` database) | Not part of core's schema; core only allocates it, and the plugin and the deployer own its tables, migrations, and cleanup | +| Other plugin-owned indexes and storage | Not part of core; managed by the plugin and the deployer | +| The local SQLite of edge mode | Holds only the edge's collection targets and scheduling state, no Sources; Sources and checkpoints live on the central Server | A target's binding ID is derived deterministically from the plugin name and the target key, using a digest when it exceeds the column width. Additions and removals ship as one new database migration whose notes list the tables, columns, and data ranges added and removed. -## 10. API changes +## 11. API changes | Change | Content | | --- | --- | | Removed | `/v1/admin/artifact-plugins` and its subpaths; the Source plugin management service, plugin instance Runtimes, and poll / reply channels; the `source load` command; MCP `pc_source_instances` and `pc_source_instance_set_enabled`; the `plugin.yml` and `powercontext.yml` manifests | | Removed | Custom family declarative indexes (`ArtifactSearchDeclaration`, `TextIndex`, and so on), `ArtifactFamily` schema declarations, `ArtifactDraft`, and the old Worker plugin executor | -| Added | `powercontext.plugins`: `SourcePlugin`, `ArtifactPlugin`, `Inputs`, `Interval`, `Cron`, `PluginContext`, and the writer and read interfaces | +| Added | `powercontext.plugins`: `SourcePlugin`, `ArtifactPlugin`, `Inputs`, `Interval`, `Cron`, `PluginContext`, `PluginStorage`, `PluginModels`, the writer and read interfaces, and the user-change request models and `InvalidContent` | +| Added | The `powercontext source-agent` edge collection mode, with `POWERCONTEXT_SOURCE_AGENT_ID` and `POWERCONTEXT_SOURCE_AGENT_TOKEN` | +| Changed | The body of `POST /v1/scopes/{scope_id}/artifacts` gains a plugin branch `{"family", "content"}`; core allocates the ID, and the plugin's `create_artifact` handles it | | Added | `GET /v1/plugins`, `GET /v1/plugins/{name}`, `GET /v1/plugins/{name}/scopes` (`server.admin`, read-only) | -| Added | `GET .../{artifact_id}/governance`, `POST /v1/scopes/{scope_id}/artifacts/{family}/merges`, `POST .../{artifact_id}/forget`, `POST .../{artifact_id}/restorations`: generic APIs on the shared merge engine registry that work for every registered family (Experience and Atomic Memory among built-ins, plus loaded plugin families). Atomic Memory's dedicated APIs are unchanged | -| Changed | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` accepts plugin families with a `{"content": {...}}` body | +| Added | `GET .../{artifact_id}/governance`, `POST /v1/scopes/{scope_id}/artifacts/{family}/merges`, `POST .../{artifact_id}/forget`, `POST .../{artifact_id}/restorations`: generic APIs on the shared merge engine registry that work for every registered family (Experience and Atomic Memory among built-ins, plus loaded plugin families, which the matching plugin methods handle). Atomic Memory's dedicated APIs are unchanged | +| Changed | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` accepts plugin families with a `{"content": {...}}` body, handled by the plugin's `update_artifact` | | Changed | The tag APIs accept plugin families: `TaggableArtifactFamily` changes from an enum to a string in the `ArtifactReadFamily` format, and Server accepts built-in families or loaded plugin families. The wire format is unchanged; generated clients lose the enum type | | Changed | The generic Artifact read and search APIs support plugin families; `GET /v1/capabilities` lists plugin families | | Changed | The write usage of `pc_source_add` uses plugin definitions, and its collect usage adds collection targets; `pc_source_capabilities` lists plugin Source types and target inputs; `pc_source_status` reports targets and their latest collection result; `pc_source_target_set_enabled` acts on targets stored by core | @@ -887,7 +1149,7 @@ ranges added and removed. After `openapi/powercontext.yaml` is updated per the table, `src/powercontext/http/_generated/` and each integration's operation table are regenerated. -## 11. Observability +## 12. Observability Plugin calls reuse the Supervisor's and Worker's existing operation vocabulary, execution identities, and outcome fields, and no request ID is generated for scheduled calls. New fields: @@ -895,13 +1157,16 @@ and no request ID is generated for scheduled calls. New fields: | Field | Meaning | | --- | --- | | `plugin_name` | Plugin name | -| `plugin_method` | `collect` or `generate` | -| `trigger` | `schedule`, `sources`, `target_added` | +| `plugin_method` | `collect`, `generate`, `resolve`, `identify_target`, `create_artifact`, `update_artifact`, `merge_artifacts`, `forget_artifact`, `restore_artifact` | +| `plugin_process` | `worker` or `host` | +| `trigger` | `schedule`, `sources`, `target_added`; `request` for synchronous calls | | `operation_count` | Write operations committed (Artifact) or Sources committed (collect) in this invocation | | `target_count` | Targets carried by this collect invocation | | `transaction_ms` | Time from the start of the write transaction block to commit or rollback, in milliseconds | -Metric labels may use `plugin_name` and `plugin_method`, never Scope IDs or target keys. Logs go through `log_safely` and +Host process starts, terminations, and restarts each record one event with `plugin_name` and the termination reason +(timeout, crash, Server stop). Metric labels may use `plugin_name`, `plugin_method`, and `plugin_process`, never Scope IDs +or target keys. Logs go through `log_safely` and exclude Source content, Projections, Artifact content, target inputs, and environment variable values; plugin exceptions are logged only by type and known error category, never by message. New fields are registered in the allowlists per the developer guide and documented in both locales' observability docs. Call outcomes go only to logs and metrics and are not @@ -909,17 +1174,23 @@ persisted, so the operational APIs report no "last call result". # Drawbacks -- **Some plugin code runs in the Server process**: Source writes must return synchronously, so a Source definition's - `resolve`, Projections, and `identify_target` bypass the Worker. They should be pure data transformations, but blocking - or exceptions still affect request handling. Server bounds these calls with timeouts and turns exceptions into request - errors. +- **Host processes hold resources**: each Server process gets one more long-lived process for each plugin that needs + synchronous calls, and the count grows with instances in multi-instance deployments. Synchronous calls add one + inter-process round trip and serialization. +- **Synchronous calls depend on the plugin being available**: while a host process is crashed or in restart backoff, the + plugin's Source writes and user changes return 503; defects in plugin code show up directly as failed user requests. - **The write transaction holds locks**: the block holds database locks, which on SQLite block other writes. A plugin that does slow work in the block slows every write in Server until the block times out and rolls back. - **Plugin-owned storage is not guaranteed consistent with core data**: they are not in the same transaction, so a failed - commit can leave entries in owned storage, and user changes do not notify the plugin. Plugins must overwrite by stable ID + commit can leave entries in owned storage. Plugins must overwrite by stable ID, keep immutable records per revision, and defer to core's current data. -- **Every call starts a Worker and loads the plugin**: process startup and factory initialization have a fixed cost that - rules out second-level high-frequency calls, and plugins cannot reuse connections or caches across calls. +- **Every background call starts a Worker and loads the plugin**: process startup and factory initialization have a fixed + cost that rules out second-level high-frequency background calls, and plugins cannot reuse connections or caches across + Worker calls. +- **Dedicated storage is isolated only by convention**: the plugin and Server use the same database login, so a plugin can + technically reach core's tables. +- **The edge and the central Server do not share a transaction**: Source identity deduplication prevents duplicates, and a + plugin whose `resolve` is unstable produces conflicts in edge mode. - **Not a security sandbox**: Workers isolate ordinary failures but do not restrict a plugin's access to files, network, or credentials, nor impose hard memory or CPU quotas. - **Dependencies must be compatible**: every plugin shares one Python environment with Server, and deployers resolve @@ -927,12 +1198,10 @@ persisted, so the operational APIs report no "last call result". - **Deploying the same code twice takes extra care**: the family is part of the declaration, so copying the same code into two directories fails the check with a family conflict. A plugin that needs several deployments should read identifiers such as its family from environment variables. -- **Content structure is not governed by core**: core does not validate a `content` schema, so plugins and applications - must keep historical revisions interpretable; content submitted with `PUT` is interpreted only when the plugin reads it - later. +- **Content structure is not governed by core**: core does not validate a `content` schema, and plugins validate + user-submitted content; plugins and applications must keep historical revisions interpretable. - **Core search is limited**: core provides only generic text search over declared fields; vector and other search requires - a plugin-owned index. -- **Collection runs with Server**: collection runs in Server's Workers, so Server needs access to the external data sources. + a plugin-owned index, which the public search APIs cannot use yet. - **Code changes require a restart**: without hot reload, Server is briefly unavailable while plugins update. # Rationale and alternatives @@ -946,18 +1215,27 @@ persisted, so the operational APIs report no "last call result". | Plugins return drafts, or the writer stages operations in memory and commits at the end | Atomic, but plugins cannot see real revisions before the commit and struggle to write their own index; merge, undo merge, and lifecycle semantics would need reimplementing in memory; search would not see staged content | | Wrap the whole `generate` in one transaction | Model calls can take minutes, and holding locks that long blocks other writes, the whole database on SQLite; plugin-owned storage is another database that one transaction cannot cover anyway | | A short write transaction block inside `generate` (this proposal) | Slow work stays outside the block; inside it, core's write operations and validation are reused directly, real revisions are returned, and reads see the block's writes; at the end it commits with the cursor and acknowledgment, and an exception rolls back. The cost is holding locks inside the block, bounded by a time limit | -| Run user commands through the plugin (queue manual requests the plugin handles in its next invocation) | Plugins must parse commands and callers can only wait asynchronously; a command that always fails blocks later generation in the Scope; and the request queue must be persisted. Forget, restore, and edit are generic operations core already has, and running them synchronously in core is simpler | +| Queue user changes for the plugin's next background call | Callers can only wait asynchronously; a change that always fails blocks later generation in the Scope; and the request queue must be persisted, while core has no task management today | +| Core executes user changes directly, without the plugin | Simple to implement, but the plugin never sees the submitted content: it cannot validate it, and its own storage and indexes catch up only after a later rebuild. Storage belongs to the plugin, so changes should go through the plugin too; built-in families likewise create and edit through their own family code | +| One `write` method for both create and edit | A create carries no ID and core allocates one, while an edit must name the ID and the expected revision; the inputs and checks differ, and one method would make the plugin tell them apart itself | +| Dispatch a Worker for each synchronous call | Reuses the existing mechanism, but each Worker starts a new Python process, and importing core modules alone takes seconds, which cannot sit on a user request path | +| Run Source `resolve` and similar code in the Server process | Saves an inter-process round trip, but a blocking or crashing plugin affects Server; a plugin's sibling modules enter `sys.modules` under top-level names, so same-named modules of different plugins collide and cannot later be reloaded reliably | +| A long-lived host process for synchronous calls and Workers for background calls (this proposal) | Synchronous calls need no process start each time, and the Server process imports no plugin code; background calls keep the Worker's per-call isolation and budgets. The cost is the resources and management of long-lived processes | +| Plugin storage inside core's transactions (plugins create tables in core's database) | Atomic commits, but core would manage plugin migrations, upgrades, and permissions on three backends, and plugin data would mix into core's schema. Plugin storage belongs to the plugin and stays out of core | +| One plugin declaring several families | Could write several families in one transaction, but declarations, permissions, and merge registration would all need a multi-family design. Cooperating cases are served by two plugins plus cross-family reads | +| Hand core's environment variables or database settings to plugins | Core's configuration does not all come from environment variables, and a plugin with core's database credentials could bypass the writer. Core provides the wrapped `ctx.storage` and `ctx.models` instead | +| Compile a separate binary for the edge | A plugin is a Python directory with dependencies; a frozen binary cannot import arbitrary plugin dependencies and would need packaging per plugin and platform. Starting the same program in `source-agent` mode is enough | | Plugin-specific trigger and management APIs | Callers would have to tell plugins from built-ins; the existing Source and Artifact APIs already cover the operations callers need | | Generic merge APIs for plugin families only | Needs a separate family allowlist in the APIs. The registry already decides which families can be changed, and a family without an adapter cannot register; built-in permission checks live in the engine, so both entry points apply the same checks | -| Resident plugin processes (like Dify's plugin daemon) | Could reuse connections and support other languages, but needs process management, a serialization protocol, and a cancellation protocol. The interfaces are request / response models, so the execution mechanism can change later without changing plugin code | -| Hot reload | Would reintroduce load states, registration generation fencing, and reclaiming old calls, which contradicts the goals | +| Hot reload in this proposal | Requires each instance to rescan on its own, fencing commits from old Workers, and switching host processes; a separate proposal is clearer | Reasons for this choice: - A plugin author writes only `plugin.py`: one declaration, one factory, and one or two methods. - Callers use only the existing Source and Artifact APIs and are unaware of plugins. - Isolation, scheduling, triggers, backoff, and leadership all reuse the Supervisor already in production. -- Writes reuse core's existing implementation, so plugin writes and user changes follow the same semantics. +- Writes reuse core's existing implementation, so plugin writes and user changes follow the same semantics; user changes + go through the plugin, so plugin storage stays in step. - Sources and Artifacts stay in core's unified storage, so the existing read APIs, lineage, merge records, and access control apply directly. - Core manages only the two core data kinds and their progress and leaves everything else to plugins. @@ -966,7 +1244,8 @@ Reasons for this choice: - **Dify plugins**: interfaces are defined per plugin type, plugins implement the matching methods, and Dify calls them when a workflow or application needs them; plugins run in a separate plugin daemon. This proposal keeps the idea that the - host defines the interfaces and when to call them, and reuses PowerContext's existing Supervisor Workers for isolation. + host defines the interfaces and when to call them: synchronous calls use long-lived host processes that core manages, + and background calls reuse PowerContext's existing Supervisor Workers. - **The Airbyte connector protocol**: a Source connector's `read` receives the previous state and emits records and a new state, which the platform persists. The collection checkpoints in this proposal split the work the same way and keep a checkpoint per target. @@ -979,16 +1258,18 @@ Reasons for this choice: [Artifact processing Supervisor](1515_artifact_processing_supervisor.md); the [shared merge methods](1859_artifact_merge.md); the write implementations of `ArtifactRepository`, `ArtifactGovernanceRepository`, handles, and tags; the generic Artifact read and write APIs; `SourceDefinition` / `SourceProjection`; the Source journal, `pc_source_cursors`, and - `list_window_for_generation`; `pc_connector_checkpoints`; and the Source management APIs with the `identify_target` - target identification rule. + `list_window_for_generation`; `pc_connector_checkpoints`; the remote ingestion APIs (`source-observations`, + `connector-checkpoints`, `source-definitions/register`); the `lineage_only` Content Source built-in families record on + edits; and the Source management APIs with the `identify_target` target identification rule. # Unresolved questions | Question | Notes | | --- | --- | -| Plugin-defined search | The existing search APIs accept only text queries, so the first version provides generic text search in core. Whether plugins may later implement the public search API depends on deciding which process runs search code | +| Plugin-defined search | The existing search APIs accept only text queries, so the first version provides generic text search in core. Search a plugin provides (for example vector or hybrid search) is left to the separate proposal for plugin-defined APIs and runs in the host process | | Default limits for the write transaction block | Whether 30 seconds is right; whether SQLite and OceanBase need different defaults; per-block operation and byte limits | -| A shared model client | Whether `PluginContext` should offer the model client Server already configures, so LLM plugins need not bring credentials and their usage counts toward the existing model usage statistics | +| Host process defaults | Default synchronous call limit, per-host-process concurrency limit, and restart backoff | +| Permission to create the dedicated database | When Server's database login cannot create databases, whether to allow falling back to tables prefixed by plugin name in core's database, besides the deployer creating the database in advance | | Deleting collection targets | The first version supports only disabling and resuming. Whether targets and their checkpoints should be deletable, and what happens to Sources already collected | | Plugin families in Recall / Prepared Context | Requires defining candidate content and budget constraints; a separate proposal is recommended | | Shrinking the window | Built-in Atomic Memory shrinks the next Source window when the model times out. Plugins cannot affect the window size, so an oversized window keeps timing out. Whether plugins may request a smaller window for the Scope with an agreed exception | @@ -999,6 +1280,8 @@ Reasons for this choice: # Future possibilities -- Resident plugin processes that reuse connections and initialization while keeping the interfaces unchanged. +- Hot reload and plugin-defined APIs (including plugin pages): built on host processes, in a separate proposal. +- Long-lived processes for background calls too, reusing connections and initialization while keeping the interfaces + unchanged. - More plugin types, such as candidate content for Recall or text fragments for Prepared Context. - Plugin packaging and installation tools, such as installing into the plugin directory from a Git URL or a wheel. diff --git a/docs/zh/rfcs/2549-plugin-interfaces.md b/docs/zh/rfcs/2549-plugin-interfaces.md index a9e23f456e..a657f34065 100644 --- a/docs/zh/rfcs/2549-plugin-interfaces.md +++ b/docs/zh/rfcs/2549-plugin-interfaces.md @@ -1,6 +1,6 @@ --- title: 插件接口 -description: Source 和 Artifact 插件各自实现少量接口，由 PC Server 通过 Supervisor 在需要时调用；调用方只使用现有的 Source 与 Artifact 接口，core 只管理两类核心数据及其消费进度，不托管插件生命周期。 +description: Source 和 Artifact 插件各自实现少量接口，由 PC Server 在需要时调用：后台调用在 Supervisor Worker 中执行，同步调用在 core 管理的插件宿主进程中执行；调用方只使用现有的 Source 与 Artifact 接口，core 只管理两类核心数据及其消费进度，不托管插件生命周期。 --- - Proposal Name: `plugin_interfaces` @@ -19,19 +19,28 @@ description: Source 和 Artifact 插件各自实现少量接口，由 PC Server `plugin`。声明写在代码里，没有单独的清单文件；部署相关的设置由插件从环境变量读取。 - **调用方式**： - 写入 Source、添加采集目标，使用现有的 Source 接口。 - - 读取、检索和修改 Artifact，使用现有的 Artifact 接口。 - - 遗忘、恢复、编辑等用户操作由 core 直接执行，不经过插件。 -- **执行**：`collect` 和 `generate` 在 Supervisor 管理的 Worker 子进程中执行，沿用现有的隔离、超时、并发预算、 - 领导租约和失败退避。 + - 读取、检索 Artifact，使用现有的 Artifact 接口。 + - 新建、编辑、合并、遗忘、恢复插件 Artifact，也使用现有的 Artifact 接口。core 负责鉴权、版本校验和溯源，再同步调用 + 插件对应的方法；插件校验内容、维护自己的存储，并通过写入器写入。插件没有实现某项操作时，该接口返回 422。 +- **执行**： + - `collect` 和 `generate` 是后台调用，在 Supervisor 管理的 Worker 子进程中执行，沿用现有的隔离、超时、并发预算、 + 领导租约和失败退避。 + - Source 写入时的 `resolve` 与 Projection、添加采集目标时的 `identify_target`、用户对插件 Artifact 的修改是同步调用， + 在 core 管理的常驻**插件宿主进程**中执行。每个 Server 进程为每个插件按需启动一个宿主进程，部署者不需要管理它。 - **写入**：Artifact 插件在 `async with writer.transaction()` 块中执行写操作，块结束时 core 把写入、消费游标和 完成确认一起提交，异常则整体回滚。插件读不到也改不了游标。 - **core 的职责**：只管理 `pc_sources`、`pc_artifacts`（含 heads、lineage、合并记录、生命周期、handle 和 tags）及其 - 消费进度。插件可以有自己的索引和存储，由插件自己管理。 -- **生命周期**：插件在 Server 启动时按目录加载，变更需要重启。Server 不保存插件的启用状态、注册代次或管理阶段。 + 消费进度。插件可以有自己的索引和存储，由插件自己管理；core 在部署使用的数据库后端上为每个插件提供一块专属存储， + 并把 Server 已配置的模型提供给插件使用。 +- **端侧采集**：Server 可以以 `source-agent` 模式启动，只运行 Source 插件的采集，把 Source 和 checkpoint 提交到远端的 + 中心 Server，用于具身智能等需要在设备上采集的场景。 +- **生命周期**：插件在 Server 启动时按目录加载，变更需要重启。每个 Server 进程只在内存中记录本进程已激活的插件， + 不持久化插件的启用状态、注册代次或管理阶段。 本提案取代 [Source 插件框架 RFC #2012](2012-source-plugin-system.md) 和 [Artifact 插件框架 RFC #2175](2175-artifact-plugin-system.md) -的插件契约与管理机制。两者的实现尚未作为稳定接口发布，本提案不保留兼容：旧的插件清单、管理接口、管理服务和插件管理相关的 -表直接移除，旧 Artifact 插件写入的自定义 Artifact 数据不迁移。 +的插件契约与管理机制。旧 Source 插件框架随 v1.5 发布，但目前没有对外发布的插件，因此本提案不保留兼容：旧的插件清单、 +管理接口、管理服务和插件管理相关的表直接移除，旧 Artifact 插件写入的自定义 Artifact 数据不迁移。旧设计中合适的部分继续 +沿用，包括 Source 管理接口（`pc_source_add` 等）及采集目标的身份规则，以及 core 已有的远程采集接口。 本文描述目标设计，不代表主线或发布版本已经可用。接口名称和代码示例用于说明契约，最终语法以实现为准。 @@ -60,12 +69,13 @@ description: Source 和 Artifact 插件各自实现少量接口，由 PC Server 1. 插件作者只需写一个 `plugin.py`：一个声明对象和几个方法，不需要理解插件管理的内部机制。 2. 调用方不感知插件：Source 和 Artifact 的现有接口对插件产生的数据同样适用。 -3. 插件代码在隔离的 Worker 进程中执行，插件故障不影响 Server 和其他插件。 +3. 插件代码只在 core 管理的子进程中执行（Worker 或插件宿主进程），插件故障不影响 Server 和其他插件。 4. 插件结果进入 `pc_sources` / `pc_artifacts`，与消费进度原子提交。 5. core 只管理这两类核心数据及其消费进度；内容结构、索引和其他存储交给插件自己。 6. 插件对核心数据的写入只经过 core 提供的写入器，覆盖 Artifact 的全部通用写操作，并沿用 core 现有的校验规则。 -7. Artifact 生成可以按调度执行，也可以由新 Source 触发；采集目标可以在运行中增减。 -8. 用户只需部署一个 Server；插件随 Server 启动加载。 +7. 用户对插件 Artifact 的修改经过插件，插件自有存储能与修改同步更新。 +8. Artifact 生成可以按调度执行，也可以由新 Source 触发；采集目标可以在运行中增减。 +9. 用户只需部署一个 Server；插件随 Server 启动加载。需要在设备上采集时，同一个程序以端侧模式运行。 # 指南级说明 @@ -80,6 +90,9 @@ flowchart LR Sup["Supervisor
调度 / 触发 / 预算 / 退避"] Core[("core 数据
pc_sources / pc_artifacts
采集目标 / checkpoint / 游标")] end + subgraph Host["插件宿主进程（常驻，core 管理）"] + Sync["resolve / Projection / identify_target
create_artifact / update_artifact / ..."] + end subgraph SW["Worker：Source 插件"] Collect["collect(request)"] end @@ -90,8 +103,11 @@ flowchart LR end Caller --> SourceAPI Caller --> ArtifactAPI - SourceAPI -->|"resolve / Projection（插件定义）"| Core - ArtifactAPI <--> Core + SourceAPI -->|"同步调用"| Sync + ArtifactAPI -->|"读取"| Core + ArtifactAPI -->|"修改，同步调用"| Sync + Sync -->|"Source 或 Artifact 写入，一次提交"| Core + Sync -.-> Own Sup -->|"目标与 checkpoint"| Collect Collect -->|"Source 与新 checkpoint，一次提交"| Core Core -->|"新 Source 触发"| Sup @@ -101,34 +117,40 @@ flowchart LR Generate -.-> Own[("插件自有存储 / 模型")] ``` -图中有两条独立的路径，一次 Worker 调用只执行其中一个插件： +图中有三条路径，一次调用只执行一个插件： - **采集路径**：Supervisor 按调度派发 Source 插件的 Worker，传入该 Scope 已启用的采集目标及其 checkpoint；插件从外部 系统读取数据并返回 Source；core 校验后把 Source 和新 checkpoint 在一个事务中保存。 - **生成路径**：新 Source 进入 journal 后，关注这些 Source 的 Artifact 插件被标记为待处理；Supervisor 派发 Worker，传入 该 Scope 尚未消费的 Source 窗口和写入器；插件在写事务块中写入 Artifact，core 连同游标和确认一起提交。 +- **同步路径**：调用方写入 Source、添加采集目标或修改插件 Artifact 时，Server 完成鉴权和版本校验后，把请求交给该插件的 + 宿主进程；插件在宿主进程中完成转换或写入，结果在一个事务中提交后同步返回给调用方。 调用方只和 core 的现有接口打交道： | 调用方要做的事 | 使用的接口 | 插件参与的方式 | | --- | --- | --- | -| 提交一条 Source | 现有 Source 写入接口 | core 调用插件定义的 `resolve` 和 Projection 完成校验与转换 | -| 添加、查看、停用采集目标 | 现有 Source 管理接口 | core 调用插件的 `identify_target` 规范化输入 | +| 提交一条 Source | 现有 Source 写入接口 | 宿主进程执行插件定义的 `resolve` 和 Projection | +| 添加、查看、停用采集目标 | 现有 Source 管理接口 | 添加时宿主进程执行插件的 `identify_target` | | 浏览、读取、检索 Artifact | 现有 Artifact 读取与检索接口 | 不参与 | -| 编辑、打 tag、合并、遗忘、恢复、撤销合并 | 现有 Artifact 写接口、tags 接口与通用合并接口 | 不参与；插件在下一次调用时看到新状态 | +| 新建、编辑、合并、遗忘、恢复、撤销合并 | 现有 Artifact 新建与编辑接口、通用合并接口 | 宿主进程执行插件对应的方法；未实现返回 422 | +| 打 tag、按 tag 查询 | 现有 tags 接口 | 不参与；tags 是 core 数据 | 职责划分： | 职责 | 归属 | | --- | --- | -| 认证、Scope 授权、公开 HTTP / MCP 接口、采集目标与 checkpoint 的保存、用户对 Artifact 的修改 | core | +| 认证、Scope 授权、公开 HTTP / MCP 接口、采集目标与 checkpoint 的保存、用户修改的版本校验与溯源记录 | core | | 何时调用：调度、触发合并、Worker 预算、超时、退避、领导租约 | core（现有 Supervisor） | -| 读取输入、提供受约束的写事务、在同一事务中提交核心数据 / 游标 / checkpoint / 完成确认 | core（Worker 中的插件执行器） | +| 宿主进程的启动、复用、超时终止与重启 | core | +| 读取输入、提供受约束的写事务、在同一事务中提交核心数据 / 游标 / checkpoint / 完成确认 | core（Worker 或宿主进程中的插件执行器） | +| 插件专属存储空间的分配、Server 已配置模型的封装 | core | | Source 定义与 Projection 计算、采集目标的识别与规范化、外部数据读取 | Source 插件 | -| Artifact 内容结构、生成算法、插件自有的索引与存储及其一致性 | Artifact 插件 | -| 插件代码、依赖、环境变量和插件自有存储的部署 | 部署者 | +| Artifact 内容结构、生成算法、用户提交内容的校验、插件自有的索引与存储及其一致性 | Artifact 插件 | +| 插件代码、依赖和环境变量的部署 | 部署者 | -插件代码只做一件事：给定输入，产出结果或抛出异常。抛出异常即失败，core 不提交任何核心数据，下次重试同一输入。 +插件代码只做一件事：给定输入，产出结果或抛出异常。抛出异常即失败，core 不提交任何核心数据；后台调用下次重试同一输入， +同步调用把错误返回给调用方。 ## 2. 插件的形态 @@ -153,8 +175,13 @@ plugin = ArtifactPlugin(family="incident-report", ..., create=create) # 或 So 声明对象回答 core 在运行插件代码之前需要知道的问题：插件提供什么（Source 类型或 Artifact family）、消费什么、何时调用、 资源上限多少。它是普通的 Python 对象，由 Pydantic 校验；没有 YAML 或其他清单文件。 -- **部署设置**：凭据、外部服务地址、模型名等随部署变化的值，由插件在 `create` 中从环境变量读取。Server 不保存、 +- **部署设置**：插件自己的凭据、外部服务地址等随部署变化的值，由插件在 `create` 中从环境变量读取。Server 不保存、 不传递也不记录这些值。 +- **core 提供的资源**：插件需要数据库和模型时，不必另行配置，通过上下文使用 core 封装好的对象（见参考级说明 1.2）： + - `ctx.storage`：部署所用数据库后端上的一块插件专属空间。SQLite 是数据目录下的一个单独文件，OceanBase 和 seekdb + 是一个单独的库。插件拿到的连接只指向这块空间，不指向 core 的表。 + - `ctx.models`：Server 已配置的生成模型和 embedding 模型。插件只能调用，拿不到模型的地址和凭据；调用计入现有的模型 + 用量统计。 - **启用与停用**：目录存在且检查通过，插件就可用；需要停用时移出目录并重启 Server。 - **检查**：Server 启动时在有时限的子进程中导入 `plugin.py`，校验声明，不调用 `create`。检查失败只影响该插件。 @@ -171,7 +198,10 @@ plugin = ArtifactPlugin(family="incident-report", ..., create=create) # 或 So | `collect(request) -> CollectResult` | 采集时必需 | 按目标和 checkpoint 读取外部数据，返回 Source 和新的 checkpoint | 只提供 `definitions` 的插件只扩展 Source 类型：调用方通过现有的 Source 写入接口提交内容，core 调用定义完成校验、 -转换和保存。 +转换和保存。`resolve`、Projection 和 `identify_target` 都在插件宿主进程中执行；`collect` 在 Worker 中执行。 + +Source 与所有内置 Source 一样只能追加，没有修改和删除；要更正内容，提交一条新的 Source。文件、图片等大块内容由插件 +存进自己的存储，Source 中只记录位置、摘要和媒体类型等元数据。 ### 3.2 声明 @@ -256,6 +286,29 @@ pc_source_target_set_enabled(scope_id="team-a", target_id="...", enabled=false) 目标。新目标保存为启用状态并立即请求一次采集，不需要重启。 - **checkpoint**：每个目标的 checkpoint 是只读输入，插件通过返回值提出新的 checkpoint。core 把本次返回的所有 Source 与各目标的新 checkpoint 在同一事务中提交；`collect` 抛出异常或超时则不提交任何内容，下一次调用仍拿到旧 checkpoint。 +- **格式校验**：`pc_source_capabilities` 发布 `target_input` 的 JSON Schema，调用方提交前就知道格式。添加时 core 先用 + `target_input` 校验，不合格返回 422 并指出字段；`identify_target` 抛出 `ValueError` 同样返回 422。外部资源是否存在、 + 是否可读，只能在 `collect` 时判断，结果在 `pc_source_status` 的最近失败中体现。 + +### 3.5 端侧采集 + +具身智能等场景需要在设备上采集，再提交到中心的 PowerContext Server。同一个程序以端侧模式启动： + +```text +powercontext source-agent --server https://memory.example.com +``` + +- 端侧只加载插件目录中带 `collect` 的 Source 插件，运行 Supervisor 调度 `collect`，不加载 Artifact 插件，也不提供 + Artifact 接口。 +- 采集目标保存在端侧本地的 SQLite 中，通过端侧本地的 `pc_source_add`、`pc_source_status`、`pc_source_target_set_enabled` + 管理。中心看不到端侧的目标，只看到提交上来的 Source 和 checkpoint。 +- 采集结果通过 core 已有的远程采集接口提交到中心：Source 走 `POST /v1/source-observations`，checkpoint 走 + `POST /v1/connector-checkpoints/commit`。`resolve` 和 Projection 在端侧执行，中心按已登记的定义清单校验，不需要 + 加载插件代码。 +- 断线、重启和重传都按幂等处理：先提交 Source，再以旧 checkpoint 为期望值提交新 checkpoint。中途失败时 checkpoint 不前进， + 下次调用从旧 checkpoint 重新采集；重新提交的 Source 身份相同、内容相同，由中心去重。 +- 离线期间的数据积压由插件自己处理。提交失败时本次调用失败，checkpoint 不前进；外部系统无法回放的数据（例如传感器流）， + 插件需要先存在自己的本地存储中，按 checkpoint 重新给出。 ## 4. Artifact 插件 @@ -263,10 +316,16 @@ pc_source_target_set_enabled(scope_id="team-a", target_id="...", enabled=false) | 接口 | 必需 | 说明 | | --- | --- | --- | -| `create(ctx) -> ArtifactGenerator` | 是 | 工厂，在 Worker 中每次调用执行一次 | +| `create(ctx) -> ArtifactGenerator` | 是 | 工厂；Worker 中每次调用执行一次，宿主进程中每个 Scope 执行一次后复用 | | `generate(request, writer) -> None` | 是 | 读取 `request.sources`（未消费的 Source 窗口），在写事务块中写入 Artifact | +| `create_artifact(request, writer) -> None` | 否 | 用户新建；`request.artifact_id` 由 core 分配 | +| `update_artifact(request, writer) -> None` | 否 | 用户编辑指定 ID 的 Artifact，`request.expected_revision` 为调用方看到的 revision | +| `merge_artifacts(request, writer) -> None` | 否 | 用户合并；结果的 ID 由 core 分配 | +| `forget_artifact(request, writer) -> None` | 否 | 用户遗忘 | +| `restore_artifact(request, writer) -> None` | 否 | 用户恢复、恢复到旧 revision 或撤销合并 | -插件只负责生成。用户对 Artifact 的修改由 core 直接执行（见 4.4），插件不需要实现命令处理。 +后五个方法处理用户修改（见 4.4），每个都是可选的：没有实现的操作，对应接口返回 422，表示这个 family 不支持该操作。 +插件在这些方法中校验用户提交的内容、更新自己的存储，并通过写入器完成 core 要求的那一项写操作。 ### 4.2 声明 @@ -281,17 +340,19 @@ pc_source_target_set_enabled(scope_id="team-a", target_id="...", enabled=false) | `timeout_seconds` | 300 | 整次调用的时限 | | `transaction_timeout_seconds` | 30 | 写事务块的时限，不超过 `timeout_seconds` | | `max_workers` | 1 | 跨 Scope 的并发上限 | +| `change_timeout_seconds` | 30 | 一次用户修改调用的时限，包括其中的写事务块 | | `create` | 必填 | 见上表 | ### 4.3 示例 下面的插件为每条工单 Source 生成一份报告：先用自己的向量索引查找相似的已有报告，命中则修订那份报告，否则新建； -同时把报告写进自己的向量索引。 +同时把报告写进自己的向量索引。它还支持用户编辑报告，没有实现的新建、合并、遗忘、恢复对该 family 返回 422。向量索引 +放在 core 分配的专属存储中，模型使用 Server 已配置的模型。 ```python -import os - -from powercontext.plugins import ArtifactPlugin, ArtifactWriter, GenerateRequest, Inputs, PluginContext +from powercontext.plugins import ( + ArtifactPlugin, ArtifactWriter, GenerateRequest, Inputs, InvalidContent, PluginContext, UpdateRequest, +) from powercontext.sources import TextEvidence @@ -320,10 +381,19 @@ class IncidentReports: await self.index.upsert(request.scope_id, written.ref.artifact_id, written.ref.revision, vector) # 块正常结束：core 提交 Artifact 写入、游标推进与完成确认 + async def update_artifact(self, request: UpdateRequest, writer: ArtifactWriter) -> None: + # 用户编辑：校验内容，重算向量，再写入 core 和自有索引 + report = Report.model_validate(request.content) # 不合格时抛出 InvalidContent，接口返回 422 + vector = await self.model.embed(report.summary) + async with writer.transaction() as tx: + written = await tx.put(request.artifact_id, content=report.model_dump(), + expected_revision=request.expected_revision) + await self.index.upsert(request.scope_id, written.ref.artifact_id, written.ref.revision, vector) + async def create(ctx: PluginContext) -> IncidentReports: - index = await ctx.enter_async_context(VectorIndex.connect(os.environ["REPORT_INDEX_URL"])) - return IncidentReports(index, Summarizer()) + index = await ctx.enter_async_context(VectorIndex.connect(ctx.storage.engine())) + return IncidentReports(index, Summarizer(ctx.models)) plugin = ArtifactPlugin( @@ -340,27 +410,28 @@ plugin = ArtifactPlugin( ### 4.4 调用方怎么使用 -调用方使用现有的 Artifact 接口访问插件 family。读取接口直接读 `pc_artifacts`，写接口由 core 执行，与插件写入复用同一套 -core 实现，都不运行插件代码： +调用方使用现有的 Artifact 接口访问插件 family。读取接口直接读 `pc_artifacts`，不运行插件代码；修改接口由 core 鉴权、 +校验版本后，同步调用插件对应的方法： -| 用途 | 接口 | -| --- | --- | -| 浏览 | `GET /v1/scopes/{scope_id}/artifacts/{family}` | -| 读取当前内容、指定 revision、revision 历史、tags | `GET .../{artifact_id}`、`GET .../{artifact_id}/revisions/{revision}`、`GET .../{artifact_id}/revisions`、`GET .../{artifact_id}/tags` | -| 文本检索 | `POST /v1/scopes/{scope_id}/artifacts/{family}/search` | -| 编辑（提交完整内容，`If-Match` 校验当前 revision） | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | -| 修改 tags | `PUT .../{artifact_id}/tags`；按 tag 查询用 `POST /v1/scopes/{scope_id}/artifact-tags/query` | -| 读取治理状态（遗忘、合并前取得 `state_version`） | `GET .../{artifact_id}/governance` | -| 合并 | `POST /v1/scopes/{scope_id}/artifacts/{family}/merges` | -| 遗忘 | `POST .../{artifact_id}/forget` | -| 恢复、恢复到旧 revision、撤销合并 | `POST .../{artifact_id}/restorations` | -| 补充知识 | 写入一条 Source，由插件在下一次调用中处理 | - -修改同步返回结果。插件在下一次调用中通过写入器读到修改后的状态。 +| 用途 | 接口 | 插件方法 | +| --- | --- | --- | +| 浏览 | `GET /v1/scopes/{scope_id}/artifacts/{family}` | 无 | +| 读取当前内容、指定 revision、revision 历史、tags | `GET .../{artifact_id}`、`GET .../{artifact_id}/revisions/{revision}`、`GET .../{artifact_id}/revisions`、`GET .../{artifact_id}/tags` | 无 | +| 文本检索 | `POST /v1/scopes/{scope_id}/artifacts/{family}/search` | 无 | +| 新建（只提交内容，ID 由 core 分配） | `POST /v1/scopes/{scope_id}/artifacts`，请求体为 `{"family": ..., "content": {...}}` | `create_artifact` | +| 编辑（提交完整内容，`If-Match` 校验当前 revision） | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | `update_artifact` | +| 读取治理状态（遗忘、合并前取得 `state_version`） | `GET .../{artifact_id}/governance` | 无 | +| 合并 | `POST /v1/scopes/{scope_id}/artifacts/{family}/merges` | `merge_artifacts` | +| 遗忘 | `POST .../{artifact_id}/forget` | `forget_artifact` | +| 恢复、恢复到旧 revision、撤销合并 | `POST .../{artifact_id}/restorations` | `restore_artifact` | +| 修改 tags | `PUT .../{artifact_id}/tags`；按 tag 查询用 `POST /v1/scopes/{scope_id}/artifact-tags/query` | 无，由 core 处理 | + +修改同步返回结果。插件拒绝内容时返回 422，插件没有实现对应方法时也返回 422，宿主进程超时或不可用时返回 503。 ## 5. 调度与触发 -插件的调度和触发都交给现有的 Artifact processing Supervisor。每个插件在启动时注册为一个 Supervisor binding， +插件的后台调用（`collect`、`generate`）的调度和触发都交给现有的 Artifact processing Supervisor；同步调用由请求触发， +不经过 Supervisor。每个插件在启动时注册为一个 Supervisor binding， binding 名为 `plugin:`，调度键是 `(binding, Scope)`。 Artifact 插件： @@ -392,7 +463,8 @@ Source 插件的 `collect` 按 `collect.schedule` 调用；添加或恢复目标 - 调用失败的原因通过日志和指标排查，不在接口中保存。 - **变更**：修改代码后重启 Server。Server 不监视文件，也不在运行中重新加载插件。采集目标是业务数据，增减不需要重启。 - **停用**：从插件目录移走后重启。已保存的 Source、Artifact、采集目标、游标和 checkpoint 保留并可继续读取；重新部署 - 同名插件时从原位置继续。插件自有存储由部署者保留或清理。 + 同名插件时从原位置继续。插件的专属存储不随之删除，由部署者保留或清理。 +- **宿主进程**：由 core 按需启动、复用和重启，部署者不需要配置或管理。宿主进程的启动、终止和重启只体现在日志和指标中。 # 参考级说明 @@ -436,14 +508,34 @@ class PluginContext: scope_id: str # 本次调用的 Scope plugin_dir: Path # 插件目录（只读约定） logger: logging.Logger # 已绑定插件名的 logger + storage: PluginStorage # 插件专属存储 + models: PluginModels | None # Server 已配置的模型；未配置时为 None async def enter_async_context(self, cm: AbstractAsyncContextManager[T]) -> T: ... + +class PluginStorage(Protocol): + backend: Literal["sqlite", "oceanbase", "seekdb"] + def engine(self) -> AsyncEngine: ... # 只指向本插件的专属空间 + +class PluginModels(Protocol): + embedding_profile: EmbeddingProfile | None # profile ID 与向量维度；未配置 embedding 时为 None + async def embed(self, texts: Sequence[str], /) -> Sequence[Sequence[float]]: ... + async def generate(self, messages: Sequence[ChatMessage], /, *, max_tokens: int | None = None) -> str: ... ``` -- 工厂在每次 Worker 调用中执行一次，实例不跨调用复用。需要关闭的资源（包括插件自有存储的连接）通过 - `ctx.enter_async_context` 注册，调用结束时由执行器释放。 -- 插件得不到 core 数据库的连接，也没有游标或 checkpoint 的写入接口。插件对核心数据的写入只有两条路径：Source 插件的 - `collect` 返回值，和 Artifact 插件的 `ArtifactWriter`。 +- 工厂在每次 Worker 调用中执行一次，实例不跨调用复用；在宿主进程中每个 Scope 执行一次，实例在宿主进程存续期间复用。 + 需要关闭的资源（包括插件自有存储的连接）通过 `ctx.enter_async_context` 注册，Worker 调用结束或宿主进程停止时由执行器释放。 +- **专属存储**：core 使用 Server 现有的数据库连接配置，在同一个后端上为插件分配一块空间，不需要额外的数据库账号： + - SQLite：数据目录下的 `plugins/<插件名>.sqlite3`。 + - OceanBase / seekdb：名为 `powercontext_plugin_<插件名>` 的库（`-` 换成 `_`），首次使用时由 core 创建。Server 的数据库 + 登录没有建库权限时，由部署者事先建好。 + - 表结构和迁移由插件自己管理，core 不读取、不迁移、不备份这块空间。它不在 core 的事务中，与核心数据的一致性按 3.5 处理。 + - 插件与 Server 使用同一个数据库登录，这一划分防止插件误写 core 的表，不防止恶意插件；插件目录本身被当作可信代码。 +- **模型**：`models` 封装 Server 的推理配置，插件拿不到模型的地址、请求头和凭据。`embedding_profile` 让插件在索引和查询时 + 使用同一个 embedding 配置。调用沿用 Server 的超时设置，并计入现有的模型用量统计。插件也可以从环境变量读取自己的 + 模型配置，自行创建客户端。 +- 插件得不到 core 数据库的连接，也没有游标或 checkpoint 的写入接口。插件对核心数据的写入只有三条路径：Source 插件的 + `collect` 返回值、宿主进程中 `resolve` 的返回值，和 Artifact 插件的 `ArtifactWriter`。 ## 2. Source 插件契约 @@ -484,7 +576,8 @@ class SourceCollector(Protocol): 冲突时检查失败。 - **写入**：调用方通过现有的 Source 写入接口（`POST /v1/scopes/{scope_id}/sources`、MCP `pc_source_add` 的 write 用法） 提交内容。core 按 `definition.input_class` 校验输入，调用 `resolve` 得到 Source，计算各 Projection 后保存，语义与内置 - Source 写入一致。这一步在 Server 进程中同步执行，有超时限制。 + Source 写入一致。校验、`resolve` 和 Projection 在该插件的宿主进程中同步执行，有超时限制。`resolve` 必须是纯函数， + 不能有复制文件等副作用；需要把外部文件纳入插件存储时，用 `collect` 完成。 - **collect**：每次调用对应一个 Scope，最多携带 `max_targets` 个已启用目标；目标较多时按目标键轮转，`has_more` 的目标 优先进入下一次调用。执行器对返回的每条 Source 执行与写入相同的校验、`resolve` 和 Projection，然后在一个事务中保存全部 Source、写入各目标的 checkpoint 和最近采集时间，并确认本次调用。任一条失败则整次不提交。条目总数超过 `max_items`、 @@ -501,8 +594,22 @@ class SourceCollector(Protocol): - 添加和变更需要调用者对该 Scope 有 Source 写权限，并且 Scope 在插件的 `scopes` 范围内。 - `identify_target` 必须是纯函数：只做校验和规范化，不访问外部系统、不检查外部授权。外部资源是否存在、是否可读， - 在 collect 时由插件判断并以异常或空结果表达。它与写入一样在 Server 进程中同步执行，有超时限制。 -- `pc_source_capabilities` 列出插件提供的 Source 类型和目标输入。 + 在 collect 时由插件判断并以异常或空结果表达。它与写入一样在宿主进程中同步执行，有超时限制。 +- `pc_source_capabilities` 列出插件提供的 Source 类型和目标输入的 JSON Schema。 + +**端侧模式**（`powercontext source-agent`）沿用同一套 Source 插件契约，只替换提交目标： + +| 项 | 端侧模式中的处理 | +| --- | --- | +| 配置 | `--server` 指定中心 Server；`POWERCONTEXT_SOURCE_AGENT_ID` 是端侧的稳定标识；`POWERCONTEXT_SOURCE_AGENT_TOKEN` 是对目标 Scope 有 `scope.contribute` 权限的凭据 | +| 本地数据 | 端侧数据目录下的 SQLite，保存采集目标（结构同 `pc_source_collect_targets`）和 Supervisor 的调度状态；不保存 Source | +| 定义清单 | 启动时把各 Source 定义的声明式清单登记到中心（`POST /v1/source-definitions/register`，需要 `server.admin`）；凭据没有该权限时，由部署者事先登记 | +| 每次 `collect` | 从中心读取各目标的 checkpoint（`/v1/connector-checkpoints/get`），调用插件，在端侧执行 `resolve` 与 Projection，逐条提交 Source（`/v1/source-observations`），全部成功后以旧值为期望提交新 checkpoint（`/v1/connector-checkpoints/commit`） | +| 目标的 binding ID | 由端侧标识、插件名和目标键确定性派生，两个端侧采集同一外部资源不会互相覆盖 checkpoint | +| 失败 | 任一步失败则本次调用失败，进入 Supervisor 退避；已提交的 Source 保留，checkpoint 不前进，下次重采由中心去重 | + +端侧和中心之间不是一个事务：可能出现 Source 已保存而 checkpoint 未前进的情况，靠 Source 身份去重保证不重复。这要求 +插件的 `resolve` 对同一外部状态给出相同的身份和内容；身份相同但内容不同时，中心返回冲突，本次调用失败。 ## 3. Artifact 插件契约 @@ -569,9 +676,10 @@ class ArtifactTransaction(ArtifactReads): 写入约束： -- 只能操作本 Scope、声明中 `family` 的 Artifact，插件不能指定其他 family 或 Scope。 -- `source_refs` 只能引用本次窗口中的 Source；`artifact_refs`、`to_revision` 和 `bind_handle` 的目标只能是本 family - 中实际存在的 revision。 +- 只能写本 Scope、声明中 `family` 的 Artifact，插件不能写其他 family 或 Scope。 +- `source_refs` 在 `generate` 中只能引用本次窗口中的 Source，在用户修改中只能引用目标 Artifact 当前 lineage 中已有的 + Source（用于保留原始输入）。`artifact_refs` 可以引用本 Scope 中任意 family 实际存在的 revision；`to_revision` 和 + `bind_handle` 的目标只能是本 family 中实际存在的 revision。 - 检索文本由 core 按声明的 `search_fields` 从内容中提取，插件不单独提供；同一份内容无论由插件还是用户写入， 检索文本都相同。 - 每个写事务块的操作数（1000）、单份内容大小（256 KiB）和总字节数（8 MiB）有上限，超出即抛出异常并回滚， @@ -581,7 +689,9 @@ class ArtifactTransaction(ArtifactReads): ### 3.4 读取 `ArtifactWriter`（事务外，只读已提交数据）和 `ArtifactTransaction`（事务内，同时看到本块已写入的内容）提供相同的读取方法， -范围固定为本 Scope、本 family： +范围默认为本 Scope、本 family。`reads(family)` 返回同一 Scope 中其他 family 的只读视图，方法相同，可以读取内置 family +和其他插件的 family（内容按原样 JSON 返回），不能写入。这让一个插件可以基于另一个插件的结果工作，例如知识库插件读取 +文档插件写入的 Artifact： ```python class ArtifactSnapshot(BaseModel): @@ -603,12 +713,13 @@ class ArtifactReads(Protocol): async def merge_inputs(self, artifact_id: str) -> Sequence[ArtifactRef]: ... async def tags(self, artifact_id: str) -> Sequence[str]: ... async def resolve_handle(self, handle_key: str) -> ArtifactRef | None: ... + def reads(self, family: str) -> ArtifactReads: ... # 同一 Scope 中其他 family 的只读视图 ``` - 快照同时携带精确 revision、lineage 和当前治理状态，插件不需要再分别读取 lineage 和生命周期。 - `get` 可以读当前内容或指定 revision；`revisions` 列出完整历史；`merge_inputs` 读取合并结果的精确输入。 - `list` 可以按生命周期状态和 tags 过滤，包括 deprecated、retired 和被合并冻结的 Artifact，用于查找之前已经存在的数据。 -- `search` 是 core 提供的文本检索（见第 6 节），只覆盖 active 的当前 revision；在写事务中能检索到本块已写入、尚未提交的内容。 +- `search` 是 core 提供的文本检索（见第 7 节），只覆盖 active 的当前 revision；在写事务中能检索到本块已写入、尚未提交的内容。 - `by_source` 反查 lineage 引用了某条 Source 的 Artifact，用于 Source 更新后找到需要修订的 Artifact。 - 读取返回的快照都带精确 revision，可以直接作为写操作的 `artifact_refs` 或 `to_revision`。 @@ -620,56 +731,114 @@ class ArtifactReads(Protocol): - **重复处理同一业务对象**：不消费 Source 的定时整理等场景会再次处理已经处理过的对象。插件用 `writer.stable_id(...)` 从业务 身份派生 Artifact ID；`put` 在内容相同时为 no-op，`set_lifecycle` 目标状态与当前相同时为 no-op，`merge` 在输入已全部 合并到同一结果时为 no-op，从而不会产生重复 Artifact 或无意义的 revision。 -- **插件自有存储**：插件的索引和其他存储与 core 是不同的数据库，不在 core 的事务中，core 不保证两者一致，由插件负责。 +- **插件自有存储**：插件的索引和其他存储（包括 core 分配的专属存储）不在 core 的事务中，core 不保证两者一致，由插件负责。 推荐做法： 1. 在写事务块内、拿到 `ArtifactSnapshot` 之后写自有存储，按 `artifact_id` upsert。这样自有存储的写入失败会让核心写入 一起回滚。 2. 写事务提交失败时，自有存储可能留下本次的条目。重试会处理同一窗口、得到同一个稳定 ID，覆盖这些条目。 - 3. 把自有索引当作候选来源：命中的 ID 用 `get` 读取 core 中的当前快照，以快照内容和状态为准。这样回滚留下的条目和用户 - 修改都不会让插件读到过期内容。插件也可以定期按 core 数据重建索引。 + 3. 把自有索引当作候选来源：命中的 ID 用 `get` 读取 core 中的当前快照，以快照内容和状态为准。这样回滚留下的条目 + 不会让插件读到过期内容。插件也可以定期按 core 数据重建索引。 + 4. 大块内容（正文、分块、文件字节）按 `(artifact_id, revision)` 存成不可变的记录，先写入自有存储，再在写事务块中提交 + 带摘要的 Artifact 内容。没有提交成功的记录不会被任何 revision 引用，读取时按 Artifact 内容中的摘要核对。 - `generate` 本身可能因失败重试而执行多次，模型调用和插件发出的外部请求不是恰好一次。 ## 4. 用户对插件 Artifact 的修改 -用户修改通过 core 的通用 Artifact 接口完成，由 Server 在短事务中同步执行，不调用插件，也不进入 Supervisor。 +用户修改使用 core 的通用 Artifact 接口。Server 完成鉴权和版本预检后，把请求交给该插件的宿主进程；插件方法在宿主进程中 +执行，通过写入器完成写入，结果在一个短事务中提交并同步返回。用户修改不进入 Supervisor，也不排队。 -**编辑与 tags** +### 4.1 请求与方法 -| 接口 | 规则 | -| --- | --- | -| `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | 请求体为 `{"content": {...}}`，content 按原样保存，检索文本按声明的 `search_fields` 提取；`If-Match` 与当前 revision 不一致返回 412；Artifact 必须已存在。与内置 family 的编辑一样，提交的内容另存为一条内部 Content Source（`lineage_only`），作为新 revision 的直接来源记入 lineage；这类 Source 不进入任何插件的输入窗口 | -| `GET` / `PUT .../{artifact_id}/tags`、`POST /v1/scopes/{scope_id}/artifact-tags/query` | 与内置 family 相同；tags 不产生新的内容 revision | +```python +class CreateRequest(BaseModel): + scope_id: str + artifact_id: str # core 分配，调用方不提交 + content: dict[str, JsonValue] + +class UpdateRequest(BaseModel): + scope_id: str + artifact_id: str + expected_revision: int + content: dict[str, JsonValue] -**合并、遗忘与恢复** +class MergeRequest(BaseModel): + scope_id: str + inputs: Sequence[MergeInput] # 每项为 artifact_id、revision、state_version + into: str # core 分配的结果 ID + content: dict[str, JsonValue] + +class ForgetRequest(BaseModel): + scope_id: str + artifact_id: str + expected_revision: int + expected_state_version: int -这几项能力由共享的合并引擎提供（`ArtifactMergeService`，[RFC #1859](1859_artifact_merge.md)）。引擎本来就按 family 无关的方式 -设计：每个 family 提供一个适配器，负责生成合并后的内容和维护自己的索引；版本校验、级联恢复、lineage 记录和权限检查都在 -引擎中完成。运行时用一个注册表（`ArtifactMergeApplication`）按 family 名找到对应的引擎实例。内置 family 目前登记了 -Experience 和 Atomic Memory；Server 加载插件时，为每个插件 family 创建一个引擎实例（插件适配器 + 共享的 Artifact 访问策略） -并登记进去。 +class RestoreRequest(BaseModel): + scope_id: str + artifact_id: str + operation: Literal["restore", "undo_merge"] + revision: int | None = None # restore 时可指定恢复到的 revision -在注册表之上提供一组不区分 family 的接口，对所有登记过的 family 生效；没有登记的 family 返回 422： +class InvalidContent(Exception): # 插件拒绝内容；接口返回 422，消息和字段返回给调用方 + def __init__(self, message: str, *, field: str | None = None) -> None: ... +``` + +| 方法 | 插件必须完成的写操作 | +| --- | --- | +| `create_artifact(request, writer)` | 恰好一次 `put(request.artifact_id, ...)`，目标此前不存在 | +| `update_artifact(request, writer)` | 恰好一次 `put(request.artifact_id, ..., expected_revision=request.expected_revision)` | +| `merge_artifacts(request, writer)` | 恰好一次 `merge(输入 ID, into=request.into, ...)`，输入与请求一致 | +| `forget_artifact(request, writer)` | 恰好一次 `set_lifecycle(request.artifact_id, "deprecated")` | +| `restore_artifact(request, writer)` | `restore` 为恰好一次 `revert` 或 `set_lifecycle(..., "active")`；`undo_merge` 为恰好一次 `undo_merge(request.artifact_id)` | + +- 这些调用中的写事务只允许上表对应的那一项写操作，以及对同一 Artifact 的 `set_tags`；写其他 Artifact 时立即抛出异常。 + 方法正常返回但没有完成要求的写操作，core 回滚并返回 502，表示插件没有按约定完成。 +- 插件可以规范化内容（例如补全默认字段），写入的内容以插件提交的为准。 +- 插件应在写事务块中、拿到快照之后更新自己的存储，做法同 3.5。 + +### 4.2 core 负责的部分 + +- **鉴权**：与内置 family 相同。新建和新建合并结果需要 Scope 的贡献权限，结果归属于调用者；修改已有 Artifact 需要该 + Artifact 的写权限（`artifact.write`）。插件在这些调用中只获得完成本次请求所需的写入范围。 +- **版本校验**：调用插件前先检查 `If-Match`、`revision` 和 `state_version`，不一致直接返回 412 或 409，不调用插件；提交时 + 在写事务中再次校验，期间被其他写入改变则回滚并返回 409。 +- **溯源**：与内置 family 相同，新建、编辑、恢复和撤销合并时，core 在同一事务中把调用方提交的内容另存为一条内部 + Content Source（`lineage_only`，操作分别为 `artifact_create`、`artifact_replace`、`artifact_restore`、`artifact_undo_merge`）， + 作为新 revision 的直接来源记入 lineage。这类 Source 由 core 写入，不进入任何插件的输入窗口。 +- **合并注册表**：通用的合并、遗忘、恢复接口建立在共享合并引擎的注册表（`ArtifactMergeApplication`， + [RFC #1859](1859_artifact_merge.md)）之上，对所有登记过的 family 生效。内置 family 登记了 Experience 和 Atomic Memory， + 由各自的适配器处理；插件 family 在加载时登记，处理方式是转交给插件对应的方法。没有登记的 family 返回 422。 +- **不提供预览**：Atomic Memory 的专用接口支持先预览恢复影响、再凭签名 token 执行。通用接口不提供预览，也就不需要签名 + 密钥；恢复和撤销合并按执行时的当前状态进行，合并和遗忘依靠 `revision` 与 `state_version` 校验。 +- **tags**：`GET` / `PUT .../{artifact_id}/tags` 和 `POST /v1/scopes/{scope_id}/artifact-tags/query` 对插件 family 与内置 + family 相同，由 core 直接处理，不调用插件，也不产生新的内容 revision。 + +### 4.3 接口 | 接口 | 规则 | | --- | --- | -| `GET .../{artifact_id}/governance` | 返回当前 revision、生命周期状态、`state_version` 和合并目标 | -| `POST /v1/scopes/{scope_id}/artifacts/{family}/merges` | 输入为至少两个当前 Artifact 的 `artifact_id`、`revision`、`state_version`，以及结果的完整内容；新建结果 Artifact 并冻结所有输入。任何输入已变化即返回 409 | +| `POST /v1/scopes/{scope_id}/artifacts` | 请求体增加插件分支 `{"family": "<插件 family>", "content": {...}}`，family 不能是内置 family 的名字；ID 由 core 分配；响应与内置 family 的新建相同，包括 `Location` 和 `ETag` | +| `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | 请求体为 `{"content": {...}}`；`If-Match` 与当前 revision 不一致返回 412；Artifact 必须已存在 | +| `GET .../{artifact_id}/governance` | 返回当前 revision、生命周期状态、`state_version` 和合并目标；由 core 处理 | +| `POST /v1/scopes/{scope_id}/artifacts/{family}/merges` | 输入为至少两个当前 Artifact 的 `artifact_id`、`revision`、`state_version`，以及结果的完整内容；新建结果 Artifact 并冻结所有输入 | | `POST .../{artifact_id}/forget` | 携带 `revision` 和 `state_version`；把 Artifact 标为 deprecated（无替代者），不追加 revision，退出检索 | | `POST .../{artifact_id}/restorations` | `operation` 为 `restore`（默认，可指定 `revision` 恢复到旧内容）或 `undo_merge`；按组执行级联恢复和退役 | -- **权限**：由引擎检查，与内置 family 相同：修改已有 Artifact 需要该 Artifact 的写权限（`artifact.write`），新建合并结果 - 需要 Scope 的贡献权限，结果归属于调用者。 - 插件调用中的写入器使用同一个引擎类和适配器，只是权限不同：插件对本 family 的 Artifact 有完整写权限，包括用户合并 - 产生的结果。 -- **不提供预览**：Atomic Memory 的专用接口支持先预览恢复影响、再凭签名 token 执行。通用接口不提供预览，也就不需要签名 - 密钥；恢复和撤销合并按执行时的当前状态进行，合并和遗忘依靠 `revision` 与 `state_version` 校验。 -- **合并内容**：插件 family 的合并内容是不透明 JSON，core 只校验 JSON 结构并按 `search_fields` 提取检索文本；内置 family - 按各自的内容模型校验。 -- **并发**：用户修改与插件调用可能同时发生。插件在事务外读取、事务内用 `expected_revision` 写入时会发现冲突并整体回滚， - 下一次调用基于修改后的状态重新决策；用户修改不会被插件静默覆盖。 -- **不通知插件**：与内置 family 一样，用户修改只记录在 lineage 和治理状态中，不通知插件。插件在下一次调用时通过写入器 - 读到新状态。 -- **新建**：公开接口不为插件 family 直接新建 Artifact（合并结果除外）。调用方要补充知识，就写入 Source。 +| 情况 | 响应 | +| --- | --- | +| 插件没有实现对应方法 | 422，说明该 family 不支持此操作 | +| 插件抛出 `InvalidContent` | 422，带插件给出的消息和字段 | +| 版本不一致 | 412（`If-Match`）或 409（`revision`、`state_version`） | +| 插件抛出其他异常 | 500，只记录异常类型，不返回异常消息 | +| 插件没有完成要求的写操作 | 502 | +| 超过 `change_timeout_seconds`、宿主进程不可用或处于重启退避 | 503 | + +- 新建请求的插件分支没有判别映射：OpenAPI 无法为任意插件 family 写映射，生成的客户端对这一分支的类型约束较弱。这与 + tags 接口把 family 改为字符串的处理相同。 +- 只有删除语义的需求（例如撤下一份文档）使用遗忘；core 不提供物理删除。插件可以在 `forget_artifact` 中清理自己存储中的 + 正文和索引，core 中的 revision 与 Source 保留。 +- **并发**：用户修改与后台的 `generate` 可能同时发生。`generate` 在事务外读取、事务内用 `expected_revision` 写入时会发现 + 冲突并整体回滚，下一次调用基于修改后的状态重新决策；用户修改不会被插件静默覆盖。 ## 5. 插件执行器与消费游标 @@ -703,7 +872,33 @@ collect 调用结构相同：输入换成该 Scope 的已启用目标及其 chec collect 不需要写事务块，由执行器根据返回值一次提交。添加采集目标通过现有的调用意图（`ArtifactProcessingIntentRepository.request`） 请求一次采集。 -## 6. 公开读取与检索 +## 6. 插件宿主进程 + +同步调用需要在请求时限内返回，不能每次都启动 Worker：Worker 用 `spawn` 启动全新的 Python 进程，在 2 核开发机上仅导入 +core 的执行模块就约需 5.5 秒、150 MB 内存。因此同步调用交给常驻的宿主进程： + +| 调用 | 执行位置 | +| --- | --- | +| Source 写入的输入校验、`resolve`、Projection | 宿主进程 | +| 添加采集目标时的 `identify_target` | 宿主进程 | +| `create_artifact`、`update_artifact`、`merge_artifacts`、`forget_artifact`、`restore_artifact` | 宿主进程 | +| `collect`、`generate` | Supervisor Worker | + +- **数量与启动**：每个 Server 进程为每个需要同步调用的插件维护至多一个宿主进程，第一次有同步调用时启动，之后复用。 + 只提供 `generate` 的 Artifact 插件不启动宿主进程。端侧模式中，`resolve` 和 Projection 在采集 Worker 内执行，不启动宿主进程。 +- **执行**：宿主进程启动时导入一次 `plugin.py` 并核对声明；与 Worker 一样，由 core 代码打开数据库连接，写入器在宿主进程中 + 执行。一个宿主进程可以并发处理多个请求，并发上限由 Server 配置决定。 +- **超时与故障**：每次调用受各自的时限约束（Source 写入与 `identify_target` 使用 Server 的同步调用时限，用户修改使用 + `change_timeout_seconds`）。超时、崩溃或失去响应时，core 终止该进程，进行中的请求返回 503，并在退避后重新启动；退避期间 + 该插件的同步调用返回 503，后台调用不受影响。 +- **代码变更**：宿主进程在存续期间不重新导入插件。修改插件后需要重启 Server，与 Worker 的规则一致。 +- **隔离**：与 Worker 相同，宿主进程隔离普通故障，不是不可信代码的沙箱。 + +**已激活插件的内存记录**：每个 Server 进程在内存中记录本进程加载的插件：插件名、目录、类型、声明摘要、检查结果，以及 +宿主进程的当前状态（未启动、运行中、重启退避）。路由用户请求、启动宿主进程和派发 Worker 都以这份记录为准。它不持久化， +也不跨进程同步；Server 重启后按插件目录重新建立。运维接口只返回各进程一致的检查结果和声明，宿主进程状态通过日志和指标查看。 + +## 7. 公开读取与检索 插件 family 的公开读取完全由 core 处理，不执行插件代码： @@ -720,30 +915,37 @@ collect 不需要写事务块，由执行器根据返回值一次提交。添加 - Review、Recall 和 Prepared Context 第一版不对插件 family 开放，对应接口返回明确的“不支持”错误；`GET /v1/capabilities` 列出插件 family。 -## 7. 加载与生命周期 +## 8. 加载与生命周期 ```text Server 启动 -> 扫描 POWERCONTEXT_SERVER_PLUGINS_DIR 的直接子目录（未设置则不加载插件） -> 对每个目录：在有时限的检查子进程中导入 plugin.py，校验声明，不调用 create - -> Source 插件：在 Server 进程导入 definitions、target_input 与 identify_target，注册 Source 类型 - -> 注册 family 的读取与检索；有调度或触发时注册 Supervisor binding + -> 检查子进程返回可序列化的声明：Source 类型及其输入的 JSON Schema、目标输入的 JSON Schema、 + family、实现了哪些用户修改方法 + -> Source 插件：注册 Source 类型；插件代码不导入 Server 进程 + -> Artifact 插件：注册 family 的读取与检索，在合并注册表中登记；有调度或触发时注册 Supervisor binding + -> 写入本进程的已激活插件记录 -> 任一步失败：记录错误，该插件标记为 failed，继续下一个 Server 停止 + -> 停止接受同步调用，等待进行中的调用结束或超时后终止宿主进程 -> 按现有 Supervisor 关闭流程停止派发并回收 Worker ``` - 插件只有 `loaded` 和 `failed` 两种结果，只存在于进程内存中。Server 不持久化插件的启用状态、注册代次或配置快照； - 每次启动按当时的目录内容重新检查。 + 每次启动按当时的目录内容重新检查。插件代码只在检查子进程、宿主进程和 Worker 中导入，Server 进程不导入插件代码。 - 没有热加载，因此不需要注册代次来屏蔽旧 Worker，现有的 Supervisor 租约 fence 已经足以屏蔽领导权变更后的旧提交。 - Worker 每次调用重新导入插件代码并核对声明。插件目录在 Server 运行期间被修改、声明随之改变时，调用失败并提示重启； 只改了实现代码时，新派发的 Worker 会使用新代码。部署者应在修改后重启 Server。 -- 只读 API 角色的 Server 实例也会检查插件并注册 family 和 Source 类型，用于读取、检索和用户修改，但不派发 Worker。 +- 只读 API 角色的 Server 实例也会检查插件并注册 family 和 Source 类型，用于读取、检索、Source 写入和用户修改；它会按需 + 启动宿主进程，但不派发 Worker。 +- 运行中增减插件（热加载）和插件自定义 API 另行提案，建立在宿主进程之上。 -## 8. 执行约束 +## 9. 执行约束 -- **隔离**：`collect` / `generate` 在 Supervisor 启动的 Worker 子进程中执行，崩溃、内存泄漏和阻塞不影响 Server。 - 子进程隔离普通故障，不是不可信代码的沙箱：插件以 Worker 的操作系统权限运行，可以访问其文件、网络和环境变量。 +- **隔离**：`collect` / `generate` 在 Supervisor 启动的 Worker 子进程中执行，同步调用在宿主进程中执行，崩溃、内存泄漏和 + 阻塞不影响 Server。子进程隔离普通故障，不是不可信代码的沙箱：插件以子进程的操作系统权限运行，可以访问其文件、网络和 + 环境变量。 - **超时与预算**：`timeout_seconds` 是 Worker 总时限，`transaction_timeout_seconds` 是写事务块时限，`max_workers` 是该插件 跨 Scope 的并发上限，分别映射到 Supervisor binding 的现有参数或由执行器强制。 - **写事务的锁影响**：写事务块持有期间会锁住涉及的 head 和游标行；在 SQLite 上，写事务会阻塞整个数据库的其他写入。 @@ -755,9 +957,10 @@ Server 停止 core 不会因为重复失败自动跳过输入；持续失败需要修复插件或数据后重启。collect 中某个目标持续失败会阻塞同一 Scope 的其他目标，可以先停用该目标。 - **多实例部署**：OceanBase 多 Server 实例时，派发只发生在持有 Supervisor 领导租约的实例上；Source 写入、目标管理、 - 读取和用户修改可以在任意实例执行。插件自有存储需要能被执行 Worker 的实例访问。SQLite 部署只有一个 Server 进程。 + 读取和用户修改可以在任意实例执行，各实例使用自己的宿主进程。插件自有存储需要能被所有实例访问；core 分配的专属存储 + 位于共享的数据库后端上，天然满足这一点。SQLite 部署只有一个 Server 进程。 -## 9. 持久化 +## 10. 持久化 | 数据 | 处理 | | --- | --- | @@ -773,21 +976,25 @@ Server 停止 | 旧 Artifact 插件写入的 Artifact、head、lineage | 删除，不迁移 | | `pc_source_write_plugins` | 移除；Source 定义改由插件目录在启动时提供 | | Source 插件管理服务的 SQLite | 随管理服务一起移除 | -| 插件自有的索引与存储 | 不属于 core，由插件和部署者管理 | +| 插件专属存储（`plugins/<插件名>.sqlite3` 或 `powercontext_plugin_<插件名>` 库） | 不属于 core 的 schema；core 只负责分配，表结构、迁移和清理由插件和部署者负责 | +| 插件自有的其他索引与存储 | 不属于 core，由插件和部署者管理 | +| 端侧模式的本地 SQLite | 只保存端侧的采集目标和调度状态，不保存 Source；Source 与 checkpoint 保存在中心 | 目标的 binding ID 由插件名和目标键确定性派生，长度超过列宽时使用其摘要。新增与移除通过一次新的数据库迁移完成， 迁移说明中列出新增、删除的表、列和数据范围。 -## 10. 接口变更 +## 11. 接口变更 | 变更 | 内容 | | --- | --- | | 移除 | `/v1/admin/artifact-plugins` 及其子路径；Source 插件管理服务、插件实例 Runtime、poll / reply 通道；`source load` 命令；MCP `pc_source_instances`、`pc_source_instance_set_enabled`；`plugin.yml` 与 `powercontext.yml` 清单 | | 移除 | 自定义 family 的声明式索引（`ArtifactSearchDeclaration`、`TextIndex` 等）、`ArtifactFamily` 的 schema 声明、`ArtifactDraft` 和旧的 Worker 插件执行器 | -| 新增 | `powercontext.plugins`：`SourcePlugin`、`ArtifactPlugin`、`Inputs`、`Interval`、`Cron`、`PluginContext`、写入器与读取接口 | +| 新增 | `powercontext.plugins`：`SourcePlugin`、`ArtifactPlugin`、`Inputs`、`Interval`、`Cron`、`PluginContext`、`PluginStorage`、`PluginModels`、写入器与读取接口、用户修改的请求模型与 `InvalidContent` | +| 新增 | `powercontext source-agent` 端侧采集模式，以及 `POWERCONTEXT_SOURCE_AGENT_ID`、`POWERCONTEXT_SOURCE_AGENT_TOKEN` | +| 调整 | `POST /v1/scopes/{scope_id}/artifacts` 的请求体增加插件分支 `{"family", "content"}`，ID 由 core 分配，由插件的 `create_artifact` 处理 | | 新增 | `GET /v1/plugins`、`GET /v1/plugins/{name}`、`GET /v1/plugins/{name}/scopes`（`server.admin`，只读） | -| 新增 | `GET .../{artifact_id}/governance`、`POST /v1/scopes/{scope_id}/artifacts/{family}/merges`、`POST .../{artifact_id}/forget`、`POST .../{artifact_id}/restorations`：基于共享合并引擎注册表的通用接口，对所有登记过的 family 生效（内置为 Experience 和 Atomic Memory，以及加载的插件 family）。Atomic Memory 的专用接口保持不变 | -| 调整 | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` 接受插件 family，请求体为 `{"content": {...}}` | +| 新增 | `GET .../{artifact_id}/governance`、`POST /v1/scopes/{scope_id}/artifacts/{family}/merges`、`POST .../{artifact_id}/forget`、`POST .../{artifact_id}/restorations`：基于共享合并引擎注册表的通用接口，对所有登记过的 family 生效（内置为 Experience 和 Atomic Memory，以及加载的插件 family；插件 family 由插件对应的方法处理）。Atomic Memory 的专用接口保持不变 | +| 调整 | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` 接受插件 family，请求体为 `{"content": {...}}`，由插件的 `update_artifact` 处理 | | 调整 | tags 接口接受插件 family：`TaggableArtifactFamily` 从枚举改为与 `ArtifactReadFamily` 相同格式的字符串，服务端校验为内置 family 或已加载的插件 family。网络上的数据格式不变，生成的客户端少了这个枚举类型 | | 调整 | 通用 Artifact 读取与检索接口支持插件 family；`GET /v1/capabilities` 列出插件 family | | 调整 | `pc_source_add` 的 write 用法使用插件定义，collect 用法添加采集目标；`pc_source_capabilities` 列出插件 Source 类型与目标输入；`pc_source_status` 报告目标及其最近采集结果；`pc_source_target_set_enabled` 作用于 core 保存的目标 | @@ -795,42 +1002,47 @@ Server 停止 `openapi/powercontext.yaml` 按上表更新后重新生成 `src/powercontext/http/_generated/` 和各集成的操作表。 -## 11. 可观测性 +## 12. 可观测性 插件调用复用 Supervisor 与 Worker 现有的操作词汇、执行身份和结果字段，不为定时调用生成 request ID。新增字段： | 字段 | 含义 | | --- | --- | | `plugin_name` | 插件名 | -| `plugin_method` | `collect` 或 `generate` | -| `trigger` | `schedule`、`sources`、`target_added` | +| `plugin_method` | `collect`、`generate`、`resolve`、`identify_target`、`create_artifact`、`update_artifact`、`merge_artifacts`、`forget_artifact`、`restore_artifact` | +| `plugin_process` | `worker` 或 `host` | +| `trigger` | `schedule`、`sources`、`target_added`；同步调用为 `request` | | `operation_count` | 本次提交的写操作数（Artifact）或 Source 条数（collect） | | `target_count` | 本次 collect 携带的目标数 | | `transaction_ms` | 写事务块从开始到提交或回滚的耗时，毫秒 | -指标标签可以使用 `plugin_name` 和 `plugin_method`，不包含 Scope ID 或目标键。日志经 `log_safely` 输出，不包含 Source +宿主进程的启动、终止和重启各记录一条事件，带 `plugin_name` 和终止原因（超时、崩溃、Server 停止）。 +指标标签可以使用 `plugin_name`、`plugin_method` 和 `plugin_process`，不包含 Scope ID 或目标键。日志经 `log_safely` 输出，不包含 Source 内容、Projection、Artifact 内容、目标输入和环境变量值；插件异常只记录类型和已知错误分类，不记录异常消息。新增字段按 开发指南登记到允许列表，并在中英文可观测性文档中说明。调用结果只进入日志和指标，不持久化，运维接口不提供“最近一次 调用结果”。 # 缺点 -- **部分插件代码在 Server 进程中执行**：Source 写入需要同步返回，Source 定义的 `resolve`、Projection 和 `identify_target` - 不经过 Worker。它们应当是纯数据转换，但阻塞或异常仍会影响请求处理。Server 对这些调用设置超时并把异常转换为请求错误。 +- **宿主进程常驻占用资源**：每个 Server 进程为每个需要同步调用的插件多一个常驻进程，多实例部署时数量随实例数增加。 + 同步调用多一次进程间往返和序列化。 +- **同步调用依赖插件可用**：宿主进程崩溃或处于重启退避时，该插件的 Source 写入和用户修改返回 503；插件代码中的缺陷会 + 直接表现为用户请求失败。 - **写事务持锁**：写事务块期间持有数据库锁，SQLite 上会阻塞其他写入。插件若在块内执行慢操作，会拖慢整个 Server 的写入， 直到块超时回滚。 -- **插件自有存储与核心数据不保证一致**：两者不在同一事务中，提交失败可能在自有存储中留下条目；用户修改也不通知插件。 - 插件需要按稳定 ID 覆盖，并以 core 的当前数据为准。 -- **每次调用都要启动 Worker 并加载插件**：子进程启动和工厂初始化有固定开销，不适合秒级高频调用；插件不能在调用之间 - 复用连接或缓存。 +- **插件自有存储与核心数据不保证一致**：两者不在同一事务中，提交失败可能在自有存储中留下条目。插件需要按稳定 ID 覆盖、 + 按 revision 保存不可变记录，并以 core 的当前数据为准。 +- **后台调用每次都要启动 Worker 并加载插件**：子进程启动和工厂初始化有固定开销，不适合秒级高频的后台调用；插件不能在 + Worker 调用之间复用连接或缓存。 +- **专属存储只按约定隔离**：插件与 Server 使用同一个数据库登录，技术上能访问 core 的表。 +- **端侧与中心之间不是一个事务**：依赖 Source 身份去重保证不重复，`resolve` 不稳定的插件在端侧模式下会产生冲突。 - **不是安全沙箱**：Worker 隔离普通故障，不限制插件的文件、网络和凭据访问，也不提供硬性的内存或 CPU 配额。 - **依赖必须兼容**：所有插件与 Server 共用一个 Python 环境，依赖冲突需要部署者解决。 - **同一份代码部署多次需要额外处理**：family 写在声明中，同一份代码复制成两个目录会因 family 冲突而检查失败。需要多次 部署时，插件应从环境变量读取 family 等标识。 -- **内容结构不受 core 治理**：core 不校验 `content` 的 schema，历史 revision 的解释需要插件和应用自行保持兼容； - 用户通过 `PUT` 提交的内容也只由插件在后续读取时解释。 -- **core 检索能力有限**：core 只提供基于声明字段的通用文本检索；向量等检索需要插件自建索引。 -- **采集与 Server 部署在一起**：采集在 Server 的 Worker 中执行，需要 Server 能访问外部数据源。 +- **内容结构不受 core 治理**：core 不校验 `content` 的 schema，用户提交的内容由插件校验；历史 revision 的解释需要插件和 + 应用自行保持兼容。 +- **core 检索能力有限**：core 只提供基于声明字段的通用文本检索；向量等检索需要插件自建索引，公开检索接口暂时用不到它。 - **代码变更需要重启**：没有热加载，插件更新期间 Server 会短暂不可用。 # 理由与备选 @@ -844,25 +1056,34 @@ Server 停止 | 插件返回草稿，或由写入器在内存中暂存操作，最后统一提交 | 可以原子提交，但插件在提交前拿不到真实的 revision，难以同步写入自有索引；合并、撤销合并、生命周期等操作需要在内存中重新实现一遍 core 的语义；检索也看不到暂存内容 | | 用一个事务包住整个 `generate` | 模型调用可能持续数分钟，长时间持锁会阻塞其他写入，SQLite 上会阻塞整个数据库；插件自有存储是另一个数据库，一个事务本来也覆盖不了 | | `generate` 内的短写事务块（本提案） | 耗时工作在块外；块内直接复用 core 的写操作和校验，拿到真实 revision，读取能看到本块写入；块结束时与游标、确认一起提交，异常即回滚。代价是块内持锁，需要时限约束 | -| 用户命令经插件执行（手动请求排队，插件在下一次调用中处理） | 插件要实现命令解析，调用方只能异步等待；一个始终失败的命令会阻塞该 Scope 的后续生成；还需要持久化请求队列。遗忘、恢复、编辑都是 core 已有的通用操作，由 core 同步执行更简单 | +| 用户修改排队，插件在下一次后台调用中处理 | 调用方只能异步等待；一个始终失败的修改会阻塞该 Scope 的后续生成；还需要持久化请求队列，而 core 目前没有任务管理机制 | +| 用户修改由 core 直接执行，不经过插件 | 实现简单，但插件拿不到用户提交的内容：无法校验，自有存储和索引只能等之后重建才能跟上。存储由插件决定，修改也应由插件处理；内置 family 的新建和编辑同样经过各自的 family 代码 | +| 用一个 `write` 方法同时处理新建和编辑 | 新建不带 ID、由 core 分配，编辑必须指定 ID 和期望 revision；两者的输入和校验不同，合成一个方法需要插件自己分辨 | +| 同步调用每次派发一个 Worker | 复用现有机制，但 Worker 每次都启动新的 Python 进程，仅导入 core 模块就需要数秒，不能放在用户请求路径上 | +| Source 的 `resolve` 等代码在 Server 进程中执行 | 省去进程间往返，但插件阻塞或崩溃会影响 Server；插件的同级模块以顶层名字进入 `sys.modules`，多个插件的同名模块会互相冲突，后续也无法可靠地重新加载 | +| 常驻宿主进程处理同步调用，Worker 处理后台调用（本提案） | 同步调用不需要每次启动进程，Server 进程不导入插件代码；后台调用保留 Worker 的按次隔离和预算。代价是常驻进程的资源占用和进程管理 | +| 插件存储进入 core 的事务（插件在 core 库中建表） | 可以原子提交，但 core 要管理插件的迁移、升级和权限，三种后端都要支持，插件数据会混入 core 的 schema。插件的存储由插件决定，不放进 core | +| 一个插件声明多个 family | 可以在一个事务中写多个 family，但声明、权限和合并注册都要按多 family 设计。需要配合的场景用两个插件加跨 family 只读即可 | +| 把 core 的环境变量或数据库配置交给插件 | core 的配置不全来自环境变量；插件拿到 core 的数据库凭据就能绕过写入器。提供封装好的 `ctx.storage` 和 `ctx.models` | +| 端侧单独编译二进制 | 插件是带依赖的 Python 目录，冻结后的二进制无法导入插件的任意依赖，需要为每个插件和平台单独打包。同一个程序以 `source-agent` 模式启动即可 | | 插件专用的触发与管理接口 | 调用方要区分插件与内置能力；现有的 Source 与 Artifact 接口已经覆盖调用方需要的操作 | | 通用合并接口只对插件 family 开放 | 需要在接口里额外维护一份 family 白名单。能操作哪些 family 本来就由注册表决定，没有适配器的 family 无法登记；内置 family 的权限检查在引擎内，两个入口走同一套检查 | -| 常驻插件进程（类似 Dify 的插件守护进程） | 可以复用连接、支持多语言，但需要进程管理、序列化协议和取消协议。接口以请求 / 响应模型定义，后续可以替换执行方式而不改插件代码 | -| 支持热加载 | 需要重新引入加载状态、注册代次屏蔽和旧调用回收，与目标冲突 | +| 本提案中支持热加载 | 需要处理多实例各自重新扫描、旧 Worker 提交的屏蔽和宿主进程切换，单独提案更清楚 | 选择这一方案的原因： - 插件作者只需写一个 `plugin.py`：一个声明、一个工厂、一两个方法。 - 调用方只用现有的 Source 与 Artifact 接口，不感知插件。 - 隔离、调度、触发、退避和领导权都复用已经在生产中使用的 Supervisor。 -- 写操作直接复用 core 现有实现，插件写入和用户修改遵循同一套语义。 +- 写操作直接复用 core 现有实现，插件写入和用户修改遵循同一套语义；用户修改经过插件，插件存储能同步更新。 - Source 与 Artifact 留在 core 的统一存储中，现有读取接口、lineage、合并记录和访问控制直接适用。 - core 只管理两类核心数据及其进度，其余交给插件。 # 先例 - **Dify 插件**：按插件类型定义接口，插件实现对应方法，Dify 在工作流或应用需要时调用；插件运行在独立的插件守护进程中。 - 本提案沿用“宿主定义接口与调用时机”的思路，执行隔离复用 PowerContext 已有的 Supervisor Worker。 + 本提案沿用“宿主定义接口与调用时机”的思路：同步调用使用 core 管理的常驻宿主进程，后台调用复用 PowerContext 已有的 + Supervisor Worker。 - **Airbyte 连接器协议**：Source 连接器的 `read` 接收上次的 state 并输出记录和新 state，由平台持久化 state。 本提案的采集 checkpoint 采用相同分工，并按目标分别保存。 - **pytest 插件与 setuptools entry points**：插件以 Python 对象声明自己提供的能力，宿主导入后读取，不需要单独的清单文件。 @@ -871,16 +1092,18 @@ Server 停止 - **PowerContext 现有基础**：[Artifact processing Supervisor](1515_artifact_processing_supervisor.md) 的调度、调用意图、 Worker 与租约 fence；[公共合并方法](1859_artifact_merge.md)；`ArtifactRepository`、`ArtifactGovernanceRepository`、 handle 与 tags 的写入实现；通用 Artifact 读写接口；`SourceDefinition` / `SourceProjection`；Source journal、 - `pc_source_cursors` 和 `list_window_for_generation`；`pc_connector_checkpoints`；Source 管理接口与 `identify_target` - 目标识别规则。 + `pc_source_cursors` 和 `list_window_for_generation`；`pc_connector_checkpoints`；远程采集接口（`source-observations`、 + `connector-checkpoints`、`source-definitions/register`）；内置 family 编辑时记录的 `lineage_only` Content Source；Source + 管理接口与 `identify_target` 目标识别规则。 # 未决问题 | 问题 | 说明 | | --- | --- | -| 插件自定义检索 | 现有检索接口只接受文本 query，第一版由 core 提供通用文本检索。后续是否允许插件为公开检索接口提供实现，需要确定检索代码在哪个进程执行 | +| 插件自定义检索 | 现有检索接口只接受文本 query，第一版由 core 提供通用文本检索。插件提供的检索（例如向量或混合检索）随插件自定义 API 另行提案，在宿主进程中执行 | | 写事务块的时限与限额默认值 | 默认 30 秒是否合适；SQLite 与 OceanBase 是否采用不同默认值；单块操作数与字节上限 | -| 共享模型客户端 | 是否在 `PluginContext` 中提供 Server 已配置的模型客户端，让 LLM 插件不必自带凭据，并计入现有模型用量统计 | +| 宿主进程的默认参数 | 同步调用时限、单个宿主进程的并发上限和重启退避的默认值 | +| 专属存储的建库权限 | Server 的数据库登录没有建库权限时，除了由部署者事先建库，是否允许退回到 core 库中按插件名前缀建表 | | 采集目标的删除 | 第一版只支持停用和恢复。是否需要彻底删除目标及其 checkpoint，以及删除后已采集 Source 的处理 | | 插件 family 接入 Recall / Prepared Context | 需要定义候选内容和预算约束，建议另行提案 | | 窗口大小的回退 | 内置 Atomic Memory 在模型超时时会缩小下一次的 Source 窗口。插件无法影响窗口大小，过大的窗口会持续超时。是否允许插件以约定的异常请求缩小该 Scope 的窗口 | @@ -891,6 +1114,7 @@ Server 停止 # 后续可能 -- 常驻插件进程，在保持接口不变的前提下复用连接和初始化结果。 +- 热加载与插件自定义 API（含插件页面）：建立在宿主进程之上，另行提案。 +- 后台调用也改用常驻进程，在保持接口不变的前提下复用连接和初始化结果。 - 更多插件类型，例如为 Recall 提供候选内容、为 Prepared Context 提供文本片段。 - 插件打包与安装工具，例如从 Git 地址或 wheel 安装到插件目录。 From 93e186aaaf19508e35904c69ae5f48ae4c45e637 Mon Sep 17 00:00:00 2001 From: jasper-yl Date: Tue, 10 Nov 2026 00:44:08 -0800 Subject: [PATCH 06/12] docs(rfc): link the plugin hot reload RFC 2579 Co-Authored-By: Claude Opus 5.5 --- docs/en/rfcs/2549-plugin-interfaces.md | 6 +++--- docs/zh/rfcs/2549-plugin-interfaces.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/en/rfcs/2549-plugin-interfaces.md b/docs/en/rfcs/2549-plugin-interfaces.md index 66534c59de..da3b3490d2 100644 --- a/docs/en/rfcs/2549-plugin-interfaces.md +++ b/docs/en/rfcs/2549-plugin-interfaces.md @@ -1077,8 +1077,8 @@ Server stops newly dispatched Workers use the new code. Deployers should restart Server after a change. - Server instances in the read-only API role also check plugins and register families and Source types for reads, search, Source writes, and user changes; they start host processes on demand but do not dispatch Workers. -- Adding or removing plugins at runtime (hot reload) and plugin-defined APIs are left to a separate proposal built on - host processes. +- Adding or removing plugins at runtime (hot reload) and plugin-defined APIs are covered by + [RFC #2579](https://github.com/oceanbase/powercontext/pull/2579), built on host processes. ## 9. Execution constraints @@ -1280,7 +1280,7 @@ Reasons for this choice: # Future possibilities -- Hot reload and plugin-defined APIs (including plugin pages): built on host processes, in a separate proposal. +- Hot reload and plugin-defined APIs (including plugin pages): built on host processes, see [RFC #2579](https://github.com/oceanbase/powercontext/pull/2579). - Long-lived processes for background calls too, reusing connections and initialization while keeping the interfaces unchanged. - More plugin types, such as candidate content for Recall or text fragments for Prepared Context. diff --git a/docs/zh/rfcs/2549-plugin-interfaces.md b/docs/zh/rfcs/2549-plugin-interfaces.md index a657f34065..62bfd44f88 100644 --- a/docs/zh/rfcs/2549-plugin-interfaces.md +++ b/docs/zh/rfcs/2549-plugin-interfaces.md @@ -939,7 +939,7 @@ Server 停止 只改了实现代码时，新派发的 Worker 会使用新代码。部署者应在修改后重启 Server。 - 只读 API 角色的 Server 实例也会检查插件并注册 family 和 Source 类型，用于读取、检索、Source 写入和用户修改；它会按需 启动宿主进程，但不派发 Worker。 -- 运行中增减插件（热加载）和插件自定义 API 另行提案，建立在宿主进程之上。 +- 运行中增减插件（热加载）和插件自定义 API 见 [RFC #2579](https://github.com/oceanbase/powercontext/pull/2579)，建立在宿主进程之上。 ## 9. 执行约束 @@ -1114,7 +1114,7 @@ Server 停止 # 后续可能 -- 热加载与插件自定义 API（含插件页面）：建立在宿主进程之上，另行提案。 +- 热加载与插件自定义 API（含插件页面）：建立在宿主进程之上，见 [RFC #2579](https://github.com/oceanbase/powercontext/pull/2579)。 - 后台调用也改用常驻进程，在保持接口不变的前提下复用连接和初始化结果。 - 更多插件类型，例如为 Recall 提供候选内容、为 Prepared Context 提供文本片段。 - 插件打包与安装工具，例如从 Git 地址或 wheel 安装到插件目录。 From 0574be762f8a4196313b742cbb44a21d3a941df8 Mon Sep 17 00:00:00 2001 From: jasper-yl Date: Thu, 12 Nov 2026 02:35:47 -0800 Subject: [PATCH 07/12] docs(rfc): refresh the plugin interfaces RFC with the implemented design Bring the RFC in line with the implementation and the decisions since the last revision: user changes run through the plugin in a host process that starts for every loaded plugin; Artifact plugins may declare their own Source Definitions; Inputs can name a Projection per Source type; plugin storage, models, and the generic create and Source write contracts are spelled out. Co-Authored-By: Claude Opus 5.5 --- docs/en/rfcs/2549-plugin-interfaces.md | 287 +++++++++++++------------ docs/zh/rfcs/2549-plugin-interfaces.md | 236 ++++++++++---------- 2 files changed, 270 insertions(+), 253 deletions(-) diff --git a/docs/en/rfcs/2549-plugin-interfaces.md b/docs/en/rfcs/2549-plugin-interfaces.md index da3b3490d2..fc571768a2 100644 --- a/docs/en/rfcs/2549-plugin-interfaces.md +++ b/docs/en/rfcs/2549-plugin-interfaces.md @@ -32,7 +32,7 @@ APIs exactly as they do for built-in capabilities. reusing its existing isolation, timeouts, concurrency budgets, leader lease, and failure backoff. - `resolve` and Projections during Source writes, `identify_target` when adding a collection target, and user changes to plugin Artifacts are synchronous calls. They run in a long-lived **plugin host process** that core manages: each - Server process starts one host process per plugin on demand, and deployers do not manage it. + Server process starts one host process for each plugin that needs synchronous calls, and deployers do not manage it. - **Writes**: an Artifact plugin performs its writes inside an `async with writer.transaction()` block. When the block ends, core commits the writes, the consumption cursor, and the completion acknowledgment together; an exception rolls everything back. Plugins can neither read nor change the cursor. @@ -235,7 +235,10 @@ write APIs, and core uses the definitions to validate, convert, and save it. `re Sources, like all built-in Sources, are append-only and cannot be changed or deleted; to correct content, submit a new Source. Large content such as files and images goes into the plugin's own storage, and the Source records only -metadata such as its location, digest, and media type. +metadata such as its location, digest, and media type. In this proposal such content is read from external systems by +`collect`; `resolve` must be a pure function and the existing Source write APIs accept only JSON, so an entry point for +callers to upload files directly comes from plugin-defined APIs, see +[RFC #2579](https://github.com/oceanbase/powercontext/pull/2579). ### 3.2 Declaration @@ -359,7 +362,7 @@ powercontext source-agent --server https://memory.example.com | Interface | Required | Description | | --- | --- | --- | -| `create(ctx) -> ArtifactGenerator` | Yes | A factory; run once per invocation in a Worker, and once per Scope and then reused in the host process | +| `create(ctx) -> ArtifactGenerator` | Yes | A factory; run once per invocation in a Worker, and only once and then reused in the host process | | `generate(request, writer) -> None` | Yes | Reads `request.sources` (the unconsumed Source window) and writes Artifacts in the write transaction block | | `create_artifact(request, writer) -> None` | No | A user create; core allocates `request.artifact_id` | | `update_artifact(request, writer) -> None` | No | A user edit of the given Artifact; `request.expected_revision` is the revision the caller saw | @@ -377,7 +380,7 @@ requires. | Field | Default | Description | | --- | --- | --- | | `family` | Required | The Artifact family the plugin owns; must not duplicate a built-in family or another plugin's | -| `inputs` | `None` | `Inputs(source_types=..., projection=..., max_sources=50)`: the Source types consumed, the Projection each Source is delivered in, and the maximum Sources per window; empty means the plugin consumes no Sources | +| `inputs` | `None` | `Inputs(source_types=..., projection=..., projections={...}, max_sources=50)`: the Source types consumed, the Projection each Source is delivered in, and the maximum Sources per window; empty means the plugin consumes no Sources. `projection` applies to every type, `projections` names a different Projection per Source type, and every type needs one or the other | | `on_sources` | `False` | Trigger after new Sources arrive; requires `inputs` | | `schedule` | `None` | `Interval(seconds=...)` or `Cron(expression=..., timezone=...)`; declare at least one of this and `on_sources` | | `scopes` | `None` | Scopes the plugin may serve; required for plugins that consume no Sources | @@ -386,6 +389,7 @@ requires. | `transaction_timeout_seconds` | 30 | Time limit for the write transaction block, at most `timeout_seconds` | | `max_workers` | 1 | Concurrency limit across Scopes | | `change_timeout_seconds` | 30 | Time limit for one user-change call, including its write transaction block | +| `definitions` | `()` | The plugin's own Source Definitions, registered when the plugin loads, under the same rules as a Source plugin's `definitions`. The plugin submits them from its routes with `ctx.sources.submit`, and callers can also submit them through the generic Source write API, so one Artifact plugin can record what users asked for without a separate Source plugin | | `create` | Required | See the table above | ### 4.3 Example @@ -518,8 +522,13 @@ invocation after it finishes. - **Changes**: restart Server after changing code. Server does not watch files or reload plugins at runtime. Collection targets are business data, so adding or removing them needs no restart. - **Disabling**: move the plugin out of the directory and restart. Saved Sources, Artifacts, collection targets, cursors, - and checkpoints remain readable; redeploying a plugin with the same name continues where it left off. The plugin's - dedicated storage is not deleted with it; the deployer keeps or cleans it up. + and checkpoints are all kept, but the plugin's family and Source types are no longer registered: reading, searching, + or changing that family through the generic APIs returns 404, as for a family that does not exist. A plugin + Artifact's content usually records only digests or locations, while bodies, chunks, and files live in the plugin's own + storage and cannot be interpreted without the plugin, so there are no reads without the plugin. New writes of the plugin's + Source types return 422. Redeploying a plugin with the same name + restores reads and continues generation from the existing cursor. The plugin's dedicated storage is not deleted with + it; the deployer keeps or cleans it up. - **Host processes**: core starts, reuses, and restarts them on demand; deployers neither configure nor manage them. Host process starts, terminations, and restarts appear only in logs and metrics. @@ -558,15 +567,16 @@ class SourcePlugin(BaseModel): # fields in guide 3.2 `scopes`, time-limit relations, and family naming. `create` must be an async function. - The parts of a declaration other than code objects such as `create`, `identify_target`, and `definitions` are serializable. Server obtains them from the check child process to register bindings and answer operational queries. - A Worker re-imports `plugin.py` on every invocation; if the declaration differs from the one checked at startup, the call - fails and asks for a Server restart. +- During the check, core records the plugin directory's **code fingerprint** (computed from file contents). Workers + and host processes check it before importing the plugin, and a mismatch fails the call and asks for a Server restart, + so calls in one Server process never mix old and new code. ### 1.2 Context ```python class PluginContext: name: str # plugin name - scope_id: str # Scope of this invocation + scope_id: str | None # Scope of a Worker invocation; None in the host process, where each request carries it plugin_dir: Path # plugin directory (read-only by convention) logger: logging.Logger # logger bound to the plugin name storage: PluginStorage # the plugin's dedicated storage @@ -578,14 +588,22 @@ class PluginStorage(Protocol): backend: Literal["sqlite", "oceanbase", "seekdb"] def engine(self) -> AsyncEngine: ... # points only to this plugin's dedicated area +class ChatMessage(BaseModel): + role: Literal["system", "user", "assistant"] + content: str + class PluginModels(Protocol): - embedding_profile: EmbeddingProfile | None # profile ID and vector dimension; None without an embedding model + embedding_available: bool # whether Server has an embedding model + embedding_dimension: int | None # vector dimension; None without an embedding model + generation_available: bool # whether Server has a generation model async def embed(self, texts: Sequence[str], /) -> Sequence[Sequence[float]]: ... async def generate(self, messages: Sequence[ChatMessage], /, *, max_tokens: int | None = None) -> str: ... + # Calling a model Server lacks raises ModelUnavailable; synchronous calls return 503 ``` - The factory runs once per Worker invocation, and instances are not reused across invocations; in the host process it - runs once per Scope, and the instance is reused for as long as the host process lives. Resources that need closing + runs only once, and the instance serves requests for every Scope for as long as the host process lives, so resource use + does not grow with the number of Scopes. Each request in the host process carries its `scope_id`. Resources that need closing (including connections to plugin-owned storage) are registered through `ctx.enter_async_context` and released by the executor when the Worker invocation ends or the host process stops. - **Dedicated storage**: core uses Server's existing database connection settings to allocate an area for the plugin on @@ -595,10 +613,16 @@ class PluginModels(Protocol): first use. When Server's database login cannot create databases, the deployer creates it in advance. - The plugin manages its own tables and migrations; core does not read, migrate, or back up this area. It is not part of core's transactions, and its consistency with core data follows 3.5. + - This area belongs to the whole plugin and is not isolated by Scope. The plugin partitions its data by `scope_id` and, + when reading, confirms that the data belongs to the request's Scope instead of returning data on a caller-supplied + ID alone. - The plugin and Server use the same database login, so this split prevents a plugin from writing core's tables by mistake but does not stop a malicious plugin; the plugin directory is treated as trusted code. - **Models**: `models` wraps Server's inference configuration; the plugin never sees model addresses, headers, or - credentials. `embedding_profile` lets the plugin use the same embedding configuration for indexing and queries. Calls + credentials. `models` is None when Server configures neither model; when only one is configured, the plugin checks + `embedding_available` and `generation_available`. Plugins should record the embedding configuration, such as + `embedding_dimension`, with their indexes and rebuild them when it changes, so vectors from different models are never + compared. Calls follow Server's timeout settings and count toward the existing model usage statistics. A plugin may also read its own model configuration from environment variables and create its own clients. - Plugins get no connection to core's database and no interface for writing cursors or checkpoints. Plugins write core @@ -706,7 +730,7 @@ class GenerateRequest(BaseModel): class GenerationSource(BaseModel): ref: SourceRef # exact Source reference position: int # the Source's journal position, for processing in collection order - projection: JsonValue # the Projection named by inputs.projection + projection: JsonValue # the Projection declared for this Source type ``` ### 3.2 Writer and write transaction @@ -751,15 +775,15 @@ Rules: The write transaction covers every generic Artifact write operation. Each reuses core's existing implementation and validation and raises immediately when a rule is violated: -| Operation | Core implementation reused | Semantics | -| --- | --- | --- | -| `put(artifact_id, *, content, source_refs=(), artifact_refs=(), expected_revision=None)` | The change path of the shared merge methods | Creates revision 1 if absent; otherwise appends a revision on the current head. With `expected_revision`, a different current revision is a conflict, enabling optimistic concurrency for reading outside the transaction and writing inside it. Identical content to the current revision is a no-op that returns the current snapshot | -| `revert(artifact_id, *, to_revision)` | The shared restoration method (selected revision) | Appends a new revision with a historical revision's content and records `restored_from_revision`; a forgotten or merge-frozen target is restored as well | -| `set_lifecycle(artifact_id, state, *, replacement_artifact_id=None)` | The shared forget and restore methods; `ArtifactGovernanceRepository.transition` for a replacement or retirement | `deprecated` forgets, `active` restores, and `retired` is irreversible. Restoring uses shared restoration: it records a new revision with the same content, and an input frozen by a merge undoes that merge as one group. Forgetting and retiring append no revision | -| `merge(inputs, *, into, content, source_refs=(), artifact_refs=())` | The shared merge method ([RFC #1859](1859_artifact_merge.md)) | Takes the current revisions of `inputs`, creates `into`, records the merge, and freezes the inputs. `into` must be a new ID; when the same inputs are merged again after an undo, the plugin should include the input revisions in `stable_id` to avoid colliding with the retired result | -| `preview_undo_merge(artifact_id)` / `undo_merge(artifact_id)` | Preview and restoration of the shared merge methods | Undoes the merge that produced the Artifact, restoring its inputs and retiring the result, and returns exact references to what was restored and retired | -| `bind_handle(handle_key, ref)` | Artifact handle binding | Binds a stable handle to the given revision | -| `set_tags(artifact_id, tags)` | Artifact tag replacement | Replaces the Artifact's tags as a whole | +| Operation | Semantics | +| --- | --- | +| `put(artifact_id, *, content, source_refs=(), artifact_refs=(), expected_revision=None)` | Creates revision 1 if absent; otherwise appends a revision on the current head. With `expected_revision`, a different current revision is a conflict; without it, the revision this invocation read outside the transaction is used (see 3.5), and nothing is checked if it was not read. Identical content to the current revision writes nothing and returns the current snapshot, and the supplied `source_refs` and `artifact_refs` are not recorded (see the write constraints below). `put` in user changes is the exception, see 4.1 | +| `revert(artifact_id, *, to_revision)` | Appends a new revision with a historical revision's content and records `restored_from_revision`; a forgotten or merge-frozen target is restored as well. Returns an `ArtifactChange`: the target's new snapshot plus exact references to the other Artifacts restored and retired with it | +| `set_lifecycle(artifact_id, state, *, replacement_artifact_id=None)` | `deprecated` forgets, `active` restores, and `retired` is irreversible. Restoring uses shared restoration: it records a new revision with the same content, and an input frozen by a merge undoes that merge as one group. Forgetting and retiring append no revision. Returns the same as `revert`, so the plugin can update entries in its own index affected by the cascade | +| `merge(inputs, *, into, content, source_refs=(), artifact_refs=())` | Follows the shared merge method ([RFC #1859](1859_artifact_merge.md)): takes the current revisions of `inputs`, creates `into`, records the merge, and freezes the inputs. `into` must be a new ID; when the same inputs are merged again after an undo, the plugin should include the input revisions in `stable_id` to avoid colliding with the retired result | +| `preview_undo_merge(artifact_id)` / `undo_merge(artifact_id)` | Undoes the merge that produced the Artifact, restoring its inputs and retiring the result, and returns exact references to what was restored and retired | +| `bind_handle(handle_key, ref)` | Binds a stable handle to the given revision | +| `set_tags(artifact_id, tags)` | Replaces the Artifact's tags as a whole | Write constraints: @@ -768,10 +792,13 @@ Write constraints: reference Sources already in the target Artifact's current lineage (to keep the original inputs). `artifact_refs` may reference existing revisions of any family in this Scope; `to_revision` and the target of `bind_handle` must be revisions that exist in this family. +- Unchanged content produces no revision, as in built-in Atomic Memory: when the model finds that an existing memory + already expresses a new Source, that memory gets no new revision and the new Source does not enter its lineage. A + plugin that needs a new Source recorded as evidence for an Artifact should change its content as well. - Core extracts search text from the content according to the declared `search_fields`; plugins do not supply it separately. The same content yields the same search text whether a plugin or a user wrote it. -- Each write transaction block has limits on operations (1000), single content size (256 KiB), and total bytes (8 MiB); - exceeding one raises and rolls back, with no truncation or partial commit. +- Each write transaction block has limits on operations, single content size, and total bytes; exceeding one raises and + rolls back, with no truncation or partial commit. - Core does not validate the business structure of `content`, nor store or pin a schema. The plugin owns how its content structure evolves and may keep a version field in `content`. @@ -793,6 +820,11 @@ class ArtifactSnapshot(BaseModel): replacement_artifact_id: str | None governance_generation: int # governance state version, to detect changes after a read +class ArtifactChange(BaseModel): # returned by revert / set_lifecycle + artifact: ArtifactSnapshot # the target's new snapshot + restored: Sequence[ArtifactRef] # other Artifacts restored with it + retired: Sequence[ArtifactRef] # other Artifacts retired with it + class ArtifactReads(Protocol): async def get(self, artifact_id: str, *, revision: int | None = None) -> ArtifactSnapshot | None: ... async def revisions(self, artifact_id: str, *, cursor: str | None = None, limit: int = 50) -> ArtifactPage: ... @@ -829,9 +861,16 @@ class ArtifactReads(Protocol): that were already processed. Plugins derive Artifact IDs from business identity with `writer.stable_id(...)`; `put` is a no-op for identical content, `set_lifecycle` is a no-op when the target state equals the current one, and `merge` is a no-op when every input is already merged into the same result. No duplicate Artifacts or meaningless revisions result. -- **Plugin-owned storage**: the plugin's indexes and other storage (including the dedicated storage core allocates) are - outside core's transaction. - Core does not guarantee their consistency with core data; the plugin does. Recommended practice: +- **Writing after a read outside the transaction**: when a plugin reads an Artifact outside the transaction and then + `put`s it in the write transaction block without `expected_revision`, core uses the revision read outside the + transaction as the expected value. If a user changed the Artifact after the read, the write conflicts, the invocation + rolls back, and it is rescheduled. This matches built-in Atomic Memory, which checks before committing whether the + memories it read have changed, so a plugin write based on old content never silently overwrites a user change. +- **Plugin-owned storage**: the plugin's indexes and other storage (the dedicated storage core allocates, or any database, + object store, or vector store the plugin connects to itself) are outside core's transaction. The plugin writes and + commits owned storage with its own clients; core takes no part and does not guarantee their consistency. Core's + transaction is the only commit point for core data: data in owned storage takes effect only once a committed revision + in core references it. This needs no two-phase commit; the following practice keeps them consistent: 1. Write owned storage inside the write transaction block after receiving the `ArtifactSnapshot`, upserting by `artifact_id`. A failed write to owned storage then rolls back the core writes too. 2. If the write transaction fails to commit, owned storage may keep entries from that attempt. The retry processes the @@ -839,10 +878,20 @@ class ArtifactReads(Protocol): 3. Treat the owned index as a source of candidates: read each hit's current snapshot from core with `get`, and use that snapshot's content and state. Entries left by rollbacks then cannot make the plugin read stale content. Plugins may also rebuild their index from core data periodically. - 4. Store large content (bodies, chunks, file bytes) as immutable records keyed by `(artifact_id, revision)`: write them - to owned storage first, then commit Artifact content that carries their digest in the write transaction block. - Records that were never committed are referenced by no revision, and reads verify them against the digest in the - Artifact content. + 4. Store large content (bodies, chunks, file bytes) as immutable records keyed by content digest: write and commit them + to owned storage first, then commit an Artifact whose content carries that digest in the write transaction block. + Do not key them by `(artifact_id, revision)`: after a write transaction rolls back, the same revision number is + allocated again to a write with different content. With content-digest keys, a retry that writes the same content + gets the same key, and a failed commit leaves only records that no revision references, which the plugin cleans up + periodically as unreferenced. +- **Requirements on plugin-owned storage**: the practice above does not depend on the kind of storage, but the storage + must meet three conditions, which the plugin confirms when choosing it and core does not check: + - It can write idempotently under a key the plugin chooses (upsert or insert-if-absent). With a system that only + assigns IDs itself, the plugin must settle the ID before writing. + - Data is durable when a write returns. + - Data is readable right after a write. With systems that have a refresh delay (for example, Elasticsearch's refresh + interval or vector stores that index asynchronously), core may already reference a record that owned storage cannot + yet return, and the plugin retries or treats it as temporarily missing when reading. - `generate` itself may run several times because of retries; model calls and external requests a plugin makes are not exactly-once. @@ -902,6 +951,9 @@ class InvalidContent(Exception): # the plugin rejects the content; the required write, core rolls back and returns 502, meaning the plugin did not complete the contract. - The plugin may normalize the content (for example, by filling default fields); the written content is what the plugin submits. +- `put` in `create_artifact` and `update_artifact` always appends a revision, even when the content equals the current + revision. This matches edits to built-in families: every user edit has a new `lineage_only` Content Source (see 4.2), + which needs a new revision to record it. - The plugin should update its own storage inside the write transaction block after receiving the snapshot, as in 3.5. ### 4.2 What core does @@ -917,7 +969,7 @@ class InvalidContent(Exception): # the plugin rejects the content; the `artifact_replace`, `artifact_restore`, or `artifact_undo_merge`) and records it in lineage as the new revision's direct source. Core writes these Sources, and they never enter a plugin's input window. - **Merge registry**: the generic merge, forget, and restoration APIs sit on the registry of the shared merge engine - (`ArtifactMergeApplication`, [RFC #1859](1859_artifact_merge.md)) and work for every registered family. Experience and + ([RFC #1859](1859_artifact_merge.md)) and work for every registered family. Experience and Atomic Memory are the registered built-in families and use their own adapters; a plugin family is registered at load and handed to the matching plugin method. Unregistered families return 422. - **No preview**: Atomic Memory's dedicated APIs can preview a restoration's effect and then execute it with a signed @@ -947,57 +999,44 @@ class InvalidContent(Exception): # the plugin rejects the content; the | The plugin did not perform the required write | 502 | | `change_timeout_seconds` is exceeded, or the host process is unavailable or in restart backoff | 503 | -- The plugin branch of the create request has no discriminator mapping: OpenAPI cannot map arbitrary plugin families, so - generated clients type this branch loosely. This matches the tag APIs, whose family became a string. +- The plugin branch of the create request has no discriminator mapping: OpenAPI cannot map arbitrary plugin families. The + built-in branch keeps its discrimination by family, the plugin branch sits beside it, and the family name selects the + branch, so a built-in family's request is never accepted by the plugin branch. This matches the tag APIs, whose family + became a string. - A need that only means removal (for example, taking down a document) uses forget; core provides no physical delete. In `forget_artifact` the plugin can clean the body and indexes out of its own storage, while the revisions and Sources in core are kept. -- **Concurrency**: user changes and background `generate` calls can overlap. A `generate` that reads outside the - transaction and writes with `expected_revision` inside it detects the conflict and rolls back; its next invocation - decides again from the changed state, so a user change is never silently overwritten by the plugin. - -## 5. Plugin executor and consumption cursor +- **Concurrency**: user changes and background `generate` calls can overlap. When a `generate` writes an + Artifact it read outside the transaction, core checks the revision it read (see 3.5); a conflict rolls the whole + invocation back, and the next invocation decides again from the changed state, so a user change is never silently + overwritten by a plugin write based on old content. -The plugin executor is core code in the Worker that implements the Supervisor's Worker entry point, replacing the old -`run_artifact_plugin_worker`. One Artifact plugin invocation looks like this: +## 5. Consumption cursor -```text -Worker receives (plugin:, scope_id, request_generation) - -> short transaction: ScopeInvocation.start; read the cursor for (scope_id, plugin:) in pc_source_cursors - -> read the journal high watermark and choose the window (after, through], through ≤ after + max_sources - -> list_window_for_generation reads the window, filters by inputs.source_types, computes inputs.projection - -> empty window with inputs declared: a short transaction does the cursor CAS and acknowledge; the plugin is not loaded - -> import plugin.py, compare the declaration, call create, build the writer, call generate (bounded by timeout_seconds) - -> reads outside the transaction: separate short read-only transactions - -> the write transaction block starts: open a transaction; guard (lease and invocation intent) - writes inside the block run in that transaction - the block ends: cursor CAS to through; acknowledge; commit - -> generate returns without opening a write transaction block: a short transaction does the cursor CAS and acknowledge -``` +Each background call of an Artifact plugin processes one Source window after the cursor in that Scope: +- The window holds at most `inputs.max_sources` Sources. When it has no matching Source, the plugin is not loaded and the + cursor advances with an acknowledgment. - Sources in the window whose type does not match are filtered out, but the cursor still advances past the whole window; a matching Source that lacks the required Projection fails the invocation instead of being skipped. -- The cursor advances only in the executor's commit transaction; plugins never see its value and have no way to write it. - The cursor CAS ensures that even if leadership changes or the invocation is dispatched twice between reading the window - and committing, the same window commits only once. -- Pending work is discovered in two ways: a Source write in the Server process marks the plugin bindings that watch the - Scope as pending, and at startup the existing `SourceProcessingPendingProvider` fills in Sources written by other - processes from the gap between the journal high watermark and the cursor. The Supervisor admits pending Scopes only when - a binding's interval or cron is due, so `on_sources` maps to an admission interval of about 2 seconds. These decisions - rest on persisted data, so pending input is not lost when Server restarts. +- The cursor advances only by CAS in the executor's commit transaction; plugins never see its value and have no way to + write it. Even if leadership changes or the invocation is dispatched twice between reading the window and committing, + the same window commits only once. +- Whether work is pending is decided from persisted data (the gap between the Source journal position and the cursor), so + pending input is not lost when Server restarts. Plugins with `on_sources` are dispatched within about 2 seconds of a new + Source write. - Without `scopes`, automatic scheduling covers every Scope whose journal is past the cursor; with `scopes`, only Scopes on the list are accepted. -A collect invocation has the same structure: its input is the Scope's enabled targets and their checkpoints, and its -commit is the Sources and every target's new checkpoint. Collect needs no write transaction block; the executor commits -the return value at once. Adding a collection target requests one collection through the existing invocation intent -(`ArtifactProcessingIntentRepository.request`). +Collect has the same structure: its input is the Scope's enabled targets and their checkpoints, and its commit is the +Sources and every target's new checkpoint, which the executor commits from the return value at once. Adding or restoring +a collection target requests one collection. ## 6. Plugin host process -Synchronous calls must return within the request's time limit, so they cannot start a Worker each time: a Worker starts a -fresh Python process with `spawn`, and on a 2-core development machine importing core's execution modules alone takes -about 5.5 seconds and 150 MB of memory. Synchronous calls therefore go to a long-lived host process: +Synchronous calls must return within the request's time limit, so they cannot start a Worker each time: every Worker is a +fresh Python process, and importing core's execution modules takes seconds. Synchronous calls therefore go to a +long-lived host process: | Call | Where it runs | | --- | --- | @@ -1006,19 +1045,20 @@ about 5.5 seconds and 150 MB of memory. Synchronous calls therefore go to a long | `create_artifact`, `update_artifact`, `merge_artifacts`, `forget_artifact`, `restore_artifact` | Host process | | `collect`, `generate` | Supervisor Worker | -- **Count and startup**: each Server process keeps at most one host process for each plugin that needs synchronous calls, - starts it on the first synchronous call, and reuses it afterward. An Artifact plugin that provides only `generate` - starts no host process. In edge mode, `resolve` and Projections run inside the collection Worker, and no host process +- **Count and startup**: each Server process keeps at most one host process for each loaded plugin, starts it in the + background once the plugin loads, and reuses it afterward. Whether a plugin implements user-change methods is known only + after `create` runs, so an Artifact plugin that provides only `generate` also starts one, at the cost of an idle process. + In edge mode, `resolve` and Projections run inside the collection Worker, and no host process starts. -- **Execution**: the host process imports `plugin.py` once at startup and compares the declaration. As in a Worker, core - code opens the database connection, and the writer runs inside the host process. One host process can serve several - requests concurrently, up to a limit set in the Server configuration. +- **Execution**: the host process imports `plugin.py` once at startup and compares the declaration, and the writer runs + inside the host process. One host process can serve several requests concurrently, up to a limit set in the Server + configuration. - **Timeouts and failures**: each call has its own time limit (Source writes and `identify_target` use Server's synchronous call limit; user changes use `change_timeout_seconds`). On a timeout, crash, or loss of response, core terminates the process, in-flight requests return 503, and the process restarts after a backoff. During the backoff the plugin's synchronous calls return 503; background calls are unaffected. -- **Code changes**: a host process does not re-import the plugin while it lives. Restart Server after changing a plugin, - the same rule as for Workers. +- **Code changes**: a host process does not re-import the plugin while it lives, and checks the code fingerprint when it + starts. Restart Server after changing a plugin, the same rule as for Workers. - **Isolation**: as with Workers, the host process isolates ordinary failures and is not a sandbox for untrusted code. **In-memory record of active plugins**: each Server process records in memory the plugins it has loaded: name, @@ -1043,40 +1083,31 @@ Core serves all public reads of plugin families without running plugin code: - On search, core matches that column and returns only active current revisions the caller may read in the requested Scope. Artifacts without searchable text do not take part. - The writer's `search` uses the same implementation. - - On SQLite, plugin families share `pc_artifact_fts` with Experience and Skill: rebuilding that helper table backfills - every head with search text, and the readiness check compares only built-in family rows. The implementation PR must - present how search works on OceanBase and validate it at scale. + - The implementation PR presents the search implementation on each database backend and validates it at scale. - Review, Recall, and Prepared Context are not open to plugin families in the first version; those APIs return an explicit "not supported" error, and `GET /v1/capabilities` lists plugin families. ## 8. Loading and lifecycle -```text -Server starts - -> scan the direct subdirectories of POWERCONTEXT_SERVER_PLUGINS_DIR (no plugins load if it is unset) - -> for each directory: import plugin.py in a time-limited check child process and validate the declaration, without calling create - -> the check child process returns a serializable declaration: Source types and the JSON Schema of their inputs, - the JSON Schema of target inputs, the family, and which user-change methods are implemented - -> Source plugins: register Source types; plugin code is not imported into the Server process - -> Artifact plugins: register the family for reads and search and in the merge registry; register a Supervisor - binding when there is a schedule or trigger - -> add the plugin to this process's record of active plugins - -> any step fails: record the error, mark the plugin failed, and continue with the next one -Server stops - -> stop accepting synchronous calls, wait for in-flight calls to finish or time out, then terminate host processes - -> stop dispatching and reclaim Workers through the existing Supervisor shutdown -``` +- At startup, Server checks each direct subdirectory of `POWERCONTEXT_SERVER_PLUGINS_DIR` (no plugins load if it is + unset): it imports `plugin.py` in a time-limited child process and validates the declaration without calling `create`. + Once the check passes, Server registers the Source types or family the plugin provides (reads, search, and the generic + merge APIs), registers a Supervisor binding when there is a schedule or trigger, and adds the plugin to this process's + record of active plugins. A failed check affects only that plugin. +- When Server stops, it stops accepting synchronous calls and ends host processes first, then stops the Supervisor and + reclaims Workers through the existing flow. - A plugin has only two outcomes, `loaded` and `failed`, kept only in process memory. Server persists no plugin enabled state, registration generation, or configuration snapshot; each startup re-checks the directory as it is. Plugin code is imported only in the check child process, host processes, and Workers, never in the Server process. - With no hot reload, no registration generation is needed to fence old Workers; the existing Supervisor lease fence is enough to fence stale commits after a leadership change. -- A Worker re-imports plugin code and compares the declaration on every invocation. If the plugin directory is modified - while Server runs and the declaration changes, the call fails and asks for a restart; if only implementation code changes, - newly dispatched Workers use the new code. Deployers should restart Server after a change. -- Server instances in the read-only API role also check plugins and register families and Source types for reads, search, - Source writes, and user changes; they start host processes on demand but do not dispatch Workers. +- A Worker re-imports plugin code on every invocation and checks the code fingerprint before importing (see 1.1). Once + the plugin directory is modified while Server runs, new Worker invocations and newly started host processes fail and + ask for a restart, so one Server never mixes old and new code. Deployers should restart Server after a change. +- Server instances in the `api` role run no background work, but they also check plugins and register families and + Source types for reads, search, Source writes, and user changes; they start host processes on demand and do not + dispatch Workers. Instances in the `background` role receive no requests and start no host processes. - Adding or removing plugins at runtime (hot reload) and plugin-defined APIs are covered by [RFC #2579](https://github.com/oceanbase/powercontext/pull/2579), built on host processes. @@ -1092,10 +1123,8 @@ Server stops - **Lock impact of the write transaction**: while the block is open it locks the heads and cursor rows involved; on SQLite a write transaction blocks all other writes to the database. The block must therefore be short, and exceeding its limit rolls it back. -- **Database access**: Workers open the database without creating tables; Server prepares the schema. The existing - database readiness check verifies that every pending invocation intent has a persistently registered binding; bindings - prefixed with `plugin:` are determined by the plugin directory at startup, so the check admits them, and intents left - behind by a removed plugin are no longer processed. +- **Leftover invocation intents**: after a plugin is removed, the invocation intents it left behind are no longer + processed and do not block the database readiness check. - **Failure and backoff**: a failed call commits no core data, and that plugin in that Scope enters the Supervisor's existing backoff without affecting other Scopes or plugins. Core never skips input automatically because of repeated failures; persistent failures require fixing the plugin or the data and restarting. A target that keeps failing in @@ -1113,9 +1142,9 @@ Server stops | `pc_artifacts`, `pc_artifact_heads`, `pc_artifact_lineage_*`, merge records, handles, tags | Kept; Artifact plugin results | | `pc_source_cursors` | Kept; Artifact plugin consumption cursors, keyed by `(scope_id, plugin:)` | | `pc_connector_checkpoints` | Kept; each collection target's checkpoint, keyed by `(scope_id, the target's binding ID)` | -| `pc_source_collect_targets` | Added; collection targets: Scope, plugin name, target key, normalized input, enabled state, creation and update times, last successful collection time, last failure category | +| `pc_source_collect_targets` | Added; collection targets with their normalized input, enabled state, and latest collection results | | Supervisor invocation intent, pending, and lease tables | Kept; plugin bindings share them with built-in bindings, and they back `GET /v1/plugins/{name}/scopes` | -| Plugin columns added to `pc_artifact_processing_binding_states` (`plugin_id`, `plugin_family`, `binding_generation`, `desired_state`, `phase`, `last_error`) | Removed | +| Columns the old plugin frameworks added to `pc_artifact_processing_binding_states` (desired state, management phase, registration generation, and so on) | Removed | | `pc_artifact_family_schemas` | Removed | | Search state and vector index data for custom family declarative indexes | Removed; the implementation PR confirms built-in families do not depend on them before dropping the tables | | Artifacts, heads, and lineage written by old Artifact plugins | Deleted, not migrated | @@ -1125,8 +1154,7 @@ Server stops | Other plugin-owned indexes and storage | Not part of core; managed by the plugin and the deployer | | The local SQLite of edge mode | Holds only the edge's collection targets and scheduling state, no Sources; Sources and checkpoints live on the central Server | -A target's binding ID is derived deterministically from the plugin name and the target key, using a digest when it exceeds -the column width. Additions and removals ship as one new database migration whose notes list the tables, columns, and data +Additions and removals ship as one new database migration whose notes list the tables, columns, and data ranges added and removed. ## 11. API changes @@ -1137,7 +1165,8 @@ ranges added and removed. | Removed | Custom family declarative indexes (`ArtifactSearchDeclaration`, `TextIndex`, and so on), `ArtifactFamily` schema declarations, `ArtifactDraft`, and the old Worker plugin executor | | Added | `powercontext.plugins`: `SourcePlugin`, `ArtifactPlugin`, `Inputs`, `Interval`, `Cron`, `PluginContext`, `PluginStorage`, `PluginModels`, the writer and read interfaces, and the user-change request models and `InvalidContent` | | Added | The `powercontext source-agent` edge collection mode, with `POWERCONTEXT_SOURCE_AGENT_ID` and `POWERCONTEXT_SOURCE_AGENT_TOKEN` | -| Changed | The body of `POST /v1/scopes/{scope_id}/artifacts` gains a plugin branch `{"family", "content"}`; core allocates the ID, and the plugin's `create_artifact` handles it | +| Changed | The body of `POST /v1/scopes/{scope_id}/artifacts` gains a plugin branch `{"family", "content"}`; core allocates the ID, and the plugin's `create_artifact` handles it. `CreateArtifactRequest` becomes a choice between the built-in branch `CreateBuiltinArtifactRequest` (the former union discriminated by family) and the plugin branch `CreatePluginArtifactRequest`, selected by family name: built-in names take the built-in branch with unchanged validation; other names take the plugin branch, and a family no plugin provides returns 422. In the generated Python client, built-in requests gain one wrapping level, and `ArtifactCreated.family` changes from the built-in family enum to a string | +| Changed | The `source_type` of `POST /v1/scopes/{scope_id}/sources` changes from an enum allowing only `content` to a string (still defaulting to `content`). It may name a Source type a plugin defines; `content` is then that type's input, which the plugin validates and resolves into a Source. Unknown types return 404. The `SourceType` enum remains | | Added | `GET /v1/plugins`, `GET /v1/plugins/{name}`, `GET /v1/plugins/{name}/scopes` (`server.admin`, read-only) | | Added | `GET .../{artifact_id}/governance`, `POST /v1/scopes/{scope_id}/artifacts/{family}/merges`, `POST .../{artifact_id}/forget`, `POST .../{artifact_id}/restorations`: generic APIs on the shared merge engine registry that work for every registered family (Experience and Atomic Memory among built-ins, plus loaded plugin families, which the matching plugin methods handle). Atomic Memory's dedicated APIs are unchanged | | Changed | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` accepts plugin families with a `{"content": {...}}` body, handled by the plugin's `update_artifact` | @@ -1152,25 +1181,15 @@ operation table are regenerated. ## 12. Observability Plugin calls reuse the Supervisor's and Worker's existing operation vocabulary, execution identities, and outcome fields, -and no request ID is generated for scheduled calls. New fields: +and no request ID is generated for scheduled calls. New fields identify the plugin name, the method called, where it ran +(Worker or host process), the trigger, and the size of the writes; host process starts, terminations, and restarts each +record one event. -| Field | Meaning | -| --- | --- | -| `plugin_name` | Plugin name | -| `plugin_method` | `collect`, `generate`, `resolve`, `identify_target`, `create_artifact`, `update_artifact`, `merge_artifacts`, `forget_artifact`, `restore_artifact` | -| `plugin_process` | `worker` or `host` | -| `trigger` | `schedule`, `sources`, `target_added`; `request` for synchronous calls | -| `operation_count` | Write operations committed (Artifact) or Sources committed (collect) in this invocation | -| `target_count` | Targets carried by this collect invocation | -| `transaction_ms` | Time from the start of the write transaction block to commit or rollback, in milliseconds | - -Host process starts, terminations, and restarts each record one event with `plugin_name` and the termination reason -(timeout, crash, Server stop). Metric labels may use `plugin_name`, `plugin_method`, and `plugin_process`, never Scope IDs -or target keys. Logs go through `log_safely` and -exclude Source content, Projections, Artifact content, target inputs, and environment variable values; plugin exceptions -are logged only by type and known error category, never by message. New fields are registered in the allowlists per the -developer guide and documented in both locales' observability docs. Call outcomes go only to logs and metrics and are not -persisted, so the operational APIs report no "last call result". +- Metric labels may use the plugin name, method, and execution location, never Scope IDs or target keys. +- Logs go through `log_safely` and exclude Source content, Projections, Artifact content, target inputs, and environment + variable values; plugin exceptions are logged only by type and known error category, never by message. +- New fields are registered in the allowlists per the developer guide and documented in both locales' observability docs. +- Call outcomes go only to logs and metrics and are not persisted, so the operational APIs report no "last call result". # Drawbacks @@ -1182,8 +1201,9 @@ persisted, so the operational APIs report no "last call result". - **The write transaction holds locks**: the block holds database locks, which on SQLite block other writes. A plugin that does slow work in the block slows every write in Server until the block times out and rolls back. - **Plugin-owned storage is not guaranteed consistent with core data**: they are not in the same transaction, so a failed - commit can leave entries in owned storage. Plugins must overwrite by stable ID, keep immutable records per revision, - and defer to core's current data. + commit can leave entries in owned storage. Plugins must overwrite by stable ID, keep immutable records keyed by + content digest, clean up unreferenced records, and defer to core's current data; owned storage must also support + idempotent writes, be durable on write, and be readable right after a write. - **Every background call starts a Worker and loads the plugin**: process startup and factory initialization have a fixed cost that rules out second-level high-frequency background calls, and plugins cannot reuse connections or caches across Worker calls. @@ -1222,6 +1242,7 @@ persisted, so the operational APIs report no "last call result". | Run Source `resolve` and similar code in the Server process | Saves an inter-process round trip, but a blocking or crashing plugin affects Server; a plugin's sibling modules enter `sys.modules` under top-level names, so same-named modules of different plugins collide and cannot later be reloaded reliably | | A long-lived host process for synchronous calls and Workers for background calls (this proposal) | Synchronous calls need no process start each time, and the Server process imports no plugin code; background calls keep the Worker's per-call isolation and budgets. The cost is the resources and management of long-lived processes | | Plugin storage inside core's transactions (plugins create tables in core's database) | Atomic commits, but core would manage plugin migrations, upgrades, and permissions on three backends, and plugin data would mix into core's schema. Plugin storage belongs to the plugin and stays out of core | +| Two-phase commit between plugin storage and core | Most plugin storage, such as object and vector stores, does not support XA, and core would need a coordinator for each kind of storage. Writing immutable content-addressed records first and making core's commit the only point where they take effect gives the consistency needed | | One plugin declaring several families | Could write several families in one transaction, but declarations, permissions, and merge registration would all need a multi-family design. Cooperating cases are served by two plugins plus cross-family reads | | Hand core's environment variables or database settings to plugins | Core's configuration does not all come from environment variables, and a plugin with core's database credentials could bypass the writer. Core provides the wrapped `ctx.storage` and `ctx.models` instead | | Compile a separate binary for the edge | A plugin is a Python directory with dependencies; a frozen binary cannot import arbitrary plugin dependencies and would need packaging per plugin and platform. Starting the same program in `source-agent` mode is enough | @@ -1256,11 +1277,9 @@ Reasons for this choice: and also commits the cursor and acknowledgment. - **Existing PowerContext foundations**: scheduling, invocation intents, Workers, and lease fences of the [Artifact processing Supervisor](1515_artifact_processing_supervisor.md); the [shared merge methods](1859_artifact_merge.md); - the write implementations of `ArtifactRepository`, `ArtifactGovernanceRepository`, handles, and tags; the generic Artifact - read and write APIs; `SourceDefinition` / `SourceProjection`; the Source journal, `pc_source_cursors`, and - `list_window_for_generation`; `pc_connector_checkpoints`; the remote ingestion APIs (`source-observations`, - `connector-checkpoints`, `source-definitions/register`); the `lineage_only` Content Source built-in families record on - edits; and the Source management APIs with the `identify_target` target identification rule. + the generic Artifact read and write APIs and the `lineage_only` Content Source built-in families record on edits; + `SourceDefinition` / `SourceProjection`, the Source journal, and consumption cursors; connector checkpoints and the + remote ingestion APIs; and the Source management APIs with their target identification rule. # Unresolved questions diff --git a/docs/zh/rfcs/2549-plugin-interfaces.md b/docs/zh/rfcs/2549-plugin-interfaces.md index 62bfd44f88..4352c64acb 100644 --- a/docs/zh/rfcs/2549-plugin-interfaces.md +++ b/docs/zh/rfcs/2549-plugin-interfaces.md @@ -26,7 +26,7 @@ description: Source 和 Artifact 插件各自实现少量接口，由 PC Server - `collect` 和 `generate` 是后台调用，在 Supervisor 管理的 Worker 子进程中执行，沿用现有的隔离、超时、并发预算、 领导租约和失败退避。 - Source 写入时的 `resolve` 与 Projection、添加采集目标时的 `identify_target`、用户对插件 Artifact 的修改是同步调用， - 在 core 管理的常驻**插件宿主进程**中执行。每个 Server 进程为每个插件按需启动一个宿主进程，部署者不需要管理它。 + 在 core 管理的常驻**插件宿主进程**中执行。每个 Server 进程为每个需要同步调用的插件启动一个宿主进程，部署者不需要管理它。 - **写入**：Artifact 插件在 `async with writer.transaction()` 块中执行写操作，块结束时 core 把写入、消费游标和 完成确认一起提交，异常则整体回滚。插件读不到也改不了游标。 - **core 的职责**：只管理 `pc_sources`、`pc_artifacts`（含 heads、lineage、合并记录、生命周期、handle 和 tags）及其 @@ -201,7 +201,9 @@ plugin = ArtifactPlugin(family="incident-report", ..., create=create) # 或 So 转换和保存。`resolve`、Projection 和 `identify_target` 都在插件宿主进程中执行；`collect` 在 Worker 中执行。 Source 与所有内置 Source 一样只能追加，没有修改和删除；要更正内容，提交一条新的 Source。文件、图片等大块内容由插件 -存进自己的存储，Source 中只记录位置、摘要和媒体类型等元数据。 +存进自己的存储，Source 中只记录位置、摘要和媒体类型等元数据。本提案中，这类内容通过 `collect` 从外部系统读入； +`resolve` 必须是纯函数，现有 Source 写入接口也只接受 JSON，调用方直接上传文件的入口由插件自定义接口提供，见 +[RFC #2579](https://github.com/oceanbase/powercontext/pull/2579)。 ### 3.2 声明 @@ -316,7 +318,7 @@ powercontext source-agent --server https://memory.example.com | 接口 | 必需 | 说明 | | --- | --- | --- | -| `create(ctx) -> ArtifactGenerator` | 是 | 工厂；Worker 中每次调用执行一次，宿主进程中每个 Scope 执行一次后复用 | +| `create(ctx) -> ArtifactGenerator` | 是 | 工厂；Worker 中每次调用执行一次，宿主进程中只执行一次后复用 | | `generate(request, writer) -> None` | 是 | 读取 `request.sources`（未消费的 Source 窗口），在写事务块中写入 Artifact | | `create_artifact(request, writer) -> None` | 否 | 用户新建；`request.artifact_id` 由 core 分配 | | `update_artifact(request, writer) -> None` | 否 | 用户编辑指定 ID 的 Artifact，`request.expected_revision` 为调用方看到的 revision | @@ -332,7 +334,7 @@ powercontext source-agent --server https://memory.example.com | 字段 | 默认值 | 说明 | | --- | --- | --- | | `family` | 必填 | 插件拥有的 Artifact family，不能与内置 family 或其他插件重复 | -| `inputs` | `None` | `Inputs(source_types=..., projection=..., max_sources=50)`：消费的 Source 类型、每条 Source 交付的 Projection、单个窗口的条数上限；为空表示不消费 Source | +| `inputs` | `None` | `Inputs(source_types=..., projection=..., projections={...}, max_sources=50)`：消费的 Source 类型、每条 Source 交付的 Projection、单个窗口的条数上限；为空表示不消费 Source。`projection` 适用于所有类型，`projections` 按 Source 类型指定不同的 Projection，每个类型必须有其一 | | `on_sources` | `False` | 新 Source 到达后触发；需要 `inputs` | | `schedule` | `None` | `Interval(seconds=...)` 或 `Cron(expression=..., timezone=...)`；与 `on_sources` 至少配置一项 | | `scopes` | `None` | 可服务的 Scope 名单；不消费 Source 的插件必须填写 | @@ -341,6 +343,7 @@ powercontext source-agent --server https://memory.example.com | `transaction_timeout_seconds` | 30 | 写事务块的时限，不超过 `timeout_seconds` | | `max_workers` | 1 | 跨 Scope 的并发上限 | | `change_timeout_seconds` | 30 | 一次用户修改调用的时限，包括其中的写事务块 | +| `definitions` | `()` | 插件自己的 Source 定义，随插件加载登记，规则与 Source 插件的 `definitions` 相同。插件在接口中用 `ctx.sources.submit` 提交，调用方也可以经通用 Source 写入接口提交；这样一个 Artifact 插件就能记录"用户要求做什么"，不需要另写一个 Source 插件 | | `create` | 必填 | 见上表 | ### 4.3 示例 @@ -462,8 +465,11 @@ Source 插件的 `collect` 按 `collect.schedule` 调用；添加或恢复目标 - `GET /v1/plugins/{name}/scopes`：每个 Scope 是否有待处理的工作、已消费到的 journal 位置和当前 journal 位置。 - 调用失败的原因通过日志和指标排查，不在接口中保存。 - **变更**：修改代码后重启 Server。Server 不监视文件，也不在运行中重新加载插件。采集目标是业务数据，增减不需要重启。 -- **停用**：从插件目录移走后重启。已保存的 Source、Artifact、采集目标、游标和 checkpoint 保留并可继续读取；重新部署 - 同名插件时从原位置继续。插件的专属存储不随之删除，由部署者保留或清理。 +- **停用**：从插件目录移走后重启。已保存的 Source、Artifact、采集目标、游标和 checkpoint 都保留，但该插件的 family 和 + Source 类型不再注册：通用接口读取、检索或修改该 family 时返回 404，与不存在的 family 相同。插件 Artifact 的内容通常只 + 记录摘要或位置，正文、分块和文件在插件自己的存储中，没有插件就无法解释，因此不提供脱离插件的读取。插件 Source 类型的 + 新写入返回 422。重新部署同名插件时，读取恢复，生成从原游标继续。插件的专属存储 + 不随之删除，由部署者保留或清理。 - **宿主进程**：由 core 按需启动、复用和重启，部署者不需要配置或管理。宿主进程的启动、终止和重启只体现在日志和指标中。 # 参考级说明 @@ -498,14 +504,16 @@ class SourcePlugin(BaseModel): # 字段见指南级说明 3.2 - 插件名取自目录名，必须匹配 `^[a-z][a-z0-9-]{0,63}$`。 - 声明在构造时校验，包括调度组合、`inputs` 与 `scopes` 的依赖、时限关系和 family 命名。`create` 必须是 async 函数。 - 声明中除 `create`、`identify_target`、`definitions` 等代码对象外的部分可以序列化，Server 在检查子进程中得到它， - 用于注册 binding 和回答运维查询；Worker 每次调用重新导入 `plugin.py`，声明与启动时不一致则调用失败，提示重启 Server。 + 用于注册 binding 和回答运维查询。 +- 检查时，core 记录插件目录的**代码指纹**（按文件内容计算）。Worker 和宿主进程导入插件前核对，不一致则调用失败并提示 + 重启 Server，同一个 Server 进程中的调用不会混用新旧代码。 ### 1.2 上下文 ```python class PluginContext: name: str # 插件名 - scope_id: str # 本次调用的 Scope + scope_id: str | None # Worker 调用的 Scope；宿主进程中为 None，Scope 随每个请求传入 plugin_dir: Path # 插件目录（只读约定） logger: logging.Logger # 已绑定插件名的 logger storage: PluginStorage # 插件专属存储 @@ -517,22 +525,33 @@ class PluginStorage(Protocol): backend: Literal["sqlite", "oceanbase", "seekdb"] def engine(self) -> AsyncEngine: ... # 只指向本插件的专属空间 +class ChatMessage(BaseModel): + role: Literal["system", "user", "assistant"] + content: str + class PluginModels(Protocol): - embedding_profile: EmbeddingProfile | None # profile ID 与向量维度；未配置 embedding 时为 None + embedding_available: bool # Server 是否配置了 embedding 模型 + embedding_dimension: int | None # 向量维度；未配置时为 None + generation_available: bool # Server 是否配置了生成模型 async def embed(self, texts: Sequence[str], /) -> Sequence[Sequence[float]]: ... async def generate(self, messages: Sequence[ChatMessage], /, *, max_tokens: int | None = None) -> str: ... + # 调用未配置的模型时抛出 ModelUnavailable；同步调用中返回 503 ``` -- 工厂在每次 Worker 调用中执行一次，实例不跨调用复用；在宿主进程中每个 Scope 执行一次，实例在宿主进程存续期间复用。 - 需要关闭的资源（包括插件自有存储的连接）通过 `ctx.enter_async_context` 注册，Worker 调用结束或宿主进程停止时由执行器释放。 +- 工厂在每次 Worker 调用中执行一次，实例不跨调用复用；在宿主进程中只执行一次，实例服务所有 Scope 的请求，在宿主进程 + 存续期间复用，资源占用不随 Scope 数量增长。宿主进程中的每个请求都带有 `scope_id`。需要关闭的资源（包括插件自有存储的 + 连接）通过 `ctx.enter_async_context` 注册，Worker 调用结束或宿主进程停止时由执行器释放。 - **专属存储**：core 使用 Server 现有的数据库连接配置，在同一个后端上为插件分配一块空间，不需要额外的数据库账号： - SQLite：数据目录下的 `plugins/<插件名>.sqlite3`。 - OceanBase / seekdb：名为 `powercontext_plugin_<插件名>` 的库（`-` 换成 `_`），首次使用时由 core 创建。Server 的数据库 登录没有建库权限时，由部署者事先建好。 - 表结构和迁移由插件自己管理，core 不读取、不迁移、不备份这块空间。它不在 core 的事务中，与核心数据的一致性按 3.5 处理。 + - 这块空间属于整个插件，不按 Scope 隔离。插件按 `scope_id` 划分自己的数据，并在读取时自行确认数据属于本次请求的 Scope， + 不能只凭调用方给出的 ID 返回数据。 - 插件与 Server 使用同一个数据库登录，这一划分防止插件误写 core 的表，不防止恶意插件；插件目录本身被当作可信代码。 -- **模型**：`models` 封装 Server 的推理配置，插件拿不到模型的地址、请求头和凭据。`embedding_profile` 让插件在索引和查询时 - 使用同一个 embedding 配置。调用沿用 Server 的超时设置，并计入现有的模型用量统计。插件也可以从环境变量读取自己的 +- **模型**：`models` 封装 Server 的推理配置，插件拿不到模型的地址、请求头和凭据。Server 两种模型都没有配置时 `models` + 为 None；只配置了其中一种时，插件用 `embedding_available`、`generation_available` 判断。插件应把 `embedding_dimension` + 等 embedding 配置记在自己的索引上，配置变化后重建，避免把不同模型的向量混在一起比较。调用沿用 Server 的超时设置，并计入现有的模型用量统计。插件也可以从环境变量读取自己的 模型配置，自行创建客户端。 - 插件得不到 core 数据库的连接，也没有游标或 checkpoint 的写入接口。插件对核心数据的写入只有三条路径：Source 插件的 `collect` 返回值、宿主进程中 `resolve` 的返回值，和 Artifact 插件的 `ArtifactWriter`。 @@ -627,7 +646,7 @@ class GenerateRequest(BaseModel): class GenerationSource(BaseModel): ref: SourceRef # 精确 Source 引用 position: int # Source 在 journal 中的位置，用于按采集顺序处理 - projection: JsonValue # inputs.projection 指定的 Projection 内容 + projection: JsonValue # 按该 Source 类型声明的 Projection 内容 ``` ### 3.2 写入器与写事务 @@ -664,15 +683,15 @@ class ArtifactTransaction(ArtifactReads): 写事务覆盖 Artifact 的全部通用写操作，每个操作复用 core 现有的实现与校验规则，违反规则时立即抛出异常： -| 操作 | 复用的 core 实现 | 语义 | -| --- | --- | --- | -| `put(artifact_id, *, content, source_refs=(), artifact_refs=(), expected_revision=None)` | 公共合并方法的修改路径 | 不存在则新建 revision 1；已存在则按当前 head 追加 revision。给出 `expected_revision` 时，当前 revision 不一致即冲突，用于事务外读取、事务内写入的乐观并发。内容与当前 revision 相同时为 no-op，返回当前快照 | -| `revert(artifact_id, *, to_revision)` | 公共恢复方法（选定 revision） | 以历史 revision 的内容追加新 revision，记录 `restored_from_revision`；目标已被遗忘或被合并冻结时一并恢复 | -| `set_lifecycle(artifact_id, state, *, replacement_artifact_id=None)` | 公共遗忘与恢复方法；指定替代者或退役时使用 `ArtifactGovernanceRepository.transition` | `deprecated` 为遗忘，`active` 为恢复，`retired` 不可逆。恢复走共享恢复流程：记录一个内容相同的新 revision，被合并冻结的输入会撤销冻结它的合并，按组恢复。遗忘和退役不追加 revision | -| `merge(inputs, *, into, content, source_refs=(), artifact_refs=())` | 公共合并方法（[RFC #1859](1859_artifact_merge.md)） | 以 `inputs` 的当前 revision 为输入，新建 `into`，记录合并关系并冻结输入。`into` 必须是新 ID；同一组输入在撤销后再次合并时，插件应把输入 revision 纳入 `stable_id`，避免与已退役的结果撞号 | -| `preview_undo_merge(artifact_id)` / `undo_merge(artifact_id)` | 公共合并方法的预览与恢复 | 撤销产生该 Artifact 的合并，恢复输入并退役结果，返回恢复与退役的精确引用 | -| `bind_handle(handle_key, ref)` | Artifact handle 绑定 | 把稳定 handle 绑定到指定 revision | -| `set_tags(artifact_id, tags)` | Artifact tags 替换 | 整体替换该 Artifact 的 tags | +| 操作 | 语义 | +| --- | --- | +| `put(artifact_id, *, content, source_refs=(), artifact_refs=(), expected_revision=None)` | 不存在则新建 revision 1；已存在则按当前 head 追加 revision。给出 `expected_revision` 时，当前 revision 不一致即冲突；不给出时，使用本次调用在事务外读到的 revision（见 3.5），没有读过则不校验。内容与当前 revision 相同时不写入，返回当前快照，本次传入的 `source_refs` 和 `artifact_refs` 不会被记录（见下方写入约束）。用户修改中的 `put` 例外，见 4.1 | +| `revert(artifact_id, *, to_revision)` | 以历史 revision 的内容追加新 revision，记录 `restored_from_revision`；目标已被遗忘或被合并冻结时一并恢复。返回 `ArtifactChange`：目标的新快照，以及一并恢复和退役的其他 Artifact 的精确引用 | +| `set_lifecycle(artifact_id, state, *, replacement_artifact_id=None)` | `deprecated` 为遗忘，`active` 为恢复，`retired` 不可逆。恢复走共享恢复流程：记录一个内容相同的新 revision，被合并冻结的输入会撤销冻结它的合并，按组恢复。遗忘和退役不追加 revision。返回值同 `revert`，插件据此同步更新自有索引中受级联影响的条目 | +| `merge(inputs, *, into, content, source_refs=(), artifact_refs=())` | 沿用公共合并方法（[RFC #1859](1859_artifact_merge.md)）：以 `inputs` 的当前 revision 为输入，新建 `into`，记录合并关系并冻结输入。`into` 必须是新 ID；同一组输入在撤销后再次合并时，插件应把输入 revision 纳入 `stable_id`，避免与已退役的结果撞号 | +| `preview_undo_merge(artifact_id)` / `undo_merge(artifact_id)` | 撤销产生该 Artifact 的合并，恢复输入并退役结果，返回恢复与退役的精确引用 | +| `bind_handle(handle_key, ref)` | 把稳定 handle 绑定到指定 revision | +| `set_tags(artifact_id, tags)` | 整体替换该 Artifact 的 tags | 写入约束： @@ -680,10 +699,11 @@ class ArtifactTransaction(ArtifactReads): - `source_refs` 在 `generate` 中只能引用本次窗口中的 Source，在用户修改中只能引用目标 Artifact 当前 lineage 中已有的 Source（用于保留原始输入）。`artifact_refs` 可以引用本 Scope 中任意 family 实际存在的 revision；`to_revision` 和 `bind_handle` 的目标只能是本 family 中实际存在的 revision。 +- 内容不变时不产生 revision，与内置 Atomic Memory 的处理一致：模型判断已有记忆已经表达了新 Source 的内容时，该记忆 + 不追加 revision，新 Source 也不进入它的 lineage。插件需要把新 Source 记为某个 Artifact 的证据时，应同时修改其内容。 - 检索文本由 core 按声明的 `search_fields` 从内容中提取，插件不单独提供；同一份内容无论由插件还是用户写入， 检索文本都相同。 -- 每个写事务块的操作数（1000）、单份内容大小（256 KiB）和总字节数（8 MiB）有上限，超出即抛出异常并回滚， - 不截断、不部分提交。 +- 每个写事务块的操作数、单份内容大小和总字节数有上限，超出即抛出异常并回滚，不截断、不部分提交。 - core 不校验 `content` 的业务结构，也不保存或固化 schema。内容结构的演进由插件负责，插件可以在 `content` 中自带版本字段。 ### 3.4 读取 @@ -703,6 +723,11 @@ class ArtifactSnapshot(BaseModel): replacement_artifact_id: str | None governance_generation: int # 治理状态版本，用于判断读取后是否被修改 +class ArtifactChange(BaseModel): # revert / set_lifecycle 的返回值 + artifact: ArtifactSnapshot # 目标的新快照 + restored: Sequence[ArtifactRef] # 一并恢复的其他 Artifact + retired: Sequence[ArtifactRef] # 一并退役的其他 Artifact + class ArtifactReads(Protocol): async def get(self, artifact_id: str, *, revision: int | None = None) -> ArtifactSnapshot | None: ... async def revisions(self, artifact_id: str, *, cursor: str | None = None, limit: int = 50) -> ArtifactPage: ... @@ -731,15 +756,26 @@ class ArtifactReads(Protocol): - **重复处理同一业务对象**：不消费 Source 的定时整理等场景会再次处理已经处理过的对象。插件用 `writer.stable_id(...)` 从业务 身份派生 Artifact ID；`put` 在内容相同时为 no-op，`set_lifecycle` 目标状态与当前相同时为 no-op，`merge` 在输入已全部 合并到同一结果时为 no-op，从而不会产生重复 Artifact 或无意义的 revision。 -- **插件自有存储**：插件的索引和其他存储（包括 core 分配的专属存储）不在 core 的事务中，core 不保证两者一致，由插件负责。 - 推荐做法： +- **事务外读取后写入**：插件在事务外读到某个 Artifact，又在写事务块中对它 `put` 而没有给出 `expected_revision` 时，core + 以事务外读到的 revision 作为期望值。读取后该 Artifact 被用户修改，写入即冲突，本次调用回滚后重新调度。这与内置 + Atomic Memory 在提交前核对所读记忆是否变化的做法一致，用户修改不会被插件基于旧内容的写入静默覆盖。 +- **插件自有存储**：插件的索引和其他存储（core 分配的专属存储，或插件自己连接的任何数据库、对象存储、向量库）不在 core + 的事务中。插件用自己的客户端写入并提交自有存储，core 不参与，也不保证两者一致。core 的事务是核心数据的唯一提交点： + 自有存储中的数据只有被 core 中已提交的 revision 引用后才算生效。这不需要两阶段提交，按下面的做法即可保持一致： 1. 在写事务块内、拿到 `ArtifactSnapshot` 之后写自有存储，按 `artifact_id` upsert。这样自有存储的写入失败会让核心写入 一起回滚。 2. 写事务提交失败时，自有存储可能留下本次的条目。重试会处理同一窗口、得到同一个稳定 ID，覆盖这些条目。 3. 把自有索引当作候选来源：命中的 ID 用 `get` 读取 core 中的当前快照，以快照内容和状态为准。这样回滚留下的条目 不会让插件读到过期内容。插件也可以定期按 core 数据重建索引。 - 4. 大块内容（正文、分块、文件字节）按 `(artifact_id, revision)` 存成不可变的记录，先写入自有存储，再在写事务块中提交 - 带摘要的 Artifact 内容。没有提交成功的记录不会被任何 revision 引用，读取时按 Artifact 内容中的摘要核对。 + 4. 大块内容（正文、分块、文件字节）按内容摘要存成不可变的记录：先写入并提交到自有存储，再在写事务块中提交内容里 + 带有该摘要的 Artifact。不能用 `(artifact_id, revision)` 作为键：写事务回滚后，同一个 revision 号会再次分配给内容 + 不同的写入。按内容摘要存储时，重试写入同样的内容得到同一个键；提交失败只会留下没有被任何 revision 引用的记录， + 由插件按"不被引用"定期清理。 +- **对插件自有存储的要求**：上述做法与存储的种类无关，但存储需要满足三点，插件选型时自行确认，core 不做检查： + - 能按插件给定的键幂等写入（upsert 或"不存在才插入"）。只能由存储分配 ID 的系统，需要插件先确定 ID 再写入。 + - 写入返回时数据已经持久化。 + - 写入后立即可读。有刷新延迟的系统（例如 Elasticsearch 的 refresh 间隔、异步建索引的向量库）可能出现 core 已经引用、 + 自有存储中暂时读不到的情况，插件读取时需要重试或按暂时缺失处理。 - `generate` 本身可能因失败重试而执行多次，模型调用和插件发出的外部请求不是恰好一次。 ## 4. 用户对插件 Artifact 的修改 @@ -794,6 +830,8 @@ class InvalidContent(Exception): # 插件拒绝内容；接口返回 42 - 这些调用中的写事务只允许上表对应的那一项写操作，以及对同一 Artifact 的 `set_tags`；写其他 Artifact 时立即抛出异常。 方法正常返回但没有完成要求的写操作，core 回滚并返回 502，表示插件没有按约定完成。 - 插件可以规范化内容（例如补全默认字段），写入的内容以插件提交的为准。 +- `create_artifact`、`update_artifact` 中的 `put` 总是追加 revision，内容与当前 revision 相同时也一样。这与内置 family 的 + 编辑一致：每次用户编辑都有一条新的 `lineage_only` Content Source（见 4.2），需要一个新 revision 记录它。 - 插件应在写事务块中、拿到快照之后更新自己的存储，做法同 3.5。 ### 4.2 core 负责的部分 @@ -805,8 +843,7 @@ class InvalidContent(Exception): # 插件拒绝内容；接口返回 42 - **溯源**：与内置 family 相同，新建、编辑、恢复和撤销合并时，core 在同一事务中把调用方提交的内容另存为一条内部 Content Source（`lineage_only`，操作分别为 `artifact_create`、`artifact_replace`、`artifact_restore`、`artifact_undo_merge`）， 作为新 revision 的直接来源记入 lineage。这类 Source 由 core 写入，不进入任何插件的输入窗口。 -- **合并注册表**：通用的合并、遗忘、恢复接口建立在共享合并引擎的注册表（`ArtifactMergeApplication`， - [RFC #1859](1859_artifact_merge.md)）之上，对所有登记过的 family 生效。内置 family 登记了 Experience 和 Atomic Memory， +- **合并注册表**：通用的合并、遗忘、恢复接口建立在共享合并引擎（[RFC #1859](1859_artifact_merge.md)）的注册表之上，对所有登记过的 family 生效。内置 family 登记了 Experience 和 Atomic Memory， 由各自的适配器处理；插件 family 在加载时登记，处理方式是转交给插件对应的方法。没有登记的 family 返回 422。 - **不提供预览**：Atomic Memory 的专用接口支持先预览恢复影响、再凭签名 token 执行。通用接口不提供预览，也就不需要签名 密钥；恢复和撤销合并按执行时的当前状态进行，合并和遗忘依靠 `revision` 与 `state_version` 校验。 @@ -833,49 +870,32 @@ class InvalidContent(Exception): # 插件拒绝内容；接口返回 42 | 插件没有完成要求的写操作 | 502 | | 超过 `change_timeout_seconds`、宿主进程不可用或处于重启退避 | 503 | -- 新建请求的插件分支没有判别映射：OpenAPI 无法为任意插件 family 写映射，生成的客户端对这一分支的类型约束较弱。这与 - tags 接口把 family 改为字符串的处理相同。 +- 新建请求的插件分支没有判别映射：OpenAPI 无法为任意插件 family 写映射。内置分支保留原来按 family 的判别，插件分支与它 + 并列，由 family 名决定走哪个分支，内置 family 的请求不会被插件分支接受。这与 tags 接口把 family 改为字符串的处理相同。 - 只有删除语义的需求（例如撤下一份文档）使用遗忘；core 不提供物理删除。插件可以在 `forget_artifact` 中清理自己存储中的 正文和索引，core 中的 revision 与 Source 保留。 -- **并发**：用户修改与后台的 `generate` 可能同时发生。`generate` 在事务外读取、事务内用 `expected_revision` 写入时会发现 - 冲突并整体回滚，下一次调用基于修改后的状态重新决策；用户修改不会被插件静默覆盖。 +- **并发**：用户修改与后台的 `generate` 可能同时发生。`generate` 对事务外读过的 Artifact 写入时，core 按读到的 revision + 校验（见 3.5），发现冲突即整体回滚，下一次调用基于修改后的状态重新决策；用户修改不会被插件基于旧内容的写入静默覆盖。 -## 5. 插件执行器与消费游标 +## 5. 消费游标 -插件执行器是 Worker 中的 core 代码，实现 Supervisor 的 Worker 入口，替代旧的 `run_artifact_plugin_worker`。 -它对 Artifact 插件的一次调用如下： - -```text -Worker 收到 (plugin:, scope_id, request_generation) - -> 短事务：ScopeInvocation.start；读取 pc_source_cursors 中 (scope_id, plugin:) 的游标 - -> 读取 journal 高水位，确定窗口 (after, through]，through ≤ after + max_sources - -> list_window_for_generation 读取窗口，按 inputs.source_types 过滤，计算 inputs.projection - -> 窗口为空且配置了 inputs：短事务完成游标 CAS 与 acknowledge，不加载插件 - -> 导入 plugin.py、核对声明、调用 create、创建写入器、调用 generate（受 timeout_seconds 约束） - -> 事务外读取：独立只读短事务 - -> 写事务块开始：开启事务；guard（租约与调用意图） - 块内写操作在该事务中执行 - 块结束：游标 CAS 到 through；acknowledge；提交 - -> generate 返回且未开启写事务块：短事务完成游标 CAS 与 acknowledge -``` +Artifact 插件的每次后台调用处理该 Scope 中游标之后的一个 Source 窗口： +- 窗口不超过 `inputs.max_sources` 条。窗口中没有匹配的 Source 时不加载插件，直接推进游标并确认。 - 窗口内类型不匹配的 Source 被过滤，但游标仍推进过整个窗口；匹配类型的 Source 缺少所需 Projection 时调用失败，不跳过。 -- 游标只在执行器的提交事务中推进，插件得不到游标值，也没有写入接口。游标 CAS 保证在读取窗口与提交之间即使发生 - 领导权变更或重复派发，同一窗口也只提交一次。 -- 待处理工作的发现有两条路径：Server 进程中的 Source 写入会把关注该 Scope 的插件 binding 标记为待处理；启动时 - 现有的 `SourceProcessingPendingProvider` 按 journal 高水位与游标的差值补齐其他进程写入的 Source。Supervisor 只在 - binding 的 interval 或 cron 到期时接纳待处理的 Scope，因此 `on_sources` 映射为约 2 秒的接纳间隔。判断基于持久化数据， - Server 重启后不会丢失待处理输入。 +- 游标只在执行器的提交事务中以 CAS 推进，插件得不到游标值，也没有写入接口。读取窗口与提交之间即使发生领导权变更或 + 重复派发，同一窗口也只提交一次。 +- 是否有待处理的工作由持久化数据判断（Source journal 位置与游标之差），Server 重启后不会丢失待处理输入。 + `on_sources` 的插件在新 Source 写入后约 2 秒内被派发。 - 未配置 `scopes` 时，自动调度对所有 journal 超过游标的 Scope 生效；配置了 `scopes` 时只接受名单内的 Scope。 -collect 调用结构相同：输入换成该 Scope 的已启用目标及其 checkpoint，提交内容换成 Source 与各目标的新 checkpoint。 -collect 不需要写事务块，由执行器根据返回值一次提交。添加采集目标通过现有的调用意图（`ArtifactProcessingIntentRepository.request`） -请求一次采集。 +collect 的结构相同：输入换成该 Scope 的已启用目标及其 checkpoint，提交内容换成 Source 与各目标的新 checkpoint，由执行器 +根据返回值一次提交。添加或恢复采集目标时请求一次采集。 ## 6. 插件宿主进程 -同步调用需要在请求时限内返回，不能每次都启动 Worker：Worker 用 `spawn` 启动全新的 Python 进程，在 2 核开发机上仅导入 -core 的执行模块就约需 5.5 秒、150 MB 内存。因此同步调用交给常驻的宿主进程： +同步调用需要在请求时限内返回，不能每次都启动 Worker：每个 Worker 都是全新的 Python 进程，导入 core 的执行模块需要数秒。 +因此同步调用交给常驻的宿主进程： | 调用 | 执行位置 | | --- | --- | @@ -884,14 +904,15 @@ core 的执行模块就约需 5.5 秒、150 MB 内存。因此同步调用交给 | `create_artifact`、`update_artifact`、`merge_artifacts`、`forget_artifact`、`restore_artifact` | 宿主进程 | | `collect`、`generate` | Supervisor Worker | -- **数量与启动**：每个 Server 进程为每个需要同步调用的插件维护至多一个宿主进程，第一次有同步调用时启动，之后复用。 - 只提供 `generate` 的 Artifact 插件不启动宿主进程。端侧模式中，`resolve` 和 Projection 在采集 Worker 内执行，不启动宿主进程。 -- **执行**：宿主进程启动时导入一次 `plugin.py` 并核对声明；与 Worker 一样，由 core 代码打开数据库连接，写入器在宿主进程中 - 执行。一个宿主进程可以并发处理多个请求，并发上限由 Server 配置决定。 +- **数量与启动**：每个 Server 进程为每个加载的插件维护至多一个宿主进程，插件加载后在后台启动，之后复用。插件是否实现了 + 用户修改方法，要调用 `create` 之后才知道，所以只提供 `generate` 的 Artifact 插件也会启动宿主进程，代价是一个空闲的进程。 + 端侧模式中，`resolve` 和 Projection 在采集 Worker 内执行，不启动宿主进程。 +- **执行**：宿主进程启动时导入一次 `plugin.py` 并核对声明，写入器在宿主进程中执行。一个宿主进程可以并发处理多个请求， + 并发上限由 Server 配置决定。 - **超时与故障**：每次调用受各自的时限约束（Source 写入与 `identify_target` 使用 Server 的同步调用时限，用户修改使用 `change_timeout_seconds`）。超时、崩溃或失去响应时，core 终止该进程，进行中的请求返回 503，并在退避后重新启动；退避期间 该插件的同步调用返回 503，后台调用不受影响。 -- **代码变更**：宿主进程在存续期间不重新导入插件。修改插件后需要重启 Server，与 Worker 的规则一致。 +- **代码变更**：宿主进程在存续期间不重新导入插件，启动时核对代码指纹。修改插件后需要重启 Server，与 Worker 的规则一致。 - **隔离**：与 Worker 相同，宿主进程隔离普通故障，不是不可信代码的沙箱。 **已激活插件的内存记录**：每个 Server 进程在内存中记录本进程加载的插件：插件名、目录、类型、声明摘要、检查结果，以及 @@ -910,35 +931,23 @@ core 的执行模块就约需 5.5 秒、150 MB 内存。因此同步调用交给 保存，恢复旧 revision 时检索文本随内容一起恢复。 - 检索时在该列上匹配，只返回请求 Scope 中调用者有权读取的 active 当前 revision。没有可检索文本的 Artifact 不参与检索。 - 写入器的 `search` 使用同一实现。 - - SQLite 上与 Experience、Skill 共用 `pc_artifact_fts`：该辅助表的重建会回填所有带检索文本的 head，就绪校验只比对内置 - family 的行。实现 PR 需要给出 OceanBase 上的检索方式和规模验证。 + - 各数据库后端上的检索实现和规模验证在实现 PR 中给出。 - Review、Recall 和 Prepared Context 第一版不对插件 family 开放，对应接口返回明确的“不支持”错误；`GET /v1/capabilities` 列出插件 family。 ## 8. 加载与生命周期 -```text -Server 启动 - -> 扫描 POWERCONTEXT_SERVER_PLUGINS_DIR 的直接子目录（未设置则不加载插件） - -> 对每个目录：在有时限的检查子进程中导入 plugin.py，校验声明，不调用 create - -> 检查子进程返回可序列化的声明：Source 类型及其输入的 JSON Schema、目标输入的 JSON Schema、 - family、实现了哪些用户修改方法 - -> Source 插件：注册 Source 类型；插件代码不导入 Server 进程 - -> Artifact 插件：注册 family 的读取与检索，在合并注册表中登记；有调度或触发时注册 Supervisor binding - -> 写入本进程的已激活插件记录 - -> 任一步失败：记录错误，该插件标记为 failed，继续下一个 -Server 停止 - -> 停止接受同步调用，等待进行中的调用结束或超时后终止宿主进程 - -> 按现有 Supervisor 关闭流程停止派发并回收 Worker -``` - +- Server 启动时检查 `POWERCONTEXT_SERVER_PLUGINS_DIR` 的每个直接子目录（未设置则不加载插件）：在有时限的子进程中导入 + `plugin.py` 并校验声明，不调用 `create`。检查通过后注册插件提供的 Source 类型或 family（读取、检索和通用合并接口）， + 有调度或触发时注册 Supervisor binding，并写入本进程的已激活插件记录。检查失败只影响该插件。 +- Server 停止时先停止接受同步调用、结束宿主进程，再按现有流程停止 Supervisor 并回收 Worker。 - 插件只有 `loaded` 和 `failed` 两种结果，只存在于进程内存中。Server 不持久化插件的启用状态、注册代次或配置快照； 每次启动按当时的目录内容重新检查。插件代码只在检查子进程、宿主进程和 Worker 中导入，Server 进程不导入插件代码。 - 没有热加载，因此不需要注册代次来屏蔽旧 Worker，现有的 Supervisor 租约 fence 已经足以屏蔽领导权变更后的旧提交。 -- Worker 每次调用重新导入插件代码并核对声明。插件目录在 Server 运行期间被修改、声明随之改变时，调用失败并提示重启； - 只改了实现代码时，新派发的 Worker 会使用新代码。部署者应在修改后重启 Server。 -- 只读 API 角色的 Server 实例也会检查插件并注册 family 和 Source 类型，用于读取、检索、Source 写入和用户修改；它会按需 - 启动宿主进程，但不派发 Worker。 +- Worker 每次调用重新导入插件代码，导入前核对代码指纹（见 1.1）。插件目录在 Server 运行期间被修改后，新的 Worker 调用 + 和新启动的宿主进程都会失败并提示重启，不会出现同一个 Server 中新旧代码混用。部署者应在修改后重启 Server。 +- `api` 角色的 Server 实例不运行后台任务，但也会检查插件并注册 family 和 Source 类型，用于读取、检索、Source 写入和用户 + 修改；它按需启动宿主进程，不派发 Worker。`background` 角色的实例不接收请求，不启动宿主进程。 - 运行中增减插件（热加载）和插件自定义 API 见 [RFC #2579](https://github.com/oceanbase/powercontext/pull/2579)，建立在宿主进程之上。 ## 9. 执行约束 @@ -950,9 +959,7 @@ Server 停止 跨 Scope 的并发上限，分别映射到 Supervisor binding 的现有参数或由执行器强制。 - **写事务的锁影响**：写事务块持有期间会锁住涉及的 head 和游标行；在 SQLite 上，写事务会阻塞整个数据库的其他写入。 因此写事务块必须短，时限超出即回滚。 -- **数据库访问**：Worker 打开数据库时不创建表，Schema 由 Server 准备。现有的数据库就绪检查会校验每个待处理调用意图的 - binding 都已持久注册；`plugin:` 前缀的 binding 由插件目录在启动时决定，就绪检查对其放行，插件移除后遗留的调用意图 - 不再被处理。 +- **遗留的调用意图**：插件移除后，它遗留的调用意图不再被处理，也不阻塞数据库就绪检查。 - **失败与退避**：调用失败时不提交任何核心数据，该插件该 Scope 进入 Supervisor 现有的退避，不影响其他 Scope 和插件。 core 不会因为重复失败自动跳过输入；持续失败需要修复插件或数据后重启。collect 中某个目标持续失败会阻塞同一 Scope 的其他目标，可以先停用该目标。 @@ -968,9 +975,9 @@ Server 停止 | `pc_artifacts`、`pc_artifact_heads`、`pc_artifact_lineage_*`、合并记录、handle、tags | 保留；Artifact 插件的结果 | | `pc_source_cursors` | 保留；Artifact 插件的消费游标，键为 `(scope_id, plugin:)` | | `pc_connector_checkpoints` | 保留；各采集目标的 checkpoint，键为 `(scope_id, 目标的 binding ID)` | -| `pc_source_collect_targets` | 新增；采集目标：Scope、插件名、目标键、规范化输入、启用状态、创建与更新时间、最近成功采集时间、最近失败分类 | +| `pc_source_collect_targets` | 新增；采集目标及其规范化输入、启用状态和最近采集结果 | | Supervisor 的调用意图、pending、租约表 | 保留；插件 binding 与内置 binding 共用，也是 `GET /v1/plugins/{name}/scopes` 的数据来源 | -| `pc_artifact_processing_binding_states` 中为插件新增的列（`plugin_id`、`plugin_family`、`binding_generation`、`desired_state`、`phase`、`last_error`） | 移除 | +| `pc_artifact_processing_binding_states` 中为旧插件框架新增的列（期望状态、管理阶段、注册代次等） | 移除 | | `pc_artifact_family_schemas` | 移除 | | 自定义 family 声明式索引使用的搜索状态与向量索引数据 | 移除；实现 PR 确认内置 family 不依赖后删除对应表 | | 旧 Artifact 插件写入的 Artifact、head、lineage | 删除，不迁移 | @@ -980,8 +987,7 @@ Server 停止 | 插件自有的其他索引与存储 | 不属于 core，由插件和部署者管理 | | 端侧模式的本地 SQLite | 只保存端侧的采集目标和调度状态，不保存 Source；Source 与 checkpoint 保存在中心 | -目标的 binding ID 由插件名和目标键确定性派生，长度超过列宽时使用其摘要。新增与移除通过一次新的数据库迁移完成， -迁移说明中列出新增、删除的表、列和数据范围。 +新增与移除通过一次新的数据库迁移完成，迁移说明中列出新增、删除的表、列和数据范围。 ## 11. 接口变更 @@ -991,7 +997,8 @@ Server 停止 | 移除 | 自定义 family 的声明式索引（`ArtifactSearchDeclaration`、`TextIndex` 等）、`ArtifactFamily` 的 schema 声明、`ArtifactDraft` 和旧的 Worker 插件执行器 | | 新增 | `powercontext.plugins`：`SourcePlugin`、`ArtifactPlugin`、`Inputs`、`Interval`、`Cron`、`PluginContext`、`PluginStorage`、`PluginModels`、写入器与读取接口、用户修改的请求模型与 `InvalidContent` | | 新增 | `powercontext source-agent` 端侧采集模式，以及 `POWERCONTEXT_SOURCE_AGENT_ID`、`POWERCONTEXT_SOURCE_AGENT_TOKEN` | -| 调整 | `POST /v1/scopes/{scope_id}/artifacts` 的请求体增加插件分支 `{"family", "content"}`，ID 由 core 分配，由插件的 `create_artifact` 处理 | +| 调整 | `POST /v1/scopes/{scope_id}/artifacts` 的请求体增加插件分支 `{"family", "content"}`，ID 由 core 分配，由插件的 `create_artifact` 处理。`CreateArtifactRequest` 改为内置分支 `CreateBuiltinArtifactRequest`（原有的按 family 区分的联合类型）与插件分支 `CreatePluginArtifactRequest` 的二选一，按 family 名选择：内置 family 名走内置分支，校验与原来相同；其他名字走插件分支，没有插件提供该 family 时返回 422。生成的 Python 客户端中，内置请求多包一层；`ArtifactCreated.family` 从内置 family 枚举改为字符串 | +| 调整 | `POST /v1/scopes/{scope_id}/sources` 的 `source_type` 从只能是 `content` 的枚举改为字符串（默认仍为 `content`），可以是插件定义的 Source 类型，此时 `content` 是该类型的输入，由插件校验并解析为 Source；未知类型返回 404。`SourceType` 枚举保留 | | 新增 | `GET /v1/plugins`、`GET /v1/plugins/{name}`、`GET /v1/plugins/{name}/scopes`（`server.admin`，只读） | | 新增 | `GET .../{artifact_id}/governance`、`POST /v1/scopes/{scope_id}/artifacts/{family}/merges`、`POST .../{artifact_id}/forget`、`POST .../{artifact_id}/restorations`：基于共享合并引擎注册表的通用接口，对所有登记过的 family 生效（内置为 Experience 和 Atomic Memory，以及加载的插件 family；插件 family 由插件对应的方法处理）。Atomic Memory 的专用接口保持不变 | | 调整 | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` 接受插件 family，请求体为 `{"content": {...}}`，由插件的 `update_artifact` 处理 | @@ -1004,23 +1011,14 @@ Server 停止 ## 12. 可观测性 -插件调用复用 Supervisor 与 Worker 现有的操作词汇、执行身份和结果字段，不为定时调用生成 request ID。新增字段： +插件调用复用 Supervisor 与 Worker 现有的操作词汇、执行身份和结果字段，不为定时调用生成 request ID。新增的字段标识 +插件名、调用的方法、执行位置（Worker 或宿主进程）、触发原因和写入规模；宿主进程的启动、终止和重启各记录一条事件。 -| 字段 | 含义 | -| --- | --- | -| `plugin_name` | 插件名 | -| `plugin_method` | `collect`、`generate`、`resolve`、`identify_target`、`create_artifact`、`update_artifact`、`merge_artifacts`、`forget_artifact`、`restore_artifact` | -| `plugin_process` | `worker` 或 `host` | -| `trigger` | `schedule`、`sources`、`target_added`；同步调用为 `request` | -| `operation_count` | 本次提交的写操作数（Artifact）或 Source 条数（collect） | -| `target_count` | 本次 collect 携带的目标数 | -| `transaction_ms` | 写事务块从开始到提交或回滚的耗时，毫秒 | - -宿主进程的启动、终止和重启各记录一条事件，带 `plugin_name` 和终止原因（超时、崩溃、Server 停止）。 -指标标签可以使用 `plugin_name`、`plugin_method` 和 `plugin_process`，不包含 Scope ID 或目标键。日志经 `log_safely` 输出，不包含 Source -内容、Projection、Artifact 内容、目标输入和环境变量值；插件异常只记录类型和已知错误分类，不记录异常消息。新增字段按 -开发指南登记到允许列表，并在中英文可观测性文档中说明。调用结果只进入日志和指标，不持久化，运维接口不提供“最近一次 -调用结果”。 +- 指标标签可以使用插件名、方法和执行位置，不包含 Scope ID 或目标键。 +- 日志经 `log_safely` 输出，不包含 Source 内容、Projection、Artifact 内容、目标输入和环境变量值；插件异常只记录类型和 + 已知错误分类，不记录异常消息。 +- 新增字段按开发指南登记到允许列表，并在中英文可观测性文档中说明。 +- 调用结果只进入日志和指标，不持久化，运维接口不提供“最近一次调用结果”。 # 缺点 @@ -1031,7 +1029,8 @@ Server 停止 - **写事务持锁**：写事务块期间持有数据库锁，SQLite 上会阻塞其他写入。插件若在块内执行慢操作，会拖慢整个 Server 的写入， 直到块超时回滚。 - **插件自有存储与核心数据不保证一致**：两者不在同一事务中，提交失败可能在自有存储中留下条目。插件需要按稳定 ID 覆盖、 - 按 revision 保存不可变记录，并以 core 的当前数据为准。 + 按内容摘要保存不可变记录、清理未被引用的记录，并以 core 的当前数据为准；自有存储还要满足幂等写入、写入即持久和 + 写后可读的要求。 - **后台调用每次都要启动 Worker 并加载插件**：子进程启动和工厂初始化有固定开销，不适合秒级高频的后台调用；插件不能在 Worker 调用之间复用连接或缓存。 - **专属存储只按约定隔离**：插件与 Server 使用同一个数据库登录，技术上能访问 core 的表。 @@ -1063,6 +1062,7 @@ Server 停止 | Source 的 `resolve` 等代码在 Server 进程中执行 | 省去进程间往返，但插件阻塞或崩溃会影响 Server；插件的同级模块以顶层名字进入 `sys.modules`，多个插件的同名模块会互相冲突，后续也无法可靠地重新加载 | | 常驻宿主进程处理同步调用，Worker 处理后台调用（本提案） | 同步调用不需要每次启动进程，Server 进程不导入插件代码；后台调用保留 Worker 的按次隔离和预算。代价是常驻进程的资源占用和进程管理 | | 插件存储进入 core 的事务（插件在 core 库中建表） | 可以原子提交，但 core 要管理插件的迁移、升级和权限，三种后端都要支持，插件数据会混入 core 的 schema。插件的存储由插件决定，不放进 core | +| 插件存储与 core 之间用两阶段提交 | 对象存储、向量库等多数插件存储不支持 XA，core 也要为每种存储实现协调者。按内容摘要先写不可变记录、以 core 的提交为唯一生效点，就能得到需要的一致性 | | 一个插件声明多个 family | 可以在一个事务中写多个 family，但声明、权限和合并注册都要按多 family 设计。需要配合的场景用两个插件加跨 family 只读即可 | | 把 core 的环境变量或数据库配置交给插件 | core 的配置不全来自环境变量；插件拿到 core 的数据库凭据就能绕过写入器。提供封装好的 `ctx.storage` 和 `ctx.models` | | 端侧单独编译二进制 | 插件是带依赖的 Python 目录，冻结后的二进制无法导入插件的任意依赖，需要为每个插件和平台单独打包。同一个程序以 `source-agent` 模式启动即可 | @@ -1090,11 +1090,9 @@ Server 停止 - **事务上下文管理器**：SQLAlchemy 的 `async with engine.begin()` 等接口在块开始时开启事务、正常结束时提交、异常时回滚。 `writer.transaction()` 沿用这一使用方式，并在提交时附带游标与调用确认。 - **PowerContext 现有基础**：[Artifact processing Supervisor](1515_artifact_processing_supervisor.md) 的调度、调用意图、 - Worker 与租约 fence；[公共合并方法](1859_artifact_merge.md)；`ArtifactRepository`、`ArtifactGovernanceRepository`、 - handle 与 tags 的写入实现；通用 Artifact 读写接口；`SourceDefinition` / `SourceProjection`；Source journal、 - `pc_source_cursors` 和 `list_window_for_generation`；`pc_connector_checkpoints`；远程采集接口（`source-observations`、 - `connector-checkpoints`、`source-definitions/register`）；内置 family 编辑时记录的 `lineage_only` Content Source；Source - 管理接口与 `identify_target` 目标识别规则。 + Worker 与租约 fence；[公共合并方法](1859_artifact_merge.md)；通用 Artifact 读写接口与内置 family 编辑时记录的 + `lineage_only` Content Source；`SourceDefinition` / `SourceProjection`、Source journal 与消费游标；connector checkpoint + 与远程采集接口；Source 管理接口与目标识别规则。 # 未决问题 From 5309c7f09ab3acbfa693b10cfacf90fff202857a Mon Sep 17 00:00:00 2001 From: jasper-yl Date: Thu, 12 Nov 2026 10:16:46 -0800 Subject: [PATCH 08/12] docs(rfc): let plugins present and search their own Artifacts, and replace the old frameworks in one migration - Rename the factory to setup; render_artifact is required, search_artifacts optional, and generate required only with a schedule or on_sources, with a methods-at-a-glance table. - Core no longer indexes plugin families: generic search passes the complete request to the plugin, and Prepared Context sections may list plugin families with optional search controls. - One new migration revision adds pc_artifacts.metadata (plugin name and version), accepts plugin: bindings without registration, keeps the FTS check to built-in rows, bumps the export record format, and refuses to upgrade while the old plugin frameworks still hold data. Co-Authored-By: Claude Opus 5.5 --- docs/en/rfcs/2549-plugin-interfaces.md | 229 ++++++++++++++++++------- docs/zh/rfcs/2549-plugin-interfaces.md | 203 ++++++++++++++++------ 2 files changed, 319 insertions(+), 113 deletions(-) diff --git a/docs/en/rfcs/2549-plugin-interfaces.md b/docs/en/rfcs/2549-plugin-interfaces.md index fc571768a2..42153e326d 100644 --- a/docs/en/rfcs/2549-plugin-interfaces.md +++ b/docs/en/rfcs/2549-plugin-interfaces.md @@ -52,7 +52,9 @@ This proposal replaces the plugin contracts and management mechanisms of the [Artifact plugin framework RFC #2175](2175-artifact-plugin-system.md). The old Source plugin framework shipped in v1.5, but no plugins for it have been published, so this proposal keeps no compatibility: the old plugin manifests, management APIs, management service, and plugin-management tables are removed, and custom Artifact data written by old Artifact -plugins is not migrated. Parts of the old design that work well are kept, including the Source management APIs +plugins is not migrated. So that no data is lost silently, the migration preflight refuses to upgrade while the old +frameworks still hold data, and the deployer exports and cleans it up on the old version first (reference section 10). +Parts of the old design that work well are kept, including the Source management APIs (`pc_source_add` and others) and their collection target identity rules, and core's existing remote ingestion APIs. This document describes the target design; it does not mean the mainline or a release already provides it. Interface @@ -196,7 +198,7 @@ is the plugin name. The directory must contain `plugin.py` and may contain other `plugin.py` assigns a declaration object to the module-level variable `plugin`: ```python -plugin = ArtifactPlugin(family="incident-report", ..., create=create) # or SourcePlugin(...) +plugin = ArtifactPlugin(family="incident-report", ..., setup=setup) # or SourcePlugin(...) ``` The declaration answers what core needs to know before it runs any plugin code: what the plugin provides (Source types or @@ -204,7 +206,7 @@ an Artifact family), what it consumes, when to call it, and its resource limits. by Pydantic; there is no YAML or other manifest file. - **Deployment settings**: the plugin's own values that vary by deployment, such as credentials and external service - addresses, are read by the plugin from environment variables in `create`. Server neither stores, passes, nor logs them. + addresses, are read by the plugin from environment variables in `setup`. Server neither stores, passes, nor logs them. - **Resources core provides**: a plugin that needs a database or models does not configure them separately; it uses objects core wraps for it through the context (see reference 1.2): - `ctx.storage`: a dedicated area for the plugin on the database backend the deployment uses. On SQLite it is a @@ -215,7 +217,7 @@ by Pydantic; there is no YAML or other manifest file. - **Enabling and disabling**: if the directory exists and passes the check, the plugin is available; to disable it, move it out of the directory and restart Server. - **Check**: at startup, Server imports `plugin.py` in a time-limited child process and validates the declaration without - calling `create`. A failed check affects only that plugin. + calling `setup`. A failed check affects only that plugin. ## 3. Source plugins @@ -226,7 +228,7 @@ by Pydantic; there is no YAML or other manifest file. | `definitions` | Yes | A sequence of `SourceDefinition`, the existing Source definition: `input_class`, `resolve`, and its Projections | | `target_input` | For collection | A Pydantic model validating the input a caller submits when adding a target | | `identify_target(value) -> TargetIdentity` | For collection | A pure function that normalizes the input into a stable target key and normalized input | -| `create(ctx) -> SourceCollector` | For collection | A factory run once per invocation in the Worker, returning an object that implements `collect` | +| `setup(ctx) -> SourceCollector` | For collection | A factory run once per invocation in the Worker, returning an object that implements `collect` | | `collect(request) -> CollectResult` | For collection | Reads external data per target and checkpoint, returning Sources and new checkpoints | A plugin that provides only `definitions` only extends Source types: callers submit content through the existing Source @@ -251,7 +253,7 @@ callers to upload files directly comes from plugin-defined APIs, see | `collect.max_items` | 200 | Maximum Sources returned per invocation | | `collect.timeout_seconds` | 120 | Total Worker time limit | | `collect.max_workers` | 1 | Concurrency limit across Scopes | -| `target_input`, `identify_target`, `create` | Required for collection | See the table above | +| `target_input`, `identify_target`, `setup` | Required for collection | See the table above | | `scopes` | `None` | Scopes the plugin may serve; empty means every Scope | ### 3.3 Example @@ -295,7 +297,7 @@ class XCollector: return result -async def create(ctx: PluginContext) -> XCollector: +async def setup(ctx: PluginContext) -> XCollector: client = await ctx.enter_async_context(XClient(os.environ["X_BEARER_TOKEN"])) return XCollector(client) @@ -305,7 +307,7 @@ plugin = SourcePlugin( collect=Collect(schedule=Interval(seconds=300), timeout_seconds=120), target_input=TargetInput, identify_target=identify_target, - create=create, + setup=setup, ) ``` @@ -362,8 +364,10 @@ powercontext source-agent --server https://memory.example.com | Interface | Required | Description | | --- | --- | --- | -| `create(ctx) -> ArtifactGenerator` | Yes | A factory; run once per invocation in a Worker, and only once and then reused in the host process | -| `generate(request, writer) -> None` | Yes | Reads `request.sources` (the unconsumed Source window) and writes Artifacts in the write transaction block | +| `setup(ctx) -> ArtifactGenerator` | Yes | The factory of the plugin object (not an Artifact create); run once per invocation in a Worker, and only once and then reused in the host process | +| `generate(request, writer) -> None` | When `schedule` or `on_sources` is set | Reads `request.sources` (the unconsumed Source window) and writes Artifacts in the write transaction block | +| `render_artifact(request) -> Rendering` | Yes | The default presentation (Markdown or plain text) of an exact revision; see reference section 7 | +| `search_artifacts(request) -> SearchResult` | No | Searches with the plugin's own indexes and returns the revisions and passages it matched; the generic search API and Prepared Context call it | | `create_artifact(request, writer) -> None` | No | A user create; core allocates `request.artifact_id` | | `update_artifact(request, writer) -> None` | No | A user edit of the given Artifact; `request.expected_revision` is the revision the caller saw | | `merge_artifacts(request, writer) -> None` | No | A user merge; core allocates the result ID | @@ -375,6 +379,11 @@ implement, the matching API returns 422, meaning the family does not support it. the content the user submitted, updates its own storage, and performs through the writer the one write operation core requires. +Core builds no index for plugin families and does not understand the structure of `content`: only the plugin knows how +its content is chunked, whether it suits full-text or vector search, and how to present it. Presentation and search +therefore belong to the plugin: `render_artifact` is required, `search_artifacts` is optional, and without it the generic +search API and Prepared Context return 422 for the family. + ### 4.2 Declaration | Field | Default | Description | @@ -382,15 +391,14 @@ requires. | `family` | Required | The Artifact family the plugin owns; must not duplicate a built-in family or another plugin's | | `inputs` | `None` | `Inputs(source_types=..., projection=..., projections={...}, max_sources=50)`: the Source types consumed, the Projection each Source is delivered in, and the maximum Sources per window; empty means the plugin consumes no Sources. `projection` applies to every type, `projections` names a different Projection per Source type, and every type needs one or the other | | `on_sources` | `False` | Trigger after new Sources arrive; requires `inputs` | -| `schedule` | `None` | `Interval(seconds=...)` or `Cron(expression=..., timezone=...)`; declare at least one of this and `on_sources` | -| `scopes` | `None` | Scopes the plugin may serve; required for plugins that consume no Sources | -| `search_fields` | `None` | Top-level content fields indexed for full-text search; empty means every top-level string field | +| `schedule` | `None` | `Interval(seconds=...)` or `Cron(expression=..., timezone=...)`. With neither `schedule` nor `on_sources`, the plugin has no background generation and need not implement `generate`; its Artifacts are written only by user changes and plugin APIs (notes, for example) | +| `scopes` | `None` | Scopes the plugin may serve; required for plugins that set `schedule` but consume no Sources | | `timeout_seconds` | 300 | Time limit for the whole invocation | | `transaction_timeout_seconds` | 30 | Time limit for the write transaction block, at most `timeout_seconds` | | `max_workers` | 1 | Concurrency limit across Scopes | | `change_timeout_seconds` | 30 | Time limit for one user-change call, including its write transaction block | | `definitions` | `()` | The plugin's own Source Definitions, registered when the plugin loads, under the same rules as a Source plugin's `definitions`. The plugin submits them from its routes with `ctx.sources.submit`, and callers can also submit them through the generic Source write API, so one Artifact plugin can record what users asked for without a separate Source plugin | -| `create` | Required | See the table above | +| `setup` | Required | See the table above | ### 4.3 Example @@ -402,7 +410,8 @@ configured. ```python from powercontext.plugins import ( - ArtifactPlugin, ArtifactWriter, GenerateRequest, Inputs, InvalidContent, PluginContext, UpdateRequest, + ArtifactPlugin, ArtifactWriter, GenerateRequest, Inputs, InvalidContent, PluginContext, Rendering, RenderRequest, + SearchHit, SearchRequest, SearchResult, UpdateRequest, ) from powercontext.sources import TextEvidence @@ -441,8 +450,20 @@ class IncidentReports: expected_revision=request.expected_revision) await self.index.upsert(request.scope_id, written.ref.artifact_id, written.ref.revision, vector) + async def render_artifact(self, request: RenderRequest) -> Rendering: + report = Report.model_validate(request.artifact.content) + return Rendering(text=f"# {report.title}\n\n{report.summary}") + + async def search_artifacts(self, request: SearchRequest) -> SearchResult: + # The generic search request arrives unchanged; this plugin supports vector search only + if request.search.mode not in (None, "vector"): + raise InvalidContent("only vector search is supported", field="mode") + vector = await self.model.embed(request.search.query) + hits = await self.index.nearest_many(request.scope_id, vector, limit=request.search.limit) + return SearchResult(hits=[SearchHit(ref=hit.ref, text=hit.summary, score=hit.score) for hit in hits]) + -async def create(ctx: PluginContext) -> IncidentReports: +async def setup(ctx: PluginContext) -> IncidentReports: index = await ctx.enter_async_context(VectorIndex.connect(ctx.storage.engine())) return IncidentReports(index, Summarizer(ctx.models)) @@ -451,8 +472,7 @@ plugin = ArtifactPlugin( family="incident-report", inputs=Inputs(source_types=("example-ticket",), projection="powercontext.text-evidence@1"), on_sources=True, - search_fields=("title", "summary"), - create=create, + setup=setup, ) ``` @@ -463,14 +483,16 @@ left by the previous attempt, and reads from the owned index defer to core's cur ### 4.4 How callers use it Callers access plugin families through the existing Artifact APIs. Read APIs read `pc_artifacts` directly and run no -plugin code; for change APIs, core checks authorization and versions and then synchronously calls the matching plugin -method: +plugin code; for presentation, search, and change APIs, core checks authorization and then synchronously calls the +matching plugin method: | Purpose | API | Plugin method | | --- | --- | --- | | Browse | `GET /v1/scopes/{scope_id}/artifacts/{family}` | None | | Read the current content, a given revision, revision history, tags | `GET .../{artifact_id}`, `GET .../{artifact_id}/revisions/{revision}`, `GET .../{artifact_id}/revisions`, `GET .../{artifact_id}/tags` | None | -| Text search | `POST /v1/scopes/{scope_id}/artifacts/{family}/search` | None | +| Default presentation | `GET .../{artifact_id}/rendering` (optionally with `revision`) | `render_artifact` | +| Search (the request is passed on unchanged; the plugin interprets the search controls) | `POST /v1/scopes/{scope_id}/artifacts/{family}/search` | `search_artifacts` | +| Prepare context for an Agent | List the plugin family in `assembly.sections` of `POST /v1/context/prepare` | `search_artifacts` | | Create (submit only content; core allocates the ID) | `POST /v1/scopes/{scope_id}/artifacts` with body `{"family": ..., "content": {...}}` | `create_artifact` | | Edit (submit complete content; `If-Match` checks the current revision) | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | `update_artifact` | | Read the governance state (the `state_version` needed to forget or merge) | `GET .../{artifact_id}/governance` | None | @@ -479,8 +501,24 @@ method: | Restore, restore an earlier revision, undo a merge | `POST .../{artifact_id}/restorations` | `restore_artifact` | | Change tags | `PUT .../{artifact_id}/tags`; query by tag with `POST /v1/scopes/{scope_id}/artifact-tags/query` | None; handled by core | -Changes return their result synchronously. A plugin rejecting the content returns 422, a plugin that does not implement -the method also returns 422, and a host process that times out or is unavailable returns 503. +Changes return their result synchronously. A plugin rejecting the content or request returns 422, a plugin that does not +implement the method also returns 422, and a host process that times out or is unavailable returns 503. + +### 4.5 Methods at a glance + +| Method | Applies to | Required | When core calls it | Where it runs | +| --- | --- | --- | --- | --- | +| `setup(ctx)` | Both plugin kinds | Yes | Once when the host process starts; once per Worker invocation | Host process, Worker | +| Source Definitions: `input_class`, `resolve`, a Projection's `project`, `read` | Source plugins, or Artifact plugins that declare `definitions` | Yes | Source writes (the generic write API, `ctx.sources.submit`); computing Projections when delivering Sources to `generate` | Host process, Worker | +| `identify_target`, `collect` | Source plugins | Yes, for continuous collection | Adding a collection target; scheduled collection | Host process, Worker | +| `generate` | Artifact plugins | When `schedule` or `on_sources` is set | On schedule, or when new Sources arrive | Worker | +| `render_artifact` | Artifact plugins | Yes | The default presentation API | Host process | +| `search_artifacts` | Artifact plugins | No | The generic search API; Prepared Context sections that list the family | Host process | +| `create_artifact`, `update_artifact`, `merge_artifacts`, `forget_artifact`, `restore_artifact` | Artifact plugins | No | The matching generic change APIs | Host process | + +Listing, reading the current content or a given revision, revision history, the governance state, and tags call no +plugin method; core serves them from the data it stores. The host process checks the required methods after running +`setup`; a plugin missing one fails to load and is `failed`. ## 5. Scheduling and triggers @@ -554,7 +592,7 @@ class ArtifactPlugin(BaseModel): # fields in guide 4.2 family: str inputs: Inputs | None = None ... - create: Callable[[PluginContext], Awaitable[ArtifactGenerator]] + setup: Callable[[PluginContext], Awaitable[ArtifactGenerator]] class SourcePlugin(BaseModel): # fields in guide 3.2 definitions: Sequence[SourceDefinition] @@ -564,8 +602,8 @@ class SourcePlugin(BaseModel): # fields in guide 3.2 - The plugin name comes from the directory name and must match `^[a-z][a-z0-9-]{0,63}$`. - A declaration is validated when constructed, including schedule combinations, the dependency between `inputs` and - `scopes`, time-limit relations, and family naming. `create` must be an async function. -- The parts of a declaration other than code objects such as `create`, `identify_target`, and `definitions` are + `scopes`, time-limit relations, and family naming. `setup` must be an async function. +- The parts of a declaration other than code objects such as `setup`, `identify_target`, and `definitions` are serializable. Server obtains them from the check child process to register bindings and answer operational queries. - During the check, core records the plugin directory's **code fingerprint** (computed from file contents). Workers and host processes check it before importing the plugin, and a mismatch fails the call and asks for a Server restart, @@ -720,6 +758,7 @@ conflict and the call fails. ```python class ArtifactGenerator(Protocol): + # required when schedule or on_sources is set async def generate(self, request: GenerateRequest, writer: ArtifactWriter, /) -> None: ... class GenerateRequest(BaseModel): @@ -795,8 +834,10 @@ Write constraints: - Unchanged content produces no revision, as in built-in Atomic Memory: when the model finds that an existing memory already expresses a new Source, that memory gets no new revision and the new Source does not enter its lineage. A plugin that needs a new Source recorded as evidence for an Artifact should change its content as well. -- Core extracts search text from the content according to the declared `search_fields`; plugins do not supply it - separately. The same content yields the same search text whether a plugin or a user wrote it. +- Core extracts no search text and builds no index for plugin families; plugins search with their own indexes (see + section 7). +- Core records, in each revision's metadata, the name and version digest of the plugin that wrote it, so when several + Servers load different versions each revision can be traced to the version that wrote it (see section 10). - Each write transaction block has limits on operations, single content size, and total bytes; exceeding one raises and rolls back, with no truncation or partial commit. - Core does not validate the business structure of `content`, nor store or pin a schema. The plugin owns how its content @@ -830,7 +871,6 @@ class ArtifactReads(Protocol): async def revisions(self, artifact_id: str, *, cursor: str | None = None, limit: int = 50) -> ArtifactPage: ... async def list(self, *, states: Collection[LifecycleState] = ("active",), tags: Collection[str] = (), cursor: str | None = None, limit: int = 50) -> ArtifactPage: ... - async def search(self, query: str, *, limit: int = 10) -> Sequence[ArtifactSnapshot]: ... async def by_source(self, ref: SourceRef) -> Sequence[ArtifactSnapshot]: ... async def merge_inputs(self, artifact_id: str) -> Sequence[ArtifactRef]: ... async def tags(self, artifact_id: str) -> Sequence[str]: ... @@ -844,8 +884,6 @@ class ArtifactReads(Protocol): inputs of a merge result. - `list` filters by lifecycle state and tags, including deprecated, retired, and merge-frozen Artifacts, for finding data that already existed. -- `search` is core's text search (see section 7) and covers only active current revisions; inside the write transaction it - also finds content the block has written but not yet committed. - `by_source` finds the Artifacts whose lineage references a given Source, for locating Artifacts to revise after a Source changes. - Every returned snapshot carries an exact revision and can be used directly as `artifact_refs` or `to_revision` in writes. @@ -1043,11 +1081,12 @@ long-lived host process: | Input validation, `resolve`, and Projections for Source writes | Host process | | `identify_target` when adding a collection target | Host process | | `create_artifact`, `update_artifact`, `merge_artifacts`, `forget_artifact`, `restore_artifact` | Host process | +| `render_artifact`, `search_artifacts` | Host process | | `collect`, `generate` | Supervisor Worker | - **Count and startup**: each Server process keeps at most one host process for each loaded plugin, starts it in the - background once the plugin loads, and reuses it afterward. Whether a plugin implements user-change methods is known only - after `create` runs, so an Artifact plugin that provides only `generate` also starts one, at the cost of an idle process. + background once the plugin loads, and reuses it afterward. Every Artifact plugin implements `render_artifact` and + Source plugin writes run `resolve`, so every loaded plugin has a host process. In edge mode, `resolve` and Projections run inside the collection Worker, and no host process starts. - **Execution**: the host process imports `plugin.py` once at startup and compares the declaration, and the writer runs @@ -1068,29 +1107,66 @@ persisted or synchronized across processes, and is rebuilt from the plugin direc operational APIs return only the check results and declarations, which are the same on every process; host process state is visible through logs and metrics. -## 7. Public reads and search +## 7. Public reads, presentation, and search -Core serves all public reads of plugin families without running plugin code: +Core serves reads directly, without running plugin code: -- list / get / revisions / tags reuse the existing generic Artifact read paths. Plugin family `content` is returned as JSON - as is, without a schema in `pc_artifact_family_schemas`. Merge-frozen, deprecated, and retired Artifacts are presented - under the shared merge and lifecycle rules. +- list / get / revisions / governance / tags reuse the existing generic Artifact read paths. Plugin family `content` is + returned as the JSON the plugin wrote, without a schema in `pc_artifact_family_schemas`. Merge-frozen, deprecated, and + retired Artifacts are presented under the shared merge and lifecycle rules. - List items carry only metadata; `title` and `summary` are empty for plugin families, and content is read with get. -- Search is core's generic text search for plugin families and, like the existing search APIs, accepts only text queries: - - On write, core takes the text of the `search_fields` from the content, analyzes it with the existing lexical analysis, - writes it to `pc_artifact_heads.searchable_text`, and keeps it with the revision, so restoring an old revision restores - its search text with its content. - - On search, core matches that column and returns only active current revisions the caller may read in the requested - Scope. Artifacts without searchable text do not take part. - - The writer's `search` uses the same implementation. - - The implementation PR presents the search implementation on each database backend and validates it at scale. -- Review, Recall, and Prepared Context are not open to plugin families in the first version; those APIs return an explicit - "not supported" error, and `GET /v1/capabilities` lists plugin families. + +A plugin may keep only digests or references in `content` and store large content in its own storage (a knowledge base +document, for example, records only the digest of its chunk set). The `content` returned as is then cannot be presented. +Presentation and search therefore belong to the plugin: + +```python +class RenderRequest(BaseModel): + scope_id: str + artifact: ArtifactSnapshot # the exact revision to present + caller: Caller | None + +class Rendering(BaseModel): + format: Literal["markdown", "text"] = "markdown" + text: str + +class SearchRequest(BaseModel): + scope_id: str + search: SearchArtifactsRequest # the complete request of the generic search API, unchanged + max_bytes: int | None = None # from Prepared Context: the byte budget left for this family + caller: Caller | None + +class SearchHit(BaseModel): + ref: ArtifactRef # the exact revision matched + text: str # a passage to use as context, such as one chunk + score: float | None = None + locator: dict[str, JsonValue] | None = None # where the passage sits in the Artifact, such as a page or chunk number + +class SearchResult(BaseModel): + hits: Sequence[SearchHit] +``` + +- **Presentation**: `GET .../{artifact_id}/rendering` calls `render_artifact` and returns the plugin's default presentation; + read APIs still return the original `content` for programs. A revision's presentation does not change under the same + plugin version, so core may cache it by family, ID, revision, and plugin version. Built-in families do not offer this API. +- **Search**: the generic search API passes every field of `SearchArtifactsRequest` (`query`, `query_vector`, `indexes`, + `mode`, `filters`, `admission`, `fusion`, `rerank`, `min_score`, and so on) unchanged to `search_artifacts`, which + interprets them against the plugin's own indexes; a plugin raises `InvalidContent` for controls it does not support, and + the API returns 422. Core keeps only matches that are active current revisions the caller may read, returns the exact + revisions in the plugin's order, and adds the plugin's scores when `include_scores` is set. A plugin therefore cannot + return data the caller cannot see through search. +- **Prepared Context**: `assembly.sections` may list plugin families. Core calls `search_artifacts` with Prepared Context's + `query`, the section's `limit`, and the remaining byte budget, and uses the matched passages as context entries that, + like built-in entries, cite exact revisions and count toward the byte budget and the receipt. A section may carry an + optional `search` with the fields of a search request other than `query` and `limit`, passed on to the plugin + unchanged; built-in families that carry `search` return 422. +- Review and Recall are not open to plugin families; those APIs return an explicit "not supported" error, and + `GET /v1/capabilities` lists plugin families. ## 8. Loading and lifecycle - At startup, Server checks each direct subdirectory of `POWERCONTEXT_SERVER_PLUGINS_DIR` (no plugins load if it is - unset): it imports `plugin.py` in a time-limited child process and validates the declaration without calling `create`. + unset): it imports `plugin.py` in a time-limited child process and validates the declaration without calling `setup`. Once the check passes, Server registers the Source types or family the plugin provides (reads, search, and the generic merge APIs), registers a Supervisor binding when there is a schedule or trigger, and adds the plugin to this process's record of active plugins. A failed check affects only that plugin. @@ -1143,19 +1219,37 @@ Core serves all public reads of plugin families without running plugin code: | `pc_source_cursors` | Kept; Artifact plugin consumption cursors, keyed by `(scope_id, plugin:)` | | `pc_connector_checkpoints` | Kept; each collection target's checkpoint, keyed by `(scope_id, the target's binding ID)` | | `pc_source_collect_targets` | Added; collection targets with their normalized input, enabled state, and latest collection results | -| Supervisor invocation intent, pending, and lease tables | Kept; plugin bindings share them with built-in bindings, and they back `GET /v1/plugins/{name}/scopes` | +| Supervisor invocation intent, pending, and lease tables | Kept; plugin bindings share them with built-in bindings, and they back `GET /v1/plugins/{name}/scopes`. Plugin bindings start with `plugin:` and are not registered in the database | +| The new `metadata` column of `pc_artifacts` | Added; nullable, stored as canonical JSON bytes like `content`, for additional information that is never searched. Revisions plugins write record the plugin name and version digest; built-in families leave it empty. It does not take part in content digests | +| `pc_artifact_fts` and `pc_artifact_heads.searchable_text` | Used only by built-in families; plugin families do not write them | | Columns the old plugin frameworks added to `pc_artifact_processing_binding_states` (desired state, management phase, registration generation, and so on) | Removed | | `pc_artifact_family_schemas` | Removed | | Search state and vector index data for custom family declarative indexes | Removed; the implementation PR confirms built-in families do not depend on them before dropping the tables | -| Artifacts, heads, and lineage written by old Artifact plugins | Deleted, not migrated | +| Artifacts, heads, and lineage written by old Artifact plugins | Not migrated; the upgrade is refused while any remain, see below | | `pc_source_write_plugins` | Removed; Source definitions come from the plugin directory at startup | | The Source plugin management service's SQLite | Removed with the management service | | Dedicated plugin storage (`plugins/.sqlite3` or the `powercontext_plugin_` database) | Not part of core's schema; core only allocates it, and the plugin and the deployer own its tables, migrations, and cleanup | | Other plugin-owned indexes and storage | Not part of core; managed by the plugin and the deployer | | The local SQLite of edge mode | Holds only the edge's collection targets and scheduling state, no Sources; Sources and checkpoints live on the central Server | -Additions and removals ship as one new database migration whose notes list the tables, columns, and data -ranges added and removed. +Additions and removals ship as **one new migration revision**; the migration resources of released versions stay +unchanged. Its notes list the tables, columns, and data ranges added and removed. The migration preflight first confirms the old plugin frameworks +left no data behind: Artifacts of old Artifact plugin families, old plugin registrations and invocation intents, and +Source plugins in `pc_source_write_plugins`. If it finds any, it refuses the upgrade, lists the plugin names, families, and +row counts, and leaves the database unchanged; the deployer exports what they need, uninstalls the old plugins, and cleans +up their data on the old version, then upgrades. The migration drops these tables only when they are empty. + +The readiness check of this revision, +which Server runs against the current database at every startup, differs from the previous one in two ways: + +- Bindings of invocation intents that start with `plugin:` count as plugin bindings and need no database registration. + Intents a removed plugin left behind are not processed, are visible in the operational APIs, and do not stop Server + from starting; they are processed again once a plugin with the same name returns. +- The full-text index check covers only built-in family data. + +The format version of Artifact revision records in export and restore (portable archives) increases by one and includes +`metadata`. Restoring an export of the old format leaves `metadata` empty; older Servers cannot restore exports of the +new format. ## 11. API changes @@ -1163,7 +1257,7 @@ ranges added and removed. | --- | --- | | Removed | `/v1/admin/artifact-plugins` and its subpaths; the Source plugin management service, plugin instance Runtimes, and poll / reply channels; the `source load` command; MCP `pc_source_instances` and `pc_source_instance_set_enabled`; the `plugin.yml` and `powercontext.yml` manifests | | Removed | Custom family declarative indexes (`ArtifactSearchDeclaration`, `TextIndex`, and so on), `ArtifactFamily` schema declarations, `ArtifactDraft`, and the old Worker plugin executor | -| Added | `powercontext.plugins`: `SourcePlugin`, `ArtifactPlugin`, `Inputs`, `Interval`, `Cron`, `PluginContext`, `PluginStorage`, `PluginModels`, the writer and read interfaces, and the user-change request models and `InvalidContent` | +| Added | `powercontext.plugins`: `SourcePlugin`, `ArtifactPlugin`, `Inputs`, `Interval`, `Cron`, `PluginContext`, `PluginStorage`, `PluginModels`, the writer and read interfaces, and the user-change request models and `InvalidContent`, and the presentation and search request and result models | | Added | The `powercontext source-agent` edge collection mode, with `POWERCONTEXT_SOURCE_AGENT_ID` and `POWERCONTEXT_SOURCE_AGENT_TOKEN` | | Changed | The body of `POST /v1/scopes/{scope_id}/artifacts` gains a plugin branch `{"family", "content"}`; core allocates the ID, and the plugin's `create_artifact` handles it. `CreateArtifactRequest` becomes a choice between the built-in branch `CreateBuiltinArtifactRequest` (the former union discriminated by family) and the plugin branch `CreatePluginArtifactRequest`, selected by family name: built-in names take the built-in branch with unchanged validation; other names take the plugin branch, and a family no plugin provides returns 422. In the generated Python client, built-in requests gain one wrapping level, and `ArtifactCreated.family` changes from the built-in family enum to a string | | Changed | The `source_type` of `POST /v1/scopes/{scope_id}/sources` changes from an enum allowing only `content` to a string (still defaulting to `content`). It may name a Source type a plugin defines; `content` is then that type's input, which the plugin validates and resolves into a Source. Unknown types return 404. The `SourceType` enum remains | @@ -1171,7 +1265,10 @@ ranges added and removed. | Added | `GET .../{artifact_id}/governance`, `POST /v1/scopes/{scope_id}/artifacts/{family}/merges`, `POST .../{artifact_id}/forget`, `POST .../{artifact_id}/restorations`: generic APIs on the shared merge engine registry that work for every registered family (Experience and Atomic Memory among built-ins, plus loaded plugin families, which the matching plugin methods handle). Atomic Memory's dedicated APIs are unchanged | | Changed | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` accepts plugin families with a `{"content": {...}}` body, handled by the plugin's `update_artifact` | | Changed | The tag APIs accept plugin families: `TaggableArtifactFamily` changes from an enum to a string in the `ArtifactReadFamily` format, and Server accepts built-in families or loaded plugin families. The wire format is unchanged; generated clients lose the enum type | -| Changed | The generic Artifact read and search APIs support plugin families; `GET /v1/capabilities` lists plugin families | +| Changed | The generic Artifact read APIs support plugin families; for plugin families, the generic search API passes the complete request to the plugin's `search_artifacts` and returns 422 when it is not implemented; `GET /v1/capabilities` lists plugin families | +| Added | `GET /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}/rendering`: the default presentation of a plugin family Artifact, provided by the plugin's `render_artifact` | +| Changed | `POST /v1/context/prepare`: `ContextAssemblySection.family` changes from an enum to a string that may name a loaded plugin family; an optional `search` (the fields of a search request other than `query` and `limit`) applies to plugin families only | +| Changed | The format version of Artifact revision records in export and restore increases by one and adds `metadata` | | Changed | The write usage of `pc_source_add` uses plugin definitions, and its collect usage adds collection targets; `pc_source_capabilities` lists plugin Source types and target inputs; `pc_source_status` reports targets and their latest collection result; `pc_source_target_set_enabled` acts on targets stored by core | | Changed | The environment variable `POWERCONTEXT_SERVER_PLUGINS_DIR` replaces `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` and the Source plugin directory setting | @@ -1220,8 +1317,13 @@ record one event. such as its family from environment variables. - **Content structure is not governed by core**: core does not validate a `content` schema, and plugins validate user-submitted content; plugins and applications must keep historical revisions interpretable. -- **Core search is limited**: core provides only generic text search over declared fields; vector and other search requires - a plugin-owned index, which the public search APIs cannot use yet. +- **Presentation and search depend on the plugin**: presentation, search, and Prepared Context for plugin families go + through the host process and return 503 when the plugin is unavailable; without `search_artifacts`, a plugin's data cannot + be used through generic search or Prepared Context. +- **Servers may load different plugin versions**: core records which plugin version wrote each revision but does not stop + instances from loading different versions; whether old and new versions can read and write the same data at once is the + plugin's responsibility. Summarizing the versions each instance has loaded needs an instance registry table, which is + left to the follow-up plugin installation proposal. - **Code changes require a restart**: without hot reload, Server is briefly unavailable while plugins update. # Rationale and alternatives @@ -1248,6 +1350,10 @@ record one event. | Compile a separate binary for the edge | A plugin is a Python directory with dependencies; a frozen binary cannot import arbitrary plugin dependencies and would need packaging per plugin and platform. Starting the same program in `source-agent` mode is enough | | Plugin-specific trigger and management APIs | Callers would have to tell plugins from built-ins; the existing Source and Artifact APIs already cover the operations callers need | | Generic merge APIs for plugin families only | Needs a separate family allowlist in the APIs. The registry already decides which families can be changed, and a family without an adapter cannot register; built-in permission checks live in the engine, so both entry points apply the same checks | +| Core builds a full-text index for plugin families from declared fields | Plugin authors need no index of their own, but core does not know how content is chunked or whether it suits full-text or vector search, so the index is often useless; it would also share index tables with built-in families, and the readiness check would have to tell them apart. Search belongs to the plugin, and core only filters by permission | +| Reads also go through the plugin (for example a `read_artifact` that expands content) | Generic reads would return complete content, but the most frequent read path would enter the host process every time, and reads would fail whenever the plugin is unavailable. Reads stay with core, and presentation has its own API | +| Register plugin bindings and families in the database (the old Artifact plugin framework's approach) | Detects another plugin taking over an existing family and leftover invocation intents, but core must store and maintain plugin state and provide a management operation to resolve conflicts. The new framework has no stable plugin identity, so a registry would catch little more; it is left to the proposal on plugin installation and distribution, to be designed with plugin identity | +| Record the plugin version in a separate table, in `created_by`, or under a reserved key of `content` | A separate table must also join export and restore; changing `created_by` or `content` alters the meaning of existing fields and the content digest. A `metadata` column needs only one added column and an export format upgrade | | Hot reload in this proposal | Requires each instance to rescan on its own, fencing commits from old Workers, and switching host processes; a separate proposal is clearer | Reasons for this choice: @@ -1285,12 +1391,14 @@ Reasons for this choice: | Question | Notes | | --- | --- | -| Plugin-defined search | The existing search APIs accept only text queries, so the first version provides generic text search in core. Search a plugin provides (for example vector or hybrid search) is left to the separate proposal for plugin-defined APIs and runs in the host process | | Default limits for the write transaction block | Whether 30 seconds is right; whether SQLite and OceanBase need different defaults; per-block operation and byte limits | | Host process defaults | Default synchronous call limit, per-host-process concurrency limit, and restart backoff | | Permission to create the dedicated database | When Server's database login cannot create databases, whether to allow falling back to tables prefixed by plugin name in core's database, besides the deployer creating the database in advance | | Deleting collection targets | The first version supports only disabling and resuming. Whether targets and their checkpoints should be deletable, and what happens to Sources already collected | -| Plugin families in Recall / Prepared Context | Requires defining candidate content and budget constraints; a separate proposal is recommended | +| Plugin families in Recall | Prepared Context is covered; whether Recall should be, and how its candidates are defined | +| Passages in generic search results | The generic search API returns only exact revisions, and the passages a plugin returns are used only by Prepared Context. Whether search results should add passages and their locations | +| Naming families and Source types | Whether they must start with the plugin name, so naming alone keeps plugins from taking over each other's families | +| Search controls in Prepared Context | Letting built-in families accept a section's `search` too is a new optional field that leaves existing calls unaffected; a separate proposal is recommended. Removing or renaming Prepared Context's other feature-specific parameters is a breaking change that needs a new API version, so v1 leaves them as they are | | Shrinking the window | Built-in Atomic Memory shrinks the next Source window when the model times out. Plugins cannot affect the window size, so an oversized window keeps timing out. Whether plugins may request a smaller window for the Scope with an agreed exception | | Windows that keep failing | A Source window that always fails blocks later invocations in the Scope. Whether to offer an operation that skips it or moves it to a dead letter, and how to record skipped input | | Consuming every Source type | `inputs.source_types` must list each type; built-in Atomic Memory consumes every Source usable for generation. Whether to support `"*"` for all types | @@ -1303,4 +1411,5 @@ Reasons for this choice: - Long-lived processes for background calls too, reusing connections and initialization while keeping the interfaces unchanged. - More plugin types, such as candidate content for Recall or text fragments for Prepared Context. -- Plugin packaging and installation tools, such as installing into the plugin directory from a Git URL or a wheel. +- Plugin installation and dependencies: installing plugins from PyPI, a Git URL, or a wheel, a separate dependency + environment per plugin, and a registry of the plugin versions each instance has loaded. diff --git a/docs/zh/rfcs/2549-plugin-interfaces.md b/docs/zh/rfcs/2549-plugin-interfaces.md index 4352c64acb..826f82750b 100644 --- a/docs/zh/rfcs/2549-plugin-interfaces.md +++ b/docs/zh/rfcs/2549-plugin-interfaces.md @@ -39,7 +39,8 @@ description: Source 和 Artifact 插件各自实现少量接口，由 PC Server 本提案取代 [Source 插件框架 RFC #2012](2012-source-plugin-system.md) 和 [Artifact 插件框架 RFC #2175](2175-artifact-plugin-system.md) 的插件契约与管理机制。旧 Source 插件框架随 v1.5 发布，但目前没有对外发布的插件，因此本提案不保留兼容：旧的插件清单、 -管理接口、管理服务和插件管理相关的表直接移除，旧 Artifact 插件写入的自定义 Artifact 数据不迁移。旧设计中合适的部分继续 +管理接口、管理服务和插件管理相关的表直接移除，旧 Artifact 插件写入的自定义 Artifact 数据不迁移。为了不悄悄丢失数据， +升级前的迁移检查发现旧框架留有数据时拒绝升级，由部署者在旧版本上先导出和清理（见参考级说明第 10 节）。旧设计中合适的部分继续 沿用，包括 Source 管理接口（`pc_source_add` 等）及采集目标的身份规则，以及 core 已有的远程采集接口。 本文描述目标设计，不代表主线或发布版本已经可用。接口名称和代码示例用于说明契约，最终语法以实现为准。 @@ -169,13 +170,13 @@ Server 从 `POWERCONTEXT_SERVER_PLUGINS_DIR` 加载插件。每个直接子目 `plugin.py` 把一个声明对象赋给模块级变量 `plugin`： ```python -plugin = ArtifactPlugin(family="incident-report", ..., create=create) # 或 SourcePlugin(...) +plugin = ArtifactPlugin(family="incident-report", ..., setup=setup) # 或 SourcePlugin(...) ``` 声明对象回答 core 在运行插件代码之前需要知道的问题：插件提供什么（Source 类型或 Artifact family）、消费什么、何时调用、 资源上限多少。它是普通的 Python 对象，由 Pydantic 校验；没有 YAML 或其他清单文件。 -- **部署设置**：插件自己的凭据、外部服务地址等随部署变化的值，由插件在 `create` 中从环境变量读取。Server 不保存、 +- **部署设置**：插件自己的凭据、外部服务地址等随部署变化的值，由插件在 `setup` 中从环境变量读取。Server 不保存、 不传递也不记录这些值。 - **core 提供的资源**：插件需要数据库和模型时，不必另行配置，通过上下文使用 core 封装好的对象（见参考级说明 1.2）： - `ctx.storage`：部署所用数据库后端上的一块插件专属空间。SQLite 是数据目录下的一个单独文件，OceanBase 和 seekdb @@ -183,7 +184,7 @@ plugin = ArtifactPlugin(family="incident-report", ..., create=create) # 或 So - `ctx.models`：Server 已配置的生成模型和 embedding 模型。插件只能调用，拿不到模型的地址和凭据；调用计入现有的模型 用量统计。 - **启用与停用**：目录存在且检查通过，插件就可用；需要停用时移出目录并重启 Server。 -- **检查**：Server 启动时在有时限的子进程中导入 `plugin.py`，校验声明，不调用 `create`。检查失败只影响该插件。 +- **检查**：Server 启动时在有时限的子进程中导入 `plugin.py`，校验声明，不调用 `setup`。检查失败只影响该插件。 ## 3. Source 插件 @@ -194,7 +195,7 @@ plugin = ArtifactPlugin(family="incident-report", ..., create=create) # 或 So | `definitions` | 是 | `SourceDefinition` 序列，沿用现有的 Source 定义：`input_class`、`resolve` 和各 Projection | | `target_input` | 采集时必需 | Pydantic 模型，校验调用方添加目标时提交的输入 | | `identify_target(value) -> TargetIdentity` | 采集时必需 | 纯函数，把输入规范化为稳定的目标键和规范化输入 | -| `create(ctx) -> SourceCollector` | 采集时必需 | 工厂，在 Worker 中每次调用执行一次，返回实现了 `collect` 的对象 | +| `setup(ctx) -> SourceCollector` | 采集时必需 | 工厂，在 Worker 中每次调用执行一次，返回实现了 `collect` 的对象 | | `collect(request) -> CollectResult` | 采集时必需 | 按目标和 checkpoint 读取外部数据，返回 Source 和新的 checkpoint | 只提供 `definitions` 的插件只扩展 Source 类型：调用方通过现有的 Source 写入接口提交内容，core 调用定义完成校验、 @@ -216,7 +217,7 @@ Source 与所有内置 Source 一样只能追加，没有修改和删除；要 | `collect.max_items` | 200 | 单次调用返回的 Source 条数上限 | | `collect.timeout_seconds` | 120 | Worker 总时限 | | `collect.max_workers` | 1 | 跨 Scope 的并发上限 | -| `target_input`、`identify_target`、`create` | 采集时必填 | 见上表 | +| `target_input`、`identify_target`、`setup` | 采集时必填 | 见上表 | | `scopes` | `None` | 可服务的 Scope 名单；为空表示所有 Scope | ### 3.3 示例 @@ -260,7 +261,7 @@ class XCollector: return result -async def create(ctx: PluginContext) -> XCollector: +async def setup(ctx: PluginContext) -> XCollector: client = await ctx.enter_async_context(XClient(os.environ["X_BEARER_TOKEN"])) return XCollector(client) @@ -270,7 +271,7 @@ plugin = SourcePlugin( collect=Collect(schedule=Interval(seconds=300), timeout_seconds=120), target_input=TargetInput, identify_target=identify_target, - create=create, + setup=setup, ) ``` @@ -318,8 +319,10 @@ powercontext source-agent --server https://memory.example.com | 接口 | 必需 | 说明 | | --- | --- | --- | -| `create(ctx) -> ArtifactGenerator` | 是 | 工厂；Worker 中每次调用执行一次，宿主进程中只执行一次后复用 | -| `generate(request, writer) -> None` | 是 | 读取 `request.sources`（未消费的 Source 窗口），在写事务块中写入 Artifact | +| `setup(ctx) -> ArtifactGenerator` | 是 | 插件对象的工厂（不是新建 Artifact）；Worker 中每次调用执行一次，宿主进程中只执行一次后复用 | +| `generate(request, writer) -> None` | 配置了 `schedule` 或 `on_sources` 时是 | 读取 `request.sources`（未消费的 Source 窗口），在写事务块中写入 Artifact | +| `render_artifact(request) -> Rendering` | 是 | 给出一个精确 revision 的默认展示内容（Markdown 或纯文本），见参考级说明第 7 节 | +| `search_artifacts(request) -> SearchResult` | 否 | 用插件自己的索引检索，返回命中的 revision 和片段；通用检索接口和 Prepared Context 调用它 | | `create_artifact(request, writer) -> None` | 否 | 用户新建；`request.artifact_id` 由 core 分配 | | `update_artifact(request, writer) -> None` | 否 | 用户编辑指定 ID 的 Artifact，`request.expected_revision` 为调用方看到的 revision | | `merge_artifacts(request, writer) -> None` | 否 | 用户合并；结果的 ID 由 core 分配 | @@ -329,6 +332,10 @@ powercontext source-agent --server https://memory.example.com 后五个方法处理用户修改（见 4.4），每个都是可选的：没有实现的操作，对应接口返回 422，表示这个 family 不支持该操作。 插件在这些方法中校验用户提交的内容、更新自己的存储，并通过写入器完成 core 要求的那一项写操作。 +core 不为插件 family 建立任何索引，也不理解 `content` 的结构：插件的内容怎样分块、是否可以全文或向量检索、怎样展示， +只有插件知道。因此展示和检索都交给插件：`render_artifact` 必须实现，`search_artifacts` 可选，未实现时通用检索接口和 +Prepared Context 对该 family 返回 422。 + ### 4.2 声明 | 字段 | 默认值 | 说明 | @@ -336,15 +343,14 @@ powercontext source-agent --server https://memory.example.com | `family` | 必填 | 插件拥有的 Artifact family，不能与内置 family 或其他插件重复 | | `inputs` | `None` | `Inputs(source_types=..., projection=..., projections={...}, max_sources=50)`：消费的 Source 类型、每条 Source 交付的 Projection、单个窗口的条数上限；为空表示不消费 Source。`projection` 适用于所有类型，`projections` 按 Source 类型指定不同的 Projection，每个类型必须有其一 | | `on_sources` | `False` | 新 Source 到达后触发；需要 `inputs` | -| `schedule` | `None` | `Interval(seconds=...)` 或 `Cron(expression=..., timezone=...)`；与 `on_sources` 至少配置一项 | -| `scopes` | `None` | 可服务的 Scope 名单；不消费 Source 的插件必须填写 | -| `search_fields` | `None` | 参与全文检索的顶层内容字段；为空表示所有顶层字符串字段 | +| `schedule` | `None` | `Interval(seconds=...)` 或 `Cron(expression=..., timezone=...)`。`schedule` 和 `on_sources` 都不配置时插件没有后台生成，不需要实现 `generate`，Artifact 只由用户修改和插件接口写入（例如笔记） | +| `scopes` | `None` | 可服务的 Scope 名单；配置了 `schedule` 但不消费 Source 的插件必须填写 | | `timeout_seconds` | 300 | 整次调用的时限 | | `transaction_timeout_seconds` | 30 | 写事务块的时限，不超过 `timeout_seconds` | | `max_workers` | 1 | 跨 Scope 的并发上限 | | `change_timeout_seconds` | 30 | 一次用户修改调用的时限，包括其中的写事务块 | | `definitions` | `()` | 插件自己的 Source 定义，随插件加载登记，规则与 Source 插件的 `definitions` 相同。插件在接口中用 `ctx.sources.submit` 提交，调用方也可以经通用 Source 写入接口提交；这样一个 Artifact 插件就能记录"用户要求做什么"，不需要另写一个 Source 插件 | -| `create` | 必填 | 见上表 | +| `setup` | 必填 | 见上表 | ### 4.3 示例 @@ -354,7 +360,8 @@ powercontext source-agent --server https://memory.example.com ```python from powercontext.plugins import ( - ArtifactPlugin, ArtifactWriter, GenerateRequest, Inputs, InvalidContent, PluginContext, UpdateRequest, + ArtifactPlugin, ArtifactWriter, GenerateRequest, Inputs, InvalidContent, PluginContext, Rendering, RenderRequest, + SearchHit, SearchRequest, SearchResult, UpdateRequest, ) from powercontext.sources import TextEvidence @@ -393,8 +400,20 @@ class IncidentReports: expected_revision=request.expected_revision) await self.index.upsert(request.scope_id, written.ref.artifact_id, written.ref.revision, vector) + async def render_artifact(self, request: RenderRequest) -> Rendering: + report = Report.model_validate(request.artifact.content) + return Rendering(text=f"# {report.title}\n\n{report.summary}") + + async def search_artifacts(self, request: SearchRequest) -> SearchResult: + # 通用检索请求原样转交；本插件只支持向量检索 + if request.search.mode not in (None, "vector"): + raise InvalidContent("only vector search is supported", field="mode") + vector = await self.model.embed(request.search.query) + hits = await self.index.nearest_many(request.scope_id, vector, limit=request.search.limit) + return SearchResult(hits=[SearchHit(ref=hit.ref, text=hit.summary, score=hit.score) for hit in hits]) + -async def create(ctx: PluginContext) -> IncidentReports: +async def setup(ctx: PluginContext) -> IncidentReports: index = await ctx.enter_async_context(VectorIndex.connect(ctx.storage.engine())) return IncidentReports(index, Summarizer(ctx.models)) @@ -403,8 +422,7 @@ plugin = ArtifactPlugin( family="incident-report", inputs=Inputs(source_types=("example-ticket",), projection="powercontext.text-evidence@1"), on_sources=True, - search_fields=("title", "summary"), - create=create, + setup=setup, ) ``` @@ -413,14 +431,16 @@ plugin = ArtifactPlugin( ### 4.4 调用方怎么使用 -调用方使用现有的 Artifact 接口访问插件 family。读取接口直接读 `pc_artifacts`，不运行插件代码；修改接口由 core 鉴权、 -校验版本后，同步调用插件对应的方法： +调用方使用现有的 Artifact 接口访问插件 family。读取接口直接读 `pc_artifacts`，不运行插件代码；展示、检索和修改接口 +由 core 鉴权后，同步调用插件对应的方法： | 用途 | 接口 | 插件方法 | | --- | --- | --- | | 浏览 | `GET /v1/scopes/{scope_id}/artifacts/{family}` | 无 | | 读取当前内容、指定 revision、revision 历史、tags | `GET .../{artifact_id}`、`GET .../{artifact_id}/revisions/{revision}`、`GET .../{artifact_id}/revisions`、`GET .../{artifact_id}/tags` | 无 | -| 文本检索 | `POST /v1/scopes/{scope_id}/artifacts/{family}/search` | 无 | +| 默认展示 | `GET .../{artifact_id}/rendering`（可带 `revision`） | `render_artifact` | +| 检索（请求原样转交，检索方式由插件解释） | `POST /v1/scopes/{scope_id}/artifacts/{family}/search` | `search_artifacts` | +| 为 Agent 准备上下文 | `POST /v1/context/prepare` 的 `assembly.sections` 中列出插件 family | `search_artifacts` | | 新建（只提交内容，ID 由 core 分配） | `POST /v1/scopes/{scope_id}/artifacts`，请求体为 `{"family": ..., "content": {...}}` | `create_artifact` | | 编辑（提交完整内容，`If-Match` 校验当前 revision） | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` | `update_artifact` | | 读取治理状态（遗忘、合并前取得 `state_version`） | `GET .../{artifact_id}/governance` | 无 | @@ -429,7 +449,22 @@ plugin = ArtifactPlugin( | 恢复、恢复到旧 revision、撤销合并 | `POST .../{artifact_id}/restorations` | `restore_artifact` | | 修改 tags | `PUT .../{artifact_id}/tags`；按 tag 查询用 `POST /v1/scopes/{scope_id}/artifact-tags/query` | 无，由 core 处理 | -修改同步返回结果。插件拒绝内容时返回 422，插件没有实现对应方法时也返回 422，宿主进程超时或不可用时返回 503。 +修改同步返回结果。插件拒绝内容或请求时返回 422，插件没有实现对应方法时也返回 422，宿主进程超时或不可用时返回 503。 + +### 4.5 方法一览 + +| 方法 | 适用 | 必须实现 | 调用时机 | 执行位置 | +| --- | --- | --- | --- | --- | +| `setup(ctx)` | 两类插件 | 是 | 宿主进程启动时一次；Worker 每次调用一次 | 宿主进程、Worker | +| Source 定义：`input_class`、`resolve`、Projection 的 `project`、`read` | Source 插件，或声明了 `definitions` 的 Artifact 插件 | 是 | Source 写入（通用写入接口、`ctx.sources.submit`）；把 Source 交付给 `generate` 时计算 Projection | 宿主进程、Worker | +| `identify_target`、`collect` | Source 插件 | 持续采集时是 | 添加采集目标；定时采集 | 宿主进程、Worker | +| `generate` | Artifact 插件 | 配置了 `schedule` 或 `on_sources` 时是 | 定时，或新 Source 到达 | Worker | +| `render_artifact` | Artifact 插件 | 是 | 默认展示接口 | 宿主进程 | +| `search_artifacts` | Artifact 插件 | 否 | 通用检索接口；Prepared Context 中列出该 family | 宿主进程 | +| `create_artifact`、`update_artifact`、`merge_artifacts`、`forget_artifact`、`restore_artifact` | Artifact 插件 | 否 | 对应的通用修改接口 | 宿主进程 | + +列出、读取当前内容或指定 revision、revision 历史、治理状态和 tags 不调用插件方法，由 core 直接从它保存的数据提供。 +必须实现的方法在宿主进程执行 `setup` 之后核对，缺少时插件加载失败，状态为 `failed`。 ## 5. 调度与触发 @@ -493,7 +528,7 @@ class ArtifactPlugin(BaseModel): # 字段见指南级说明 4.2 family: str inputs: Inputs | None = None ... - create: Callable[[PluginContext], Awaitable[ArtifactGenerator]] + setup: Callable[[PluginContext], Awaitable[ArtifactGenerator]] class SourcePlugin(BaseModel): # 字段见指南级说明 3.2 definitions: Sequence[SourceDefinition] @@ -502,8 +537,8 @@ class SourcePlugin(BaseModel): # 字段见指南级说明 3.2 ``` - 插件名取自目录名，必须匹配 `^[a-z][a-z0-9-]{0,63}$`。 -- 声明在构造时校验，包括调度组合、`inputs` 与 `scopes` 的依赖、时限关系和 family 命名。`create` 必须是 async 函数。 -- 声明中除 `create`、`identify_target`、`definitions` 等代码对象外的部分可以序列化，Server 在检查子进程中得到它， +- 声明在构造时校验，包括调度组合、`inputs` 与 `scopes` 的依赖、时限关系和 family 命名。`setup` 必须是 async 函数。 +- 声明中除 `setup`、`identify_target`、`definitions` 等代码对象外的部分可以序列化，Server 在检查子进程中得到它， 用于注册 binding 和回答运维查询。 - 检查时，core 记录插件目录的**代码指纹**（按文件内容计算）。Worker 和宿主进程导入插件前核对，不一致则调用失败并提示 重启 Server，同一个 Server 进程中的调用不会混用新旧代码。 @@ -636,6 +671,7 @@ class SourceCollector(Protocol): ```python class ArtifactGenerator(Protocol): + # 配置了 schedule 或 on_sources 时必须实现 async def generate(self, request: GenerateRequest, writer: ArtifactWriter, /) -> None: ... class GenerateRequest(BaseModel): @@ -701,8 +737,9 @@ class ArtifactTransaction(ArtifactReads): `bind_handle` 的目标只能是本 family 中实际存在的 revision。 - 内容不变时不产生 revision，与内置 Atomic Memory 的处理一致：模型判断已有记忆已经表达了新 Source 的内容时，该记忆 不追加 revision，新 Source 也不进入它的 lineage。插件需要把新 Source 记为某个 Artifact 的证据时，应同时修改其内容。 -- 检索文本由 core 按声明的 `search_fields` 从内容中提取，插件不单独提供；同一份内容无论由插件还是用户写入， - 检索文本都相同。 +- core 不为插件 family 提取检索文本或建立索引，检索由插件用自己的索引完成（见第 7 节）。 +- core 在每个 revision 的元数据中记录写入它的插件名和插件版本摘要，便于在多个 Server 加载了不同版本时追溯每个 revision + 由哪个版本写入（见第 10 节）。 - 每个写事务块的操作数、单份内容大小和总字节数有上限，超出即抛出异常并回滚，不截断、不部分提交。 - core 不校验 `content` 的业务结构，也不保存或固化 schema。内容结构的演进由插件负责，插件可以在 `content` 中自带版本字段。 @@ -733,7 +770,6 @@ class ArtifactReads(Protocol): async def revisions(self, artifact_id: str, *, cursor: str | None = None, limit: int = 50) -> ArtifactPage: ... async def list(self, *, states: Collection[LifecycleState] = ("active",), tags: Collection[str] = (), cursor: str | None = None, limit: int = 50) -> ArtifactPage: ... - async def search(self, query: str, *, limit: int = 10) -> Sequence[ArtifactSnapshot]: ... async def by_source(self, ref: SourceRef) -> Sequence[ArtifactSnapshot]: ... async def merge_inputs(self, artifact_id: str) -> Sequence[ArtifactRef]: ... async def tags(self, artifact_id: str) -> Sequence[str]: ... @@ -744,7 +780,6 @@ class ArtifactReads(Protocol): - 快照同时携带精确 revision、lineage 和当前治理状态，插件不需要再分别读取 lineage 和生命周期。 - `get` 可以读当前内容或指定 revision；`revisions` 列出完整历史；`merge_inputs` 读取合并结果的精确输入。 - `list` 可以按生命周期状态和 tags 过滤，包括 deprecated、retired 和被合并冻结的 Artifact，用于查找之前已经存在的数据。 -- `search` 是 core 提供的文本检索（见第 7 节），只覆盖 active 的当前 revision；在写事务中能检索到本块已写入、尚未提交的内容。 - `by_source` 反查 lineage 引用了某条 Source 的 Artifact，用于 Source 更新后找到需要修订的 Artifact。 - 读取返回的快照都带精确 revision，可以直接作为写操作的 `artifact_refs` 或 `to_revision`。 @@ -902,10 +937,11 @@ collect 的结构相同：输入换成该 Scope 的已启用目标及其 checkpo | Source 写入的输入校验、`resolve`、Projection | 宿主进程 | | 添加采集目标时的 `identify_target` | 宿主进程 | | `create_artifact`、`update_artifact`、`merge_artifacts`、`forget_artifact`、`restore_artifact` | 宿主进程 | +| `render_artifact`、`search_artifacts` | 宿主进程 | | `collect`、`generate` | Supervisor Worker | -- **数量与启动**：每个 Server 进程为每个加载的插件维护至多一个宿主进程，插件加载后在后台启动，之后复用。插件是否实现了 - 用户修改方法，要调用 `create` 之后才知道，所以只提供 `generate` 的 Artifact 插件也会启动宿主进程，代价是一个空闲的进程。 +- **数量与启动**：每个 Server 进程为每个加载的插件维护至多一个宿主进程，插件加载后在后台启动，之后复用。Artifact 插件 + 都要实现 `render_artifact`，Source 插件的写入要执行 `resolve`，所以每个加载的插件都有宿主进程。 端侧模式中，`resolve` 和 Projection 在采集 Worker 内执行，不启动宿主进程。 - **执行**：宿主进程启动时导入一次 `plugin.py` 并核对声明，写入器在宿主进程中执行。一个宿主进程可以并发处理多个请求， 并发上限由 Server 配置决定。 @@ -919,26 +955,61 @@ collect 的结构相同：输入换成该 Scope 的已启用目标及其 checkpo 宿主进程的当前状态（未启动、运行中、重启退避）。路由用户请求、启动宿主进程和派发 Worker 都以这份记录为准。它不持久化， 也不跨进程同步；Server 重启后按插件目录重新建立。运维接口只返回各进程一致的检查结果和声明，宿主进程状态通过日志和指标查看。 -## 7. 公开读取与检索 +## 7. 公开读取、展示与检索 -插件 family 的公开读取完全由 core 处理，不执行插件代码： +读取由 core 直接提供，不执行插件代码： -- list / get / revisions / tags 复用现有的通用 Artifact 读取路径。插件 family 的 `content` 按原样 JSON 返回，不需要 - `pc_artifact_family_schemas` 中的 schema。被合并冻结、deprecated 和 retired 的 Artifact 按公共合并与生命周期规则展示。 +- list / get / revisions / governance / tags 复用现有的通用 Artifact 读取路径。插件 family 的 `content` 按插件写入时的 + JSON 原样返回，不需要 `pc_artifact_family_schemas` 中的 schema。被合并冻结、deprecated 和 retired 的 Artifact 按公共 + 合并与生命周期规则展示。 - list 返回的条目只含元数据，`title` 和 `summary` 对插件 family 为空；内容需要通过 get 读取。 -- search 是 core 为插件 family 提供的通用文本检索，与现有检索接口一样只接受文本 query： - - 写入时，core 按 `search_fields` 从内容中取出文本，经现有词法分析写入 `pc_artifact_heads.searchable_text`，并随 revision - 保存，恢复旧 revision 时检索文本随内容一起恢复。 - - 检索时在该列上匹配，只返回请求 Scope 中调用者有权读取的 active 当前 revision。没有可检索文本的 Artifact 不参与检索。 - - 写入器的 `search` 使用同一实现。 - - 各数据库后端上的检索实现和规模验证在实现 PR 中给出。 -- Review、Recall 和 Prepared Context 第一版不对插件 family 开放，对应接口返回明确的“不支持”错误；`GET /v1/capabilities` - 列出插件 family。 + +插件可能只在 `content` 中保存摘要或引用，把大块内容放在自己的存储中（例如知识库文档只记录切块组的摘要），这时原样返回 +的 `content` 不足以展示。展示和检索因此交给插件： + +```python +class RenderRequest(BaseModel): + scope_id: str + artifact: ArtifactSnapshot # 要展示的精确 revision + caller: Caller | None + +class Rendering(BaseModel): + format: Literal["markdown", "text"] = "markdown" + text: str + +class SearchRequest(BaseModel): + scope_id: str + search: SearchArtifactsRequest # 通用检索接口的完整请求，原样转交 + max_bytes: int | None = None # 来自 Prepared Context 时，本 family 可用的字节预算 + caller: Caller | None + +class SearchHit(BaseModel): + ref: ArtifactRef # 命中的精确 revision + text: str # 用作上下文的片段，例如一个切块 + score: float | None = None + locator: dict[str, JsonValue] | None = None # 片段在 Artifact 中的位置，例如页码、切块序号 + +class SearchResult(BaseModel): + hits: Sequence[SearchHit] +``` + +- **展示**：`GET .../{artifact_id}/rendering` 调用 `render_artifact`，返回插件给出的默认展示内容；原始 `content` 仍由读取 + 接口返回，供程序使用。同一 revision 在同一插件版本下的展示结果不变，core 可以按 family、ID、revision 和插件版本缓存。 + 内置 family 不提供这个接口。 +- **检索**：通用检索接口把 `SearchArtifactsRequest` 的全部字段（`query`、`query_vector`、`indexes`、`mode`、`filters`、 + `admission`、`fusion`、`rerank`、`min_score` 等）原样交给 `search_artifacts`，由插件按自己的索引解释；插件不支持的参数 + 抛出 `InvalidContent`，接口返回 422。core 只保留命中中调用者有权读取的 active 当前 revision，按插件给出的顺序返回 + 精确 revision，`include_scores` 时附带插件给出的分数。插件因此不能借检索返回调用者看不到的数据。 +- **Prepared Context**：`assembly.sections` 可以列出插件 family。core 以 Prepared Context 的 `query` 和该项的 `limit`、 + 剩余字节预算调用 `search_artifacts`，把命中的片段作为上下文条目，与内置 family 一样带精确 revision 引用、计入字节预算 + 和回执。每项可以携带可选的 `search`，内容是检索请求中除 `query`、`limit` 以外的字段，原样转交插件；内置 family 携带 + `search` 时返回 422。 +- Review 和 Recall 不对插件 family 开放，对应接口返回明确的“不支持”错误；`GET /v1/capabilities` 列出插件 family。 ## 8. 加载与生命周期 - Server 启动时检查 `POWERCONTEXT_SERVER_PLUGINS_DIR` 的每个直接子目录（未设置则不加载插件）：在有时限的子进程中导入 - `plugin.py` 并校验声明，不调用 `create`。检查通过后注册插件提供的 Source 类型或 family（读取、检索和通用合并接口）， + `plugin.py` 并校验声明，不调用 `setup`。检查通过后注册插件提供的 Source 类型或 family（读取、检索和通用合并接口）， 有调度或触发时注册 Supervisor binding，并写入本进程的已激活插件记录。检查失败只影响该插件。 - Server 停止时先停止接受同步调用、结束宿主进程，再按现有流程停止 Supervisor 并回收 Worker。 - 插件只有 `loaded` 和 `failed` 两种结果，只存在于进程内存中。Server 不持久化插件的启用状态、注册代次或配置快照； @@ -976,18 +1047,32 @@ collect 的结构相同：输入换成该 Scope 的已启用目标及其 checkpo | `pc_source_cursors` | 保留；Artifact 插件的消费游标，键为 `(scope_id, plugin:)` | | `pc_connector_checkpoints` | 保留；各采集目标的 checkpoint，键为 `(scope_id, 目标的 binding ID)` | | `pc_source_collect_targets` | 新增；采集目标及其规范化输入、启用状态和最近采集结果 | -| Supervisor 的调用意图、pending、租约表 | 保留；插件 binding 与内置 binding 共用，也是 `GET /v1/plugins/{name}/scopes` 的数据来源 | +| Supervisor 的调用意图、pending、租约表 | 保留；插件 binding 与内置 binding 共用，也是 `GET /v1/plugins/{name}/scopes` 的数据来源。插件 binding 以 `plugin:` 开头，不在数据库中登记 | +| `pc_artifacts` 的新列 `metadata` | 新增；可为空，按与 `content` 相同的方式保存规范化 JSON 字节，存放不参与检索的附加信息。插件写入的 revision 记录插件名和插件版本摘要，内置 family 为空。不参与内容摘要的计算 | +| `pc_artifact_fts` 与 `pc_artifact_heads.searchable_text` | 只用于内置 family；插件 family 不写入 | | `pc_artifact_processing_binding_states` 中为旧插件框架新增的列（期望状态、管理阶段、注册代次等） | 移除 | | `pc_artifact_family_schemas` | 移除 | | 自定义 family 声明式索引使用的搜索状态与向量索引数据 | 移除；实现 PR 确认内置 family 不依赖后删除对应表 | -| 旧 Artifact 插件写入的 Artifact、head、lineage | 删除，不迁移 | +| 旧 Artifact 插件写入的 Artifact、head、lineage | 不迁移；留有数据时拒绝升级，见下文 | | `pc_source_write_plugins` | 移除；Source 定义改由插件目录在启动时提供 | | Source 插件管理服务的 SQLite | 随管理服务一起移除 | | 插件专属存储（`plugins/<插件名>.sqlite3` 或 `powercontext_plugin_<插件名>` 库） | 不属于 core 的 schema；core 只负责分配，表结构、迁移和清理由插件和部署者负责 | | 插件自有的其他索引与存储 | 不属于 core，由插件和部署者管理 | | 端侧模式的本地 SQLite | 只保存端侧的采集目标和调度状态，不保存 Source；Source 与 checkpoint 保存在中心 | -新增与移除通过一次新的数据库迁移完成，迁移说明中列出新增、删除的表、列和数据范围。 +新增与移除通过**一个新的迁移版本**完成，已发布版本的迁移资源保持不变。迁移说明中列出新增、删除的表、列和数据范围。 +迁移检查（preflight）先确认旧插件框架没有留下数据：旧 Artifact 插件 family 的 Artifact、旧插件的登记与调用意图、 +`pc_source_write_plugins` 中的 Source 插件。发现任何一项时拒绝升级，列出插件名、family 和行数，数据库保持不变；部署者 +在旧版本上导出所需内容、卸载旧插件并清理数据后再升级。只有这些表为空时，迁移才删除它们。 + +这个版本的就绪检查（Server 每次启动时对当前数据库执行）与旧版本有两点不同： + +- 调用意图中以 `plugin:` 开头的 binding 视为插件 binding，不要求在数据库中登记。插件被移除后遗留的意图不处理、 + 可以在运维接口中看到，不阻止 Server 启动；同名插件重新放入后继续处理。 +- 全文索引的核对只覆盖内置 family 的数据。 + +导出与恢复（portable archive）中 Artifact revision 记录的格式版本加一，包含 `metadata`。恢复旧格式的导出时 `metadata` +为空；旧版本的 Server 不能恢复新格式的导出。 ## 11. 接口变更 @@ -995,7 +1080,7 @@ collect 的结构相同：输入换成该 Scope 的已启用目标及其 checkpo | --- | --- | | 移除 | `/v1/admin/artifact-plugins` 及其子路径；Source 插件管理服务、插件实例 Runtime、poll / reply 通道；`source load` 命令；MCP `pc_source_instances`、`pc_source_instance_set_enabled`；`plugin.yml` 与 `powercontext.yml` 清单 | | 移除 | 自定义 family 的声明式索引（`ArtifactSearchDeclaration`、`TextIndex` 等）、`ArtifactFamily` 的 schema 声明、`ArtifactDraft` 和旧的 Worker 插件执行器 | -| 新增 | `powercontext.plugins`：`SourcePlugin`、`ArtifactPlugin`、`Inputs`、`Interval`、`Cron`、`PluginContext`、`PluginStorage`、`PluginModels`、写入器与读取接口、用户修改的请求模型与 `InvalidContent` | +| 新增 | `powercontext.plugins`：`SourcePlugin`、`ArtifactPlugin`、`Inputs`、`Interval`、`Cron`、`PluginContext`、`PluginStorage`、`PluginModels`、写入器与读取接口、用户修改的请求模型与 `InvalidContent`、展示与检索的请求和结果模型 | | 新增 | `powercontext source-agent` 端侧采集模式，以及 `POWERCONTEXT_SOURCE_AGENT_ID`、`POWERCONTEXT_SOURCE_AGENT_TOKEN` | | 调整 | `POST /v1/scopes/{scope_id}/artifacts` 的请求体增加插件分支 `{"family", "content"}`，ID 由 core 分配，由插件的 `create_artifact` 处理。`CreateArtifactRequest` 改为内置分支 `CreateBuiltinArtifactRequest`（原有的按 family 区分的联合类型）与插件分支 `CreatePluginArtifactRequest` 的二选一，按 family 名选择：内置 family 名走内置分支，校验与原来相同；其他名字走插件分支，没有插件提供该 family 时返回 422。生成的 Python 客户端中，内置请求多包一层；`ArtifactCreated.family` 从内置 family 枚举改为字符串 | | 调整 | `POST /v1/scopes/{scope_id}/sources` 的 `source_type` 从只能是 `content` 的枚举改为字符串（默认仍为 `content`），可以是插件定义的 Source 类型，此时 `content` 是该类型的输入，由插件校验并解析为 Source；未知类型返回 404。`SourceType` 枚举保留 | @@ -1003,7 +1088,10 @@ collect 的结构相同：输入换成该 Scope 的已启用目标及其 checkpo | 新增 | `GET .../{artifact_id}/governance`、`POST /v1/scopes/{scope_id}/artifacts/{family}/merges`、`POST .../{artifact_id}/forget`、`POST .../{artifact_id}/restorations`：基于共享合并引擎注册表的通用接口，对所有登记过的 family 生效（内置为 Experience 和 Atomic Memory，以及加载的插件 family；插件 family 由插件对应的方法处理）。Atomic Memory 的专用接口保持不变 | | 调整 | `PUT /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}` 接受插件 family，请求体为 `{"content": {...}}`，由插件的 `update_artifact` 处理 | | 调整 | tags 接口接受插件 family：`TaggableArtifactFamily` 从枚举改为与 `ArtifactReadFamily` 相同格式的字符串，服务端校验为内置 family 或已加载的插件 family。网络上的数据格式不变，生成的客户端少了这个枚举类型 | -| 调整 | 通用 Artifact 读取与检索接口支持插件 family；`GET /v1/capabilities` 列出插件 family | +| 调整 | 通用 Artifact 读取接口支持插件 family；通用检索接口对插件 family 把完整请求交给插件的 `search_artifacts`，未实现时返回 422；`GET /v1/capabilities` 列出插件 family | +| 新增 | `GET /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}/rendering`：插件 family 的默认展示内容，由插件的 `render_artifact` 提供 | +| 调整 | `POST /v1/context/prepare`：`ContextAssemblySection.family` 从枚举改为字符串，可以是已加载的插件 family；新增可选的 `search`（检索请求中除 `query`、`limit` 以外的字段），只对插件 family 生效 | +| 调整 | 导出与恢复的 Artifact revision 记录格式版本加一，增加 `metadata` | | 调整 | `pc_source_add` 的 write 用法使用插件定义，collect 用法添加采集目标；`pc_source_capabilities` 列出插件 Source 类型与目标输入；`pc_source_status` 报告目标及其最近采集结果；`pc_source_target_set_enabled` 作用于 core 保存的目标 | | 调整 | 环境变量 `POWERCONTEXT_SERVER_PLUGINS_DIR` 取代 `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` 和 Source 插件目录配置 | @@ -1041,7 +1129,10 @@ collect 的结构相同：输入换成该 Scope 的已启用目标及其 checkpo 部署时，插件应从环境变量读取 family 等标识。 - **内容结构不受 core 治理**：core 不校验 `content` 的 schema，用户提交的内容由插件校验；历史 revision 的解释需要插件和 应用自行保持兼容。 -- **core 检索能力有限**：core 只提供基于声明字段的通用文本检索；向量等检索需要插件自建索引，公开检索接口暂时用不到它。 +- **展示和检索依赖插件**：插件 family 的展示、检索和 Prepared Context 都要进入宿主进程，插件不可用时这些接口返回 503； + 插件没有实现 `search_artifacts` 时，它的数据不能通过通用检索和 Prepared Context 使用。 +- **多个 Server 的插件版本可能不一致**：core 记录每个 revision 由哪个插件版本写入，但不阻止不同实例加载不同版本；新旧 + 版本能否同时读写同一份数据，由插件负责。汇总各实例当前加载的版本需要维护一张实例登记表，留给插件安装的后续提案。 - **代码变更需要重启**：没有热加载，插件更新期间 Server 会短暂不可用。 # 理由与备选 @@ -1068,6 +1159,10 @@ collect 的结构相同：输入换成该 Scope 的已启用目标及其 checkpo | 端侧单独编译二进制 | 插件是带依赖的 Python 目录，冻结后的二进制无法导入插件的任意依赖，需要为每个插件和平台单独打包。同一个程序以 `source-agent` 模式启动即可 | | 插件专用的触发与管理接口 | 调用方要区分插件与内置能力；现有的 Source 与 Artifact 接口已经覆盖调用方需要的操作 | | 通用合并接口只对插件 family 开放 | 需要在接口里额外维护一份 family 白名单。能操作哪些 family 本来就由注册表决定，没有适配器的 family 无法登记；内置 family 的权限检查在引擎内，两个入口走同一套检查 | +| core 按声明的字段为插件 family 建全文索引 | 插件作者不用自建索引，但 core 不知道内容怎样分块、是否适合全文或向量检索，建出的索引常常用不上；还要与内置 family 共用索引表，就绪检查也要区分两者。检索交给插件，core 只负责权限过滤 | +| 读取也经过插件（例如 `read_artifact` 展开内容） | 通用读取就能拿到完整内容，但最常用的读取路径每次都要进入宿主进程，插件不可用时连读取都失败。读取保持由 core 提供，展示另设接口 | +| 在数据库中登记插件 binding 与 family（旧 Artifact 插件框架的做法） | 能发现其他插件冒用已有 family 和遗留的调用意图，但 core 要保存并维护插件状态，还要提供解除冲突的管理操作。新框架没有稳定的插件身份，登记表能多发现的问题有限；留到插件安装与分发的提案中，与插件身份一起设计 | +| 插件版本记在单独的表、`created_by` 或 `content` 的保留键中 | 单独的表同样要进入导出与恢复；改动 `created_by` 或 `content` 会改变已有字段的含义和内容摘要。`metadata` 列只需一次加列和导出格式升级 | | 本提案中支持热加载 | 需要处理多实例各自重新扫描、旧 Worker 提交的屏蔽和宿主进程切换，单独提案更清楚 | 选择这一方案的原因： @@ -1098,12 +1193,14 @@ collect 的结构相同：输入换成该 Scope 的已启用目标及其 checkpo | 问题 | 说明 | | --- | --- | -| 插件自定义检索 | 现有检索接口只接受文本 query，第一版由 core 提供通用文本检索。插件提供的检索（例如向量或混合检索）随插件自定义 API 另行提案，在宿主进程中执行 | | 写事务块的时限与限额默认值 | 默认 30 秒是否合适；SQLite 与 OceanBase 是否采用不同默认值；单块操作数与字节上限 | | 宿主进程的默认参数 | 同步调用时限、单个宿主进程的并发上限和重启退避的默认值 | | 专属存储的建库权限 | Server 的数据库登录没有建库权限时，除了由部署者事先建库，是否允许退回到 core 库中按插件名前缀建表 | | 采集目标的删除 | 第一版只支持停用和恢复。是否需要彻底删除目标及其 checkpoint，以及删除后已采集 Source 的处理 | -| 插件 family 接入 Recall / Prepared Context | 需要定义候选内容和预算约束，建议另行提案 | +| 插件 family 接入 Recall | Prepared Context 已接入，Recall 是否需要接入、候选内容怎样定义 | +| 通用检索结果带片段 | 通用检索接口目前只返回精确 revision，插件返回的片段只用于 Prepared Context。是否在检索结果中增加片段和位置 | +| family 与 Source 类型的命名 | 是否要求以插件名为前缀，从命名上避免不同插件冒用同一个 family | +| Prepared Context 的检索控制 | 内置 family 也接受每项的 `search`，是新增的可选字段，不影响现有调用，建议另行提案。Prepared Context 中其他按功能定制的参数，移除或改名是不兼容变更，需要新的 API 版本，不在 v1 中整理 | | 窗口大小的回退 | 内置 Atomic Memory 在模型超时时会缩小下一次的 Source 窗口。插件无法影响窗口大小，过大的窗口会持续超时。是否允许插件以约定的异常请求缩小该 Scope 的窗口 | | 持续失败的窗口 | 始终失败的 Source 窗口会阻塞该 Scope 的后续调用。是否提供跳过或转入死信的运维操作，以及如何记录被跳过的输入 | | 消费全部 Source 类型 | `inputs.source_types` 需要逐一列出类型；内置 Atomic Memory 消费所有可用于生成的 Source。是否支持 `"*"` 表示全部类型 | @@ -1115,4 +1212,4 @@ collect 的结构相同：输入换成该 Scope 的已启用目标及其 checkpo - 热加载与插件自定义 API（含插件页面）：建立在宿主进程之上，见 [RFC #2579](https://github.com/oceanbase/powercontext/pull/2579)。 - 后台调用也改用常驻进程，在保持接口不变的前提下复用连接和初始化结果。 - 更多插件类型，例如为 Recall 提供候选内容、为 Prepared Context 提供文本片段。 -- 插件打包与安装工具，例如从 Git 地址或 wheel 安装到插件目录。 +- 插件安装与依赖：从 PyPI、Git 地址或 wheel 安装插件，每个插件独立的依赖环境，以及记录各实例加载的插件版本的登记表。 From c7c94edb29493884655458f8d3709ee5ea8dacb1 Mon Sep 17 00:00:00 2001 From: jasper-yl Date: Thu, 12 Nov 2026 12:06:08 -0800 Subject: [PATCH 09/12] docs(rfc): keep the plugin interfaces RFC to Artifact plugins and their own Source types Standalone Source plugins, collection targets, and edge collection move to a follow-up proposal, which will replace the old Source plugin framework; this RFC replaces only the old Artifact plugin framework, and the old Source plugin framework stays unchanged until then. Co-Authored-By: Claude Opus 5.5 --- docs/en/rfcs/2549-plugin-interfaces.md | 453 +++++++------------------ docs/zh/rfcs/2549-plugin-interfaces.md | 394 ++++++--------------- 2 files changed, 215 insertions(+), 632 deletions(-) diff --git a/docs/en/rfcs/2549-plugin-interfaces.md b/docs/en/rfcs/2549-plugin-interfaces.md index 42153e326d..220233230f 100644 --- a/docs/en/rfcs/2549-plugin-interfaces.md +++ b/docs/en/rfcs/2549-plugin-interfaces.md @@ -1,6 +1,6 @@ --- title: Plugin interfaces -description: Source and Artifact plugins each implement a few interfaces that PC Server calls when needed, with background calls in Supervisor Workers and synchronous calls in plugin host processes that core manages; callers use only the existing Source and Artifact APIs, and core manages only the two core data kinds and their consumption progress, without managing plugin lifecycle. +description: Artifact plugins implement a few interfaces that PC Server calls when needed, with background calls in Supervisor Workers and synchronous calls in plugin host processes that core manages; callers use only the existing Source and Artifact APIs, and core manages only the two core data kinds and their consumption progress, without managing plugin lifecycle. --- - Proposal Name: `plugin_interfaces` @@ -13,26 +13,25 @@ This proposal redefines PowerContext's third-party plugin model: plugins impleme PC Server calls them when it needs them. Callers never call plugins directly; they use the existing Source and Artifact APIs exactly as they do for built-in capabilities. -- **Two plugin kinds**: - - **Source plugins** provide Source definitions, and implement `identify_target` and `collect` when they need to - collect continuously. - - **Artifact plugins** implement `generate`, which turns unconsumed Sources into Artifacts. -- **Plugin shape**: a plugin is a directory containing `plugin.py`, which assigns a declaration object (`SourcePlugin` - or `ArtifactPlugin`) to `plugin`. The declaration lives in code; there is no separate manifest file. The plugin reads +- **Artifact plugins**: a plugin owns one Artifact family, implements `generate` to turn unconsumed Sources into + Artifacts and `render_artifact` to give the default presentation, and may implement search and user changes. A plugin + may bring its own Source types to record what users submit to it. +- **Plugin shape**: a plugin is a directory containing `plugin.py`, which assigns an `ArtifactPlugin` declaration object + to `plugin`. The declaration lives in code; there is no separate manifest file. The plugin reads deployment-specific settings from environment variables. - **How callers use plugins**: - - Writing Sources and adding collection targets use the existing Source APIs. + - Writing Sources of a plugin's own types uses the existing Source write API. - Reading and searching Artifacts use the existing Artifact APIs. - Creating, editing, merging, forgetting, and restoring plugin Artifacts also use the existing Artifact APIs. Core handles authorization, version checks, and provenance, then synchronously calls the matching plugin method; the plugin validates the content, maintains its own storage, and writes through the writer. An operation the plugin does not implement returns 422. - **Execution**: - - `collect` and `generate` are background calls. They run in Worker child processes managed by the Supervisor, - reusing its existing isolation, timeouts, concurrency budgets, leader lease, and failure backoff. - - `resolve` and Projections during Source writes, `identify_target` when adding a collection target, and user changes - to plugin Artifacts are synchronous calls. They run in a long-lived **plugin host process** that core manages: each - Server process starts one host process for each plugin that needs synchronous calls, and deployers do not manage it. + - `generate` is a background call. It runs in Worker child processes managed by the Supervisor, reusing its existing + isolation, timeouts, concurrency budgets, leader lease, and failure backoff. + - `resolve` and Projections during Source writes, presentation, search, and user changes to plugin Artifacts are + synchronous calls. They run in a long-lived **plugin host process** that core manages: each Server process starts one + host process for each loaded plugin, and deployers do not manage it. - **Writes**: an Artifact plugin performs its writes inside an `async with writer.transaction()` block. When the block ends, core commits the writes, the consumption cursor, and the completion acknowledgment together; an exception rolls everything back. Plugins can neither read nor change the cursor. @@ -40,41 +39,39 @@ APIs exactly as they do for built-in capabilities. tags), and their consumption progress. Plugins may have their own indexes and storage and manage them themselves. Core gives each plugin a dedicated storage area on the database backend the deployment uses, and lets plugins use the models Server has configured. -- **Edge collection**: Server can start in `source-agent` mode, which only runs Source plugin collection and submits - Sources and checkpoints to a remote central Server. This serves scenarios such as embodied AI, where data is collected - on the device. - **Lifecycle**: plugins load from a directory at Server startup, and changes require a restart. Each Server process records its active plugins only in memory and persists no plugin enabled state, registration generation, or management phase. This proposal replaces the plugin contracts and management mechanisms of the -[Source plugin framework RFC #2012](2012-source-plugin-system.md) and the -[Artifact plugin framework RFC #2175](2175-artifact-plugin-system.md). The old Source plugin framework shipped in v1.5, -but no plugins for it have been published, so this proposal keeps no compatibility: the old plugin manifests, management -APIs, management service, and plugin-management tables are removed, and custom Artifact data written by old Artifact -plugins is not migrated. So that no data is lost silently, the migration preflight refuses to upgrade while the old -frameworks still hold data, and the deployer exports and cleans it up on the old version first (reference section 10). -Parts of the old design that work well are kept, including the Source management APIs -(`pc_source_add` and others) and their collection target identity rules, and core's existing remote ingestion APIs. +[Artifact plugin framework RFC #2175](2175-artifact-plugin-system.md). The old framework shipped in v1.5, but no plugins +for it have been published, so this proposal keeps no compatibility: the old plugin manifests, management APIs, and +plugin-management tables are removed, and custom Artifact data written by old plugins is not migrated. So that no data is +lost silently, the migration preflight refuses to upgrade while the old framework still holds data, and the deployer +exports and cleans it up on the old version first (reference section 10). + +Standalone Source plugins, which only provide Source types or collect continuously from external systems (collection +targets, checkpoints, collecting on a device and reporting to a central Server), are out of scope. A follow-up proposal +defines them and replaces the [Source plugin framework RFC #2012](2012-source-plugin-system.md); until then the old Source +plugin framework stays unchanged. This document describes the target design; it does not mean the mainline or a release already provides it. Interface names and code examples illustrate the contract; the implementation determines the final syntax. # Motivation -Both existing plugin frameworks make PowerContext carry a large amount of lifecycle management for third-party plugins: +The existing Artifact plugin framework makes PowerContext carry a large amount of lifecycle management for third-party +plugins: | Framework | Work Server carries for plugins | | --- | --- | | Artifact plugins (#2175) | `desired_state`, `phase`, `binding_generation`, and `last_error` in `pc_artifact_processing_binding_states`; seven management states and their transitions; the `enable-all / reload / enable / disable / unload` management APIs with `expected_generation` concurrency checks; registration generations that fence old Worker commits; content schemas and index definitions pinned in `pc_artifact_family_schemas`, with 409 on change; the framework builds, rebuilds, and tracks availability of custom declarative indexes | -| Source plugins (#2012) | A separate plugin management service with its own SQLite (instances, capabilities, targets, desired state, commit records); one Runtime process per plugin instance; poll / reply channels and capability catalog sync between the service and Server; the `source load` deployment command; MCP tools for starting and stopping instances | Everyone pays for these mechanisms: -- **Plugin authors** must understand framework concepts such as bindings, generations, schema pinning rules, index - declarations, and management channels before they can write a plugin. -- **Deployers** must run several processes, maintain two databases, write manifest files, and use management APIs to get - plugins into a runnable state. +- **Plugin authors** must understand framework concepts such as bindings, generations, schema pinning rules, and index + declarations before they can write a plugin. +- **Deployers** must write manifest files and use management APIs to get plugins into a runnable state. - **Callers** must tell built-in capabilities from plugin capabilities and use different APIs for them. - **Maintainers** must maintain state machines, concurrency fencing, schema and index governance, and recovery paths for third-party code; this code and its test surface are already larger than the plugins' own business logic. @@ -98,10 +95,8 @@ Goals: 6. Plugin writes to core data go only through a writer core provides, which covers every generic Artifact write operation and keeps core's existing validation rules. 7. User changes to plugin Artifacts go through the plugin, so plugin-owned storage is updated together with the change. -8. Artifact generation can run on a schedule or be triggered by new Sources; collection targets can be added and removed - at runtime. -9. Users deploy a single Server; plugins load when Server starts. When data must be collected on a device, the same - program runs in edge mode. +8. Artifact generation can run on a schedule or be triggered by new Sources. +9. Users deploy a single Server; plugins load when Server starts. # Guide-level explanation @@ -111,16 +106,13 @@ Goals: flowchart LR Caller["Application / Agent / Host"] subgraph Server["PC Server (core)"] - SourceAPI["Existing Source APIs
write Sources, add collection targets"] + SourceAPI["Existing Source APIs
write Sources"] ArtifactAPI["Existing Artifact APIs
read, search, change"] Sup["Supervisor
schedules / triggers / budgets / backoff"] - Core[("Core data
pc_sources / pc_artifacts
collection targets / checkpoints / cursors")] + Core[("Core data
pc_sources / pc_artifacts / cursors")] end subgraph Host["Plugin host process (long-lived, managed by core)"] - Sync["resolve / Projections / identify_target
create_artifact / update_artifact / ..."] - end - subgraph SW["Worker: Source plugin"] - Collect["collect(request)"] + Sync["resolve / Projections / render_artifact
search_artifacts / create_artifact / ..."] end subgraph AW["Worker: Artifact plugin"] Generate["generate(request, writer)"] @@ -131,27 +123,23 @@ flowchart LR Caller --> ArtifactAPI SourceAPI -->|"synchronous call"| Sync ArtifactAPI -->|"reads"| Core + ArtifactAPI -->|"presentation, search, synchronous call"| Sync ArtifactAPI -->|"changes, synchronous call"| Sync Sync -->|"Source or Artifact writes, one commit"| Core Sync -.-> Own - Sup -->|"targets and checkpoints"| Collect - Collect -->|"Sources and new checkpoints, one commit"| Core Core -->|"new Sources trigger"| Sup Sup -->|"unconsumed Source window"| Generate Tx -->|"Artifact writes, cursor, acknowledgment, one commit"| Core - Collect -.-> Ext["External systems"] Generate -.-> Own[("Plugin-owned storage / models")] ``` -The diagram has three paths, and one call runs only one plugin: +The diagram has two paths, and one call runs only one plugin: -- **Collection path**: the Supervisor dispatches a Source plugin's Worker on schedule with the Scope's enabled collection - targets and their checkpoints. The plugin reads from the external system and returns Sources; core validates them and - saves the Sources and the new checkpoints in one transaction. - **Generation path**: when new Sources enter the journal, Artifact plugins that consume them are marked as pending. The Supervisor dispatches a Worker with the Scope's unconsumed Source window and a writer; the plugin writes Artifacts in a write transaction block, and core commits them together with the cursor and the acknowledgment. -- **Synchronous path**: when a caller writes a Source, adds a collection target, or changes a plugin Artifact, Server +- **Synchronous path**: when a caller writes a Source of a plugin type, presents or searches plugin Artifacts, or + changes a plugin Artifact, Server checks authorization and versions and then hands the request to the plugin's host process. The plugin performs the conversion or write there, and the result is committed in one transaction and returned to the caller. @@ -159,9 +147,9 @@ Callers deal only with core's existing APIs: | What the caller does | API | How the plugin takes part | | --- | --- | --- | -| Submit a Source | Existing Source write API | The host process runs the plugin's `resolve` and Projections | -| Add, view, or disable a collection target | Existing Source management APIs | On add, the host process runs the plugin's `identify_target` | -| Browse, read, or search Artifacts | Existing Artifact read and search APIs | Not involved | +| Submit a Source of a plugin type | Existing Source write API | The host process runs the plugin's `resolve` and Projections | +| Browse or read Artifacts | Existing Artifact read APIs | Not involved | +| Present or search plugin Artifacts | The presentation endpoint, the existing search API, Prepared Context | The host process runs `render_artifact` or `search_artifacts` | | Create, edit, merge, forget, restore, undo a merge | Existing Artifact create and edit APIs, and the generic merge APIs | The host process runs the matching plugin method; an unimplemented one returns 422 | | Tag, query by tag | Existing tag APIs | Not involved; tags are core data | @@ -169,13 +157,12 @@ Responsibilities: | Responsibility | Owner | | --- | --- | -| Authentication, Scope authorization, public HTTP / MCP APIs, storing collection targets and checkpoints, version checks and provenance for user changes | Core | +| Authentication, Scope authorization, public HTTP / MCP APIs, version checks and provenance for user changes | Core | | When to call: schedules, trigger coalescing, Worker budgets, timeouts, backoff, leader lease | Core (the existing Supervisor) | | Starting, reusing, terminating on timeout, and restarting host processes | Core | -| Reading inputs, providing the constrained write transaction, committing core data / cursor / checkpoints / acknowledgment in one transaction | Core (the plugin executor in the Worker or host process) | +| Reading inputs, providing the constrained write transaction, committing core data / cursor / acknowledgment in one transaction | Core (the plugin executor in the Worker or host process) | | Allocating dedicated plugin storage, wrapping the models Server has configured | Core | -| Source definitions and Projection computation, identifying and normalizing collection targets, reading external data | Source plugin | -| Artifact content structure, generation algorithm, validating user-submitted content, plugin-owned indexes and storage and their consistency | Artifact plugin | +| Definitions and Projection computation for the plugin's own Source types; Artifact content structure, generation algorithm, presentation, search, validating user-submitted content, plugin-owned indexes and storage and their consistency | Plugin | | Deploying plugin code, dependencies, and environment variables | Deployer | Plugin code does one thing: given an input, produce a result or raise. Raising means failure and core commits no core @@ -188,9 +175,9 @@ is the plugin name. The directory must contain `plugin.py` and may contain other ```text /opt/powercontext/plugins/ -├── x-posts/ +├── knowledge-base/ │ ├── plugin.py -│ └── definition.py +│ └── parsers.py └── incident-report/ └── plugin.py ``` @@ -198,11 +185,11 @@ is the plugin name. The directory must contain `plugin.py` and may contain other `plugin.py` assigns a declaration object to the module-level variable `plugin`: ```python -plugin = ArtifactPlugin(family="incident-report", ..., setup=setup) # or SourcePlugin(...) +plugin = ArtifactPlugin(family="incident-report", ..., setup=setup) ``` -The declaration answers what core needs to know before it runs any plugin code: what the plugin provides (Source types or -an Artifact family), what it consumes, when to call it, and its resource limits. It is an ordinary Python object validated +The declaration answers what core needs to know before it runs any plugin code: what the plugin provides (an Artifact +family and its own Source types), what it consumes, when to call it, and its resource limits. It is an ordinary Python object validated by Pydantic; there is no YAML or other manifest file. - **Deployment settings**: the plugin's own values that vary by deployment, such as credentials and external service @@ -219,144 +206,34 @@ by Pydantic; there is no YAML or other manifest file. - **Check**: at startup, Server imports `plugin.py` in a time-limited child process and validates the declaration without calling `setup`. A failed check affects only that plugin. -## 3. Source plugins - -### 3.1 Interfaces to implement - -| Interface | Required | Description | -| --- | --- | --- | -| `definitions` | Yes | A sequence of `SourceDefinition`, the existing Source definition: `input_class`, `resolve`, and its Projections | -| `target_input` | For collection | A Pydantic model validating the input a caller submits when adding a target | -| `identify_target(value) -> TargetIdentity` | For collection | A pure function that normalizes the input into a stable target key and normalized input | -| `setup(ctx) -> SourceCollector` | For collection | A factory run once per invocation in the Worker, returning an object that implements `collect` | -| `collect(request) -> CollectResult` | For collection | Reads external data per target and checkpoint, returning Sources and new checkpoints | - -A plugin that provides only `definitions` only extends Source types: callers submit content through the existing Source -write APIs, and core uses the definitions to validate, convert, and save it. `resolve`, Projections, and -`identify_target` all run in the plugin host process; `collect` runs in a Worker. - -Sources, like all built-in Sources, are append-only and cannot be changed or deleted; to correct content, submit a new -Source. Large content such as files and images goes into the plugin's own storage, and the Source records only -metadata such as its location, digest, and media type. In this proposal such content is read from external systems by -`collect`; `resolve` must be a pure function and the existing Source write APIs accept only JSON, so an entry point for -callers to upload files directly comes from plugin-defined APIs, see -[RFC #2579](https://github.com/oceanbase/powercontext/pull/2579). +## 3. A plugin's own Source types -### 3.2 Declaration - -| Field | Default | Description | -| --- | --- | --- | -| `definitions` | Required | See the table above | -| `collect` | `None` | Collection settings; empty means the plugin only provides definitions | -| `collect.schedule` | Required | `Interval(seconds=...)` or `Cron(expression=..., timezone=...)` | -| `collect.max_targets` | 50 | Maximum targets per invocation | -| `collect.max_items` | 200 | Maximum Sources returned per invocation | -| `collect.timeout_seconds` | 120 | Total Worker time limit | -| `collect.max_workers` | 1 | Concurrency limit across Scopes | -| `target_input`, `identify_target`, `setup` | Required for collection | See the table above | -| `scopes` | `None` | Scopes the plugin may serve; empty means every Scope | - -### 3.3 Example - -The plugin below collects new posts from each target account on X every 5 minutes: +An Artifact plugin can provide its own Source types in the `definitions` of its declaration (see 4.2). The plugin uses them +to record what users asked for, such as uploading a document or rewriting a chunk, and then turns those Sources into +Artifacts in `generate`. Every change thus has a matching Source and traceable provenance, without a separate Source +plugin. ```python -import os - -from pydantic import BaseModel - -from powercontext.plugins import ( - Collect, CollectedItem, CollectRequest, CollectResult, Interval, PluginContext, SourcePlugin, TargetIdentity, -) - -from definition import PostCapture, XPostDefinition - - -class TargetInput(BaseModel): - account: str - - -def identify_target(value: TargetInput) -> TargetIdentity: - account = value.account.strip().removeprefix("@").lower() - if not account: - raise ValueError("account must not be blank") - return TargetIdentity(key=f"account:{account}", input={"account": account}) - - -class XCollector: - def __init__(self, client: XClient) -> None: - self.client = client - - async def collect(self, request: CollectRequest) -> CollectResult: - result = CollectResult() - for target in request.targets: - since = (target.checkpoint or {}).get("since_id") - page = await self.client.user_posts(target.input["account"], since_id=since) - result.add(target, [CollectedItem(definition="x-post", value=PostCapture(**p)) for p in page.posts], - checkpoint={"since_id": page.newest_id or since}) - return result - - -async def setup(ctx: PluginContext) -> XCollector: - client = await ctx.enter_async_context(XClient(os.environ["X_BEARER_TOKEN"])) - return XCollector(client) - - -plugin = SourcePlugin( - definitions=(XPostDefinition(),), - collect=Collect(schedule=Interval(seconds=300), timeout_seconds=120), - target_input=TargetInput, - identify_target=identify_target, +plugin = ArtifactPlugin( + family="knowledge-base", + definitions=(DocumentUploadDefinition(),), # Source type kb-document + inputs=Inputs(source_types=("kb-document",), projection="knowledge-base.upload@1"), + on_sources=True, setup=setup, ) ``` -### 3.4 How callers use it - -Callers use the existing Source management APIs and do not need to know a plugin provides the capability: - -```text -pc_source_add(scope_id="team-a", capability="x-posts", input={"account": "@PowerContextAI"}) -pc_source_status(scope_id="team-a", target_id="...") -pc_source_target_set_enabled(scope_id="team-a", target_id="...", enabled=false) -``` - -- **Adding**: core validates the input with `target_input` and calls `identify_target` to get the key and normalized - input. The same account yields the same key and reuses the existing target. A new target is saved as enabled and one - collection is requested immediately, with no restart. -- **Checkpoints**: each target's checkpoint is a read-only input, and the plugin proposes a new checkpoint through its - return value. Core commits all returned Sources and every target's new checkpoint in one transaction; if `collect` - raises or times out, nothing is committed and the next invocation still receives the old checkpoint. -- **Format validation**: `pc_source_capabilities` publishes the JSON Schema of `target_input`, so callers know the format - before they submit. On add, core first validates with `target_input` and returns 422 naming the field on failure; a - `ValueError` from `identify_target` also returns 422. Whether the external resource exists or is readable can only be - known during `collect`, and the result shows up as the latest failure in `pc_source_status`. - -### 3.5 Edge collection - -Scenarios such as embodied AI collect data on the device and submit it to a central PowerContext Server. The same -program starts in edge mode: - -```text -powercontext source-agent --server https://memory.example.com -``` - -- The edge loads only the Source plugins in the plugin directory that declare `collect`, and runs the Supervisor to - schedule `collect`. It loads no Artifact plugins and serves no Artifact APIs. -- Collection targets are stored in a local SQLite database on the edge and managed through the edge's local - `pc_source_add`, `pc_source_status`, and `pc_source_target_set_enabled`. The central Server does not see edge targets; - it sees only the submitted Sources and checkpoints. -- Results are submitted to the central Server through core's existing remote ingestion APIs: Sources through - `POST /v1/source-observations`, and checkpoints through `POST /v1/connector-checkpoints/commit`. `resolve` and - Projections run on the edge; the central Server validates against the registered definition manifests and does not - need to load plugin code. -- Disconnects, restarts, and resubmission are idempotent: Sources are submitted first, then the new checkpoint is - committed with the old checkpoint as the expected value. If anything fails midway, the checkpoint does not advance and - the next call collects again from the old checkpoint; resubmitted Sources have the same identity and content, and the - central Server deduplicates them. -- The plugin handles any backlog that builds up while offline. When a submission fails, the call fails and the - checkpoint does not advance. For data the external system cannot replay, such as sensor streams, the plugin first - keeps it in its own local storage and provides it again by checkpoint. +- **Definitions**: the existing `SourceDefinition` applies: `input_class` validates the input, `resolve` turns it into a + Source, and Projections give the shapes consumers read. +- **Writes**: callers submit through the existing Source write API (`POST /v1/scopes/{scope_id}/sources` with the plugin's + type as `source_type`), and the plugin's own code submits with `ctx.sources.submit`. Validation, `resolve`, and + Projections run in the plugin's host process, and saving follows the same semantics as built-in Sources. +- **Append-only**: like every built-in Source, these have no updates or deletes; to correct content, submit a new Source. +- **Large content**: `resolve` must be a pure function, and the existing Source write APIs accept only JSON. Files and + images come in through plugin-defined routes (see [RFC #2579](https://github.com/oceanbase/powercontext/pull/2579)); the + plugin stores them in its own storage, and the Source records only metadata such as the location, digest, and media type. +- **Consumption**: the plugin consumes these types by listing them in `inputs.source_types`; other Artifact plugins can + consume them too. ## 4. Artifact plugins @@ -397,7 +274,7 @@ search API and Prepared Context return 422 for the family. | `transaction_timeout_seconds` | 30 | Time limit for the write transaction block, at most `timeout_seconds` | | `max_workers` | 1 | Concurrency limit across Scopes | | `change_timeout_seconds` | 30 | Time limit for one user-change call, including its write transaction block | -| `definitions` | `()` | The plugin's own Source Definitions, registered when the plugin loads, under the same rules as a Source plugin's `definitions`. The plugin submits them from its routes with `ctx.sources.submit`, and callers can also submit them through the generic Source write API, so one Artifact plugin can record what users asked for without a separate Source plugin | +| `definitions` | `()` | The plugin's own Source types, registered when the plugin loads; see section 3 and reference section 2 | | `setup` | Required | See the table above | ### 4.3 Example @@ -508,9 +385,8 @@ implement the method also returns 422, and a host process that times out or is u | Method | Applies to | Required | When core calls it | Where it runs | | --- | --- | --- | --- | --- | -| `setup(ctx)` | Both plugin kinds | Yes | Once when the host process starts; once per Worker invocation | Host process, Worker | -| Source Definitions: `input_class`, `resolve`, a Projection's `project`, `read` | Source plugins, or Artifact plugins that declare `definitions` | Yes | Source writes (the generic write API, `ctx.sources.submit`); computing Projections when delivering Sources to `generate` | Host process, Worker | -| `identify_target`, `collect` | Source plugins | Yes, for continuous collection | Adding a collection target; scheduled collection | Host process, Worker | +| `setup(ctx)` | Every plugin | Yes | Once when the host process starts; once per Worker invocation | Host process, Worker | +| Source Definitions: `input_class`, `resolve`, a Projection's `project`, `read` | Plugins that declare `definitions` | Yes | Source writes (the generic write API, `ctx.sources.submit`); computing Projections when delivering Sources to `generate` | Host process, Worker | | `generate` | Artifact plugins | When `schedule` or `on_sources` is set | On schedule, or when new Sources arrive | Worker | | `render_artifact` | Artifact plugins | Yes | The default presentation API | Host process | | `search_artifacts` | Artifact plugins | No | The generic search API; Prepared Context sections that list the family | Host process | @@ -522,7 +398,7 @@ plugin method; core serves them from the data it stores. The host process checks ## 5. Scheduling and triggers -Scheduling and triggering of plugins' background calls (`collect`, `generate`) are delegated to the existing Artifact +Scheduling and triggering of plugins' background calls (`generate`) are delegated to the existing Artifact processing Supervisor; synchronous calls are triggered by requests and do not go through the Supervisor. At startup each plugin registers as one Supervisor binding named `plugin:`, and the scheduling key is `(binding, Scope)`. @@ -540,8 +416,6 @@ Artifact plugins: - `Cron` is allowed only for plugins with `inputs` and cannot be combined with `on_sources`: each Supervisor binding accepts a single interval or cron policy. -A Source plugin's `collect` is called on `collect.schedule`; adding or resuming a target also requests one collection. - At most one Worker runs per plugin per Scope. Triggers that arrive during a run are coalesced by the Supervisor into one invocation after it finishes. @@ -557,10 +431,8 @@ invocation after it finishes. - `GET /v1/plugins/{name}/scopes`: for each Scope, whether work is pending, the journal position consumed through, and the current journal position. - Reasons for failed calls are diagnosed through logs and metrics; the API does not store them. -- **Changes**: restart Server after changing code. Server does not watch files or reload plugins at runtime. Collection - targets are business data, so adding or removing them needs no restart. -- **Disabling**: move the plugin out of the directory and restart. Saved Sources, Artifacts, collection targets, cursors, - and checkpoints are all kept, but the plugin's family and Source types are no longer registered: reading, searching, +- **Changes**: restart Server after changing code. Server does not watch files or reload plugins at runtime. +- **Disabling**: move the plugin out of the directory and restart. Saved Sources, Artifacts, and cursors are all kept, but the plugin's family and Source types are no longer registered: reading, searching, or changing that family through the generic APIs returns 404, as for a family that does not exist. A plugin Artifact's content usually records only digests or locations, while bodies, chunks, and files live in the plugin's own storage and cannot be interpreted without the plugin, so there are no reads without the plugin. New writes of the plugin's @@ -574,8 +446,8 @@ invocation after it finishes. ## 1. Shared contract -The public interfaces live in a new module, `powercontext.plugins`, replacing the plugin-author-facing parts of the existing -`powercontext.artifact_plugins` and `powercontext.source_management`. Interfaces are expressed as `typing.Protocol` and +The public interfaces live in a new module, `powercontext.plugins`, replacing the existing +`powercontext.artifact_plugins`. Interfaces are expressed as `typing.Protocol` and Pydantic models. ### 1.1 Declarations and loading @@ -593,17 +465,12 @@ class ArtifactPlugin(BaseModel): # fields in guide 4.2 inputs: Inputs | None = None ... setup: Callable[[PluginContext], Awaitable[ArtifactGenerator]] - -class SourcePlugin(BaseModel): # fields in guide 3.2 - definitions: Sequence[SourceDefinition] - collect: Collect | None = None - ... ``` - The plugin name comes from the directory name and must match `^[a-z][a-z0-9-]{0,63}$`. - A declaration is validated when constructed, including schedule combinations, the dependency between `inputs` and `scopes`, time-limit relations, and family naming. `setup` must be an async function. -- The parts of a declaration other than code objects such as `setup`, `identify_target`, and `definitions` are +- The parts of a declaration other than code objects such as `setup` and `definitions` are serializable. Server obtains them from the check child process to register bindings and answer operational queries. - During the check, core records the plugin directory's **code fingerprint** (computed from file contents). Workers and host processes check it before importing the plugin, and a mismatch fails the call and asks for a Server restart, @@ -663,94 +530,23 @@ class PluginModels(Protocol): compared. Calls follow Server's timeout settings and count toward the existing model usage statistics. A plugin may also read its own model configuration from environment variables and create its own clients. -- Plugins get no connection to core's database and no interface for writing cursors or checkpoints. Plugins write core - data through exactly three paths: a Source plugin's `collect` return value, the return value of `resolve` in the host - process, and an Artifact plugin's `ArtifactWriter`. - -## 2. Source plugin contract +- Plugins get no connection to core's database and no interface for writing cursors. Plugins write core data through + exactly two paths: the return value of `resolve` in the host process, and the `ArtifactWriter`. -```python -class TargetIdentity(BaseModel): - key: str # stable target key, always the same for the same external resource - input: dict[str, JsonValue] # normalized input keeping every parameter that affects collection - -class CollectTarget(BaseModel): - key: str - input: dict[str, JsonValue] - checkpoint: JsonValue | None # the target's last committed checkpoint; None the first time; read-only - -class CollectRequest(BaseModel): - scope_id: str - reason: Literal["schedule", "target_added"] - targets: Sequence[CollectTarget] # enabled targets to collect in this invocation - max_items: int - -class CollectedItem(BaseModel): - definition: str # name of one of the definitions - value: BaseModel # an instance of that definition's input_class - -class TargetResult(BaseModel): - items: Sequence[CollectedItem] - checkpoint: JsonValue | None - has_more: bool = False - -class CollectResult(BaseModel): - targets: dict[str, TargetResult] = {} # keyed by target key - def add(self, target: CollectTarget, items: Sequence[CollectedItem], *, checkpoint: JsonValue | None) -> None: ... - -class SourceCollector(Protocol): - async def collect(self, request: CollectRequest, /) -> CollectResult: ... -``` +## 2. Plugin Source type contract - **Definition registration**: the Source types and Projections in `definitions` are registered in the existing Source - definition registry at startup; a conflict with a built-in type or another plugin fails the check. -- **Writes**: callers submit content through the existing Source write APIs (`POST /v1/scopes/{scope_id}/sources`, or the - write usage of MCP `pc_source_add`). Core validates the input with `definition.input_class`, calls `resolve` to obtain - the Source, computes each Projection, and saves it, with the same semantics as built-in Source writes. Validation, - `resolve`, and Projections run synchronously in the plugin's host process with a timeout. `resolve` must be a pure - function with no side effects such as copying files; to bring external files into plugin storage, use `collect`. -- **collect**: each invocation covers one Scope and carries at most `max_targets` enabled targets; when there are more, - targets rotate by key, and targets that returned `has_more` go first in the next invocation. The executor applies the - same validation, `resolve`, and Projections as writes to every returned Source, then in one transaction saves all - Sources, writes each target's checkpoint and last collection time, and acknowledges the invocation. If any item fails, - nothing is committed. Exceeding `max_items` in total, or returning a target key absent from the request, is a failure; - nothing is truncated. Targets absent from the result keep their checkpoint. -- **Source identity**: determined by `resolve`. When repeated collection of the same external state yields the same - identity, existing deduplication rules apply and no duplicate Source is created. - -**Collection targets** are business data stored by core and, like Sources, are not plugin lifecycle state. Management reuses -the existing Source management APIs: - -| Operation | API | Description | -| --- | --- | --- | -| Add | MCP / HTTP `pc_source_add` (`capability` is the plugin name) | Validates the input with `target_input` and calls `identify_target` to get the key and normalized input. The same key with the same normalized input returns the existing target; the same key with a different input is a conflict. A new target is saved as enabled and one collection is requested | -| View | MCP / HTTP `pc_source_status` | Target key, normalized input, enabled state, last successful collection time, last failure category | -| Disable / resume | MCP / HTTP `pc_source_target_set_enabled` | A disabled target is left out of collect requests, and its checkpoint and collected Sources are kept; resuming continues from the saved checkpoint | - -- Adding and changing targets require Source write permission on the Scope, and the Scope must be within the plugin's + definition manifests (`pc_source_definition_manifests`) when the plugin loads. A conflict with a built-in type, a type + registered by the old Source plugin framework, or another plugin fails the check. +- **Writes**: core validates the input with `definition.input_class`, calls `resolve` to get the Source, computes each + Projection, and saves it, with the same semantics as built-in Source writes. Validation, `resolve`, and Projections run + synchronously in the plugin's host process under Server's synchronous call limit. `resolve` must be a pure function with + no side effects such as copying files. +- **Source identity**: determined by `resolve`. Resubmitting the same content yields the same identity and follows the + existing deduplication rules, so no duplicate Source is created; the same identity with different content is a conflict. +- **Permissions**: writes require Source write permission on the Scope, and the Scope must be within the plugin's `scopes`. -- `identify_target` must be a pure function: it only validates and normalizes, without contacting external systems or - checking external authorization. Whether an external resource exists or is readable is decided by the plugin during - collect and expressed as an exception or an empty result. Like writes, it runs synchronously in the host process with a - timeout. -- `pc_source_capabilities` lists the Source types plugins provide and the JSON Schema of their target inputs. - -**Edge mode** (`powercontext source-agent`) keeps the same Source plugin contract and only replaces where results are -committed: - -| Item | Handling in edge mode | -| --- | --- | -| Configuration | `--server` names the central Server; `POWERCONTEXT_SOURCE_AGENT_ID` is the edge's stable identifier; `POWERCONTEXT_SOURCE_AGENT_TOKEN` is a credential with `scope.contribute` on the target Scopes | -| Local data | A SQLite database in the edge's data directory holds collection targets (with the structure of `pc_source_collect_targets`) and Supervisor scheduling state; it holds no Sources | -| Definition manifests | At startup, the declarative manifest of each Source definition is registered with the central Server (`POST /v1/source-definitions/register`, which requires `server.admin`); if the credential lacks that permission, the deployer registers them in advance | -| Each `collect` | Reads each target's checkpoint from the central Server (`/v1/connector-checkpoints/get`), calls the plugin, runs `resolve` and Projections on the edge, submits the Sources one by one (`/v1/source-observations`), and after all succeed commits the new checkpoint with the old one as the expected value (`/v1/connector-checkpoints/commit`) | -| Target binding ID | Derived deterministically from the edge identifier, the plugin name, and the target key, so two edges collecting the same external resource do not overwrite each other's checkpoints | -| Failure | Any failed step fails the call, which enters the Supervisor's backoff; already submitted Sources are kept, the checkpoint does not advance, and the central Server deduplicates the next collection | - -The edge and the central Server do not share a transaction: a Source can be saved while the checkpoint has not advanced, -and Source identity deduplication prevents duplicates. This requires the plugin's `resolve` to give the same identity and -content for the same external state; when the identity matches but the content differs, the central Server returns a -conflict and the call fails. +- **Discovery**: `GET /v1/capabilities` and `pc_source_capabilities` list the Source types of loaded plugins. ## 3. Artifact plugin contract @@ -768,7 +564,7 @@ class GenerateRequest(BaseModel): class GenerationSource(BaseModel): ref: SourceRef # exact Source reference - position: int # the Source's journal position, for processing in collection order + position: int # the Source's journal position, for processing in write order projection: JsonValue # the Projection declared for this Source type ``` @@ -1066,10 +862,6 @@ Each background call of an Artifact plugin processes one Source window after the - Without `scopes`, automatic scheduling covers every Scope whose journal is past the cursor; with `scopes`, only Scopes on the list are accepted. -Collect has the same structure: its input is the Scope's enabled targets and their checkpoints, and its commit is the -Sources and every target's new checkpoint, which the executor commits from the return value at once. Adding or restoring -a collection target requests one collection. - ## 6. Plugin host process Synchronous calls must return within the request's time limit, so they cannot start a Worker each time: every Worker is a @@ -1079,21 +871,18 @@ long-lived host process: | Call | Where it runs | | --- | --- | | Input validation, `resolve`, and Projections for Source writes | Host process | -| `identify_target` when adding a collection target | Host process | | `create_artifact`, `update_artifact`, `merge_artifacts`, `forget_artifact`, `restore_artifact` | Host process | | `render_artifact`, `search_artifacts` | Host process | -| `collect`, `generate` | Supervisor Worker | +| `generate` | Supervisor Worker | - **Count and startup**: each Server process keeps at most one host process for each loaded plugin, starts it in the - background once the plugin loads, and reuses it afterward. Every Artifact plugin implements `render_artifact` and - Source plugin writes run `resolve`, so every loaded plugin has a host process. - In edge mode, `resolve` and Projections run inside the collection Worker, and no host process - starts. + background once the plugin loads, and reuses it afterward. Every plugin implements `render_artifact`, so every + loaded plugin has a host process. - **Execution**: the host process imports `plugin.py` once at startup and compares the declaration, and the writer runs inside the host process. One host process can serve several requests concurrently, up to a limit set in the Server configuration. -- **Timeouts and failures**: each call has its own time limit (Source writes and `identify_target` use Server's - synchronous call limit; user changes use `change_timeout_seconds`). On a timeout, crash, or loss of response, core +- **Timeouts and failures**: each call has its own time limit (Source writes, presentation, and search use + Server's synchronous call limit; user changes use `change_timeout_seconds`). On a timeout, crash, or loss of response, core terminates the process, in-flight requests return 503, and the process restarts after a backoff. During the backoff the plugin's synchronous calls return 503; background calls are unaffected. - **Code changes**: a host process does not re-import the plugin while it lives, and checks the code fingerprint when it @@ -1189,7 +978,7 @@ class SearchResult(BaseModel): ## 9. Execution constraints -- **Isolation**: `collect` / `generate` run in Worker child processes started by the Supervisor and synchronous calls run in +- **Isolation**: `generate` runs in Worker child processes started by the Supervisor and synchronous calls run in host processes, so crashes, memory leaks, and blocking do not affect Server. Child processes isolate ordinary failures; they are not a sandbox for untrusted code: a plugin runs with the child process's operating-system privileges and can access its files, network, and environment variables. @@ -1203,10 +992,9 @@ class SearchResult(BaseModel): processed and do not block the database readiness check. - **Failure and backoff**: a failed call commits no core data, and that plugin in that Scope enters the Supervisor's existing backoff without affecting other Scopes or plugins. Core never skips input automatically because of repeated - failures; persistent failures require fixing the plugin or the data and restarting. A target that keeps failing in - collect blocks the other targets in the same Scope; disable it first. + failures; persistent failures require fixing the plugin or the data and restarting. - **Multi-instance deployments**: with several Server instances on OceanBase, dispatch happens only on the instance holding - the Supervisor leader lease; Source writes, target management, reads, and user changes can run on any instance, each using + the Supervisor leader lease; Source writes, reads, presentation, search, and user changes can run on any instance, each using its own host processes. Plugin-owned storage must be reachable from every instance; the dedicated storage core allocates lives on the shared database backend and meets this naturally. A SQLite deployment has a single Server process. @@ -1214,11 +1002,9 @@ class SearchResult(BaseModel): | Data | Handling | | --- | --- | -| `pc_sources`, Projections, the Source journal, `pc_source_definition_manifests` | Kept; Source plugin results | +| `pc_sources`, Projections, the Source journal, `pc_source_definition_manifests` | Kept; definitions and written results of plugin Source types | | `pc_artifacts`, `pc_artifact_heads`, `pc_artifact_lineage_*`, merge records, handles, tags | Kept; Artifact plugin results | | `pc_source_cursors` | Kept; Artifact plugin consumption cursors, keyed by `(scope_id, plugin:)` | -| `pc_connector_checkpoints` | Kept; each collection target's checkpoint, keyed by `(scope_id, the target's binding ID)` | -| `pc_source_collect_targets` | Added; collection targets with their normalized input, enabled state, and latest collection results | | Supervisor invocation intent, pending, and lease tables | Kept; plugin bindings share them with built-in bindings, and they back `GET /v1/plugins/{name}/scopes`. Plugin bindings start with `plugin:` and are not registered in the database | | The new `metadata` column of `pc_artifacts` | Added; nullable, stored as canonical JSON bytes like `content`, for additional information that is never searched. Revisions plugins write record the plugin name and version digest; built-in families leave it empty. It does not take part in content digests | | `pc_artifact_fts` and `pc_artifact_heads.searchable_text` | Used only by built-in families; plugin families do not write them | @@ -1226,16 +1012,13 @@ class SearchResult(BaseModel): | `pc_artifact_family_schemas` | Removed | | Search state and vector index data for custom family declarative indexes | Removed; the implementation PR confirms built-in families do not depend on them before dropping the tables | | Artifacts, heads, and lineage written by old Artifact plugins | Not migrated; the upgrade is refused while any remain, see below | -| `pc_source_write_plugins` | Removed; Source definitions come from the plugin directory at startup | -| The Source plugin management service's SQLite | Removed with the management service | | Dedicated plugin storage (`plugins/.sqlite3` or the `powercontext_plugin_` database) | Not part of core's schema; core only allocates it, and the plugin and the deployer own its tables, migrations, and cleanup | | Other plugin-owned indexes and storage | Not part of core; managed by the plugin and the deployer | -| The local SQLite of edge mode | Holds only the edge's collection targets and scheduling state, no Sources; Sources and checkpoints live on the central Server | Additions and removals ship as **one new migration revision**; the migration resources of released versions stay -unchanged. Its notes list the tables, columns, and data ranges added and removed. The migration preflight first confirms the old plugin frameworks -left no data behind: Artifacts of old Artifact plugin families, old plugin registrations and invocation intents, and -Source plugins in `pc_source_write_plugins`. If it finds any, it refuses the upgrade, lists the plugin names, families, and +unchanged. Its notes list the tables, columns, and data ranges added and removed. The migration preflight first confirms the old Artifact plugin +framework left no data behind: Artifacts of old plugin families, and old plugin registrations and invocation intents. +If it finds any, it refuses the upgrade, lists the plugin names, families, and row counts, and leaves the database unchanged; the deployer exports what they need, uninstalls the old plugins, and cleans up their data on the old version, then upgrades. The migration drops these tables only when they are empty. @@ -1255,10 +1038,9 @@ new format. | Change | Content | | --- | --- | -| Removed | `/v1/admin/artifact-plugins` and its subpaths; the Source plugin management service, plugin instance Runtimes, and poll / reply channels; the `source load` command; MCP `pc_source_instances` and `pc_source_instance_set_enabled`; the `plugin.yml` and `powercontext.yml` manifests | +| Removed | `/v1/admin/artifact-plugins` and its subpaths; the old Artifact plugins' `plugin.yml` manifest | | Removed | Custom family declarative indexes (`ArtifactSearchDeclaration`, `TextIndex`, and so on), `ArtifactFamily` schema declarations, `ArtifactDraft`, and the old Worker plugin executor | -| Added | `powercontext.plugins`: `SourcePlugin`, `ArtifactPlugin`, `Inputs`, `Interval`, `Cron`, `PluginContext`, `PluginStorage`, `PluginModels`, the writer and read interfaces, and the user-change request models and `InvalidContent`, and the presentation and search request and result models | -| Added | The `powercontext source-agent` edge collection mode, with `POWERCONTEXT_SOURCE_AGENT_ID` and `POWERCONTEXT_SOURCE_AGENT_TOKEN` | +| Added | `powercontext.plugins`: `ArtifactPlugin`, `Inputs`, `Interval`, `Cron`, `PluginContext`, `PluginStorage`, `PluginModels`, the writer and read interfaces, and the user-change request models and `InvalidContent`, and the presentation and search request and result models | | Changed | The body of `POST /v1/scopes/{scope_id}/artifacts` gains a plugin branch `{"family", "content"}`; core allocates the ID, and the plugin's `create_artifact` handles it. `CreateArtifactRequest` becomes a choice between the built-in branch `CreateBuiltinArtifactRequest` (the former union discriminated by family) and the plugin branch `CreatePluginArtifactRequest`, selected by family name: built-in names take the built-in branch with unchanged validation; other names take the plugin branch, and a family no plugin provides returns 422. In the generated Python client, built-in requests gain one wrapping level, and `ArtifactCreated.family` changes from the built-in family enum to a string | | Changed | The `source_type` of `POST /v1/scopes/{scope_id}/sources` changes from an enum allowing only `content` to a string (still defaulting to `content`). It may name a Source type a plugin defines; `content` is then that type's input, which the plugin validates and resolves into a Source. Unknown types return 404. The `SourceType` enum remains | | Added | `GET /v1/plugins`, `GET /v1/plugins/{name}`, `GET /v1/plugins/{name}/scopes` (`server.admin`, read-only) | @@ -1269,8 +1051,7 @@ new format. | Added | `GET /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}/rendering`: the default presentation of a plugin family Artifact, provided by the plugin's `render_artifact` | | Changed | `POST /v1/context/prepare`: `ContextAssemblySection.family` changes from an enum to a string that may name a loaded plugin family; an optional `search` (the fields of a search request other than `query` and `limit`) applies to plugin families only | | Changed | The format version of Artifact revision records in export and restore increases by one and adds `metadata` | -| Changed | The write usage of `pc_source_add` uses plugin definitions, and its collect usage adds collection targets; `pc_source_capabilities` lists plugin Source types and target inputs; `pc_source_status` reports targets and their latest collection result; `pc_source_target_set_enabled` acts on targets stored by core | -| Changed | The environment variable `POWERCONTEXT_SERVER_PLUGINS_DIR` replaces `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` and the Source plugin directory setting | +| Changed | The environment variable `POWERCONTEXT_SERVER_PLUGINS_DIR` replaces `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` | After `openapi/powercontext.yaml` is updated per the table, `src/powercontext/http/_generated/` and each integration's operation table are regenerated. @@ -1282,9 +1063,9 @@ and no request ID is generated for scheduled calls. New fields identify the plug (Worker or host process), the trigger, and the size of the writes; host process starts, terminations, and restarts each record one event. -- Metric labels may use the plugin name, method, and execution location, never Scope IDs or target keys. -- Logs go through `log_safely` and exclude Source content, Projections, Artifact content, target inputs, and environment - variable values; plugin exceptions are logged only by type and known error category, never by message. +- Metric labels may use the plugin name, method, and execution location, never Scope IDs. +- Logs go through `log_safely` and exclude Source content, Projections, Artifact content, and environment variable + values; plugin exceptions are logged only by type and known error category, never by message. - New fields are registered in the allowlists per the developer guide and documented in both locales' observability docs. - Call outcomes go only to logs and metrics and are not persisted, so the operational APIs report no "last call result". @@ -1306,8 +1087,9 @@ record one event. Worker calls. - **Dedicated storage is isolated only by convention**: the plugin and Server use the same database login, so a plugin can technically reach core's tables. -- **The edge and the central Server do not share a transaction**: Source identity deduplication prevents duplicates, and a - plugin whose `resolve` is unstable produces conflicts in edge mode. +- **Old and new ways to add Source types coexist**: until the follow-up proposal lands, Source types can come from a + plugin's own definitions in this proposal or from write plugins of the old Source plugin framework. Both register in + the same definition manifests, and names must not collide. - **Not a security sandbox**: Workers isolate ordinary failures but do not restrict a plugin's access to files, network, or credentials, nor impose hard memory or CPU quotas. - **Dependencies must be compatible**: every plugin shares one Python environment with Server, and deployers resolve @@ -1347,7 +1129,6 @@ record one event. | Two-phase commit between plugin storage and core | Most plugin storage, such as object and vector stores, does not support XA, and core would need a coordinator for each kind of storage. Writing immutable content-addressed records first and making core's commit the only point where they take effect gives the consistency needed | | One plugin declaring several families | Could write several families in one transaction, but declarations, permissions, and merge registration would all need a multi-family design. Cooperating cases are served by two plugins plus cross-family reads | | Hand core's environment variables or database settings to plugins | Core's configuration does not all come from environment variables, and a plugin with core's database credentials could bypass the writer. Core provides the wrapped `ctx.storage` and `ctx.models` instead | -| Compile a separate binary for the edge | A plugin is a Python directory with dependencies; a frozen binary cannot import arbitrary plugin dependencies and would need packaging per plugin and platform. Starting the same program in `source-agent` mode is enough | | Plugin-specific trigger and management APIs | Callers would have to tell plugins from built-ins; the existing Source and Artifact APIs already cover the operations callers need | | Generic merge APIs for plugin families only | Needs a separate family allowlist in the APIs. The registry already decides which families can be changed, and a family without an adapter cannot register; built-in permission checks live in the engine, so both entry points apply the same checks | | Core builds a full-text index for plugin families from declared fields | Plugin authors need no index of their own, but core does not know how content is chunked or whether it suits full-text or vector search, so the index is often useless; it would also share index tables with built-in families, and the readiness check would have to tell them apart. Search belongs to the plugin, and core only filters by permission | @@ -1373,9 +1154,6 @@ Reasons for this choice: when a workflow or application needs them; plugins run in a separate plugin daemon. This proposal keeps the idea that the host defines the interfaces and when to call them: synchronous calls use long-lived host processes that core manages, and background calls reuse PowerContext's existing Supervisor Workers. -- **The Airbyte connector protocol**: a Source connector's `read` receives the previous state and emits records and a new - state, which the platform persists. The collection checkpoints in this proposal split the work the same way and keep a - checkpoint per target. - **pytest plugins and setuptools entry points**: plugins declare what they provide as Python objects that the host reads after importing them, with no separate manifest file. - **Transaction context managers**: interfaces such as SQLAlchemy's `async with engine.begin()` open a transaction when the @@ -1384,8 +1162,7 @@ Reasons for this choice: - **Existing PowerContext foundations**: scheduling, invocation intents, Workers, and lease fences of the [Artifact processing Supervisor](1515_artifact_processing_supervisor.md); the [shared merge methods](1859_artifact_merge.md); the generic Artifact read and write APIs and the `lineage_only` Content Source built-in families record on edits; - `SourceDefinition` / `SourceProjection`, the Source journal, and consumption cursors; connector checkpoints and the - remote ingestion APIs; and the Source management APIs with their target identification rule. + `SourceDefinition` / `SourceProjection`, the Source journal, and consumption cursors. # Unresolved questions @@ -1394,7 +1171,6 @@ Reasons for this choice: | Default limits for the write transaction block | Whether 30 seconds is right; whether SQLite and OceanBase need different defaults; per-block operation and byte limits | | Host process defaults | Default synchronous call limit, per-host-process concurrency limit, and restart backoff | | Permission to create the dedicated database | When Server's database login cannot create databases, whether to allow falling back to tables prefixed by plugin name in core's database, besides the deployer creating the database in advance | -| Deleting collection targets | The first version supports only disabling and resuming. Whether targets and their checkpoints should be deletable, and what happens to Sources already collected | | Plugin families in Recall | Prepared Context is covered; whether Recall should be, and how its candidates are defined | | Passages in generic search results | The generic search API returns only exact revisions, and the passages a plugin returns are used only by Prepared Context. Whether search results should add passages and their locations | | Naming families and Source types | Whether they must start with the plugin name, so naming alone keeps plugins from taking over each other's families | @@ -1408,6 +1184,9 @@ Reasons for this choice: # Future possibilities - Hot reload and plugin-defined APIs (including plugin pages): built on host processes, see [RFC #2579](https://github.com/oceanbase/powercontext/pull/2579). +- Standalone Source plugins and collection: plugins that only provide Source types; central configuration of collection + targets that devices pull and sync; running collection on a device and reporting Sources to a central Server; moving + large files. A follow-up proposal defines them and replaces the old Source plugin framework. - Long-lived processes for background calls too, reusing connections and initialization while keeping the interfaces unchanged. - More plugin types, such as candidate content for Recall or text fragments for Prepared Context. diff --git a/docs/zh/rfcs/2549-plugin-interfaces.md b/docs/zh/rfcs/2549-plugin-interfaces.md index 826f82750b..7cc54249e8 100644 --- a/docs/zh/rfcs/2549-plugin-interfaces.md +++ b/docs/zh/rfcs/2549-plugin-interfaces.md @@ -1,6 +1,6 @@ --- title: 插件接口 -description: Source 和 Artifact 插件各自实现少量接口，由 PC Server 在需要时调用：后台调用在 Supervisor Worker 中执行，同步调用在 core 管理的插件宿主进程中执行；调用方只使用现有的 Source 与 Artifact 接口，core 只管理两类核心数据及其消费进度，不托管插件生命周期。 +description: Artifact 插件实现少量接口，由 PC Server 在需要时调用：后台调用在 Supervisor Worker 中执行，同步调用在 core 管理的插件宿主进程中执行；调用方只使用现有的 Source 与 Artifact 接口，core 只管理两类核心数据及其消费进度，不托管插件生命周期。 --- - Proposal Name: `plugin_interfaces` @@ -12,52 +12,50 @@ description: Source 和 Artifact 插件各自实现少量接口，由 PC Server 本提案重新定义 PowerContext 的第三方插件模型：插件只实现几个固定接口，PC Server 在需要时调用；调用方不直接调用插件， 而是像使用内置能力一样使用现有的 Source 和 Artifact 接口。 -- **两类插件**： - - **Source 插件**提供 Source 定义，需要持续采集时再实现 `identify_target` 和 `collect`。 - - **Artifact 插件**实现 `generate`，把尚未消费的 Source 加工成 Artifact。 -- **插件形态**：一个插件是一个目录，包含 `plugin.py`，其中把一个声明对象（`SourcePlugin` 或 `ArtifactPlugin`）赋给 - `plugin`。声明写在代码里，没有单独的清单文件；部署相关的设置由插件从环境变量读取。 +- **Artifact 插件**：拥有一个 Artifact family，实现 `generate` 把尚未消费的 Source 加工成 Artifact，实现 + `render_artifact` 给出默认展示，可选实现检索和用户修改。插件可以自带 Source 类型，用来记录用户提交给它的内容。 +- **插件形态**：一个插件是一个目录，包含 `plugin.py`，其中把一个声明对象 `ArtifactPlugin` 赋给 `plugin`。 + 声明写在代码里，没有单独的清单文件；部署相关的设置由插件从环境变量读取。 - **调用方式**： - - 写入 Source、添加采集目标，使用现有的 Source 接口。 + - 写入插件自带类型的 Source，使用现有的 Source 写入接口。 - 读取、检索 Artifact，使用现有的 Artifact 接口。 - 新建、编辑、合并、遗忘、恢复插件 Artifact，也使用现有的 Artifact 接口。core 负责鉴权、版本校验和溯源，再同步调用 插件对应的方法；插件校验内容、维护自己的存储，并通过写入器写入。插件没有实现某项操作时，该接口返回 422。 - **执行**： - - `collect` 和 `generate` 是后台调用，在 Supervisor 管理的 Worker 子进程中执行，沿用现有的隔离、超时、并发预算、 - 领导租约和失败退避。 - - Source 写入时的 `resolve` 与 Projection、添加采集目标时的 `identify_target`、用户对插件 Artifact 的修改是同步调用， - 在 core 管理的常驻**插件宿主进程**中执行。每个 Server 进程为每个需要同步调用的插件启动一个宿主进程，部署者不需要管理它。 + - `generate` 是后台调用，在 Supervisor 管理的 Worker 子进程中执行，沿用现有的隔离、超时、并发预算、领导租约和失败退避。 + - Source 写入时的 `resolve` 与 Projection、展示、检索和用户对插件 Artifact 的修改是同步调用，在 core 管理的常驻 + **插件宿主进程**中执行。每个 Server 进程为每个加载的插件启动一个宿主进程，部署者不需要管理它。 - **写入**：Artifact 插件在 `async with writer.transaction()` 块中执行写操作，块结束时 core 把写入、消费游标和 完成确认一起提交，异常则整体回滚。插件读不到也改不了游标。 - **core 的职责**：只管理 `pc_sources`、`pc_artifacts`（含 heads、lineage、合并记录、生命周期、handle 和 tags）及其 消费进度。插件可以有自己的索引和存储，由插件自己管理；core 在部署使用的数据库后端上为每个插件提供一块专属存储， 并把 Server 已配置的模型提供给插件使用。 -- **端侧采集**：Server 可以以 `source-agent` 模式启动，只运行 Source 插件的采集，把 Source 和 checkpoint 提交到远端的 - 中心 Server，用于具身智能等需要在设备上采集的场景。 - **生命周期**：插件在 Server 启动时按目录加载，变更需要重启。每个 Server 进程只在内存中记录本进程已激活的插件， 不持久化插件的启用状态、注册代次或管理阶段。 -本提案取代 [Source 插件框架 RFC #2012](2012-source-plugin-system.md) 和 [Artifact 插件框架 RFC #2175](2175-artifact-plugin-system.md) -的插件契约与管理机制。旧 Source 插件框架随 v1.5 发布，但目前没有对外发布的插件，因此本提案不保留兼容：旧的插件清单、 -管理接口、管理服务和插件管理相关的表直接移除，旧 Artifact 插件写入的自定义 Artifact 数据不迁移。为了不悄悄丢失数据， -升级前的迁移检查发现旧框架留有数据时拒绝升级，由部署者在旧版本上先导出和清理（见参考级说明第 10 节）。旧设计中合适的部分继续 -沿用，包括 Source 管理接口（`pc_source_add` 等）及采集目标的身份规则，以及 core 已有的远程采集接口。 +本提案取代 [Artifact 插件框架 RFC #2175](2175-artifact-plugin-system.md) 的插件契约与管理机制。旧框架随 v1.5 发布， +但目前没有对外发布的插件，因此本提案不保留兼容：旧的插件清单、管理接口和插件管理相关的表直接移除，旧插件写入的自定义 +Artifact 数据不迁移。为了不悄悄丢失数据，升级前的迁移检查发现旧框架留有数据时拒绝升级，由部署者在旧版本上先导出和 +清理（见参考级说明第 10 节）。 + +独立的 Source 插件，即只提供 Source 类型，或从外部系统持续采集（采集目标、checkpoint、在设备上采集并汇报到中心）， +不在本提案范围内，由后续提案定义并取代 [Source 插件框架 RFC #2012](2012-source-plugin-system.md)。在那之前，旧 Source +插件框架保持不变。 本文描述目标设计，不代表主线或发布版本已经可用。接口名称和代码示例用于说明契约，最终语法以实现为准。 # 动机 -现有两套插件框架都让 PowerContext 替第三方插件承担了大量生命周期管理： +现有的 Artifact 插件框架让 PowerContext 替第三方插件承担了大量生命周期管理： | 框架 | Server 为插件承担的工作 | | --- | --- | | Artifact 插件（#2175） | 在 `pc_artifact_processing_binding_states` 中维护 `desired_state`、`phase`、`binding_generation`、`last_error`；七种管理状态及其转换；`enable-all / reload / enable / disable / unload` 管理接口与 `expected_generation` 并发校验；用注册代次屏蔽旧 Worker 提交；在 `pc_artifact_family_schemas` 中固化内容 schema 和索引定义，变更返回 409；框架负责自定义声明式索引的构建、重建和可用状态 | -| Source 插件（#2012） | 独立的插件管理服务及其 SQLite（实例、capability、target、期望状态、提交记录）；每个插件实例一个 Runtime 进程；管理服务与 Server 之间的 poll / reply 通道和能力目录同步；`source load` 部署命令；实例启停 MCP 工具 | 这些机制的代价由所有人承担： -- **插件作者**需要理解 binding、generation、schema 固化规则、索引声明和管理通道等框架概念，才能写出一个插件。 -- **部署者**需要运行多个进程、维护两个数据库、编写清单文件，并通过管理 API 让插件进入可运行状态。 +- **插件作者**需要理解 binding、generation、schema 固化规则和索引声明等框架概念，才能写出一个插件。 +- **部署者**需要编写清单文件，并通过管理 API 让插件进入可运行状态。 - **调用方**需要区分内置能力和插件能力，使用不同的接口。 - **维护者**需要为第三方代码维护状态机、并发屏蔽、schema 和索引治理以及恢复路径，这部分代码的规模和测试面已经超过 插件业务本身。 @@ -75,8 +73,8 @@ description: Source 和 Artifact 插件各自实现少量接口，由 PC Server 5. core 只管理这两类核心数据及其消费进度；内容结构、索引和其他存储交给插件自己。 6. 插件对核心数据的写入只经过 core 提供的写入器，覆盖 Artifact 的全部通用写操作，并沿用 core 现有的校验规则。 7. 用户对插件 Artifact 的修改经过插件，插件自有存储能与修改同步更新。 -8. Artifact 生成可以按调度执行，也可以由新 Source 触发；采集目标可以在运行中增减。 -9. 用户只需部署一个 Server；插件随 Server 启动加载。需要在设备上采集时，同一个程序以端侧模式运行。 +8. Artifact 生成可以按调度执行，也可以由新 Source 触发。 +9. 用户只需部署一个 Server；插件随 Server 启动加载。 # 指南级说明 @@ -86,16 +84,13 @@ description: Source 和 Artifact 插件各自实现少量接口，由 PC Server flowchart LR Caller["应用 / Agent / 宿主"] subgraph Server["PC Server（core）"] - SourceAPI["现有 Source 接口
写入 Source、添加采集目标"] + SourceAPI["现有 Source 接口
写入 Source"] ArtifactAPI["现有 Artifact 接口
读取、检索、修改"] Sup["Supervisor
调度 / 触发 / 预算 / 退避"] - Core[("core 数据
pc_sources / pc_artifacts
采集目标 / checkpoint / 游标")] + Core[("core 数据
pc_sources / pc_artifacts / 游标")] end subgraph Host["插件宿主进程（常驻，core 管理）"] - Sync["resolve / Projection / identify_target
create_artifact / update_artifact / ..."] - end - subgraph SW["Worker：Source 插件"] - Collect["collect(request)"] + Sync["resolve / Projection / render_artifact
search_artifacts / create_artifact / ..."] end subgraph AW["Worker：Artifact 插件"] Generate["generate(request, writer)"] @@ -106,34 +101,30 @@ flowchart LR Caller --> ArtifactAPI SourceAPI -->|"同步调用"| Sync ArtifactAPI -->|"读取"| Core + ArtifactAPI -->|"展示、检索，同步调用"| Sync ArtifactAPI -->|"修改，同步调用"| Sync Sync -->|"Source 或 Artifact 写入，一次提交"| Core Sync -.-> Own - Sup -->|"目标与 checkpoint"| Collect - Collect -->|"Source 与新 checkpoint，一次提交"| Core Core -->|"新 Source 触发"| Sup Sup -->|"未消费的 Source 窗口"| Generate Tx -->|"Artifact 写入、游标、确认，一次提交"| Core - Collect -.-> Ext["外部系统"] Generate -.-> Own[("插件自有存储 / 模型")] ``` -图中有三条路径，一次调用只执行一个插件： +图中有两条路径，一次调用只执行一个插件： -- **采集路径**：Supervisor 按调度派发 Source 插件的 Worker，传入该 Scope 已启用的采集目标及其 checkpoint；插件从外部 - 系统读取数据并返回 Source；core 校验后把 Source 和新 checkpoint 在一个事务中保存。 - **生成路径**：新 Source 进入 journal 后，关注这些 Source 的 Artifact 插件被标记为待处理；Supervisor 派发 Worker，传入 该 Scope 尚未消费的 Source 窗口和写入器；插件在写事务块中写入 Artifact，core 连同游标和确认一起提交。 -- **同步路径**：调用方写入 Source、添加采集目标或修改插件 Artifact 时，Server 完成鉴权和版本校验后，把请求交给该插件的 +- **同步路径**：调用方写入插件类型的 Source、展示或检索插件 Artifact、修改插件 Artifact 时，Server 完成鉴权和版本校验后，把请求交给该插件的 宿主进程；插件在宿主进程中完成转换或写入，结果在一个事务中提交后同步返回给调用方。 调用方只和 core 的现有接口打交道： | 调用方要做的事 | 使用的接口 | 插件参与的方式 | | --- | --- | --- | -| 提交一条 Source | 现有 Source 写入接口 | 宿主进程执行插件定义的 `resolve` 和 Projection | -| 添加、查看、停用采集目标 | 现有 Source 管理接口 | 添加时宿主进程执行插件的 `identify_target` | -| 浏览、读取、检索 Artifact | 现有 Artifact 读取与检索接口 | 不参与 | +| 提交一条插件类型的 Source | 现有 Source 写入接口 | 宿主进程执行插件定义的 `resolve` 和 Projection | +| 浏览、读取 Artifact | 现有 Artifact 读取接口 | 不参与 | +| 展示、检索插件 Artifact | 展示接口、现有检索接口、Prepared Context | 宿主进程执行 `render_artifact`、`search_artifacts` | | 新建、编辑、合并、遗忘、恢复、撤销合并 | 现有 Artifact 新建与编辑接口、通用合并接口 | 宿主进程执行插件对应的方法；未实现返回 422 | | 打 tag、按 tag 查询 | 现有 tags 接口 | 不参与；tags 是 core 数据 | @@ -141,13 +132,12 @@ flowchart LR | 职责 | 归属 | | --- | --- | -| 认证、Scope 授权、公开 HTTP / MCP 接口、采集目标与 checkpoint 的保存、用户修改的版本校验与溯源记录 | core | +| 认证、Scope 授权、公开 HTTP / MCP 接口、用户修改的版本校验与溯源记录 | core | | 何时调用：调度、触发合并、Worker 预算、超时、退避、领导租约 | core（现有 Supervisor） | | 宿主进程的启动、复用、超时终止与重启 | core | -| 读取输入、提供受约束的写事务、在同一事务中提交核心数据 / 游标 / checkpoint / 完成确认 | core（Worker 或宿主进程中的插件执行器） | +| 读取输入、提供受约束的写事务、在同一事务中提交核心数据 / 游标 / 完成确认 | core（Worker 或宿主进程中的插件执行器） | | 插件专属存储空间的分配、Server 已配置模型的封装 | core | -| Source 定义与 Projection 计算、采集目标的识别与规范化、外部数据读取 | Source 插件 | -| Artifact 内容结构、生成算法、用户提交内容的校验、插件自有的索引与存储及其一致性 | Artifact 插件 | +| 插件自带 Source 类型的定义与 Projection 计算；Artifact 内容结构、生成算法、展示、检索、用户提交内容的校验、插件自有的索引与存储及其一致性 | 插件 | | 插件代码、依赖和环境变量的部署 | 部署者 | 插件代码只做一件事：给定输入，产出结果或抛出异常。抛出异常即失败，core 不提交任何核心数据；后台调用下次重试同一输入， @@ -160,9 +150,9 @@ Server 从 `POWERCONTEXT_SERVER_PLUGINS_DIR` 加载插件。每个直接子目 ```text /opt/powercontext/plugins/ -├── x-posts/ +├── knowledge-base/ │ ├── plugin.py -│ └── definition.py +│ └── parsers.py └── incident-report/ └── plugin.py ``` @@ -170,10 +160,10 @@ Server 从 `POWERCONTEXT_SERVER_PLUGINS_DIR` 加载插件。每个直接子目 `plugin.py` 把一个声明对象赋给模块级变量 `plugin`： ```python -plugin = ArtifactPlugin(family="incident-report", ..., setup=setup) # 或 SourcePlugin(...) +plugin = ArtifactPlugin(family="incident-report", ..., setup=setup) ``` -声明对象回答 core 在运行插件代码之前需要知道的问题：插件提供什么（Source 类型或 Artifact family）、消费什么、何时调用、 +声明对象回答 core 在运行插件代码之前需要知道的问题：插件提供什么（Artifact family 和自带的 Source 类型）、消费什么、何时调用、 资源上限多少。它是普通的 Python 对象，由 Pydantic 校验；没有 YAML 或其他清单文件。 - **部署设置**：插件自己的凭据、外部服务地址等随部署变化的值，由插件在 `setup` 中从环境变量读取。Server 不保存、 @@ -186,132 +176,32 @@ plugin = ArtifactPlugin(family="incident-report", ..., setup=setup) # 或 Sour - **启用与停用**：目录存在且检查通过，插件就可用；需要停用时移出目录并重启 Server。 - **检查**：Server 启动时在有时限的子进程中导入 `plugin.py`，校验声明，不调用 `setup`。检查失败只影响该插件。 -## 3. Source 插件 - -### 3.1 要实现的接口 - -| 接口 | 必需 | 说明 | -| --- | --- | --- | -| `definitions` | 是 | `SourceDefinition` 序列，沿用现有的 Source 定义：`input_class`、`resolve` 和各 Projection | -| `target_input` | 采集时必需 | Pydantic 模型，校验调用方添加目标时提交的输入 | -| `identify_target(value) -> TargetIdentity` | 采集时必需 | 纯函数，把输入规范化为稳定的目标键和规范化输入 | -| `setup(ctx) -> SourceCollector` | 采集时必需 | 工厂，在 Worker 中每次调用执行一次，返回实现了 `collect` 的对象 | -| `collect(request) -> CollectResult` | 采集时必需 | 按目标和 checkpoint 读取外部数据，返回 Source 和新的 checkpoint | - -只提供 `definitions` 的插件只扩展 Source 类型：调用方通过现有的 Source 写入接口提交内容，core 调用定义完成校验、 -转换和保存。`resolve`、Projection 和 `identify_target` 都在插件宿主进程中执行；`collect` 在 Worker 中执行。 - -Source 与所有内置 Source 一样只能追加，没有修改和删除；要更正内容，提交一条新的 Source。文件、图片等大块内容由插件 -存进自己的存储，Source 中只记录位置、摘要和媒体类型等元数据。本提案中，这类内容通过 `collect` 从外部系统读入； -`resolve` 必须是纯函数，现有 Source 写入接口也只接受 JSON，调用方直接上传文件的入口由插件自定义接口提供，见 -[RFC #2579](https://github.com/oceanbase/powercontext/pull/2579)。 +## 3. 插件自带的 Source 类型 -### 3.2 声明 - -| 字段 | 默认值 | 说明 | -| --- | --- | --- | -| `definitions` | 必填 | 见上表 | -| `collect` | `None` | 采集配置；为空表示只提供定义 | -| `collect.schedule` | 必填 | `Interval(seconds=...)` 或 `Cron(expression=..., timezone=...)` | -| `collect.max_targets` | 50 | 单次调用携带的目标数上限 | -| `collect.max_items` | 200 | 单次调用返回的 Source 条数上限 | -| `collect.timeout_seconds` | 120 | Worker 总时限 | -| `collect.max_workers` | 1 | 跨 Scope 的并发上限 | -| `target_input`、`identify_target`、`setup` | 采集时必填 | 见上表 | -| `scopes` | `None` | 可服务的 Scope 名单；为空表示所有 Scope | - -### 3.3 示例 - -下面的插件每 5 分钟采集各目标账号在 X 上的新帖子： +Artifact 插件可以在声明的 `definitions` 中提供自己的 Source 类型（见 4.2）。插件用它记录"用户要求做什么"，例如上传了 +一份文档、改写了一个分块，再在 `generate` 中把这些 Source 加工成 Artifact。这样每次变化都有对应的 Source，来源可以追溯， +而不需要另写一个 Source 插件。 ```python -import os - -from pydantic import BaseModel - -from powercontext.plugins import ( - Collect, CollectedItem, CollectRequest, CollectResult, Interval, PluginContext, SourcePlugin, TargetIdentity, -) - -from definition import PostCapture, XPostDefinition - - -class TargetInput(BaseModel): - account: str - - -def identify_target(value: TargetInput) -> TargetIdentity: - account = value.account.strip().removeprefix("@").lower() - if not account: - raise ValueError("account must not be blank") - return TargetIdentity(key=f"account:{account}", input={"account": account}) - - -class XCollector: - def __init__(self, client: XClient) -> None: - self.client = client - - async def collect(self, request: CollectRequest) -> CollectResult: - result = CollectResult() - for target in request.targets: - since = (target.checkpoint or {}).get("since_id") - page = await self.client.user_posts(target.input["account"], since_id=since) - result.add(target, [CollectedItem(definition="x-post", value=PostCapture(**p)) for p in page.posts], - checkpoint={"since_id": page.newest_id or since}) - return result - - -async def setup(ctx: PluginContext) -> XCollector: - client = await ctx.enter_async_context(XClient(os.environ["X_BEARER_TOKEN"])) - return XCollector(client) - - -plugin = SourcePlugin( - definitions=(XPostDefinition(),), - collect=Collect(schedule=Interval(seconds=300), timeout_seconds=120), - target_input=TargetInput, - identify_target=identify_target, +plugin = ArtifactPlugin( + family="knowledge-base", + definitions=(DocumentUploadDefinition(),), # Source 类型 kb-document + inputs=Inputs(source_types=("kb-document",), projection="knowledge-base.upload@1"), + on_sources=True, setup=setup, ) ``` -### 3.4 调用方怎么使用 - -调用方使用现有的 Source 管理接口，不需要知道这是插件提供的能力： - -```text -pc_source_add(scope_id="team-a", capability="x-posts", input={"account": "@PowerContextAI"}) -pc_source_status(scope_id="team-a", target_id="...") -pc_source_target_set_enabled(scope_id="team-a", target_id="...", enabled=false) -``` - -- **添加**：core 用 `target_input` 校验输入，调用 `identify_target` 得到键和规范化输入。同一账号得到同一个键，复用已有 - 目标。新目标保存为启用状态并立即请求一次采集，不需要重启。 -- **checkpoint**：每个目标的 checkpoint 是只读输入，插件通过返回值提出新的 checkpoint。core 把本次返回的所有 Source - 与各目标的新 checkpoint 在同一事务中提交；`collect` 抛出异常或超时则不提交任何内容，下一次调用仍拿到旧 checkpoint。 -- **格式校验**：`pc_source_capabilities` 发布 `target_input` 的 JSON Schema，调用方提交前就知道格式。添加时 core 先用 - `target_input` 校验，不合格返回 422 并指出字段；`identify_target` 抛出 `ValueError` 同样返回 422。外部资源是否存在、 - 是否可读，只能在 `collect` 时判断，结果在 `pc_source_status` 的最近失败中体现。 - -### 3.5 端侧采集 - -具身智能等场景需要在设备上采集，再提交到中心的 PowerContext Server。同一个程序以端侧模式启动： - -```text -powercontext source-agent --server https://memory.example.com -``` - -- 端侧只加载插件目录中带 `collect` 的 Source 插件，运行 Supervisor 调度 `collect`，不加载 Artifact 插件，也不提供 - Artifact 接口。 -- 采集目标保存在端侧本地的 SQLite 中，通过端侧本地的 `pc_source_add`、`pc_source_status`、`pc_source_target_set_enabled` - 管理。中心看不到端侧的目标，只看到提交上来的 Source 和 checkpoint。 -- 采集结果通过 core 已有的远程采集接口提交到中心：Source 走 `POST /v1/source-observations`，checkpoint 走 - `POST /v1/connector-checkpoints/commit`。`resolve` 和 Projection 在端侧执行，中心按已登记的定义清单校验，不需要 - 加载插件代码。 -- 断线、重启和重传都按幂等处理：先提交 Source，再以旧 checkpoint 为期望值提交新 checkpoint。中途失败时 checkpoint 不前进， - 下次调用从旧 checkpoint 重新采集；重新提交的 Source 身份相同、内容相同，由中心去重。 -- 离线期间的数据积压由插件自己处理。提交失败时本次调用失败，checkpoint 不前进；外部系统无法回放的数据（例如传感器流）， - 插件需要先存在自己的本地存储中，按 checkpoint 重新给出。 +- **定义**：沿用现有的 `SourceDefinition`：`input_class` 校验输入，`resolve` 把输入解析为 Source，Projection 给出消费方 + 读取的形态。 +- **写入**：调用方通过现有的 Source 写入接口提交（`POST /v1/scopes/{scope_id}/sources`，`source_type` 为插件定义的类型）， + 插件自己的代码通过 `ctx.sources.submit` 提交。校验、`resolve` 和 Projection 在插件的宿主进程中执行，保存语义与内置 + Source 一致。 +- **只能追加**：与所有内置 Source 一样，没有修改和删除；要更正内容，提交一条新的 Source。 +- **大块内容**：`resolve` 必须是纯函数，现有 Source 写入接口也只接受 JSON。文件、图片由插件自定义接口接收 + （见 [RFC #2579](https://github.com/oceanbase/powercontext/pull/2579)），插件存进自己的存储，Source 中只记录位置、摘要 + 和媒体类型等元数据。 +- **消费**：插件在 `inputs.source_types` 中列出这些类型即可消费；其他 Artifact 插件也可以消费它们。 ## 4. Artifact 插件 @@ -349,7 +239,7 @@ Prepared Context 对该 family 返回 422。 | `transaction_timeout_seconds` | 30 | 写事务块的时限，不超过 `timeout_seconds` | | `max_workers` | 1 | 跨 Scope 的并发上限 | | `change_timeout_seconds` | 30 | 一次用户修改调用的时限，包括其中的写事务块 | -| `definitions` | `()` | 插件自己的 Source 定义，随插件加载登记，规则与 Source 插件的 `definitions` 相同。插件在接口中用 `ctx.sources.submit` 提交，调用方也可以经通用 Source 写入接口提交；这样一个 Artifact 插件就能记录"用户要求做什么"，不需要另写一个 Source 插件 | +| `definitions` | `()` | 插件自带的 Source 类型，随插件加载登记，见第 3 节和参考级说明第 2 节 | | `setup` | 必填 | 见上表 | ### 4.3 示例 @@ -455,9 +345,8 @@ plugin = ArtifactPlugin( | 方法 | 适用 | 必须实现 | 调用时机 | 执行位置 | | --- | --- | --- | --- | --- | -| `setup(ctx)` | 两类插件 | 是 | 宿主进程启动时一次；Worker 每次调用一次 | 宿主进程、Worker | -| Source 定义：`input_class`、`resolve`、Projection 的 `project`、`read` | Source 插件，或声明了 `definitions` 的 Artifact 插件 | 是 | Source 写入（通用写入接口、`ctx.sources.submit`）；把 Source 交付给 `generate` 时计算 Projection | 宿主进程、Worker | -| `identify_target`、`collect` | Source 插件 | 持续采集时是 | 添加采集目标；定时采集 | 宿主进程、Worker | +| `setup(ctx)` | 所有插件 | 是 | 宿主进程启动时一次；Worker 每次调用一次 | 宿主进程、Worker | +| Source 定义：`input_class`、`resolve`、Projection 的 `project`、`read` | 声明了 `definitions` 的插件 | 是 | Source 写入（通用写入接口、`ctx.sources.submit`）；把 Source 交付给 `generate` 时计算 Projection | 宿主进程、Worker | | `generate` | Artifact 插件 | 配置了 `schedule` 或 `on_sources` 时是 | 定时，或新 Source 到达 | Worker | | `render_artifact` | Artifact 插件 | 是 | 默认展示接口 | 宿主进程 | | `search_artifacts` | Artifact 插件 | 否 | 通用检索接口；Prepared Context 中列出该 family | 宿主进程 | @@ -468,7 +357,7 @@ plugin = ArtifactPlugin( ## 5. 调度与触发 -插件的后台调用（`collect`、`generate`）的调度和触发都交给现有的 Artifact processing Supervisor；同步调用由请求触发， +插件的后台调用（`generate`）的调度和触发都交给现有的 Artifact processing Supervisor；同步调用由请求触发， 不经过 Supervisor。每个插件在启动时注册为一个 Supervisor binding， binding 名为 `plugin:`，调度键是 `(binding, Scope)`。 @@ -486,8 +375,6 @@ Artifact 插件： - `Cron` 只能用于配置了 `inputs` 的插件，且不能与 `on_sources` 同时使用：Supervisor 的每个 binding 只接受一种 interval 或 cron 策略。 -Source 插件的 `collect` 按 `collect.schedule` 调用；添加或恢复目标时也会请求一次采集。 - 同一插件、同一 Scope 同时最多运行一个 Worker。运行期间到达的触发由 Supervisor 合并成结束后的一次调用。 ## 6. 运维 @@ -499,8 +386,8 @@ Source 插件的 `collect` 按 `collect.schedule` 调用；添加或恢复目标 触发方式和 Scope 范围。 - `GET /v1/plugins/{name}/scopes`：每个 Scope 是否有待处理的工作、已消费到的 journal 位置和当前 journal 位置。 - 调用失败的原因通过日志和指标排查，不在接口中保存。 -- **变更**：修改代码后重启 Server。Server 不监视文件，也不在运行中重新加载插件。采集目标是业务数据，增减不需要重启。 -- **停用**：从插件目录移走后重启。已保存的 Source、Artifact、采集目标、游标和 checkpoint 都保留，但该插件的 family 和 +- **变更**：修改代码后重启 Server。Server 不监视文件，也不在运行中重新加载插件。 +- **停用**：从插件目录移走后重启。已保存的 Source、Artifact 和游标都保留，但该插件的 family 和 Source 类型不再注册：通用接口读取、检索或修改该 family 时返回 404，与不存在的 family 相同。插件 Artifact 的内容通常只 记录摘要或位置，正文、分块和文件在插件自己的存储中，没有插件就无法解释，因此不提供脱离插件的读取。插件 Source 类型的 新写入返回 422。重新部署同名插件时，读取恢复，生成从原游标继续。插件的专属存储 @@ -511,8 +398,7 @@ Source 插件的 `collect` 按 `collect.schedule` 调用；添加或恢复目标 ## 1. 公共契约 -公开接口位于新模块 `powercontext.plugins`，替换现有的 `powercontext.artifact_plugins` 和 `powercontext.source_management` -中面向插件作者的部分。接口以 `typing.Protocol` 和 Pydantic 模型表达。 +公开接口位于新模块 `powercontext.plugins`，替换现有的 `powercontext.artifact_plugins`。接口以 `typing.Protocol` 和 Pydantic 模型表达。 ### 1.1 声明与加载 @@ -529,16 +415,11 @@ class ArtifactPlugin(BaseModel): # 字段见指南级说明 4.2 inputs: Inputs | None = None ... setup: Callable[[PluginContext], Awaitable[ArtifactGenerator]] - -class SourcePlugin(BaseModel): # 字段见指南级说明 3.2 - definitions: Sequence[SourceDefinition] - collect: Collect | None = None - ... ``` - 插件名取自目录名，必须匹配 `^[a-z][a-z0-9-]{0,63}$`。 - 声明在构造时校验，包括调度组合、`inputs` 与 `scopes` 的依赖、时限关系和 family 命名。`setup` 必须是 async 函数。 -- 声明中除 `setup`、`identify_target`、`definitions` 等代码对象外的部分可以序列化，Server 在检查子进程中得到它， +- 声明中除 `setup`、`definitions` 等代码对象外的部分可以序列化，Server 在检查子进程中得到它， 用于注册 binding 和回答运维查询。 - 检查时，core 记录插件目录的**代码指纹**（按文件内容计算）。Worker 和宿主进程导入插件前核对，不一致则调用失败并提示 重启 Server，同一个 Server 进程中的调用不会混用新旧代码。 @@ -588,82 +469,20 @@ class PluginModels(Protocol): 为 None；只配置了其中一种时，插件用 `embedding_available`、`generation_available` 判断。插件应把 `embedding_dimension` 等 embedding 配置记在自己的索引上，配置变化后重建，避免把不同模型的向量混在一起比较。调用沿用 Server 的超时设置，并计入现有的模型用量统计。插件也可以从环境变量读取自己的 模型配置，自行创建客户端。 -- 插件得不到 core 数据库的连接，也没有游标或 checkpoint 的写入接口。插件对核心数据的写入只有三条路径：Source 插件的 - `collect` 返回值、宿主进程中 `resolve` 的返回值，和 Artifact 插件的 `ArtifactWriter`。 +- 插件得不到 core 数据库的连接，也没有游标的写入接口。插件对核心数据的写入只有两条路径：宿主进程中 `resolve` 的 + 返回值，和 `ArtifactWriter`。 -## 2. Source 插件契约 - -```python -class TargetIdentity(BaseModel): - key: str # 稳定的目标键，同一外部资源始终相同 - input: dict[str, JsonValue] # 规范化后的输入，保留所有影响采集的参数 +## 2. 插件 Source 类型契约 -class CollectTarget(BaseModel): - key: str - input: dict[str, JsonValue] - checkpoint: JsonValue | None # 该目标上次成功提交的 checkpoint；首次为 None；只读 - -class CollectRequest(BaseModel): - scope_id: str - reason: Literal["schedule", "target_added"] - targets: Sequence[CollectTarget] # 本次要采集的已启用目标 - max_items: int - -class CollectedItem(BaseModel): - definition: str # definitions 中某个定义的 name - value: BaseModel # 该定义 input_class 的实例 - -class TargetResult(BaseModel): - items: Sequence[CollectedItem] - checkpoint: JsonValue | None - has_more: bool = False - -class CollectResult(BaseModel): - targets: dict[str, TargetResult] = {} # 键为目标键 - def add(self, target: CollectTarget, items: Sequence[CollectedItem], *, checkpoint: JsonValue | None) -> None: ... - -class SourceCollector(Protocol): - async def collect(self, request: CollectRequest, /) -> CollectResult: ... -``` - -- **定义注册**：`definitions` 中的 Source 类型与 Projection 在启动时注册到现有的 Source 定义清单，与内置类型和其他插件 - 冲突时检查失败。 -- **写入**：调用方通过现有的 Source 写入接口（`POST /v1/scopes/{scope_id}/sources`、MCP `pc_source_add` 的 write 用法） - 提交内容。core 按 `definition.input_class` 校验输入，调用 `resolve` 得到 Source，计算各 Projection 后保存，语义与内置 - Source 写入一致。校验、`resolve` 和 Projection 在该插件的宿主进程中同步执行，有超时限制。`resolve` 必须是纯函数， - 不能有复制文件等副作用；需要把外部文件纳入插件存储时，用 `collect` 完成。 -- **collect**：每次调用对应一个 Scope，最多携带 `max_targets` 个已启用目标；目标较多时按目标键轮转，`has_more` 的目标 - 优先进入下一次调用。执行器对返回的每条 Source 执行与写入相同的校验、`resolve` 和 Projection，然后在一个事务中保存全部 - Source、写入各目标的 checkpoint 和最近采集时间，并确认本次调用。任一条失败则整次不提交。条目总数超过 `max_items`、 - 返回了请求中不存在的目标键，都视为失败，不截断。结果中没有出现的目标，checkpoint 保持不变。 -- **Source 身份**：由 `resolve` 决定。重复采集同一外部状态得到相同身份时，沿用现有去重规则，不产生重复 Source。 - -**采集目标**是 core 保存的业务数据，与 Source 一样不属于插件生命周期状态。管理接口复用现有的 Source 管理接口： - -| 操作 | 接口 | 说明 | -| --- | --- | --- | -| 添加 | MCP / HTTP `pc_source_add`（`capability` 为插件名） | 用 `target_input` 校验输入，调用 `identify_target` 得到键和规范化输入。同键且规范化输入相同则返回已有目标；同键但输入不同返回冲突。新目标保存为启用状态并请求一次采集 | -| 查看 | MCP / HTTP `pc_source_status` | 目标键、规范化输入、启用状态、最近成功采集时间、最近失败分类 | -| 停用 / 恢复 | MCP / HTTP `pc_source_target_set_enabled` | 停用后不再进入 collect 请求，checkpoint 和已采集的 Source 保留；恢复后从原 checkpoint 继续 | - -- 添加和变更需要调用者对该 Scope 有 Source 写权限，并且 Scope 在插件的 `scopes` 范围内。 -- `identify_target` 必须是纯函数：只做校验和规范化，不访问外部系统、不检查外部授权。外部资源是否存在、是否可读， - 在 collect 时由插件判断并以异常或空结果表达。它与写入一样在宿主进程中同步执行，有超时限制。 -- `pc_source_capabilities` 列出插件提供的 Source 类型和目标输入的 JSON Schema。 - -**端侧模式**（`powercontext source-agent`）沿用同一套 Source 插件契约，只替换提交目标： - -| 项 | 端侧模式中的处理 | -| --- | --- | -| 配置 | `--server` 指定中心 Server；`POWERCONTEXT_SOURCE_AGENT_ID` 是端侧的稳定标识；`POWERCONTEXT_SOURCE_AGENT_TOKEN` 是对目标 Scope 有 `scope.contribute` 权限的凭据 | -| 本地数据 | 端侧数据目录下的 SQLite，保存采集目标（结构同 `pc_source_collect_targets`）和 Supervisor 的调度状态；不保存 Source | -| 定义清单 | 启动时把各 Source 定义的声明式清单登记到中心（`POST /v1/source-definitions/register`，需要 `server.admin`）；凭据没有该权限时，由部署者事先登记 | -| 每次 `collect` | 从中心读取各目标的 checkpoint（`/v1/connector-checkpoints/get`），调用插件，在端侧执行 `resolve` 与 Projection，逐条提交 Source（`/v1/source-observations`），全部成功后以旧值为期望提交新 checkpoint（`/v1/connector-checkpoints/commit`） | -| 目标的 binding ID | 由端侧标识、插件名和目标键确定性派生，两个端侧采集同一外部资源不会互相覆盖 checkpoint | -| 失败 | 任一步失败则本次调用失败，进入 Supervisor 退避；已提交的 Source 保留，checkpoint 不前进，下次重采由中心去重 | - -端侧和中心之间不是一个事务：可能出现 Source 已保存而 checkpoint 未前进的情况，靠 Source 身份去重保证不重复。这要求 -插件的 `resolve` 对同一外部状态给出相同的身份和内容；身份相同但内容不同时，中心返回冲突，本次调用失败。 +- **定义注册**：`definitions` 中的 Source 类型与 Projection 在加载时登记到现有的 Source 定义清单 + （`pc_source_definition_manifests`）。与内置类型、旧 Source 插件框架登记的类型或其他插件冲突时，检查失败。 +- **写入**：core 按 `definition.input_class` 校验输入，调用 `resolve` 得到 Source，计算各 Projection 后保存，语义与内置 + Source 写入一致。校验、`resolve` 和 Projection 在该插件的宿主进程中同步执行，使用 Server 的同步调用时限。`resolve` + 必须是纯函数，不能有复制文件等副作用。 +- **Source 身份**：由 `resolve` 决定。重复提交同一内容得到相同身份时，沿用现有去重规则，不产生重复 Source；身份相同但 + 内容不同时返回冲突。 +- **权限**：写入需要调用者对该 Scope 有 Source 写权限，并且 Scope 在插件的 `scopes` 范围内。 +- **发现**：`GET /v1/capabilities` 与 `pc_source_capabilities` 列出已加载插件的 Source 类型。 ## 3. Artifact 插件契约 @@ -681,7 +500,7 @@ class GenerateRequest(BaseModel): class GenerationSource(BaseModel): ref: SourceRef # 精确 Source 引用 - position: int # Source 在 journal 中的位置，用于按采集顺序处理 + position: int # Source 在 journal 中的位置，用于按写入顺序处理 projection: JsonValue # 按该 Source 类型声明的 Projection 内容 ``` @@ -924,9 +743,6 @@ Artifact 插件的每次后台调用处理该 Scope 中游标之后的一个 Sou `on_sources` 的插件在新 Source 写入后约 2 秒内被派发。 - 未配置 `scopes` 时，自动调度对所有 journal 超过游标的 Scope 生效；配置了 `scopes` 时只接受名单内的 Scope。 -collect 的结构相同：输入换成该 Scope 的已启用目标及其 checkpoint，提交内容换成 Source 与各目标的新 checkpoint，由执行器 -根据返回值一次提交。添加或恢复采集目标时请求一次采集。 - ## 6. 插件宿主进程 同步调用需要在请求时限内返回，不能每次都启动 Worker：每个 Worker 都是全新的 Python 进程，导入 core 的执行模块需要数秒。 @@ -935,17 +751,15 @@ collect 的结构相同：输入换成该 Scope 的已启用目标及其 checkpo | 调用 | 执行位置 | | --- | --- | | Source 写入的输入校验、`resolve`、Projection | 宿主进程 | -| 添加采集目标时的 `identify_target` | 宿主进程 | | `create_artifact`、`update_artifact`、`merge_artifacts`、`forget_artifact`、`restore_artifact` | 宿主进程 | | `render_artifact`、`search_artifacts` | 宿主进程 | -| `collect`、`generate` | Supervisor Worker | +| `generate` | Supervisor Worker | -- **数量与启动**：每个 Server 进程为每个加载的插件维护至多一个宿主进程，插件加载后在后台启动，之后复用。Artifact 插件 - 都要实现 `render_artifact`，Source 插件的写入要执行 `resolve`，所以每个加载的插件都有宿主进程。 - 端侧模式中，`resolve` 和 Projection 在采集 Worker 内执行，不启动宿主进程。 +- **数量与启动**：每个 Server 进程为每个加载的插件维护至多一个宿主进程，插件加载后在后台启动，之后复用。插件都要实现 + `render_artifact`，所以每个加载的插件都有宿主进程。 - **执行**：宿主进程启动时导入一次 `plugin.py` 并核对声明，写入器在宿主进程中执行。一个宿主进程可以并发处理多个请求， 并发上限由 Server 配置决定。 -- **超时与故障**：每次调用受各自的时限约束（Source 写入与 `identify_target` 使用 Server 的同步调用时限，用户修改使用 +- **超时与故障**：每次调用受各自的时限约束（Source 写入、展示和检索使用 Server 的同步调用时限，用户修改使用 `change_timeout_seconds`）。超时、崩溃或失去响应时，core 终止该进程，进行中的请求返回 503，并在退避后重新启动；退避期间 该插件的同步调用返回 503，后台调用不受影响。 - **代码变更**：宿主进程在存续期间不重新导入插件，启动时核对代码指纹。修改插件后需要重启 Server，与 Worker 的规则一致。 @@ -1023,7 +837,7 @@ class SearchResult(BaseModel): ## 9. 执行约束 -- **隔离**：`collect` / `generate` 在 Supervisor 启动的 Worker 子进程中执行，同步调用在宿主进程中执行，崩溃、内存泄漏和 +- **隔离**：`generate` 在 Supervisor 启动的 Worker 子进程中执行，同步调用在宿主进程中执行，崩溃、内存泄漏和 阻塞不影响 Server。子进程隔离普通故障，不是不可信代码的沙箱：插件以子进程的操作系统权限运行，可以访问其文件、网络和 环境变量。 - **超时与预算**：`timeout_seconds` 是 Worker 总时限，`transaction_timeout_seconds` 是写事务块时限，`max_workers` 是该插件 @@ -1032,21 +846,18 @@ class SearchResult(BaseModel): 因此写事务块必须短，时限超出即回滚。 - **遗留的调用意图**：插件移除后，它遗留的调用意图不再被处理，也不阻塞数据库就绪检查。 - **失败与退避**：调用失败时不提交任何核心数据，该插件该 Scope 进入 Supervisor 现有的退避，不影响其他 Scope 和插件。 - core 不会因为重复失败自动跳过输入；持续失败需要修复插件或数据后重启。collect 中某个目标持续失败会阻塞同一 Scope - 的其他目标，可以先停用该目标。 -- **多实例部署**：OceanBase 多 Server 实例时，派发只发生在持有 Supervisor 领导租约的实例上；Source 写入、目标管理、 - 读取和用户修改可以在任意实例执行，各实例使用自己的宿主进程。插件自有存储需要能被所有实例访问；core 分配的专属存储 + core 不会因为重复失败自动跳过输入；持续失败需要修复插件或数据后重启。 +- **多实例部署**：OceanBase 多 Server 实例时，派发只发生在持有 Supervisor 领导租约的实例上；Source 写入、读取、 + 展示、检索和用户修改可以在任意实例执行，各实例使用自己的宿主进程。插件自有存储需要能被所有实例访问；core 分配的专属存储 位于共享的数据库后端上，天然满足这一点。SQLite 部署只有一个 Server 进程。 ## 10. 持久化 | 数据 | 处理 | | --- | --- | -| `pc_sources`、Projection、Source journal、`pc_source_definition_manifests` | 保留；Source 插件的结果 | +| `pc_sources`、Projection、Source journal、`pc_source_definition_manifests` | 保留；插件 Source 类型的定义与写入结果 | | `pc_artifacts`、`pc_artifact_heads`、`pc_artifact_lineage_*`、合并记录、handle、tags | 保留；Artifact 插件的结果 | | `pc_source_cursors` | 保留；Artifact 插件的消费游标，键为 `(scope_id, plugin:)` | -| `pc_connector_checkpoints` | 保留；各采集目标的 checkpoint，键为 `(scope_id, 目标的 binding ID)` | -| `pc_source_collect_targets` | 新增；采集目标及其规范化输入、启用状态和最近采集结果 | | Supervisor 的调用意图、pending、租约表 | 保留；插件 binding 与内置 binding 共用，也是 `GET /v1/plugins/{name}/scopes` 的数据来源。插件 binding 以 `plugin:` 开头，不在数据库中登记 | | `pc_artifacts` 的新列 `metadata` | 新增；可为空，按与 `content` 相同的方式保存规范化 JSON 字节，存放不参与检索的附加信息。插件写入的 revision 记录插件名和插件版本摘要，内置 family 为空。不参与内容摘要的计算 | | `pc_artifact_fts` 与 `pc_artifact_heads.searchable_text` | 只用于内置 family；插件 family 不写入 | @@ -1054,15 +865,12 @@ class SearchResult(BaseModel): | `pc_artifact_family_schemas` | 移除 | | 自定义 family 声明式索引使用的搜索状态与向量索引数据 | 移除；实现 PR 确认内置 family 不依赖后删除对应表 | | 旧 Artifact 插件写入的 Artifact、head、lineage | 不迁移；留有数据时拒绝升级，见下文 | -| `pc_source_write_plugins` | 移除；Source 定义改由插件目录在启动时提供 | -| Source 插件管理服务的 SQLite | 随管理服务一起移除 | | 插件专属存储（`plugins/<插件名>.sqlite3` 或 `powercontext_plugin_<插件名>` 库） | 不属于 core 的 schema；core 只负责分配，表结构、迁移和清理由插件和部署者负责 | | 插件自有的其他索引与存储 | 不属于 core，由插件和部署者管理 | -| 端侧模式的本地 SQLite | 只保存端侧的采集目标和调度状态，不保存 Source；Source 与 checkpoint 保存在中心 | 新增与移除通过**一个新的迁移版本**完成，已发布版本的迁移资源保持不变。迁移说明中列出新增、删除的表、列和数据范围。 -迁移检查（preflight）先确认旧插件框架没有留下数据：旧 Artifact 插件 family 的 Artifact、旧插件的登记与调用意图、 -`pc_source_write_plugins` 中的 Source 插件。发现任何一项时拒绝升级，列出插件名、family 和行数，数据库保持不变；部署者 +迁移检查（preflight）先确认旧 Artifact 插件框架没有留下数据：旧插件 family 的 Artifact、旧插件的登记与调用意图。 +发现任何一项时拒绝升级，列出插件名、family 和行数，数据库保持不变；部署者 在旧版本上导出所需内容、卸载旧插件并清理数据后再升级。只有这些表为空时，迁移才删除它们。 这个版本的就绪检查（Server 每次启动时对当前数据库执行）与旧版本有两点不同： @@ -1078,10 +886,9 @@ class SearchResult(BaseModel): | 变更 | 内容 | | --- | --- | -| 移除 | `/v1/admin/artifact-plugins` 及其子路径；Source 插件管理服务、插件实例 Runtime、poll / reply 通道；`source load` 命令；MCP `pc_source_instances`、`pc_source_instance_set_enabled`；`plugin.yml` 与 `powercontext.yml` 清单 | +| 移除 | `/v1/admin/artifact-plugins` 及其子路径；旧 Artifact 插件的 `plugin.yml` 清单 | | 移除 | 自定义 family 的声明式索引（`ArtifactSearchDeclaration`、`TextIndex` 等）、`ArtifactFamily` 的 schema 声明、`ArtifactDraft` 和旧的 Worker 插件执行器 | -| 新增 | `powercontext.plugins`：`SourcePlugin`、`ArtifactPlugin`、`Inputs`、`Interval`、`Cron`、`PluginContext`、`PluginStorage`、`PluginModels`、写入器与读取接口、用户修改的请求模型与 `InvalidContent`、展示与检索的请求和结果模型 | -| 新增 | `powercontext source-agent` 端侧采集模式，以及 `POWERCONTEXT_SOURCE_AGENT_ID`、`POWERCONTEXT_SOURCE_AGENT_TOKEN` | +| 新增 | `powercontext.plugins`：`ArtifactPlugin`、`Inputs`、`Interval`、`Cron`、`PluginContext`、`PluginStorage`、`PluginModels`、写入器与读取接口、用户修改的请求模型与 `InvalidContent`、展示与检索的请求和结果模型 | | 调整 | `POST /v1/scopes/{scope_id}/artifacts` 的请求体增加插件分支 `{"family", "content"}`，ID 由 core 分配，由插件的 `create_artifact` 处理。`CreateArtifactRequest` 改为内置分支 `CreateBuiltinArtifactRequest`（原有的按 family 区分的联合类型）与插件分支 `CreatePluginArtifactRequest` 的二选一，按 family 名选择：内置 family 名走内置分支，校验与原来相同；其他名字走插件分支，没有插件提供该 family 时返回 422。生成的 Python 客户端中，内置请求多包一层；`ArtifactCreated.family` 从内置 family 枚举改为字符串 | | 调整 | `POST /v1/scopes/{scope_id}/sources` 的 `source_type` 从只能是 `content` 的枚举改为字符串（默认仍为 `content`），可以是插件定义的 Source 类型，此时 `content` 是该类型的输入，由插件校验并解析为 Source；未知类型返回 404。`SourceType` 枚举保留 | | 新增 | `GET /v1/plugins`、`GET /v1/plugins/{name}`、`GET /v1/plugins/{name}/scopes`（`server.admin`，只读） | @@ -1092,8 +899,7 @@ class SearchResult(BaseModel): | 新增 | `GET /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}/rendering`：插件 family 的默认展示内容，由插件的 `render_artifact` 提供 | | 调整 | `POST /v1/context/prepare`：`ContextAssemblySection.family` 从枚举改为字符串，可以是已加载的插件 family；新增可选的 `search`（检索请求中除 `query`、`limit` 以外的字段），只对插件 family 生效 | | 调整 | 导出与恢复的 Artifact revision 记录格式版本加一，增加 `metadata` | -| 调整 | `pc_source_add` 的 write 用法使用插件定义，collect 用法添加采集目标；`pc_source_capabilities` 列出插件 Source 类型与目标输入；`pc_source_status` 报告目标及其最近采集结果；`pc_source_target_set_enabled` 作用于 core 保存的目标 | -| 调整 | 环境变量 `POWERCONTEXT_SERVER_PLUGINS_DIR` 取代 `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` 和 Source 插件目录配置 | +| 调整 | 环境变量 `POWERCONTEXT_SERVER_PLUGINS_DIR` 取代 `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` | `openapi/powercontext.yaml` 按上表更新后重新生成 `src/powercontext/http/_generated/` 和各集成的操作表。 @@ -1102,8 +908,8 @@ class SearchResult(BaseModel): 插件调用复用 Supervisor 与 Worker 现有的操作词汇、执行身份和结果字段，不为定时调用生成 request ID。新增的字段标识 插件名、调用的方法、执行位置（Worker 或宿主进程）、触发原因和写入规模；宿主进程的启动、终止和重启各记录一条事件。 -- 指标标签可以使用插件名、方法和执行位置，不包含 Scope ID 或目标键。 -- 日志经 `log_safely` 输出，不包含 Source 内容、Projection、Artifact 内容、目标输入和环境变量值；插件异常只记录类型和 +- 指标标签可以使用插件名、方法和执行位置，不包含 Scope ID。 +- 日志经 `log_safely` 输出，不包含 Source 内容、Projection、Artifact 内容和环境变量值；插件异常只记录类型和 已知错误分类，不记录异常消息。 - 新增字段按开发指南登记到允许列表，并在中英文可观测性文档中说明。 - 调用结果只进入日志和指标，不持久化，运维接口不提供“最近一次调用结果”。 @@ -1122,7 +928,8 @@ class SearchResult(BaseModel): - **后台调用每次都要启动 Worker 并加载插件**：子进程启动和工厂初始化有固定开销，不适合秒级高频的后台调用；插件不能在 Worker 调用之间复用连接或缓存。 - **专属存储只按约定隔离**：插件与 Server 使用同一个数据库登录，技术上能访问 core 的表。 -- **端侧与中心之间不是一个事务**：依赖 Source 身份去重保证不重复，`resolve` 不稳定的插件在端侧模式下会产生冲突。 +- **新旧 Source 扩展方式并存**：在后续提案落地前，添加 Source 类型有两种方式：本提案中插件自带的 Source 类型，以及旧 + Source 插件框架的 write 插件。两者登记到同一份定义清单，名字不能重复。 - **不是安全沙箱**：Worker 隔离普通故障，不限制插件的文件、网络和凭据访问，也不提供硬性的内存或 CPU 配额。 - **依赖必须兼容**：所有插件与 Server 共用一个 Python 环境，依赖冲突需要部署者解决。 - **同一份代码部署多次需要额外处理**：family 写在声明中，同一份代码复制成两个目录会因 family 冲突而检查失败。需要多次 @@ -1156,7 +963,6 @@ class SearchResult(BaseModel): | 插件存储与 core 之间用两阶段提交 | 对象存储、向量库等多数插件存储不支持 XA，core 也要为每种存储实现协调者。按内容摘要先写不可变记录、以 core 的提交为唯一生效点，就能得到需要的一致性 | | 一个插件声明多个 family | 可以在一个事务中写多个 family，但声明、权限和合并注册都要按多 family 设计。需要配合的场景用两个插件加跨 family 只读即可 | | 把 core 的环境变量或数据库配置交给插件 | core 的配置不全来自环境变量；插件拿到 core 的数据库凭据就能绕过写入器。提供封装好的 `ctx.storage` 和 `ctx.models` | -| 端侧单独编译二进制 | 插件是带依赖的 Python 目录，冻结后的二进制无法导入插件的任意依赖，需要为每个插件和平台单独打包。同一个程序以 `source-agent` 模式启动即可 | | 插件专用的触发与管理接口 | 调用方要区分插件与内置能力；现有的 Source 与 Artifact 接口已经覆盖调用方需要的操作 | | 通用合并接口只对插件 family 开放 | 需要在接口里额外维护一份 family 白名单。能操作哪些 family 本来就由注册表决定，没有适配器的 family 无法登记；内置 family 的权限检查在引擎内，两个入口走同一套检查 | | core 按声明的字段为插件 family 建全文索引 | 插件作者不用自建索引，但 core 不知道内容怎样分块、是否适合全文或向量检索，建出的索引常常用不上；还要与内置 family 共用索引表，就绪检查也要区分两者。检索交给插件，core 只负责权限过滤 | @@ -1179,15 +985,12 @@ class SearchResult(BaseModel): - **Dify 插件**：按插件类型定义接口，插件实现对应方法，Dify 在工作流或应用需要时调用；插件运行在独立的插件守护进程中。 本提案沿用“宿主定义接口与调用时机”的思路：同步调用使用 core 管理的常驻宿主进程，后台调用复用 PowerContext 已有的 Supervisor Worker。 -- **Airbyte 连接器协议**：Source 连接器的 `read` 接收上次的 state 并输出记录和新 state，由平台持久化 state。 - 本提案的采集 checkpoint 采用相同分工，并按目标分别保存。 - **pytest 插件与 setuptools entry points**：插件以 Python 对象声明自己提供的能力，宿主导入后读取，不需要单独的清单文件。 - **事务上下文管理器**：SQLAlchemy 的 `async with engine.begin()` 等接口在块开始时开启事务、正常结束时提交、异常时回滚。 `writer.transaction()` 沿用这一使用方式，并在提交时附带游标与调用确认。 - **PowerContext 现有基础**：[Artifact processing Supervisor](1515_artifact_processing_supervisor.md) 的调度、调用意图、 Worker 与租约 fence；[公共合并方法](1859_artifact_merge.md)；通用 Artifact 读写接口与内置 family 编辑时记录的 - `lineage_only` Content Source；`SourceDefinition` / `SourceProjection`、Source journal 与消费游标；connector checkpoint - 与远程采集接口；Source 管理接口与目标识别规则。 + `lineage_only` Content Source；`SourceDefinition` / `SourceProjection`、Source journal 与消费游标。 # 未决问题 @@ -1196,7 +999,6 @@ class SearchResult(BaseModel): | 写事务块的时限与限额默认值 | 默认 30 秒是否合适；SQLite 与 OceanBase 是否采用不同默认值；单块操作数与字节上限 | | 宿主进程的默认参数 | 同步调用时限、单个宿主进程的并发上限和重启退避的默认值 | | 专属存储的建库权限 | Server 的数据库登录没有建库权限时，除了由部署者事先建库，是否允许退回到 core 库中按插件名前缀建表 | -| 采集目标的删除 | 第一版只支持停用和恢复。是否需要彻底删除目标及其 checkpoint，以及删除后已采集 Source 的处理 | | 插件 family 接入 Recall | Prepared Context 已接入，Recall 是否需要接入、候选内容怎样定义 | | 通用检索结果带片段 | 通用检索接口目前只返回精确 revision，插件返回的片段只用于 Prepared Context。是否在检索结果中增加片段和位置 | | family 与 Source 类型的命名 | 是否要求以插件名为前缀，从命名上避免不同插件冒用同一个 family | @@ -1210,6 +1012,8 @@ class SearchResult(BaseModel): # 后续可能 - 热加载与插件自定义 API（含插件页面）：建立在宿主进程之上，见 [RFC #2579](https://github.com/oceanbase/powercontext/pull/2579)。 +- 独立的 Source 插件与采集：只提供 Source 类型的插件；采集目标的集中配置与设备拉取同步；在设备上运行采集并把 Source + 汇报到中心；大文件的传输。由后续提案定义，并取代旧 Source 插件框架。 - 后台调用也改用常驻进程，在保持接口不变的前提下复用连接和初始化结果。 - 更多插件类型，例如为 Recall 提供候选内容、为 Prepared Context 提供文本片段。 - 插件安装与依赖：从 PyPI、Git 地址或 wheel 安装插件，每个插件独立的依赖环境，以及记录各实例加载的插件版本的登记表。 From 169083b754991c3f49d8cb1d379a2de6e7b20be7 Mon Sep 17 00:00:00 2001 From: jasper-yl Date: Thu, 12 Nov 2026 14:55:00 -0800 Subject: [PATCH 10/12] docs(rfc): describe the plugin search request as JSON and the Prepared Context budget as the request's Prepared Context passes the request's byte budget and fills plugin entries in section order, and receipt families become strings so plugin entries count. Co-Authored-By: Claude Opus 5.5 --- docs/en/rfcs/2549-plugin-interfaces.md | 11 ++++++----- docs/zh/rfcs/2549-plugin-interfaces.md | 12 ++++++------ 2 files changed, 12 insertions(+), 11 deletions(-) diff --git a/docs/en/rfcs/2549-plugin-interfaces.md b/docs/en/rfcs/2549-plugin-interfaces.md index 220233230f..a404404476 100644 --- a/docs/en/rfcs/2549-plugin-interfaces.md +++ b/docs/en/rfcs/2549-plugin-interfaces.md @@ -921,8 +921,8 @@ class Rendering(BaseModel): class SearchRequest(BaseModel): scope_id: str - search: SearchArtifactsRequest # the complete request of the generic search API, unchanged - max_bytes: int | None = None # from Prepared Context: the byte budget left for this family + search: dict[str, JsonValue] # the complete request of the generic search API as JSON, unchanged + max_bytes: int | None = None # from Prepared Context: the request's byte budget caller: Caller | None class SearchHit(BaseModel): @@ -945,8 +945,9 @@ class SearchResult(BaseModel): revisions in the plugin's order, and adds the plugin's scores when `include_scores` is set. A plugin therefore cannot return data the caller cannot see through search. - **Prepared Context**: `assembly.sections` may list plugin families. Core calls `search_artifacts` with Prepared Context's - `query`, the section's `limit`, and the remaining byte budget, and uses the matched passages as context entries that, - like built-in entries, cite exact revisions and count toward the byte budget and the receipt. A section may carry an + `query`, the section's `limit`, and the request's byte budget, and fills the matched passages in section order as + context entries that, like built-in entries, cite exact revisions and count toward the byte budget and the receipt; + entries beyond the budget are truncated or left out. A section may carry an optional `search` with the fields of a search request other than `query` and `limit`, passed on to the plugin unchanged; built-in families that carry `search` return 422. - Review and Recall are not open to plugin families; those APIs return an explicit "not supported" error, and @@ -1049,7 +1050,7 @@ new format. | Changed | The tag APIs accept plugin families: `TaggableArtifactFamily` changes from an enum to a string in the `ArtifactReadFamily` format, and Server accepts built-in families or loaded plugin families. The wire format is unchanged; generated clients lose the enum type | | Changed | The generic Artifact read APIs support plugin families; for plugin families, the generic search API passes the complete request to the plugin's `search_artifacts` and returns 422 when it is not implemented; `GET /v1/capabilities` lists plugin families | | Added | `GET /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}/rendering`: the default presentation of a plugin family Artifact, provided by the plugin's `render_artifact` | -| Changed | `POST /v1/context/prepare`: `ContextAssemblySection.family` changes from an enum to a string that may name a loaded plugin family; an optional `search` (the fields of a search request other than `query` and `limit`) applies to plugin families only | +| Changed | `POST /v1/context/prepare`: `ContextAssemblySection.family` changes from an enum to a string that may name a loaded plugin family; an optional `search` (the fields of a search request other than `query` and `limit`) applies to plugin families only; `PreparedContextFamily` in the receipt changes from an enum to a string that may name a plugin family | | Changed | The format version of Artifact revision records in export and restore increases by one and adds `metadata` | | Changed | The environment variable `POWERCONTEXT_SERVER_PLUGINS_DIR` replaces `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` | diff --git a/docs/zh/rfcs/2549-plugin-interfaces.md b/docs/zh/rfcs/2549-plugin-interfaces.md index 7cc54249e8..761e2aa364 100644 --- a/docs/zh/rfcs/2549-plugin-interfaces.md +++ b/docs/zh/rfcs/2549-plugin-interfaces.md @@ -793,8 +793,8 @@ class Rendering(BaseModel): class SearchRequest(BaseModel): scope_id: str - search: SearchArtifactsRequest # 通用检索接口的完整请求，原样转交 - max_bytes: int | None = None # 来自 Prepared Context 时，本 family 可用的字节预算 + search: dict[str, JsonValue] # 通用检索接口的完整请求（JSON），原样转交 + max_bytes: int | None = None # 来自 Prepared Context 时，请求的字节预算 caller: Caller | None class SearchHit(BaseModel): @@ -814,9 +814,9 @@ class SearchResult(BaseModel): `admission`、`fusion`、`rerank`、`min_score` 等）原样交给 `search_artifacts`，由插件按自己的索引解释；插件不支持的参数 抛出 `InvalidContent`，接口返回 422。core 只保留命中中调用者有权读取的 active 当前 revision，按插件给出的顺序返回 精确 revision，`include_scores` 时附带插件给出的分数。插件因此不能借检索返回调用者看不到的数据。 -- **Prepared Context**：`assembly.sections` 可以列出插件 family。core 以 Prepared Context 的 `query` 和该项的 `limit`、 - 剩余字节预算调用 `search_artifacts`，把命中的片段作为上下文条目，与内置 family 一样带精确 revision 引用、计入字节预算 - 和回执。每项可以携带可选的 `search`，内容是检索请求中除 `query`、`limit` 以外的字段，原样转交插件；内置 family 携带 +- **Prepared Context**：`assembly.sections` 可以列出插件 family。core 以 Prepared Context 的 `query`、该项的 `limit` 和 + 请求的字节预算调用 `search_artifacts`，把命中的片段作为上下文条目按 section 顺序装填，与内置 family 一样带精确 revision + 引用、计入字节预算和回执，超出预算的条目被截断或省略。每项可以携带可选的 `search`，内容是检索请求中除 `query`、`limit` 以外的字段，原样转交插件；内置 family 携带 `search` 时返回 422。 - Review 和 Recall 不对插件 family 开放，对应接口返回明确的“不支持”错误；`GET /v1/capabilities` 列出插件 family。 @@ -897,7 +897,7 @@ class SearchResult(BaseModel): | 调整 | tags 接口接受插件 family：`TaggableArtifactFamily` 从枚举改为与 `ArtifactReadFamily` 相同格式的字符串，服务端校验为内置 family 或已加载的插件 family。网络上的数据格式不变，生成的客户端少了这个枚举类型 | | 调整 | 通用 Artifact 读取接口支持插件 family；通用检索接口对插件 family 把完整请求交给插件的 `search_artifacts`，未实现时返回 422；`GET /v1/capabilities` 列出插件 family | | 新增 | `GET /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}/rendering`：插件 family 的默认展示内容，由插件的 `render_artifact` 提供 | -| 调整 | `POST /v1/context/prepare`：`ContextAssemblySection.family` 从枚举改为字符串，可以是已加载的插件 family；新增可选的 `search`（检索请求中除 `query`、`limit` 以外的字段），只对插件 family 生效 | +| 调整 | `POST /v1/context/prepare`：`ContextAssemblySection.family` 从枚举改为字符串，可以是已加载的插件 family；新增可选的 `search`（检索请求中除 `query`、`limit` 以外的字段），只对插件 family 生效；回执中的 `PreparedContextFamily` 从枚举改为字符串，可以是插件 family | | 调整 | 导出与恢复的 Artifact revision 记录格式版本加一，增加 `metadata` | | 调整 | 环境变量 `POWERCONTEXT_SERVER_PLUGINS_DIR` 取代 `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` | From 0bd06ac7986a8dc54d462432c11c30d5feef79ec Mon Sep 17 00:00:00 2001 From: jasper-yl Date: Thu, 12 Nov 2026 15:12:54 -0800 Subject: [PATCH 11/12] docs(rfc): leave include_scores to core when passing a search to the plugin Co-Authored-By: Claude Opus 5.5 --- docs/en/rfcs/2549-plugin-interfaces.md | 3 ++- docs/zh/rfcs/2549-plugin-interfaces.md | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/en/rfcs/2549-plugin-interfaces.md b/docs/en/rfcs/2549-plugin-interfaces.md index a404404476..1cc5c9d172 100644 --- a/docs/en/rfcs/2549-plugin-interfaces.md +++ b/docs/en/rfcs/2549-plugin-interfaces.md @@ -939,7 +939,8 @@ class SearchResult(BaseModel): read APIs still return the original `content` for programs. A revision's presentation does not change under the same plugin version, so core may cache it by family, ID, revision, and plugin version. Built-in families do not offer this API. - **Search**: the generic search API passes every field of `SearchArtifactsRequest` (`query`, `query_vector`, `indexes`, - `mode`, `filters`, `admission`, `fusion`, `rerank`, `min_score`, and so on) unchanged to `search_artifacts`, which + `mode`, `filters`, `admission`, `fusion`, `rerank`, `min_score`, and so on, except `include_scores`, which only decides + whether core returns scores) unchanged to `search_artifacts`, which interprets them against the plugin's own indexes; a plugin raises `InvalidContent` for controls it does not support, and the API returns 422. Core keeps only matches that are active current revisions the caller may read, returns the exact revisions in the plugin's order, and adds the plugin's scores when `include_scores` is set. A plugin therefore cannot diff --git a/docs/zh/rfcs/2549-plugin-interfaces.md b/docs/zh/rfcs/2549-plugin-interfaces.md index 761e2aa364..90623745c5 100644 --- a/docs/zh/rfcs/2549-plugin-interfaces.md +++ b/docs/zh/rfcs/2549-plugin-interfaces.md @@ -811,7 +811,8 @@ class SearchResult(BaseModel): 接口返回，供程序使用。同一 revision 在同一插件版本下的展示结果不变，core 可以按 family、ID、revision 和插件版本缓存。 内置 family 不提供这个接口。 - **检索**：通用检索接口把 `SearchArtifactsRequest` 的全部字段（`query`、`query_vector`、`indexes`、`mode`、`filters`、 - `admission`、`fusion`、`rerank`、`min_score` 等）原样交给 `search_artifacts`，由插件按自己的索引解释；插件不支持的参数 + `admission`、`fusion`、`rerank`、`min_score` 等，`include_scores` 除外，它只决定 core 是否返回分数）原样交给 + `search_artifacts`，由插件按自己的索引解释；插件不支持的参数 抛出 `InvalidContent`，接口返回 422。core 只保留命中中调用者有权读取的 active 当前 revision，按插件给出的顺序返回 精确 revision，`include_scores` 时附带插件给出的分数。插件因此不能借检索返回调用者看不到的数据。 - **Prepared Context**：`assembly.sections` 可以列出插件 family。core 以 Prepared Context 的 `query`、该项的 `limit` 和 From 1624dafdafc2585aa6346edc25d4dad63a0655b0 Mon Sep 17 00:00:00 2001 From: jasper-yl Date: Thu, 12 Nov 2026 16:47:49 -0800 Subject: [PATCH 12/12] docs(rfc): call the receipt change by its effect, not a schema name Co-Authored-By: Claude Opus 5.5 --- docs/en/rfcs/2549-plugin-interfaces.md | 2 +- docs/zh/rfcs/2549-plugin-interfaces.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/rfcs/2549-plugin-interfaces.md b/docs/en/rfcs/2549-plugin-interfaces.md index 1cc5c9d172..fd35f6ccbe 100644 --- a/docs/en/rfcs/2549-plugin-interfaces.md +++ b/docs/en/rfcs/2549-plugin-interfaces.md @@ -1051,7 +1051,7 @@ new format. | Changed | The tag APIs accept plugin families: `TaggableArtifactFamily` changes from an enum to a string in the `ArtifactReadFamily` format, and Server accepts built-in families or loaded plugin families. The wire format is unchanged; generated clients lose the enum type | | Changed | The generic Artifact read APIs support plugin families; for plugin families, the generic search API passes the complete request to the plugin's `search_artifacts` and returns 422 when it is not implemented; `GET /v1/capabilities` lists plugin families | | Added | `GET /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}/rendering`: the default presentation of a plugin family Artifact, provided by the plugin's `render_artifact` | -| Changed | `POST /v1/context/prepare`: `ContextAssemblySection.family` changes from an enum to a string that may name a loaded plugin family; an optional `search` (the fields of a search request other than `query` and `limit`) applies to plugin families only; `PreparedContextFamily` in the receipt changes from an enum to a string that may name a plugin family | +| Changed | `POST /v1/context/prepare`: `ContextAssemblySection.family` changes from an enum to a string that may name a loaded plugin family; an optional `search` (the fields of a search request other than `query` and `limit`) applies to plugin families only; the family of receipt entries changes from an enum to a string that may name a plugin family | | Changed | The format version of Artifact revision records in export and restore increases by one and adds `metadata` | | Changed | The environment variable `POWERCONTEXT_SERVER_PLUGINS_DIR` replaces `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` | diff --git a/docs/zh/rfcs/2549-plugin-interfaces.md b/docs/zh/rfcs/2549-plugin-interfaces.md index 90623745c5..f9a6170c79 100644 --- a/docs/zh/rfcs/2549-plugin-interfaces.md +++ b/docs/zh/rfcs/2549-plugin-interfaces.md @@ -898,7 +898,7 @@ class SearchResult(BaseModel): | 调整 | tags 接口接受插件 family：`TaggableArtifactFamily` 从枚举改为与 `ArtifactReadFamily` 相同格式的字符串，服务端校验为内置 family 或已加载的插件 family。网络上的数据格式不变，生成的客户端少了这个枚举类型 | | 调整 | 通用 Artifact 读取接口支持插件 family；通用检索接口对插件 family 把完整请求交给插件的 `search_artifacts`，未实现时返回 422；`GET /v1/capabilities` 列出插件 family | | 新增 | `GET /v1/scopes/{scope_id}/artifacts/{family}/{artifact_id}/rendering`：插件 family 的默认展示内容，由插件的 `render_artifact` 提供 | -| 调整 | `POST /v1/context/prepare`：`ContextAssemblySection.family` 从枚举改为字符串，可以是已加载的插件 family；新增可选的 `search`（检索请求中除 `query`、`limit` 以外的字段），只对插件 family 生效；回执中的 `PreparedContextFamily` 从枚举改为字符串，可以是插件 family | +| 调整 | `POST /v1/context/prepare`：`ContextAssemblySection.family` 从枚举改为字符串，可以是已加载的插件 family；新增可选的 `search`（检索请求中除 `query`、`limit` 以外的字段），只对插件 family 生效；回执中条目的 family 从枚举改为字符串，可以是插件 family | | 调整 | 导出与恢复的 Artifact revision 记录格式版本加一，增加 `metadata` | | 调整 | 环境变量 `POWERCONTEXT_SERVER_PLUGINS_DIR` 取代 `POWERCONTEXT_SERVER_ARTIFACT_PLUGINS_DIR` |