Skip to content

docs(rfc): propose Source plugins with central collection targets pulled by runners - #2617

Open
jasper-yl wants to merge 4 commits into
oceanbase:masterfrom
jasper-yl:rfc/source-plugins
Open

jasper-yl wants to merge 4 commits into
oceanbase:masterfrom
jasper-yl:rfc/source-plugins

Conversation

@jasper-yl

@jasper-yl jasper-yl commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Which issue or RFC does this PR close?

This PR proposes a follow-up RFC to RFC #2549 (plugin interfaces), which leaves standalone Source plugins and collection to a separate proposal. It replaces the contracts and management mechanisms of the Source plugin framework RFC #2012; the implementation of RFC #2549 is in #2616. It builds on the plugin host processes of RFC #2549 and the plugin routes of RFC #2579.

Rationale for this change

The old Source plugin framework runs a separate management service with its own SQLite, one Runtime process per plugin instance, and poll / reply reverse channels between the service and Server. Embodied and edge deployments still need what it offers: operators configure collection for a fleet of devices from the central Server, and devices behind NAT only connect out. This proposal keeps that capability with less machinery: targets live in two central tables, and runners pull them.

What changes are included in this PR?

New RFC docs/{en,zh}/rfcs/2617-source-plugins.md (Chinese first, English translation in sync):

  • Source plugins: a SourcePlugin declaration with definitions, and for collection collect, a TargetSpec (parameter model plus a declarative identity expression), and setup. The plugin name is the capability name.
  • Runners: the Server's built-in server runner and powercontext source-agent on devices, named after the credential's subject. Runners sync on a schedule: they report their plugins with parameter schemas and identity expressions, and fetch every target assigned to them across Scopes. The Server never connects to devices.
  • Central targets: pc_source_collect_targets (Scope, runner, capability, normalized key, parameters, enabled, and runner-written status columns) and pc_source_runners (capabilities and last_seen_at). Adding a target validates its parameters and deduplicates by the identity expression immediately. User writes and runner writes touch separate columns.
  • Results: Sources go through POST /v1/source-observations, then the checkpoint through POST /v1/connector-checkpoints/commit against its old value, carrying the target's status. Failures repeat the same Sources, which the Server deduplicates.
  • Large content: inline, the plugin's object storage, or the same Source plugin on the central Server receiving files through plugin routes and reading them in its Projections; devices call their central routes through ctx.center.
  • Migration and mapping: one new migration revision adds the two tables and drops pc_source_write_plugins only when it is empty; the management service, instance registration, reverse channels, and source load are removed, with a table mapping each old mechanism to its replacement.

A design demo on the author's fork (https://github.com/jasper-yl/powercontext/tree/feat/source-plugins-demo, not for merging) runs a simulated robot camera through a real Server: a device runner reports its plugin and parameter schema on sync; targets added centrally are validated and deduplicated by the identity expression (same parameters return the existing target, a normalized duplicate with other parameters conflicts, invalid parameters and unknown runners return 422); the runner collects, submits Sources, and advances the checkpoint; a lost connection before the checkpoint commit leaves it unchanged and the retry repeats the same Sources without duplicates; status and failures are reported per target; disabled targets leave the sync; a target in a new Scope is collected on the next sync without a restart. The demo keeps its tables in a separate SQLite file and does not cover the Server runner, large files through a central plugin, the migration, or the Dashboard form.

Are there any user-facing changes?

This PR changes documentation only. The proposed design replaces the /v1/source-management/* instance, configuration, channel, and write-plugin APIs with runner and target APIs, keeps the MCP tools pc_source_add, pc_source_status, and pc_source_target_set_enabled, and adds pc_source_runners, pc_source_target_delete, and powercontext source-agent.

How was this change tested?

  • uv run prek run --files docs/en/rfcs/2617-source-plugins.md docs/zh/rfcs/2617-source-plugins.md
  • Checked that both locales have matching sections, tables, and code blocks.
  • Ran the design demo's acceptance scenario tests/e2e/test_source_collection_demo.py on the fork branch against a real Server process.

AI usage statement

Drafted with Claude Code (Claude Opus 5.5) under the author's direction and review; the design decisions were made in discussion with the author.

🤖 Generated with Claude Code

…led by runners

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jasper-yl added a commit to jasper-yl/powercontext that referenced this pull request Nov 13, 2026
RFC oceanbase#2549 renamed the factory to setup and requires render_artifact; standalone
Source plugins moved to RFC oceanbase#2617, so the examples declare routes and pages on
an Artifact plugin that brings its own Source type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@Zxf-xufeng Zxf-xufeng left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design review at d00cbb05. This PR contains an RFC, so the findings below concern the proposed contract rather than a merged collector implementation. I have not run the fork demo.

[P1] Fence collection commits against target configuration changes and deletion

The RFC allows target parameters to change, but remote result submission only compares the previous checkpoint. That checkpoint need not change when the parameters change. A runner can therefore collect using the old parameters and subsequently advance the checkpoint after the target has been edited, causing the new configuration to resume at an incompatible position and potentially skip data.

Deletion has a related gap. The existing checkpoint repository permits insertion when the checkpoint is missing and the expected value is None. A delayed first collection can therefore recreate a checkpoint after its target has been deleted unless the target is checked as part of the commit.

Please specify that a checkpoint/status commit validates the target's existence, runner ownership, enabled state, and configuration revision or digest in the same transaction as the update. Define what happens to the checkpoint when parameters change the collection identity: reject the in-place change and require a new target, or explicitly reset it with stale-attempt protection. Also state the allowed behavior for Sources already accepted by the separate observations request; checkpoint validation alone cannot undo those writes.

Acceptance should cover parameter edits, disable/delete, and delayed commits, including a first attempt with an expected None checkpoint. This can use the proposed target table rather than introducing another table.

[P2] Persist the authenticated owner of a custom runner name

The runner section permits --name, including multiple devices sharing a credential. The API section defines ownership as either a matching authenticated subject or the subject recorded when the runner was registered. However, the proposed pc_source_runners schema contains no owner-principal field or equivalent binding.

Please define and persist the authenticated owner for custom runner names, including name uniqueness/conflicts and the rule that an ordinary sync cannot replace that owner. Otherwise the specified authorization cannot reliably distinguish the owner from another principal attempting to sync under the same name and overwrite capabilities or retrieve assigned targets. Alternatively, restrict the initial implementation to names derived from the authenticated principal.

Acceptance should verify that another principal cannot claim or sync an existing custom runner name, while the original owner can reconnect.

Reference: RFC: runners, result submission, persistence, and API authorization.

…rsion

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed a7c1224b (RFC only). The two contract gaps in the existing review remain: checkpoint/status commits are not fenced against target edits or deletion, and custom runner names have no persisted authenticated owner. The version-compatibility update does not address either.

I checked the reused checkpoint store with SQLite: removing a checkpoint and then submitting a delayed first commit with expected=None recreates it. Please specify transactional target/configuration validation and durable runner ownership in both locales before accepting the design. This verifies the existing persistence behavior; I have not run the fork demo or a proposed runner implementation.

…per-target collection failures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| `POST /v1/source-observations` | Existing | Requests from a runner gain the plugin name and version, checked against the compatible range |
| `POST /v1/connector-checkpoints/commit` | Existing | The request gains an optional `status`: latest attempt, success, error category, and message; requests from a runner gain the plugin name and version, checked against the compatible range |

"The runner itself" means the authenticated subject equals the runner name, or the subject recorded when the runner first

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Authorize registered runners only through their bound subject

If subject fleet-service first registers runner robot-7 using --name, this OR rule also authorizes the distinct subject robot-7 to retrieve its targets and checkpoints, violating section 3's first-sync ownership binding. Require the persisted bound subject for subsequent syncs, document that binding in runner persistence, and correct both locales.

hidb4ai pushed a commit that referenced this pull request Nov 14, 2026
* docs(rfc): propose plugin hot reload and plugin-defined APIs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rfc): number plugin hot reload RFC as 2579

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rfc): import plugins from version snapshots and mount routes as a group

Pin every call to a read-only snapshot named by the content digest so a
draining host never mixes versions and a failed update keeps the old one,
detect changes by content instead of size and mtime, mount plugin routes
under their own group with per-route Scope authorization, let plugins
create Scopes with core's existing permission, and state page isolation
as session isolation, stopping forwarding once the iframe navigates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rfc): keep the plugin reload RFC at the contract level

Move scan, host replacement, and hosting page flows and the literal CSP header to the implementation design.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rfc): let plugin routes opt in as MCP tools

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rfc): let plugins upgrade their own data through a load hook

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rfc): confirm write access in handlers, declare route body limits, and record API changes as Sources

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rfc): name the factory setup and show routes on an Artifact plugin

RFC #2549 renamed the factory to setup and requires render_artifact; standalone
Source plugins moved to RFC #2617, so the examples declare routes and pages on
an Artifact plugin that brings its own Source type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(rfc): run upgrade on every role, keep it small, and name MCP tools without collisions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
jasper-yl added a commit to jasper-yl/powercontext that referenced this pull request Nov 14, 2026
…r and device runners

Implements RFC oceanbase#2617 and replaces the previous Source plugin framework.

- `SourcePlugin` (version, min_compatible_version, definitions, collect, target, setup,
  api, pages, upgrade) loads from the plugin directory alongside Artifact plugins. A
  plugin with only definitions extends the Source write API; one with collect runs
  through runners.
- Revision v1_5_0_006 adds pc_source_runners and pc_source_collect_targets and drops
  pc_source_write_plugins, refusing the upgrade while it still lists plugins. Portable
  bundles carry runners (name and bound subject) and targets without runner status.
- Target API and MCP tools: pc_source_runners, pc_source_add, pc_source_status,
  pc_source_target_set_enabled, pc_source_target_delete; runners sync through
  POST /v1/source-runners/{runner}/sync. Targets of the server runner need
  server.admin; identity parameters are declared, normalized, and immutable.
- The server runner collects in Supervisor Workers and commits Sources, checkpoints,
  and target status in one transaction; CollectResult.fail reports one target.
- `powercontext source-agent` collects on devices: it submits Sources and then
  checkpoints with status. Runner requests carry the plugin version, accepted within
  the Server plugin's compatible range; the Server's plugin resolves and projects them.
- Removed: /v1/source-management/*, `powercontext source load`, the source management
  settings, and the old examples, which are rewritten as Source plugins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jasper-yl added a commit to jasper-yl/powercontext that referenced this pull request Nov 14, 2026
…r and device runners

Implements RFC oceanbase#2617 and replaces the previous Source plugin framework.

- `SourcePlugin` (version, min_compatible_version, definitions, collect, target, setup,
  api, pages, upgrade) loads from the plugin directory alongside Artifact plugins. A
  plugin with only definitions extends the Source write API; one with collect runs
  through runners.
- Revision v1_5_0_006 adds pc_source_runners and pc_source_collect_targets and drops
  pc_source_write_plugins, refusing the upgrade while it still lists plugins. Portable
  bundles carry runners (name and bound subject) and targets without runner status.
- Target API and MCP tools: pc_source_runners, pc_source_add, pc_source_status,
  pc_source_target_set_enabled, pc_source_target_delete; runners sync through
  POST /v1/source-runners/{runner}/sync. Targets of the server runner need
  server.admin; identity parameters are declared, normalized, and immutable.
- The server runner collects in Supervisor Workers and commits Sources, checkpoints,
  and target status in one transaction; CollectResult.fail reports one target.
- `powercontext source-agent` collects on devices: it submits Sources and then
  checkpoints with status. Runner requests carry the plugin version, accepted within
  the Server plugin's compatible range; the Server's plugin resolves and projects them.
- Removed: /v1/source-management/*, `powercontext source load`, the source management
  settings, and the old examples, which are rewritten as Source plugins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants