[jasondeng1997]

Which issue or RFC does this PR close?

Tracks RFC 1430 — distributed Server workers /
#1430. This is the runtime Work Ledger rebased onto current
master; it is deliberately narrowed to the ledger, its roles, and the transport that makes durable work observable.

Rationale for this change

Background processing had two problems. Every process that wanted to run Memory, Profile or Experience work needed its
own APScheduler sidecar and SQLite state, so two Servers could not coordinate; and a caller had no way to see what a
bounded Source window was doing, stop it, or recover it after a failure. The Work Ledger makes one logical Source window
a database row with a fenced lease, so any process can observe it and exactly one process can act on it at a time.

What changes are included in this PR?

Persistence. builtin/persistence/work.py adds the work item, lane and logical-key tables with an optimistic
state_version, database-time leases, fences, and the operations that move a window through queued → running → retry_wait → succeeded/failed/cancelled. coordination.py adds leader election and membership heartbeats.
migrations/server_resources/schemas/mysql-v1_4_1_006.json carries the schema (v1_4_1_006 avoids the number already
taken by #2192).

Runtime. DurableScheduler scans keyset-paginated discovery pages as a fenced leader; DurableWorker claims with
capacity, renews leases and recovers abandoned attempts; work_handlers.py implements the Memory, Profile and Experience
source-window handlers and the maintenance handler; composition.py assembles components by deployment role
(single_node keeps the APScheduler supervisors, distributed splits api / scheduler / worker / all), with each
Family owned by exactly one path. operations.py exposes the OperationManager facade.

Server. Deployment-role validation (including the third BuiltinConfig construction site in cli/config.py), the
scoped durable operation endpoints described below, a shared rate-limit middleware, and worker observability (OTLP spans
and model usage per attempt).

Docs. RFC 1430 (en/zh), deployment, configuration and HTTP API pages; the operation endpoints are documented with an
explicit statement of what is still proposal material.

Are there any user-facing changes?

Yes, three groups:

1. New endpoints — durable operations are served under the owning Scope, so authorization resolves from the request
   and another Scope's row cannot be probed:

   • GET /v1/scopes/{scope_id}/operations — page public operations, filter by kind / status (scope.read)
   • GET /v1/scopes/{scope_id}/operations/{operation_id} (scope.read)
   • POST /v1/scopes/{scope_id}/operations/{operation_id}/cancel — expected_version required (scope.contribute)
   • POST /v1/scopes/{scope_id}/operations/{operation_id}/retry — failed operations only (scope.contribute)

   A stale expected_version returns 409 operation_conflict and leaves the row untouched. Another Scope's operation,
   and internal maintenance work, return 404.

2. New configuration sections — deployment, coordination, worker, operations, rate_limit. A
   distributed deployment requires OceanBase for coordination and a generation_model on worker; host-local
   external Skill targets are rejected because they cannot be reproduced on another replica.

3. Behavior — in single_node nothing changes: the APScheduler supervisors still own processing and the ledger is
   not opened. In distributed the ledger owns Memory, Profile and Experience windows. POST /v1/memory/flush is still
   synchronous (200); the 202 / Prefer: respond-async handle described in RFC 1430 is not implemented, and the
   docs now say so.

How was this change tested?

• make contract-test → 56 passed (contract, generated Python transport, JS operations, Desktop digest consistent)
• tests/test_operation_routes.py (new) → 2 passed: paging and filters, cross-Scope 404, maintenance 404,
  stale-version 409, cancel → cancelled, cancelled-is-not-retryable, and a genuinely failed window retrying to
  queued with cleared error fields
• tests/test_server.py + tests/e2e/test_operation_api.py + docs contract tests → 70 passed
• tests/builtin/runtime + tests/builtin/persistence + server metric/tracing/health suites → 1055 passed / 1 skipped
• tests/test_config_cli.py → 45 passed (role validation)
• tests/e2e/test_zcode_service_chain.py → passed repeatedly; the boundary-flush assertion now re-issues the bounded
  Stop flush instead of racing the hook's 800 ms budget
• DSH plugin: pnpm test 428 passed, test:e2e 9 passed; the committed bundle is rebuilt from the regenerated catalog

AI usage statement

Implementation, tests and documentation were drafted with an AI coding assistant (WorkBuddy / CodeBuddy) under author
review; every command in the list above was run by the author on this head, and the contract, generated transport and
Desktop digest were regenerated and re-verified rather than hand-edited.

Rebase rounds after this description was written

The branch was rebased onto master 2291237a as 7d716b64, which brought in the 1.5.1 contract. The Desktop artifacts
and the protocol qualification were regenerated from that merge (253ee2ad, desktop/VALIDATION.md), not carried over.
Every review finding since then has its own commit and its own regression test; the commit-by-commit table, and the CI
attribution for the checks that are red on master itself, are in the newest review comment.

Verification that this host can run at 7db9ebeb: ruff check and ruff format --check over e2e/bub (44 files), the
generated ACP pin command executed end to end against a stub interpreter (shell quoting and probe compilation), and
git diff --check. The suites, the contract gates and the Docker-based acceptance run in CI: this host has no Docker
runtime, and local pytest is not usable in this environment.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[hireview001]

• CI failed: Upgrade release gate (failure).
• CI failed: upgrade-acceptance / Prepare upgrade artifacts and matrix (failure).
• CI failed: Atomic Memory clean migration (SeekDB) (failure).
• CI failed: tests (3.12, e2e) (failure).
• CI failed: tests (3.14, e2e) (failure).
• CI failed: tests (3.11, unit) (failure).
• CI failed: tests (3.13, unit) (failure).
• CI failed: tests (3.13, e2e) (failure).
• CI failed: tests (3.14, unit) (failure).
• CI failed: tests (3.12, unit) (failure).
• Additional CI issues: PR checks.

[hireview001]

[P1] Wire deployment roles into server and runtime composition

create_server_app reconstructs BuiltinConfig without the new deployment settings, and open_builtin_runtime never starts DurableScheduler, DurableWorker, or RuntimeMembership, so distributed roles retain the existing process-local execution behavior and workers expose data-plane routes. Propagate the configuration and compose the promised role-specific services and routes.

[jasondeng1997]

Fixed in b311e14a. The wiring now exists end to end:

• ServerSettings accepts deployment, coordination, worker, operations and rate_limit (src/powercontext/server/settings.py:238-242). It previously had none of them, so no environment variable could reach the ledger.
• create_server_app passes all five into BuiltinConfig (server/factory.py:169-182), and so do the background runner and the offline migration command (server/cli.py).
• open_builtin_runtime opens the ledger (composition.py:900), implemented by _open_work_background (composition.py:1241) with role gating:

scheduler = DurableScheduler(...) if role in ("all", "scheduler") else None
worker = DurableWorker(...) if role in ("all", "worker") else None

api therefore never scans or claims, scheduler never claims, and worker never scans. RuntimeMembership starts for every role that runs something, and shutdown stops the worker, then the scheduler, then membership, so an in-flight claim still commits under a valid lease.

One deliberate scope decision: a single_node/all deployment starts the ledger only when a background schedule is configured (_work_background_requested, composition.py:1145). An unconfigured install keeps exactly today's behaviour instead of acquiring an ownership lease and polling an empty queue. OperationManager is exposed as runtime.operations in every role, because reading, cancelling and retrying history is not role-specific.

Covered by tests/builtin/runtime/test_work_composition.py (role gating, discoverer and handler registration, a second runtime rejected with DuplicateSingleNodeError, clean shutdown).

[hidb4ai]

Rechecked at 886a0cfb: configuration propagation and ledger component construction are present, but the route-isolation part of this finding is still reproducible. create_server_app() with deployment.mode="distributed" and either role="scheduler" or role="worker" still registers /v1/memory/flush and /v1/scopes/{scope_id}/operations, even with Dashboard and MCP disabled. The factory unconditionally builds the full data-plane app. This conflicts with the health/metrics-only role boundary in RFC 1430 and leaves a worker's provider-backed business routes reachable on its listener. Please gate the application/routes by deployment role and cover the actual factory output, rather than only testing _work_role_components() in isolation.

[jasondeng1997]

Closed by 33eb25e9 (route isolation). Distributed scheduler/worker now get the health route profile only — /health/live, /health/ready, /metrics — so neither /v1/memory/flush nor any business route remains reachable on those roles, and the health profile rejects Dashboard/MCP at composition time. tests/test_server_role_routes.py pins the exact route sets for every role.

[hireview001]

[P1] Implement the advertised durable operation endpoints

The generated schema advertises operation listing, retrieval, cancellation, and retry, but the canonical OpenAPI contract and server route registration contain none of them, so these requests return 404. Add the canonical contract and handlers, connect OperationManager to memory flush, and regenerate the transport definitions.

[jasondeng1997]

Still open — this is the one substantive item I have not closed yet, and I would rather say so than claim it.

State at b311e14a: the runtime side is now ready. OperationManager is composed for every role and exposed as runtime.operations (builtin/runtime/composition.py:1121), and MemoryWorkHandler drives durable Memory work through the ledger, so there is finally something to inspect, cancel and retry. What is still missing is the transport: openapi/powercontext.yaml has no /v1/operations paths, so scripts/generate_api.py --check regenerates nothing and server/app.py registers no handler.

The work I plan, in this order:

1. Add the four paths to the canonical contract with Operation, OperationPage and an expected_version mutation body, plus the 202 + Location + Retry-After response on POST /v1/memory/flush that the docs already describe.
2. Regenerate _generated/{models,operations,schema}.py and keep --check clean.
3. Register the handlers in server/app.py through the same _add_route path as the other operations, mapping StoredWork to the response and keeping the _PUBLIC_KINDS whitelist so maintenance work stays private.
4. Enqueue from the flush handler when Prefer: respond-async is requested, and notify the local worker.

One design question I do not want to guess on: these paths address work by operation_id, while the Scope lives on the stored row, so access cannot be resolved from the request alone. I plan to add a resolver that loads the operation and authorizes against its scope_id (scope.read for get/list, scope.contribute for cancel/retry). If you would rather keep the existing resolver vocabulary and require scope_id as a query parameter instead, say so and I will do that — it changes the contract shape, which is exactly the kind of thing I should not decide unilaterally.

[jasondeng1997]

Implemented in 11f4319c. The four endpoints are now canonical contract, generated transport, and live routes.

Route	Access
GET /v1/scopes/{scope_id}/operations	scope.read
GET /v1/scopes/{scope_id}/operations/{operation_id}	scope.read
POST /v1/scopes/{scope_id}/operations/{operation_id}/cancel	scope.contribute
POST /v1/scopes/{scope_id}/operations/{operation_id}/retry	scope.contribute

The paths are Scope-scoped, and that was the open design question from my last comment. You did not answer it, so I picked the shape that does not weaken authorization: a durable row carries its own scope_id, and every access resolver in server/app.py is a pure function of path/query/body. Authorizing /v1/operations/{id} from the request alone is therefore impossible today — it would need a database read inside _authorization_dependency, or an unauthenticated identifier space. Under the owning Scope, the existing resolver vocabulary works unchanged and the path Scope becomes an ownership assertion in the handler.

What that buys, concretely:

• another Scope's operation and internal maintenance work both read as 404, so ids cannot be probed for existence;
• _PUBLIC_KINDS (builtin/runtime/operations.py) now covers Memory and Profile source-window families in addition to Experience, so a Scope owner can act on the work its own Sources produced while maintenance work stays invisible;
• the response projects a safe subset of StoredWork — payload, logical key and lease internals do not leave the ledger;
• cancel and retry take expected_version; a lost optimistic race is 409 operation_conflict and leaves the original state untouched (_map_governance_error now maps WorkStateConflictError and StaleWorkClaimError).

On "connect OperationManager to memory flush": I did not do that, and I corrected the docs instead of claiming it. The runtime side is ready — runtime.operations is composed for every role — but wiring flush to it means changing POST /v1/memory/flush from a synchronous 200 FlushMemoryResponse to a 202 handle with Location/Retry-After, which is a client-visible response-model change (the generator only accepts multiple success codes when they share one model). That is a contract decision I did not want to slip into a rebase PR. So:

• docs/{en,zh}/docs/develop/http-api.md now documents exactly these four endpoints and states that flush still answers 200 synchronously;
• docs/{en,zh}/rfcs/1430_distributed_server_workers.md keeps the proposal but now carries an implementation-status paragraph naming what is not shipped: 202 OperationAccepted, Prefer: respond-async, Prefer: wait=N, 409 operation_blocked, and the client submit_memory_flush() / polling helpers.

Evidence at this head:

• tests/test_operation_routes.py (new, 2 tests): list paging and filters, kind/status scoping, cross-Scope 404, maintenance 404, stale-version 409, cancel → cancelled, cancelled-is-not-retryable 409, and retry of a genuinely failed window returning to queued with the error fields cleared;
• make contract-test → 56 passed (contract, generated Python, JS operations, Desktop digest all consistent);
• tests/test_server.py + tests/e2e/test_operation_api.py + docs contract tests → 70 passed;
• DSH plugin pnpm test green after the regenerated catalog.

Two generated-artifact notes, in case they look surprising in the diff: the schema is named DurableOperation rather than Operation because a model named Operation collides with the generator's own route generic (Operation[Request, Response]) and silently pointed response_type at the generic; and OperationStatus is a named schema so the generator cannot invent a colliding Status2 — the first attempt did exactly that and renamed an unrelated model's enum.

[hireview001]

[P1] Adapt durable memory work to the current Atomic Memory service

For an unconsumed source window, MemoryWorkHandler calls services.memory, which does not exist on _ScopedServices; its subsequent memory_artifact_id and collection planning assumptions also belong to the previous memory implementation. Prepare and commit through the current Atomic Memory service so durable memory work can complete instead of exhausting retries.

[jasondeng1997]

Restored in b311e14a. MemoryWorkDiscoverer (work_handlers.py:110), MemoryWorkHandler (work_handlers.py:267) and memory_work_spec (work_handlers.py:540) are back, and they no longer touch _ScopedServices.memory, memory_artifact_id or the old collection planning.

The claimed window now runs through RelationalContexts.process_memory (runtime/relational.py:1544), which is the current Atomic Memory path: it takes the per-scope activation lock and calls AtomicMemorySourceWindowProcessor.flush, so the model calls stay outside the transaction and the processor keeps the single transaction that commits the cursor and the artifacts. MemoryWorkHandler.prepare only decides the outcome:

• already_committed when the cursor already passed the claimed window,
• superseded when a racing window consumed the prefix (the key is released instead of retried),
• processed/idle from the MemoryFlushResult,
• memory_conflict as a retryable WorkExecutionError for AtomicMemoryConflictError/GenerationConflictError/RevisionConflictError.

A retried attempt re-reads the cursor, so a window that already committed retires rather than repeating the domain write — at-least-once model calls, at-most-once commit, as the RFC states.

Covered by tests/builtin/runtime/test_memory_work.py: the spec tracks the unconsumed window, the handler drives the flush once, and a second attempt reports already_committed after the cursor advanced.

[hireview001]

[P2] Provide the scope pagination method used by discovery

MemoryWorkDiscoverer and ExperienceWorkDiscoverer call scope_ids_page, but RelationalContexts has no such method, so their first discovery page fails before enqueueing any work. Implement bounded scope pagination or adapt both discoverers to a supported interface.

[jasondeng1997]

Already satisfied at this head: RelationalContexts.scope_ids_page exists (runtime/relational.py:1402) and both MemoryWorkDiscoverer (work_handlers.py:131) and ExperienceWorkDiscoverer (work_handlers.py:170) use it for their first page. tests/builtin/runtime/test_work_composition.py now asserts that both discoverers register, so an interface regression fails a unit test rather than the first discovery page.

[hireview001]

[P2] Install the limiter when rate limiting is enabled

ServerSettings accepts rate_limit.enabled=True, but create_server_app neither installs SharedRateLimitMiddleware nor binds SharedRateLimiter, leaving requests unlimited regardless of the configured budget. Connect the limiter to the application lifecycle and run it after principal binding.

[jasondeng1997]

Fixed in b311e14a. SharedRateLimitMiddleware is new (server/rate_limit.py:43) and create_server_app installs it when rate_limit.enabled is set (server/factory.py:368), appended after the authentication middleware so current_principal() is already bound.

Behaviour: the fixed window lives in the deployment database (RateLimitRepository.consume), so replicas share one budget; the key is a SHA-256 of principal.type:principal.id, falling back to the client host when no principal is bound; /health/live, /health/ready and /metrics are exempt; a rejected request returns 429 rate_limited with Retry-After.

tests/test_server_rate_limit.py drives a real app: requests=1 allows the first request and rejects the second with a positive Retry-After, the readiness probe still returns 200, and with enabled=False three requests are never rejected.

[hireview001]

[P2] Update the remaining caller of the server helper

The helper now accepts one positional argument, but test_context_text_assembly.py still passes settings and a scheduler path, preventing that configured acceptance scenario from starting its server. Update the remaining caller or preserve the compatible signature.

[jasondeng1997]

This one does not reproduce at this head. _start_configured_server still takes two positional parameters — (settings: ServerSettings, scheduler_path: Path) (tests/e2e/real_experience_skill/harness.py:2027) — and every caller passes both, including test_context_text_assembly.py:111, test_profile_context_assembly.py:83 and test_native_code.py:376,483. The rebase kept the signature and the call sites consistent, so no acceptance scenario is blocked here. If you see the failure again, the job log would help me find the path I missed.

[hidb4ai]

Reviewed 1ee3cdc47b055f734ca001830f44cb5a9e07d6f5. This is not ready to merge.

I reproduced the existing role composition issue, missing Operation routes, obsolete Memory service calls, missing scope pagination, and inactive rate limiting. Operation GET/list return 404; discovery and Memory preparation raise AttributeError; a one-request rate limit still permits the second request. I am keeping those findings in their existing threads.

The inline comment adds a separate migration-test collection failure. scripts/generate_api.py --check also fails because the checked-in generated API does not match the canonical contract. Please complete the runtime/transport wiring and migration acceptance before seeking approval. Validation here used isolated SQLite and ASGI; it does not establish real OceanBase multiprocess acceptance.

[hidb4ai]

[P2] Use a migration module that exists in the package

python -m pytest --collect-only tests/test_server_migration_cli.py fails at this import with ModuleNotFoundError: No module named 'powercontext.builtin.persistence.migration'. This PR adds the test but no such module, so a normal test-suite collection aborts before migration behavior can be checked. Update the test to use the shipped migration API and revision contract, or include the missing implementation, so the acceptance scenario can actually run. The new Alembic migrations/env.py and versions/0001_baseline.py also import this absent module.

The invocation on line 39 also exits with code 2: No such command 'migrate'. The CLI registers server db-migrate, so updating only the import would still leave this scenario unable to exercise a migration.

[jasondeng1997]

Fixed by the rebase. tests/test_server_migration_cli.py no longer imports powercontext.builtin.persistence.migration; it uses the shipped API (powercontext.cli.app.create_cli plus powercontext.server.cli.app) and invokes the registered command, server db-migrate plan / apply (tests/test_server_migration_cli.py:15-16,39). There is no 0001_baseline.py in the tree, and neither server_resources/env.py nor mysql/env.py imports the absent module — they only configure Alembic against the executor's pinned connection.

python -m pytest tests/test_server_migration_cli.py passes here (it is part of the 46-test migration run), and scripts/generate_api.py --check is also clean at this head.

[hidb4ai]

The test import and its db-migrate plan/apply invocation are fixed and pass at 886a0cfb. One shipped caller still needs the same correction: docker/compose.distributed.yaml:18 runs ["server", "migrate"], and both deployment guides tell operators to run that command. The exact-head CLI exits 2 with No such command 'migrate'. Every API/Scheduler/Worker service depends on that migrator completing successfully, so the documented Compose topology cannot start. Please adapt the migrator to the supported explicit db-migrate plan/apply workflow, including its acceptance requirements, and update both guides and the Docker contract test that currently asserts the invalid command.

[jasondeng1997]

Adapted in c9934b701, and it is now covered end to end instead of by string matching:

• docker/compose.distributed.yaml:15-45 — the one-shot migrate service runs powercontext server db-migrate plan with --evidence-dir /var/lib/powercontext/migration-state --lock-coordination single-host --backup <policy>, reads plan_id from its JSON output, then exec powercontext server db-migrate apply --plan-id ... --maintenance-confirmed --yes. The backup policy defaults to manual and a skip run has to pass --accept-no-backup, so a missing backup cannot pass silently. Container-side shell variables are escaped as $$, because Compose otherwise interpolates $policy to an empty string before the script runs.
• docs/en/docs/operate/deploy-server.md, docs/zh/docs/operate/deploy-server.md, and both operate/database-migrations.md — the operator instructions use the same plan/apply pair.
• tests/test_docker_contract.py:76-86 — asserts the migrator is /bin/sh -ec running server db-migrate plan followed by server db-migrate apply with --plan-id, the evidence directory, --lock-coordination single-host, --maintenance-confirmed and --yes.
• tests/test_docker_contract.py:130-190 — renders the Compose script the way Compose does (including host interpolation) and executes it against a stub powercontext binary, then asserts the argv it receives: server db-migrate apply, --plan-id plan-abc, --backup manual --backup-confirmed by default and --backup skip under POWERCONTEXT_MIGRATION_BACKUP_POLICY=skip. This is the regression that fails if the script regresses to server migrate.

Head a25b6adf3.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[jasondeng1997]

Scope update after the rebase: this PR is now the runtime Work Ledger

Rebasing onto current master forced two structural calls, both recorded in the latest commit:

1. The memory work kind is removed. Master now owns Source-to-Memory activation in the atomic memory pipeline (AtomicMemorySourceWindowProcessor), which opens its own transactions inside flush(). That is incompatible with this PR's contract — extraction must happen outside the fenced transaction and commit inside it, sharing the caller's transaction with the ledger write. Bridging would have meant reimplementing the deleted two-phase MemoryService, fighting master's architecture. The ledger's semantics don't depend on which work kind drives them: the Experience handler shares the same cursor helpers, the same superseded-snapshot retirement rule, and the same prepare/commit split, so the end-to-end tests (test_durable_experience_work.py) now ride Experience. Profile remains the second integrated kind.

2. The operations HTTP layer is deferred. The /v1/operations spec paths, generated clients, rate-limit middleware and server-role settings all assumed app/factory wiring that master has rewritten (that wiring is the remaining integration work). Shipping spec endpoints no server mounts would be dishonest, so the spec, generated code, JS integrations and the desktop contract are restored to master verbatim. OperationManager survives as the runtime control plane — submit/wait/list/cancel/retry with optimistic versioning — now covered directly by test_operation_api.py at the runtime level. The server mounting comes back as its own commit once the composition/factory assembly lands.

What the ledger itself gains from master's rework: scope_ids_page is restored for paged discovery, the two database primitives the ledger claims on (database_now, insert_if_absent) are restored beside master's engine layer with their semantics documented, the dead Alembic scaffolding is removed in favor of the frozen v1_4_1_005 bundle revision, and the tracing/metrics instrumentation is re-applied onto master's rewritten server/tracing.py and server/metrics.py.

3. The revision is v1_4_1_005, not s1403. Master replaced Alembic revision discovery with frozen bundles that require one linear chain, and its v1_4_1_004 owner-recovery revision already occupies the slot after s1402. The ledger therefore appends behind v1_4_1_004, and it carries a release-based name because that is the naming rule master introduced with v1_4_1_004: a legacy-style id would leave server db-migrate status unable to describe the target. The SQLite fingerprint is measured by replaying s1300 through the new head, so it changed only because the recorded layout now includes pc_access_owner_recovery.

Local verification on the rebased commit: make check green (prek, ruff, ty, upgrade policy, integration manifest 61 passed), tests/builtin/persistence 691 passed / 110 skipped, tests/builtin/runtime 961 passed, contract + ledger e2e 53 passed, migration/revision-name/maintenance CLI suites 37 passed. The operations HTTP layer, its spec and the server-role wiring come back as separate commits; nothing in this PR mounts routes it cannot serve.

[AlexStocks]

[P1] 迁移版本号 v1_4_1_005 与并行 PR #2192 撞号，两者是内容不同的 revision

本行注册的 v1_4_1_005（work ledger）与 #2192（head 27d499e0）注册的 v1_4_1_005（atomic memory description）同号不同内容：

项	本 PR（head dd80249b）	#2192（head 27d499e0）
版本模块	versions/v1_4_1_005_work_ledger.py	versions/v1_4_1_005_atomic_memory_description.py
sqlite_fingerprints["v1_4_1_005"]	92eaad28512ba8a03917acdd9a0996523a67450092e03161eca68e4e0f9d4cb6	2e24be9f699c202cb9d7985690b2be947bdc98f8f64c1c4e4b8e16e35a580fc4
release/upgrade-support.json 的 persistence_sha256	581696bc…（本 head 复算一致）	cbbf2858…（同 head 复算一致）

两个 PR 还同时改 migrations/server_sources/1.4.0/manifest.json 的同一处追加，以及 release/upgrade-support.json 的同一个摘要值。

影响：任一方先合，另一方 rebase 时这三处必然文本冲突；若按“两边都保留”解决，同一个 revision 名会对应两份不同 DDL 与两个不同 fingerprint，离线迁移校验按 revision 名取 fingerprint，升级链基线取到哪一份取决于合并顺序。两边的 CI 各自为绿，缺陷只在合并后出现。

建议：本 PR 或 #2192 任一方改用下一个未占用编号（如 v1_4_1_006），同步 server_sources/1.4.0/manifest.json 与 sqlite_fingerprints，并按 scripts/upgrade_release_gate.py::managed_digest 重算 persistence_sha256；合入顺序确定后另一方 rebase 复算。

本次已核对的部分（不构成 Finding）：本 head 的 managed_digest 与 release/upgrade-support.json 的声明值相等；tests/test_migration_revision_names.py、test_full_server_migrations.py、test_portable_storage_migration.py 在 exact head 副本上 27 passed，迁移资源自身自洽。

[jasondeng1997]

Renamed to v1_4_1_006 (9683d25e). #2192 opened on 2026-10-22 and is 29/30 green, so it keeps 005; this PR takes the next free number.

Item	Now
Version module	versions/v1_4_1_006_work_ledger.py
MySQL fragment	mysql/versions/v1_4_1_006.py
Frozen schemas	schemas/{sqlite,mysql}-v1_4_1_006.json
down_revision	unchanged: v1_4_1_004
sqlite_fingerprints["v1_4_1_006"]	92eaad28… (unchanged, see below)
persistence_sha256	fcc35b91… → 96fc678e14fa0cc85ee6668169b860bd345d0bc40f7f31e11c82291d5fdfaf56

Two details worth recording:

• The SQLite fingerprint did not change. sqlite_inventory excludes pc_schema_revision and the retained objects, so the revision name never reaches the digest. I re-verified by replaying s1300 → head through the real bundle and comparing every recorded value: all six match, including v1_4_1_006.
• down_revision deliberately stays at v1_4_1_004. If feat(recall): add bounded atomic memory manifest #2192 lands first, rebasing will conflict here and I will point it at that v1_4_1_005 instead of merging both into a two-child graph.

tests/test_migration_revision_names.py (target now sequence: 6), test_full_server_migrations.py, test_portable_storage_migration.py, test_owner_recovery_migration.py, test_release140_server_migrations.py and test_server_migration_cli.py pass (46 passed), and scripts/generate_api.py --check is clean.

[AlexStocks]

[P1] 改号到 v1_4_1_006 后，该编号已被 master 占用：合并结果是 add/add 冲突

回复里的判断在 2026-10-29 成立，但 master 随后合入了 source-write-plugins，v1_4_1_006 已被占用。同一根因在当前 head 上仍然存在。

项	本 PR（head 886a0cfb）	master 176b3f92
版本模块	versions/v1_4_1_006_work_ledger.py	versions/v1_4_1_006_source_write_plugins.py
down_revision	v1_4_1_004	v1_4_1_005
sqlite_fingerprints["v1_4_1_006"]	92eaad28512ba8a03917acdd9a0996523a67450092e03161eca68e4e0f9d4cb6	cacce80bdfb14aff889b7babebc9ab8487b7f0a11d2dbbd3d77a3717a23df74c
revision_resources["v1_4_1_006"]	schemas/{sqlite,mysql}-v1_4_1_006.json	同名同键
链尾	停在 v1_4_1_006	另有 v1_5_0_001（down_revision = v1_4_1_006）

证据按固定 OID 取得，未用工作树判断行尾：

• git merge-tree --write-tree master refs/pr/2270（master 176b3f92，head 886a0cfb，merge-base bdf99789）报 31 处冲突，其中 CONFLICT (add/add) 三处正是 mysql/versions/v1_4_1_006.py、schemas/mysql-v1_4_1_006.json、schemas/sqlite-v1_4_1_006.json；manifest.json 与 release/upgrade-support.json 为 content 冲突。
• 两边 revision_resources 与 sqlite_fingerprints 都以 v1_4_1_006 为键。按「两边都保留」解冲突后，后写覆盖前写，同一 revision 名会对应两份不同 DDL 与两个不同 fingerprint——正是原 Finding 里的第 ① ② 条失效机制，不会因为改名一次而消失。
• 链形也有断点：master 为 004 → 005 → 006 → v1_5_0_001，本 PR 为 004 → 006。若只解文本冲突而不改链，本 PR 的 v1_4_1_006 不在 master 那条链上。

本 head 自身仍然自洽，所以缺陷只在合并后出现，与本 PR 的 CI 结果无关：tests/test_migration_revision_names.py + tests/builtin/persistence/test_full_server_migrations.py 在本 head 的 exact 副本上 23 passed；scripts/upgrade_release_gate.py::managed_digest 复算 = 96fc678e14fa0cc85ee6668169b860bd345d0bc40f7f31e11c82291d5fdfaf56，与 release/upgrade-support.json 声明值相等。

修复方向：改号到当前未占用的 id（v1_4_1 系列下一个空号 v1_4_1_007，或按发布语义取 v1_5_0_002），down_revision 指向 master tip v1_5_0_001；同步 versions/、mysql/versions/、两个 schemas/*、manifest.json 的 revision_resources / sqlite_fingerprints / affected_objects，按 managed_digest 重算 persistence_sha256，并同步 tests/test_migration_revision_names.py 里 sequence 与待应用列表的期望值。

[AlexStocks]

本条状态更新（head 5fa749d8）：d9f33be3 把 ledger 从 v1_4_1_006 改成 v1_4_1_007，与本条指出的“编号已被 master 占用 → add/add 冲突”这一半已经解除，manifest.json 里也不再有重名注册。

但改号只解决了“名字撞车”，没有解决“位置”：v1_5_0_001 是 master 当前的已登记 head，而本 PR 把它的 down_revision 从 v1_4_1_006 改成 v1_4_1_007，等于在已发布 head 之前插入一个新 revision。已经 stamp 成 v1_5_0_001 的库仍是 head，v1_4_1_007 永远不会执行。这条的完整证据与建议写在 v1_5_0_001_artifact_storage.py:23 的线程里（Copilot 提出、我补充了 master/head 对照与 #2395 的发布目标）。

本条按“重名已解”记录，不再单独挂 P1；剩下的链位置问题以那条线程为准。

[jasondeng1997]

Resolved for good in 20bd4624. Master also took v1_4_1_006 (source-write-plugins) and then v1_5_0_001 (artifact storage), so the ledger no longer fights for a number at all: it is renamed to v1_5_0_002 and appended after v1_5_0_001 instead of being inserted before it.

Item	Now
Version module	versions/v1_5_0_002_work_ledger.py
MySQL fragment	mysql/versions/v1_5_0_002.py
Frozen schemas	schemas/sqlite-v1_5_0_002.json / schemas/mysql-v1_5_0_002.json
down_revision	v1_5_0_001

Both manifests chain v1_5_0_001 -> v1_5_0_002; the v1_5_0_001 fingerprints are restored to master's 8f2d4a32…, and v1_5_0_002 carries the ledger's 3cf07deb…. test_migration_revision_names.py asserts pending("v1_5_0_001") == ("v1_5_0_002",) and the new head.

Reproduction with the real resources: a database built from base 745ebf2e and stamped at v1_5_0_001 verifies ready with no pc_work_items; upgrading through this bundle lands exactly one pending revision (v1_5_0_002) and apply returns state=ready, revision=v1_5_0_002, server_ready=true with all eight ledger tables present.

[AlexStocks]

本条的「编号与链位置」在 20bd4624 之后已解除，核对如下（head a8cb90ee）：

• versions/v1_5_0_001_artifact_storage.py 的 blob 与 master 63e9665c 完全相同（a8dd25e4…），即没有再改写已发布 head 的 down_revision。
• server_resources/manifest.json 相对 master 只多出 v1_5_0_002 一个键；master 已有键逐条不变（v1_5_0_001 = 8f2d4a32…、v1_4_1_006 = cacce80b…），v1_4_1_007 在 manifest 中已 0 命中。
• 按 revision/down_revision 静态走链为单线：s1300 → s1400 → s1401 → s1402 → v1_4_1_004 → v1_4_1_005 → v1_4_1_006 → v1_5_0_001 → v1_5_0_002，v1_5_0_001 的后继只有一个；tests/test_migration_revision_names.py 在 exact head 上 7 passed。

本条按「已解」结束。同一改名引入的另一处问题不在迁移链上：release/upgrade-support.json 的 persistence_sha256 没有重算（managed_digest 复算 1b482cd3… ≠ 声明 0577c14e…，load_policy() 抛 GateError），已在新线程 4463726794（release/upgrade-support.json:6）留证，请按那条处理。

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[jasondeng1997]

Status at a25b6adf3

Fixed in this round, each with the test that holds it

commit	change	regression proof
c8b62cf6	narrow the Memory publication context so ty can see principal	tests/builtin/runtime/test_memory_work.py (isinstance narrowing)
fb2a984b	wait for the first cron boundary instead of running discovery at startup	test_durable_scheduler.py::test_cron_discoverer_waits_for_its_first_boundary_instead_of_running_at_startup
0bb9eda8, 0a478b2a	accept APScheduler's last day-of-month marker, validation still unconditional	5 cases in tests/builtin/runtime/test_cron.py
c9dd22b0, e7679469	repeated cancellation with the row's current version is idempotent	tests/builtin/persistence/test_work.py replay case, tests/test_operation_routes.py::test_repeated_cancellation_of_a_running_operation_is_idempotent (200/200/409 over HTTP)
ed3accd6	declare persistence_sha256 = 054731df… for the cancellation change	scripts/upgrade_release_gate.py → upgrade-policy: upgrade-required
bfbd2e95	pin griffe<2 for the website generator (fumapy imports a symbol griffe 2 moved to griffelib)	check-website green on run 42647706651
5a8dd237	pin the standalone Connector type checker to ty==0.0.56 (the locked version)	opendal-package
a25b6adf3	name the ACP SDK in Harbor's runner venv and log the version it resolves	e2e/bub/tests/test_harbor_agent.py::test_runner_venv_is_pinned_to_the_sdk_the_acp_server_names

Red checks on the previous head, and where they come from

tests (3.12, unit) — tests/builtin/inference/test_codex_credentials.py::test_codex_workers_and_restart_observe_cli_renewal asserts process.exitcode == 0 right after process.join(10), and the spawned worker had not exited yet (assert None == 0). The test arrived on master today in #2432 (6d781dcfe); the same job passes on master's own run 42615177843, so this is the timing assumption in the new test under a loaded matrix, not a Work Ledger path. Nothing in this branch touches it.

Upgrade release gate — a consequence, not a second failure: upgrade-acceptance is gated on needs.tests.result in {success, skipped}, so the unit failure skipped it and the gate saw UPGRADE_SELECTED=true / UPGRADE_RESULT=skipped. The digest itself is consistent — scripts/upgrade_release_gate.py prints upgrade-policy: upgrade-required at this head.

harness-validation / Acceptance (sqlite) — run 42687710011, job 129376714407. All four steps ended at No ACP summary file found; the step's own agent/acp.txt is the cause:

File "/installed-agent/acp_runner.py", line 16, in <module>
    from acp import PROTOCOL_VERSION, RequestError, spawn_agent_process, text_block
ImportError: cannot import name 'text_block' from 'acp' (/opt/harbor-acp-venv/...)

acp_runner.py belongs to Harbor, and Harbor builds the venv that runs it with a bare pip install agent-client-protocol. This branch changes nothing under e2e/ before a25b6adf3 (git diff --stat 2291237a..5a8dd237 -- e2e/ is empty), and the same reusable workflow passed on master on 2026-11-07 (runs 42365885788, 42366219630) before that unpinned resolve started returning a release without the symbol. a25b6adf3 names the SDK this repository already pins for bub-acp-server==0.1.3 (agent-client-protocol==1.0.0rc2) in that venv, and prints the resolved version plus the symbols the runner imports, so a mismatch shows up in the setup log instead of as four missing summaries.

Master is red on the same workflow today as well: run 42615177843 fails Acceptance (sqlite) earlier with exec: "powercontext-e2e-container": executable file not found in $PATH, Acceptance (oceanbase) on its 50-minute step limit, and Recall effort (OceanBase) with exit code 124. That work is tracked by #2537, so the container-side outcome of a25b6adf3 is verified by this branch's next run rather than assumed here.

Threads answered in this round

• docker/compose.distributed.yaml migrator → db-migrate plan/apply (c9934b701, plus tests/test_docker_contract.py executing the rendered script against a stub CLI)
• tests/test_server_migration_cli.py env-file path → database.as_posix() (182106be3)
• docker/compose.distributed.yaml generation model → the Server-prefixed variable in both worker definitions (44cc6b02e)

All of the above is at head a25b6adf3. Requesting re-review from @hidb4ai, @AlexStocks and @knqiufan; the remaining review items are answered in their threads.

[hidb4ai]

Re-reviewed 9c43653957483ede58223867e81472d49823a104. One P1 remains: Profile and Experience still publish after contribution is revoked, without ownership records. Please address the remaining paths in that existing thread before merging.

The six other reported counterexamples no longer reproduce: internal maintenance, slow discovery, the initial Profile cron boundary, legacy last cron syntax, cancellation replay, and Windows env-file path serialization. Memory now carries the configured background identity.

Validation used SQLite runtime/domain-policy probes and ASGI HTTP routes; the Memory probe deferred audit writes around the Worker transaction for SQLite. This does not establish distributed OceanBase or native Windows acceptance. CI is still incomplete.

[jasondeng1997]

fix it done

[Zxf-xufeng]

Reviewed 9c43653957483ede58223867e81472d49823a104. The two inline findings below are new; I checked the existing review threads and have not repeated previously reported issues. Please address the normal-progress lane blockage and the missing membership compatibility gate before merging.

Validation: 38 related existing tests passed, and two focused reproductions exposed these failures using the actual ledger, handlers/runtime composition, and SQLite. The reproductions used locally available dependencies because downloading the locked environment failed; they do not establish OceanBase multiprocess acceptance.

[Zxf-xufeng]

[P1] Retire superseded windows after normal cursor generation advances

SourceCursorRepository.save() increments generation on every successful cursor update, not only on resets. Once the cursor exists, this equality therefore rejects an ordinary partial advance. For example, start at (sequence=1, generation=1), enqueue a window through source 2, then capture source 3 and take a discovery snapshot (after=1, through=3, cursor_generation=1) while the earlier window is pending. If the earlier window commits before that snapshot is enqueued, the cursor becomes (2, 2). The snapshot is then failed with non-retryable cursor_changed instead of retiring as superseded. Its failed lane head blocks subsequent work indefinitely.

A real Experience handler/worker reproduction with the same window limit for both snapshots produced failed / cursor_changed, followed by a queued remainder and zero claims. No timeout or window reduction is involved. Please distinguish normal cursor advancement from resets and retire the superseded snapshot without blocking the lane; add coverage after the first cursor generation, since the special 0 -> 1 case does not cover this race.

[jasondeng1997]

Fixed in 8bc8aff23, on head 8099c4c9c. You are right that the equality only ever matched the 0 -> 1 creation: SourceCursorRepository.save() bumps generation on every successful save, so an ordinary partial advance leaves the snapshot's cursor_generation=1 behind a persisted (2, 2).

_is_superseded_window (work_handlers.py:884) now decides on forward movement alone:

return payload.after < sequence < payload.through

A sequence strictly inside the window is by definition a racing commit, whatever the generation says. A reset leaves the sequence at or before the snapshot's prefix, so it never satisfies the predicate — it falls through to _require_exact_cursor and stays a terminal cursor_changed. The docstring states that boundary explicitly so the next change does not re-add the generation comparison.

Coverage after the first cursor generation, following your sequence exactly — tests/e2e/test_durable_experience_work.py::test_snapshot_taken_after_the_first_generation_retires_when_a_racing_window_advances:

1. the first window commits, so the cursor row exists at (1, 1);
2. the window through source 2 is enqueued and left pending;
3. source 3 arrives, and the snapshot is taken as (after=1, through=3, cursor_generation=1);
4. the pending window commits, the cursor becomes (2, 2);
5. the snapshot is enqueued and claimed.

Before this change the snapshot's Work ended FAILED / cursor_changed and blocked its lane head. It now ends SUCCEEDED / superseded, and the lane keeps moving: the next discovery snapshot processes the remainder with candidate_count == 1 and the cursor reaches 3, so nothing is stuck behind the retired row.

[Zxf-xufeng]

[P2] Connect membership compatibility to the worker claim gate

The production DurableWorker is constructed without claim_readiness, so RuntimeMembership.readiness() is never consulted before claiming work. When live nodes advertise different behavior_revision values, membership correctly reports misconfigured, but the worker still reports ready and executes queued work. This leaves the advertised compatibility check ineffective during an incompatible rollout.

I reproduced this through _open_work_background with two live memberships advertising revisions a and b: membership readiness was misconfigured, worker readiness was ready, and an enqueued maintenance task reached succeeded. Please connect the membership compatibility check to the worker claim gate and ensure the failing check is reflected in readiness. Add a composition-level test proving an incompatible worker leaves queued work unclaimed.

[jasondeng1997]

Fixed in 6a889ff09, on head 8099c4c9c. _open_work_background() now passes claim_readiness=membership.readiness to DurableWorker (composition.py:1562), wiring the gate the worker already applies in run_once() and in its own readiness() to the membership composition actually started.

Your reproduction now lands the other way. test_membership_gate_leaves_queued_work_unclaimed (tests/builtin/runtime/test_work_publication_security.py) starts a compatible and an incompatible member against one ledger, enqueues Experience work, and asserts:

• membership.readiness() == "misconfigured",
• worker.readiness() == "misconfigured" (previously ready),
• worker.run_once() == 0, with the queued row still left for later rather than executed.

Both members then stop heartbeating, their rows expire with the TTL, and the same worker instance claims and processes that row — so the gate reopens on its own and nothing was failed while it was closed. test_work_composition.py additionally asserts the composed worker's claim_readiness is bound to the started membership, which is what would have caught the missing wire.

[AlexStocks]

[P2] 本次提交只补了 JS fixture，同根因的 Python fixture 仍使 dsh-package 第 11 步在本 head 红

本 head 上 dsh-package 仍是 failure：run 42991410345 / job 130469094641，步骤 11 Verify DSH setup with customized profiles 失败，步骤 12 Verify DSH guidance adapter requests and results 被 skip。

失败用例与断言（tests/test_dsh_transport.py:1072）：

tests/test_dsh_transport.py::test_explicit_web_doctor_does_not_read_desktop_environment_profile
E  assert 1 == 0
E  "plugin_memory_protocol": {"ok": false, "status": "degraded",
E                            "checks": {"memory_protocol": "unverified"}}

成因（源码级判断）：该用例的 profile 由 dsh_profile（tests/test_dsh_transport.py:36-55）建成 {"dsh": {"profile": {"bundles": []}}}，没有任何 PowerContext bundle；_run_dsh 被打桩返回 id: powercontext-dsh，于是 plugin 检查报 registration: present，但 read_dsh_settings(..., require_installed=True) 组合出的 settings 里没有 memoryProtocol，installed_memory_protocol(None, "dsh") 落到 DiagnosticStatus.DEGRADED（src/powercontext/cli/memory_protocol.py:36-52），doctor 退出码 1。同一形态的第二处是 plugin_source（tests/test_dsh_transport.py:133-148）：它写的 package.json 只有 name 与 dsh.bundle.patch，同样没有 powercontext.memoryProtocol。

即：本次提交为本行 fixture 补上了 memoryProtocol 声明，Python 侧同形 fixture 未补，#2495 的新判定在这条路径上仍然把宿主判成 degraded。

建议：给 plugin_source 与需要 doctor 成功断言的那些 dsh_profile 用例的 package.json 加上 "powercontext": {"memoryProtocol": "atomic-artifact-v1"}（与真实插件 integrations/dsh/plugins/powercontext/package.json 一致），或把该用例的期望改为显式的 degraded 断言。两种改法都会改变断言语义，请选择与 doctor 契约一致的那种，并在改后重跑 dsh-package。

归因边界（避免误读）：master 上同一 job 在第 6 步就红（tests/status-http.spec.ts 断言 expected 'skipped' to be 'appended'），第 11 步在 master 侧被 skip，因此本条的对照只有本 head 的 95e0a357，不主张 master 上必然同样失败。

[jasondeng1997]

Thanks, and agreed on the root cause — the Python-side setup fixtures needed the same powercontext.memoryProtocol declaration as the real plugin, not a weakened doctor expectation. That is now in the branch: 05a50973e (#2569, already on this line before the merge) adds {"powercontext": {"memoryProtocol": "atomic-artifact-v1"}} to the synthetic plugin_source manifest in tests/test_dsh_transport.py:1071-1074, matching integrations/dsh/plugins/powercontext/package.json; 230efe1a (#2580, merged in 8d9d107a4) adds the corresponding JS fixture declaration in setup.test.mjs and asserts plugin_memory_protocol.status == "ok". On the current head c08c8c630 the previously failing test_explicit_web_doctor_does_not_read_desktop_environment_profile therefore asserts a healthy Memory protocol instead of a degraded one, and no assertion was changed to accept degraded.

Your baseline 95e0a357 had zero memoryProtocol occurrences in that Python test file; the current head has the declaration plus the stronger status assertion. dsh-package step 11 is being re-run on the re-triggered CI for c08c8c630; I will report the job result here once it completes.

Update: the final head is db2c2cf62 — it only pins an explicit anchor for the new desktop/VALIDATION.md section; CI is running on that head.

[AlexStocks]

@jasondeng1997 the github ci process failed. please fix this problem.

[hidb4ai]

Re-reviewed eb6a9dc788d643e9a0bcebaded89f4319254243d. One P1 remains: Profile publication still bypasses revoked Scope contribution. The production-provider replay succeeds after revocation for initial creation, an owned Profile update, and a review-required Candidate. Please add the missing transactional contribution check while retaining the new Head and ownership hooks.

Experience revocation/ownership, cursor-progress retirement, and the membership claim gate now pass the affected checks. Validation used SQLite runtime/domain-policy probes, with audit writes deferred around the publication transaction; it does not establish OceanBase multiprocess acceptance.

Separately, required CI remains red: the Python 3.13 e2e job fails in test_load_real_background_processes_and_preserves_disabled_state[True] because a Source process did not become ready. Upgrade acceptance was consequently skipped. I have not established that this startup failure was introduced by this PR.