Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 71 additions & 23 deletions app/spec/worker_coordination/worker_api.mli
Original file line number Diff line number Diff line change
@@ -1,7 +1,20 @@
(** App/Worker coordination boundary. Pure data belongs to [Worker_pure_data];
sealed authority is issued by the trusted
Worker runner. Abstraction is not authentication: every message, continuation,
receipt and resource is validated against its actual channel and owner. *)
(** {1 TLDR}

Define the typed App/Worker boundary: read DTOs, durable intent, observations,
resource leases and host handoff capabilities. The trusted runner issues
sealed authority; [Worker_pure_coordinator.request] plans execution and
[Worker_effect_runner.submit] interprets it. This module itself performs no
transport I/O. Abstraction is not authentication: runtime checks still bind
every message, continuation, receipt and lease to its real channel and owner.
Queue admission, durable commitment and presentation confirmation are separate.

The query/request GADTs depend on the capability families below. Their actual
public shapes include [Query (budget, query)], [Observe (budget, query)],
[Mutate intent], [Confirm (presentation, receipt)] and [Release observation].
Replies distinguish [Not_admitted], [Answered] and [Interrupted]; interruption
after admission does not prove rollback. Sessions survive resident focus
changes but close/reopen, rebinding and account changes revoke execution. *)

module Graph = Logseq_db_types.Graph_types

module Uuid = Graph.Uuid
Expand All @@ -19,6 +32,44 @@ val equal_binding : binding -> binding -> bool
type session = Worker_effect_authority.session
type presentation = Worker_effect_authority.presentation

(** Sessions survive focus changes while resident. Close/reopen, account changes,
removal and rebinding revoke execution authority. Presentation has a separate
lifetime. Admission and completion recheck all identities, not just UUID. *)
type error = Worker_effect_authority.error =
| Revoked_session
| Invalid_input of string
| Response_budget_exceeded
| Stale_read of string
| Resync_required of string
| Host_change_in_progress
| Protocol_mismatch of string
| Db_error of Logseq_db_worker.Error.t
| Service_failed of string
| Unsupported_operation of string

(** {1 Minimal usage example}

The caller supplies a coordinator [state], a [session] obtained from a Ready
projection's Open resident, a typed [query], and the runner's [interpret].
This example returns its ticket so the caller can later consume the reply:
{[
match read_budget session ~maximum_items:1 ~maximum_bytes:4096 with
| Error error -> Error error
| Ok budget ->
match Worker_pure_coordinator.request !state session (Query (budget, query)) with
| Error error -> Error error
| Ok (ticket, transition) ->
state := transition.next;
List.iter interpret transition.instructions;
Ok ticket
]}
[interpret] queues completion events for later serialized turns. Inspect
[Worker_pure_coordinator.reply] for admission, operation and interruption
outcomes, then feed [Forget ticket] through that coordinator. An observation
or file lease additionally requires its actual release acknowledgement. *)

(** {1 Construction and validation} *)

val account_generation : session -> account_generation
val graph_generation : session -> graph_generation
val presentation_generation : presentation -> presentation_generation
Expand All @@ -36,21 +87,6 @@ val worker_incarnation
-> worker_generation:Journal_worker_ids.Worker.generation
-> worker_incarnation

(** Sessions survive focus changes while resident. Close/reopen, account changes,
removal and rebinding revoke execution authority. Presentation has a separate
lifetime. Admission and completion recheck all identities, not just UUID. *)
type error = Worker_effect_authority.error =
| Revoked_session
| Invalid_input of string
| Response_budget_exceeded
| Stale_read of string
| Resync_required of string
| Host_change_in_progress
| Protocol_mismatch of string
| Db_error of Logseq_db_worker.Error.t
| Service_failed of string
| Unsupported_operation of string

type request_id = Worker_effect_authority.request_id

val request_id : Uuid.t -> request_id
Expand Down Expand Up @@ -83,6 +119,8 @@ type journal_day = Worker_effect_authority.journal_day

val journal_day : int -> (journal_day, error) result

(** {1 Capability details} *)

type page = Worker_effect_authority.page
type block = Worker_effect_authority.block

Expand Down Expand Up @@ -282,6 +320,8 @@ val change_ack_session : change_ack -> session
val change_ack_generation : change_ack -> string
val change_ack_through : change_ack -> string

(** {2 Durable mutations} *)

type append_intent = Worker_effect_authority.append_intent

(** Persist the complete bounded root/child/status payload and normalized stable
Expand Down Expand Up @@ -357,7 +397,6 @@ val durable_intent_id : durable_intent -> Uuid.t
val encode_durable_intent : durable_intent -> bytes
val restore_durable_intent : session -> bytes -> (durable_intent, error) result


(** General mutations preserve today's exact preconditions and mutation IDs.
Restores validate logical origin/account/graph and all bytes, allowing fresh
execution sessions only for the same logical graph. Same ID/different payload
Expand Down Expand Up @@ -458,6 +497,8 @@ type 'a reply = 'a Worker_effect_authority.reply =
non-commit evidence cannot become Rejected. Replay must be durable/idempotent
across restart, not merely a process-local duplicate cache. *)

(** {2 Observations and change windows} *)

type changefeed_notice = Worker_effect_authority.changefeed_notice = private
{ notice_session : session
; notice : Worker_pure_data.changefeed_signal
Expand Down Expand Up @@ -509,6 +550,8 @@ type observation_notice = Worker_effect_authority.observation_notice = private
}
| Observations_resync of binding

(** {2 Assets and import sources} *)

type asset_consumer = Worker_effect_authority.asset_consumer
type asset_demand = Worker_effect_authority.asset_demand
type asset_file = Worker_effect_authority.asset_file
Expand Down Expand Up @@ -619,6 +662,8 @@ val upload_operation : asset_notice -> upload_operation option
replaces an underlying successful resource result and App never receives the
handle. App-only late-reply cleanup cannot prove this; backend acquisition/
import ledger or atomic rollback must settle every Accepted operation. *)
(** {2 Typed execution requests} *)

type 'a request = 'a Worker_effect_authority.request =
| Query : read_budget * 'a query -> 'a request
| Observe : read_budget * 'a query -> 'a observed request
Expand Down Expand Up @@ -658,6 +703,8 @@ type 'a request = 'a Worker_effect_authority.request =
fail before enqueue, without a fabricated receipt/registration/lease ACK. *)
val validate_request : session -> 'a request -> (unit, error) result

(** {1 Lifecycle and recovery} *)

type observation_update = Worker_effect_authority.observation_update = Update : 'a observed -> observation_update
type projection_version = Worker_effect_authority.projection_version

Expand Down Expand Up @@ -748,6 +795,8 @@ type bootstrap_state = Worker_effect_authority.bootstrap_state =
state. The host must perform a handoff before retiring a graph/account/client;
focus alone revokes presentation, not resident execution. The authoritative
projection, not optimistic UI flags, admits the next session. *)
(** {2 Host handoff} *)

type host_change = Worker_effect_authority.host_change =
| Account_will_change
| Graph_will_close of Uuid.t
Expand Down Expand Up @@ -816,6 +865,5 @@ type handoff_state = Worker_effect_authority.handoff_state =
and removal uses the bound existing deletion owner. Do not dispatch a second
legacy Close/Delete after that ACK. Auth/UI/client lifecycle stay with host. *)

(** Connection-bound validation used inside the direct effect runner. Methods below
map concrete Service messages, not fictional bytes or a separate adapter. The
factory is trusted integration, not proof of an honest backend. *)
(** Connection-bound validation remains the direct effect runner's responsibility.
The trusted integration is not itself proof of an honest backend. *)
42 changes: 37 additions & 5 deletions app/spec/worker_coordination/worker_effect_runner.mli
Original file line number Diff line number Diff line change
@@ -1,9 +1,40 @@
(** Direct interpreter against the existing dbworker Service/client. No adapter,
second Worker client is introduced; the host selects its implementation explicitly. *)
(** {1 TLDR}

Interpret the coordinator's private instructions through the existing Worker
client and its single host pump. {!create} borrows that client/account and
queues binding/bootstrap events; success does not imply Ready. {!submit}
performs scoped I/O, while {!on_event} routes actual mailbox completions.
The host owns serialization, UI/business state and shared client lifetime.
{!shutdown} returns [Draining], [Stopped] or [Shutdown_failed]; only [Stopped]
proves this runner's cleanup has reached its terminal fence. *)

module Worker = Worker_api

type client = (Logseq_db_worker_lui.Logseq_db_worker_lui_service.request, Logseq_db_worker_lui.Logseq_db_worker_lui_service.response, Logseq_db_worker_lui.Logseq_db_worker_lui_service.push) Logseq_db_worker_lui.Journal_worker.client
type t
type disposition = Consumed | Unowned_response | Broadcast | Dropped_envelope
type delivery = private { disposition : disposition; error : Worker.error option }

type shutdown_state = Draining | Stopped | Shutdown_failed of Worker.error

(** {1 Minimal usage example}

The caller supplies the existing [client], verified [managed_origin]/[user_id],
[post] that queues serialized coordinator events, and the host's [publish]:
{[
match create ~client ~managed_origin ~user_id ~post ~publish with
| Error error -> Error error
| Ok runner -> Ok runner
]}
Keep the returned runner in the host. For each coordinator transition install
[next] first, then [List.iter (submit runner) transition.instructions]. Route
the existing pump through [on_event] before legacy dispatch and honor its
[disposition]. During retirement keep delivering real cleanup events and call
[shutdown runner] after progress until [Stopped]; retain failures for retry.
Closing the shared client also waits for every other owner. *)

(** {1 Construction and validation} *)

val create
: client:client -> managed_origin:string -> user_id:string
-> post:(Worker_pure_coordinator.event -> unit)
Expand All @@ -23,8 +54,8 @@ val create
turns; publish does not own UI/business state. Ready comes only from the
correlated registered Service baseline, never from independent legacy pushes. *)

type disposition = Consumed | Unowned_response | Broadcast | Dropped_envelope
type delivery = private { disposition : disposition; error : Worker.error option }
(** {1 Capability details} *)

val on_event : t -> (Logseq_db_worker_lui.Logseq_db_worker_lui_service.response, Logseq_db_worker_lui.Logseq_db_worker_lui_service.push) Logseq_db_worker_lui.Journal_worker.event -> delivery

(** One shared host router calls this BEFORE legacy response dispatch. Ownership
Expand Down Expand Up @@ -82,7 +113,8 @@ val submit : t -> Worker_pure_coordinator.instruction -> unit
UI/auth/client continuation. Unknown outcomes retain their original permission
and cleanup owner; they are never replayed with a fresh receipt. *)

type shutdown_state = Draining | Stopped | Shutdown_failed of Worker.error
(** {1 Lifecycle and recovery} *)

val shutdown : t -> shutdown_state

(** Nonblocking and idempotent. Revoke ordinary admission, cancel/reconcile owned
Expand Down
52 changes: 49 additions & 3 deletions app/spec/worker_coordination/worker_pure_coordinator.mli
Original file line number Diff line number Diff line change
@@ -1,6 +1,23 @@
(** Pure Worker request/observation/resource coordination, not the whole App.
Navigation/feed/Favorites algorithms, Capture editing and media UI stay with
their current business owners. step performs no DB/network/clock/callback IO. *)
(** {1 TLDR}

Coordinate App requests, observations and resource ownership for one Worker
binding. The caller owns {!t}; {!step} and {!request} perform no I/O or
persistence. Each returns a private {!transition}: install [next] before
interpreting [instructions] in order. The runner queues later typed events.
Queue admission, durable outcomes and presentation confirmation are separate;
cancellation does not prove rollback. {!shutdown_state} reaches [Stopped]
only after real cleanup acknowledgements, not merely [Shutdown_requested].

The indexed event/instruction families depend on the capabilities below.
These are excerpts of the actual entry points, not additional APIs:
{[
val initial : client_id:Worker.Uuid.t -> t
val step : t -> event -> transition
val request : t -> Worker.session -> 'a Worker.request
-> ('a ticket * transition, Worker.error) result
val reply : view -> 'a ticket -> 'a Worker.reply option
]} *)

module Worker = Worker_api

type t
Expand All @@ -13,6 +30,27 @@ val view : t -> view
val bootstrap : view -> Worker.bootstrap_state
val projection : view -> Worker.projection option

(** {1 Minimal usage example}

The caller supplies a coordinator [state] that has installed its real binding
and Ready bootstrap projection, an Open resident [session], validated [budget]
and [query], and the runner's [interpret]:
{[
let apply transition =
state := transition.next;
List.iter interpret transition.instructions
in
match request !state session (Worker.Query (budget, query)) with
| Error error -> Error error
| Ok (ticket, transition) -> apply transition; Ok ticket
]}
The host establishes [state] by applying the real [Worker_bound] and bootstrap
events through [step] before this flow. [interpret] queues all callbacks for
later serialized turns. Read [reply (view !state) ticket] after
completion; feed [Forget ticket] once its result has been consumed. *)

(** {1 Capability details} *)

(** Latest original notifications accepted by Core for current residents since
the previous view. Stale/foreign notices never reach this outlet; it retains
no delivery history. Queued consumers still validate the notice session. *)
Expand Down Expand Up @@ -98,6 +136,8 @@ val change_application_ready : view -> Worker.change_batch -> bool
val change_application_error : view -> Worker.change_batch -> Worker.error option
val equal_view : view -> view -> bool

(** {1 Lifecycle and recovery} *)

type shutdown_state =
| Active
| Draining
Expand Down Expand Up @@ -181,6 +221,8 @@ type event =
(* Return means no presentation consumer retains the capability. It is not an
I/O ACK: failed/in-flight cleanup and metadata finalization stay owned until
their matching real completions. Cleanup retries remain possible while owned. *)
(** {2 Cleanup instructions} *)

(** Cleanup identity allocated by Core before sending a replacement. It records
an obligation, not successful registration. Runner may inspect it but cannot
construct or retarget it; cancellation can clean up even if no handle arrived. *)
Expand Down Expand Up @@ -257,6 +299,8 @@ val request
-> 'a Worker.request
-> ('a ticket * transition, Worker.error) result

(** {2 Mutation reconciliation} *)

type mutation_reconciliation =
| Submission_ticket of Worker.submission_outcome ticket
| Confirmation_ticket of Worker.confirmation ticket
Expand All @@ -280,6 +324,8 @@ val settle_applied_mutation
: t -> presentation:Worker.presentation -> applied_presentation:Worker.presentation -> Worker.durable_intent
-> (mutation_settlement * transition, Worker.error) result

(** {2 Applying change windows} *)

(** Start application of an actually delivered batch. An active successful
attempt cannot be replaced by a duplicate pull; an explicit retry after a
failure gets a new token. The root itself is applied only after its UI state. *)
Expand Down
48 changes: 43 additions & 5 deletions app/spec/worker_coordination/worker_pure_data.mli
Original file line number Diff line number Diff line change
@@ -1,7 +1,41 @@
(** Pure query values shared by the coordinator and Worker. Constructor tags
preserve the canonical JSON identity of persisted mutation payloads. *)
(** {1 TLDR}

Define pure query DTOs shared by the App coordinator and Worker.
Constructor tags preserve canonical JSON identity for persisted mutation
payloads; this module performs no I/O and grants no execution authority.
The mutually recursive data family below includes revision preconditions,
explicit present/missing lookups, bounded page members and change windows.
{!valid_append_node} validates append trees; requests execute through
[Worker_api.Query] and [Worker_pure_coordinator.request], not through DTOs. *)

open Logseq_db_types.Graph_types

type changefeed_signal =
| Changes_available of { generation : string; through : string }
| Resync_required of { generation : string; reason : string }

(** {1 Minimal usage example}

The caller supplies a [session], a [budget] validated by [Worker_api.read_budget],
the coordinator [state] and the runner's [interpret]. This read consumes the
DTOs through their public owner:
{[
match Worker_pure_coordinator.request !state session
(Worker_api.Query (budget, Worker_api.Admission)) with
| Error error -> Error error
| Ok (ticket, transition) ->
state := transition.next;
List.iter interpret transition.instructions;
Ok ticket
]}
[interpret] queues completion events for later serialized turns. The caller
reads the eventual typed reply through [Worker_pure_coordinator.reply]. *)

(** {1 Construction and validation}

These bounds are part of request, response and tree validation. A valid
DTO alone does not prove that its session, revisions or budgets are current. *)

val maximum_request_bytes : int
val maximum_response_bytes : int
val maximum_push_bytes : int
Expand All @@ -15,6 +49,12 @@ val maximum_changed_uuids : int
val maximum_property_values : int
val maximum_filter_values : int

(** {1 Capability details}

This recursive family is kept together to preserve its generated JSON
representation. Lookups distinguish missing records from an empty page;
version strings and preconditions must be passed back to their owning API. *)

type precondition_scope = V2_children_scope of Uuid.t

and task_status = Uuid.t
Expand Down Expand Up @@ -144,8 +184,6 @@ and admission_inspection =
}
[@@deriving yojson]

type changefeed_signal =
| Changes_available of { generation : string; through : string }
| Resync_required of { generation : string; reason : string }
(** {1 Observation and helpers} *)

val valid_append_node : append_node -> bool
Loading
Loading