Storage / Backend blueprints
+ +# Your database. Triplex's memory. + +Build on the storage primitives you already trust. Keep facts, time, versioned rules, and derived work behind one TypeScript API.
+ +Triplex separates its application model from storage and query execution. These blueprints show +how a backend can provide that foundation: the data representation, the transaction boundary, +the query path, and the checks an implementation needs to pass. + +Backend blueprint / Datomic
+ +# Building Triplex on Datomic + +A fact-oriented foundation for a TypeScript application that needs to remember which rules governed every decision.
+ +Proposed blueprint · No adapter exists
+ +This page proposes an implementation to investigate. There is no Datomic package, running +integration, or compatibility claim. The mappings below are design choices to validate against +Triplex's public adapter contract. + + + +## Why Datomic + +Datomic offers datoms, Datalog, transaction entities, immutable database values, historical views, +and ordered log access. Those primitives are useful ingredients for a Triplex adapter. +[Datomic's best practices](https://docs.datomic.com/reference/best.html) describe their use for +consistent reads and audit queries. + +On top of an adapter, Triplex would supply its TypeScript domain model, immutable configuration +releases, pinned command provenance, and derivation reconciliation. Datomic would own persistence +and native query execution. A TypeScript-to-JVM client boundary would be part of the deployment; +choose and document that bridge before promising an embedded integration. + +## Storage mapping + +The proposed starting point is **one Datomic entity per Triplex assertion**, with a fixed adapter +schema. A direct mapping from a domain triple to one native datom cannot by itself preserve +independent `TripleId`s and valid-time intervals for repeated assertions of the same value. + +| Triplex contract | Proposed representation | +| ---------------------------- | ---------------------------------------------------------------------------------------------------- | +| Stable entity and triple IDs | Original strings stored as adapter identity attributes; native entity IDs stay internal | +| Fact value | Assertion entity with domain entity ID, attribute keyword, entity type, value tag, and typed payload | +| Assertion history | Immutable assertion identity and payload; explicit assertion transaction and recorded position | +| Retraction | Retraction metadata on the assertion entity, retaining the original payload and valid interval | +| Valid time | Explicit `validFrom` / `validTo` fields filtered on every relevant read | +| Command receipt | A scoped command identity claimed atomically without overwriting an earlier receipt | +| Commit order | A per-logical-database position counter advanced in the same transaction as facts and journal | +| Causal journal | Triplex transaction envelopes and typed changes stored atomically with the command | +| Query execution | Translate validated Triplex Datalog to joins over assertion entities and shared temporal filters | + +Keep Triplex's recorded timestamp and commit position explicit. Datomic's native transaction +coordinate can help identify an exact database value, but it is not a drop-in implementation of +`nextCommitPosition()`, which Triplex calls before inserting the transaction's rows. + +## Bridge the transaction models + +Triplex's `StorageAdapter.withTransaction` accepts an Effect callback whose reads and writes must +share an atomic view, including pending changes. Datomic accepts declarative transaction data; +its transaction functions inspect `db-before` and return additional data, without seeing one +another's output. An arbitrary TypeScript callback cannot simply become a native transaction +function. [Datomic transaction functions](https://docs.datomic.com/transactions/transaction-functions.html) +define that boundary. + +One candidate design is an optimistic staging adapter: + +1. Acquire one immutable database value, read the logical database's position and existing receipt, + and run the callback against that value plus an in-memory overlay of pending mutations. +2. Stage constraint checks, a new receipt, assertion/retraction metadata, the next position, and + the complete journal. Reads during the callback must see the overlay consistently. +3. Submit the staged changes with a compare-and-swap from the observed position to the next one. + Every writer of that logical database's source data must participate in this guard. +4. On a conflict, discard the staged result, acquire a fresh database value, and re-evaluate. + For a duplicate, `claimCommand` must return the original transaction ID so core raises + `CommandAlreadyCommittedError`; callers can load its receipt with `transactionByCommand`. + Publish a successful callback result only after confirmed commit. + +This is a proposal, not an implemented algorithm. Test retries, typed failures, defects, +interruption, and uncertain submission outcomes before calling it conformant. Constraint queries +must read the same protected basis; direct writes that bypass the guard would invalidate the +design. A native transaction function could instead perform compatible checks, but translating +Triplex's enforcement logic would be additional work. + +## Preserve both clocks and the query contract + +For Maria's correction, stage a retraction of the March 1 assertion and a new March 5 assertion, +with distinct triple IDs, in the same command. Retain the old payload. The earlier recorded basis +must still expose the original training interval, while current knowledge exposes the gap on +March 3. Datomic's historical view alone does not implement the explicit valid-time filtering in +this proposed assertion schema. + +Start with patterns, joins, and temporal filters, then implement the complete validated Triplex +query surface. Verify typed scalar comparisons, negation, aggregates, the constrained recursive +form, deterministic ordering, pagination, and `explain`. Identical-looking Datalog clauses do not +prove equivalent execution. + +Pinned pages need the original query, temporal basis, database scope, and exact commit boundary. +Use retained assertion/retraction positions or a verified mapping to native database values; +later corrections must not change which rows an earlier cursor sees. Keep the Triplex journal's +typed changes and provenance even if native log access is used to locate commits. + +## Build the first slice + +1. Specify the JVM bridge, assertion schema, typed codecs, and physical database isolation. +2. Prototype the staged transaction boundary first: one correction, two competing writers, one + duplicate command, and one rejected constraint. Establish that failures leave no partial state. +3. Implement `StorageAdapter` and `QueryExecutor`, then compose them with `Runtime.define` and + `Capabilities.none`. See the [checked runtime skeleton](/custom-runtimes#runtime-definitions). +4. Run [runtime conformance](/custom-runtimes#correctness-contract-and-conformance) on a fresh + database. Extend it with bridge disconnects, lost commit acknowledgements, process restarts, + and schema upgrades. + +Configuration and derivation services can then use the same `Triples` layer. Durable entity +snapshots and change emitters are separate capabilities to implement and verify. + +[Open a Datomic design issue](https://github.com/bjacobso/triplex/issues/new) with a transaction +prototype and its test results, or compare the [working SQLite reference](/sqlite) and +[XTDB proposal](/xtdb). diff --git a/docs/foundationdb.md b/docs/foundationdb.md new file mode 100644 index 0000000..8d274ff --- /dev/null +++ b/docs/foundationdb.md @@ -0,0 +1,94 @@ +--- +title: Building Triplex on FoundationDB +description: An experimental Triplex storage blueprint that maps ordered byte keys and atomic FoundationDB transactions to the existing hexastore runtime. +pageClass: backend-blueprint +--- + +Backend blueprint / FoundationDB
+ +# Building Triplex on FoundationDB + +Let FoundationDB keep the ordered keys. Let Triplex turn them into facts, history, and explainable work.
+ +Experimental · Private workspace package
+ +An adapter already exists in this repository. It requires compatible native FoundationDB client +libraries and is held from npm publication. Native integration tests are opt-in; it is outside the +default CI matrix. Evaluate from a source checkout with locked dependencies. + + + +## Why these primitives fit + +FoundationDB exposes an ordered key-value store with atomic transactions and range reads. +Transaction retries handle optimistic conflicts; transaction size and lifetime limits shape how +much work can fit in one command. See its [developer guide](https://apple.github.io/foundationdb/developer-guide.html). + +That matches Triplex's `KvBackend` path. The backend handles bytes and atomicity; Triplex's existing +hexastore handles fact indexes, temporal visibility, Datalog, command receipts, and the journal. +The design avoids implementing a second query engine. + +## Storage mapping + +| Triplex needs | FoundationDB adapter responsibility | +| -------------------------------------------- | -------------------------------------------------------------------------------- | +| Ordered byte keys | Preserve lexicographic ordering within an isolated subspace | +| Pattern index scans | Implement inclusive start, exclusive end, direction, and limit | +| Atomic fact and journal writes | Run every operation in the callback on the same native transaction | +| Ordered commit positions and unique receipts | Preserve the KV engine's counter and claim read/write conflicts | +| Abort behavior | Discard all pending mutations when the callback fails | +| Large reads | Page range reads without changing the requested scan semantics | +| Invalidation hints | Optional FoundationDB watches; consumers still catch up from the durable journal | + +Hexastore keys and their encoding are implementation details. A backend implements `KvBackend`; +it does not introduce a portable on-disk hexastore format. Persistent-data upgrades need their own +rehearsal. + +## Compose the runtime + +This checked example uses existing repository exports. Run it only in a checkout with the native +client and a configured FoundationDB cluster. It is not an npm installation recipe. + +<<< @/snippets/foundationdb.ts + +The existing `FdbTriples.layer({ subspace })` is also a convenience composition. A nonempty +subspace is required by default, and clears stay within it. Choose physically disjoint subspaces +for separate databases; `DatabaseScope` binds cursors and does not partition cluster keys. + +`Capabilities.none` deliberately selects the base store. Durable KV entity snapshot persistence +is not supplied automatically. Add a database-owned snapshot provider or a change emitter only +after testing those capabilities separately. + +## One correction across every index + +Maria's correction touches the old assertion's visibility, the new assertion, pattern indexes, +the command claim, the commit position, and the journal. All those mutations must share one +FoundationDB transaction, so a failed correction leaves the old belief intact. + +A retried callback must keep effects inside storage. Sending an email or advancing a host workflow +inside it would repeat on conflict. Use the committed journal to drive host-owned effects. +Splitting one logical command across native transactions would lose Triplex's atomicity. + +The adapter's bulk `setAll` helper can chunk large loads; those chunks are separate commits. +It must not be used to make an oversized atomic Triplex command appear supported. Establish and +measure command limits before production use. + +## Verify and extend the adapter + +Start with the [KV adapter](https://github.com/bjacobso/triplex/blob/main/packages/foundationdb/src/FdbKvBackend.ts) +and [package guide](https://github.com/bjacobso/triplex/blob/main/packages/foundationdb/README.md). +Run `pnpm test:foundationdb:integration` with the native client and a disposable cluster. Use the +[shared runtime conformance helper](/custom-runtimes#correctness-contract-and-conformance) against +an isolated subspace as well. + +Before promoting this backend, establish multi-process contention behavior, interruption and +lost-acknowledgement handling, crash recovery, persistent-key upgrades, and measured transaction +limits. A command interrupted during submission must not leave an unexplained commit; receipts +must resolve any uncertain outcome. These are additional gates beyond the shared corpus. + +See [current state](/current-state#backend-maturity) for maturity and [custom runtimes](/custom-runtimes) +for the complete KV and capability contract. diff --git a/docs/index.md b/docs/index.md index 151e916..767f29b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -237,6 +237,8 @@ npm install @triplex-build/triplex effect@4.0.0-rc.112 | Cloudflare Durable Objects | private workspace package | Experimental | | FoundationDB | private workspace package | Experimental | +Want to build on another storage engine? Explore the backend blueprints: working SQLite, experimental FoundationDB, and proposed Datomic and XTDB adapters. Each maps storage primitives to the Triplex contract and identifies what still needs to be built.
+Also in the box: a JSON-first CLI for agents, a browser data explorer, a configuration-derived HTTP API with OpenAPI, and an in-browser playground.
@@ -256,7 +258,7 @@ npm install @triplex-build/triplex effect@4.0.0-rc.112Datomic and XTDB share the model of immutable facts, Datalog, and time travel; Triplex embeds in TypeScript and adds versioned rules and reconciled work. Event sourcing gives you history, but you hand-write every projection. Workflow engines run durable code; Triplex decides what work should exist, and the two compose.
+Datomic offers facts, Datalog, and historical database views. Current XTDB offers bitemporal data through SQL and XTQL. Triplex embeds in TypeScript and connects facts to versioned rules and reconciled work. Event sourcing gives you history, but you hand-write every projection. Workflow engines run durable code; Triplex decides what work should exist, and the two compose.
Backend blueprint / SQLite
+ +# Building Triplex on SQLite + +An embedded database underneath an embedded fact model. Durable history, attributed commands, and versioned rules in one local file.
+ +Supported durable baseline · Existing adapter
+ +SQLite is the working reference for Triplex's SQL storage path. Use it for local applications, +single-process services, and a durable first implementation of a back-office domain. It is in the +default shared conformance suite. Triplex remains pre-1.0; see [current state](/current-state). + + + +## Run it + +Install the SQLite adapter with the compatible Effect version on Node.js 22 or newer: + +```sh +npm install @triplex-build/triplex @triplex-build/triplex-sqlite effect@4.0.0-rc.112 +``` + +This writes a training fact and reads it through the same `Triples` service used by other backends: + +```ts check +import { Effect } from "effect"; +import { EntityId, Triples, string } from "@triplex-build/triplex"; +import { SqliteTriples } from "@triplex-build/triplex-sqlite"; + +const DatabaseLive = SqliteTriples.layer({ filename: "training.db" }); +const maria = EntityId.make("worker:maria"); + +const program = Effect.gen(function* () { + const triples = yield* Triples; + const receipt = yield* triples.transact( + [ + { + op: "assert", + entityId: maria, + attribute: ":evidence/site-safety", + value: string("site:harbor"), + validFrom: Date.parse("2026-03-01T00:00:00Z"), + validTo: Date.parse("2026-06-01T00:00:00Z"), + }, + ], + { actor: "user:dana", commandId: "training:maria-harbor:1" }, + ); + const facts = yield* triples.entity(maria, { + validAt: Date.parse("2026-03-03T00:00:00Z"), + }); + return { position: receipt.position, facts }; +}); + +await Effect.runPromise(program.pipe(Effect.provide(DatabaseLive))); +``` + +Re-running this command raises `CommandAlreadyCommittedError` with the original transaction ID. +Handle that tag and call `transactionByCommand` with the same command ID to load the durable +receipt; the retry writes no second fact or journal entry. + +Create the layer once for the database's intended lifetime. The convenience layer owns the SQLite +client, configures WAL mode, and runs the ordered Triplex migrations. Production hosts that own +DDL can use `makeSqliteLayerUnmigrated`, `makeSqliteAdapter({ autoMigrate: false })`, and the shared +SQL migration entrypoints. See [host integration](/host-integration). + +## Storage mapping + +| Triplex contract | Existing SQLite representation | +| ----------------------------- | -------------------------------------------------------------------------------------------- | +| Fact identity and typed value | One `triples` row per `TripleId`, with entity, attribute, value tag, and typed value columns | +| Recorded and valid time | `recorded_at`, `retracted_at`, `valid_from`, and `valid_to` | +| Exact commit visibility | `recorded_position` and `retracted_position` on each row | +| Ordered command boundary | Transactional `triplex_commit_position` row | +| Retry identity | Unique command claim in `triplex_command_receipts` | +| Attributed journal | `_Transaction` facts and typed changes stored with the application writes | +| Entity projections | Content-addressed blobs and entity snapshot tables, when composed with snapshot services | +| Datalog | Shared SQL compiler/executor with the SQLite dialect | + +The physical schema stores each assertion separately. Domain attributes such as +`:evidence/site-safety` remain data, so applications do not need a physical table per entity type. +Indexes cover entity lookup, attribute/value lookup, references, temporal reads, and commit +visibility. The additive v2 migration adds the shared numeric range index. + +## One command, one SQL transaction + +The adapter executes Triplex's transaction callback on the ambient Effect SQL transaction. +Command claiming, position allocation, constraint checks, fact changes, and journal changes belong +to that same boundary. A rejected command must leave none of them committed. + +For Maria's correction, the adapter marks the old assertion's retraction time, transaction, and +position, then inserts the corrected assertion and journal changes. The assertion row is retained. +Historical reads filter both clocks; query pages additionally use the pinned commit position so +later retractions cannot remove a fact from an older page. + +Keeping timestamps and positions separate matters when multiple commands share a recorded +millisecond. The cursor contract relies on exact commit visibility. + +## Read the implementation + +Follow these modules in order: + +1. [SQLite layer and resource lifetime](https://github.com/bjacobso/triplex/blob/main/packages/sqlite/src/SqliteLayer.ts) +2. [SQLite storage adapter](https://github.com/bjacobso/triplex/blob/main/packages/sqlite/src/SqliteAdapter.ts) +3. [Shared SQL schema](https://github.com/bjacobso/triplex/blob/main/packages/sql/src/schema.ts) and + [migrations](https://github.com/bjacobso/triplex/blob/main/packages/sql/src/migrations.ts) +4. [Triples composition](https://github.com/bjacobso/triplex/blob/main/packages/sqlite/src/SqliteTriples.ts) + +Run `pnpm --filter @triplex-build/triplex-sqlite test` from a repository checkout for the adapter's +tests; `pnpm --filter triplex-integration-tests test` runs the shared backend corpus. Conformance is a +behavioral boundary, not a promise about every workload or storage failure. Backup, restore, +deployment migrations, and process lifecycle belong to your application. + +Continue with the [SQLite quickstart](/getting-started#use-durable-sqlite), or use this implementation +as a reference while [building a custom runtime](/custom-runtimes). diff --git a/docs/xtdb.md b/docs/xtdb.md new file mode 100644 index 0000000..b8bfe4c --- /dev/null +++ b/docs/xtdb.md @@ -0,0 +1,126 @@ +--- +title: Building Triplex on XTDB +description: A proposed Triplex adapter blueprint for XTDB's current SQL and bitemporal primitives, including non-interactive writes and stable read boundaries. +pageClass: backend-blueprint +--- + +Backend blueprint / XTDB
+ +# Building Triplex on XTDB + +A temporal database underneath a temporal application model. Keep the history, then connect it to the rules and work it explains.
+ +Proposed blueprint · No adapter exists
+ +This is a design to investigate for current XTDB, using SQL and its temporal primitives. +It is not an XTDB 1.x Datalog integration. No Triplex XTDB adapter or installable package exists; +the mappings below need a prototype and conformance evidence. + + + +## Why XTDB + +XTDB tracks valid and system time and exposes SQL and XTQL for querying historical data. +That makes late evidence and corrections a natural area to explore. +[XTDB's overview](https://docs.xtdb.com/intro/what-is-xtdb.html) describes its temporal and relational +model. + +Triplex would add its TypeScript-facing fact identity, configuration releases, attributed +command receipts, decision provenance, and derived work on top of an adapter. The shared temporal +vocabulary is a starting point; the exact representation and transaction behavior still need +to match Triplex. + +## Storage mapping + +Start with a compatibility representation: one row per Triplex assertion, retaining explicit +Triplex time fields and retraction metadata. Explore native temporal pushdown after that version +passes the behavioral corpus. + +| Triplex contract | Proposed XTDB representation | +| ---------------------------- | --------------------------------------------------------------------------------------------------- | +| Fact identity | Assertion row keyed by `TripleId`, retaining the original domain `EntityId` | +| Typed values | Fixed assertion columns for the value tag and typed payload | +| Recorded and valid intervals | Explicit Triplex timestamp fields initially; native temporal optimization requires a proven mapping | +| Retractions and history | Retained assertion payload with retraction transaction, timestamp, and position | +| Ordered commit boundary | Per-logical-database position record advanced with an atomic expected-position check | +| Retry identity | Scoped command-receipt record whose absence is checked in the submitted command | +| Attributed journal | Triplex transaction envelope and typed changes written alongside source facts | +| Datalog | A dedicated translator/executor from validated Triplex queries to XTDB SQL | + +The first version should keep adapter records visible independently of the domain fact's valid +interval. Applying native default valid-time filtering to a journal or receipt could hide it when +a user queries a different business date. Choose explicit temporal clauses for every adapter +table and test historical reads; do not rely on the backend's implicit "now" basis. + +An existing PostgreSQL client connection does not establish compatibility with Triplex's +PostgreSQL adapter. XTDB needs its own storage implementation, transaction handling, and verified +query dialect. The [SQLite reference](/sqlite) illustrates the responsibilities to cover. + +## Submit one atomic command + +XTDB write transactions are non-interactive: DML and assertions form a submitted atomic batch, +and cannot be interleaved with result-returning `SELECT`s. Triplex's Effect callback expects +reads and pending writes on one view. See [XTDB transaction consistency](https://docs.xtdb.com/about/txs-in-xtdb.html). + +One proposed bridge is to evaluate the callback against a stable read basis plus a staged overlay, +then submit an atomic batch: + +1. Read the logical database's position and command receipt on one consistent basis. +2. Evaluate enforcement and pending reads using that basis plus all staged assertion and + retraction changes. Build the journal and callback result without publishing them. +3. Submit an `ASSERT` guarding the expected position and command claim, then the position update, + receipt, facts, and journal in the same write batch. XTDB's + [SQL transaction reference](https://docs.xtdb.com/reference/main/sql/txs.html) describes `ASSERT`. +4. Confirm successful processing before returning the receipt. On a conflicting position, + reacquire a basis and re-evaluate. An existing command claim must return its original + transaction ID to core, which raises `CommandAlreadyCommittedError`; callers then resolve + the durable receipt with `transactionByCommand`. + +The guard must cover every source writer for that logical database. Prototype overlay reads, +retry behavior, interruption, and rejection before treating this approach as an implementation +of `withTransaction`. Native serial submission alone does not protect a constraint checked on an +older client-side read. + +## Preserve a stable read boundary + +XTDB distinguishes read snapshot tokens from await tokens used to ensure a node has processed +earlier writes. Reusing a snapshot and clock fixes the read basis; waiting for a write alone does +not pin future reads. [XTDB's basis documentation](https://docs.xtdb.com/about/txs-in-xtdb.html#repeatable-queries-basis) +explains the distinction. + +Map Triplex's commit position to a reproducible source boundary, keep that boundary on every +page, and ensure pooled connections observe confirmed commits. Define how a position is mapped +to a native snapshot or reproduced from retained assertion/retraction positions. Timestamp-only +pagination cannot establish the same contract. + +Maria's correction is the first useful exercise. Retain the original March 1 assertion, mark its +retraction, and assert the March 5 replacement atomically. Earlier recorded knowledge must still +show training on March 3, current knowledge must show the gap, and a cursor opened before the +correction must keep its original results. The governing release stays on the placement receipt. + +Native system time and Triplex's recorded time must be mapped explicitly, including timestamp +precision, same-millisecond commits, open interval ends, and imported or scheduled valid times. +Any native temporal optimization must preserve the reference behavior of direct reads, Datalog, +history, and derivation wakeups. + +## Build the first slice + +1. Specify the supported XTDB version, driver, typed codecs, table temporal policy, and physical + database isolation. Record the mapping of Triplex positions to consistent read boundaries. +2. Prototype atomic command submission before the query compiler: correction, failed assertion, + competing writers, duplicate commands, and read-after-write across two connections. +3. Implement `StorageAdapter` and a dedicated `QueryExecutor`, including validation, joins, + negation, aggregation, constrained recursion, stable ordering, pagination, and `explain`. +4. Compose with `Runtime.define` and `Capabilities.none`, then run + [runtime conformance](/custom-runtimes#correctness-contract-and-conformance) against fresh + storage. Add delayed processing, lost acknowledgements, restart, and upgrade exercises. + +Once the base layer is coherent, Triplex's configuration and derivation services can reuse it. +Snapshot and emission capabilities need their own providers and failure tests. + +[Open an XTDB design issue](https://github.com/bjacobso/triplex/issues/new) with the submission +prototype and read-boundary evidence, or compare the [Datomic proposal](/datomic). diff --git a/e2e/docs-home.spec.ts-snapshots/home-dark.png b/e2e/docs-home.spec.ts-snapshots/home-dark.png index 0b962a0..13b5502 100644 Binary files a/e2e/docs-home.spec.ts-snapshots/home-dark.png and b/e2e/docs-home.spec.ts-snapshots/home-dark.png differ diff --git a/e2e/docs-home.spec.ts-snapshots/home-light.png b/e2e/docs-home.spec.ts-snapshots/home-light.png index b3e67fb..209b2cd 100644 Binary files a/e2e/docs-home.spec.ts-snapshots/home-light.png and b/e2e/docs-home.spec.ts-snapshots/home-light.png differ