Upgrade the complete OpenFn pilot to Registry Stack 0.38

.github/workflows/ci.yml

@@ -25,7 +25,7 @@ jobs:
           cache: npm
       - name: Install the committed npm dependency graph
         run: npm ci --ignore-scripts --no-audit --no-fund
-      - name: Check all four 0.37 adaptors, event services and compiled jobs
+      - name: Check all four 0.38 adaptors, event services and compiled jobs
         run: npm run check
       - name: Check adaptor package contents
         run: npm run pack:dry-run
@@ -36,13 +36,17 @@ jobs:
     timeout-minutes: 60
     steps:
       - uses: actions/checkout@v4
-      - name: Build frozen 0.27 images, execute the native worker gate and set up the legacy pilot
+      - name: Build pinned 0.38 images, execute the native worker gate and set up the isolated pilot
         # Setup owns the build and suppresses raw upstream subprocess output.
         run: ./deployment/pilot.sh setup
       - name: Check private mount ownership
         run: ./deployment/check-permissions.sh
       - name: Run registration, governed correction and verified destination acceptance
         run: python3 pilot/smoke.py --timeout 300
+      - name: Run native agriculture authorization and minimized assertion acceptance
+        run: ./deployment/compose.sh run --rm --no-deps tools python3 /opt/pilot/agriculture/smoke.py
+      - name: Run native explicit Evidence SDK acceptance
+        run: ./deployment/compose.sh run --rm --no-deps sdk-tools
       - name: Verify retained records and exact replay after service restart
         run: python3 pilot/smoke.py --restart-recovery --timeout 300
       - name: Verify delivery retry, failed-step replay and queued-work recovery
@@ -51,14 +55,14 @@ jobs:
         if: failure()
         shell: bash
         run: |
-          if [ -f pilot/agriculture/.runtime/prepared.json ]; then
+          if [ -f pilot/agriculture/.runtime-0.38/prepared.json ]; then
             ./deployment/pilot.sh status
           fi
       - name: Stop application services and retain isolated test data until runner disposal
         if: always()
         shell: bash
         run: |
-          if [ -f pilot/agriculture/.runtime/prepared.json ]; then
+          if [ -f pilot/agriculture/.runtime-0.38/prepared.json ]; then
             if ! ./deployment/pilot.sh stop >/dev/null 2>&1; then
               echo 'Pilot stop did not complete. The isolated CI runner will be disposed.'
             fi

.gitignore

@@ -4,7 +4,11 @@ npm-debug.log*
 .DS_Store
 coverage/
 pilot/agriculture/.runtime/
+pilot/agriculture/.runtime-0.37/
+pilot/agriculture/.runtime-0.38/
 deployment/.env
+deployment/.env-0.37
+deployment/.env-0.38
 deployment/.build/
 deployment/.runtime/
 __pycache__/

CHANGELOG.md

@@ -14,10 +14,16 @@
 ### Changed
 
 - The Evidence, BREG, Casework and Relay V2 adaptors pin the published
-  `@registrystack/client` 0.37.0. Relay uses governed V2 resources and native
+  `@registrystack/client` 0.38.0. Relay uses governed V2 resources and native
   continuations; Casework uses current review-request bindings and results.
-  The retained pilot's Registry Stack images, bootstrap tools and separately
-  locked client remain at 0.27.0.
+- The container pilot uses matching Registry Stack 0.38 clients, tools and
+  packages, ThunderID identity, Casework correction review, and explicit BREG
+  application. Evidence uses persistent OpenBao Transit signing and local TLS
+  origins. Its new configuration and volumes are separate from the retained
+  0.27 data.
+- The event bridge's explicit `hook-envelope-v1` mode validates canonical
+  0.38 events and returns the native notification acknowledgement after
+  durable OpenFn acceptance. The legacy body mode remains available.
 
 ### Removed
 

README.md

@@ -4,34 +4,34 @@
 configuration may change before 1.0.
 
 The BREG, Evidence, Relay V2 and Casework adaptors target the published
-`@registrystack/client@0.37.0`. Use matching 0.37 runtimes and tools for new
+`@registrystack/client@0.38.0`. Use matching 0.38 runtimes and tools for new
 integrations. Run `npm run check` to verify all four adaptors and their OpenFn
 composition. Relay and Casework now use their current native APIs; consult their
 package READMEs when migrating older jobs.
 
-The existing container pilot below is a retained **Registry Stack 0.27.0**
-deployment with its own [dependency lock](deployment/package-lock.json).
-It demonstrates the older Mint and BREG-owned review contracts. Its build does
-not install the current 0.37 client graph. Upgrading that deployment requires
-ThunderID, Casework review, and regenerated runtime/package configuration;
-changing image tags alone is insufficient. Existing pilot data is preserved.
+The container pilot uses matching **Registry Stack 0.38.0** clients, runtimes,
+tools and authored packages, ThunderID for identity, and Casework for correction
+review. It creates a fresh deployment with separate databases and private
+configuration. The previous 0.27 pilot's `pilot/agriculture/.runtime` directory
+and `registry-openfn-pilot` volumes are retained and are not migrated or reused.
 
 A self-hosted synthetic pilot connecting upstream OpenFn Lightning to Registry
 Stack Base Registry Engine (BREG) and signed Evidence. It demonstrates registration,
 reviewed correction, and an idempotent downstream update containing a verified
 Boolean. All supplied people, holding names, identifiers and credentials are for
 an isolated local pilot.
 
-The retained 0.27 pilot uses:
+The pilot uses:
 
 - [Registry Evidence adaptor](packages/registry-evidence), using the `evidence`
-  namespace of the published `@registrystack/client@0.27.0`.
+  namespace of the published `@registrystack/client@0.38.0`.
 - [Registry BREG adaptor](packages/registry-breg), using the same package's `breg`
   namespace for metadata-selected writes, exact lookups and lifecycle actions.
 - Upstream Lightning 2.18.2 and websocket worker 1.29.0, packaged with Node
   24.19.0 on glibc and local adaptors. No OpenFn source patch is required.
-- Separate BREG and Lightning PostgreSQL databases, Mint, Evidence, an
-  authenticated event bridge and a small SQLite-backed destination.
+- Separate BREG, Casework and Lightning PostgreSQL databases, ThunderID,
+  Evidence with an OpenBao Transit signer, an authenticated event bridge and a
+  small SQLite-backed destination.
 
 [Image identities](deployment/images.lock.json), the npm lockfile, and bootstrap
 [tool checksums](deployment/tools.sha256) pin the delivered dependencies.
@@ -41,11 +41,12 @@ The retained 0.27 pilot uses:
 | Workflow | Authoritative result | Response and boundary |
 | --- | --- | --- |
 | Registration | Creates one Farm using a stable submission key. | The authenticated synchronous webhook returns its UUID. An exact retry returns the same record; changed content with the same key conflicts. |
-| Correction | Creates and submits a name-correction request. | Returns its request UUID. A separate reviewer approves, then explicitly applies it. Submission and approval leave the Farm unchanged. |
+| Correction | Creates and submits a name-correction request. | Returns its request UUID. A separate reviewer approves in Casework, then explicitly applies through BREG. Submission and approval leave the Farm unchanged. |
 | Committed Farm event | Verifies HMAC delivery, requests signed registration Evidence, then updates the destination. | Asynchronous. Only the verified `registered: true`, record reference and revision reach the destination. |
 
-BREG owns records, permissions, review, revisions, ETags and the transactional
-outbox. OpenFn orchestrates these contracts. The Evidence source performs an
+BREG owns records, permissions, proposals, revisions, ETags and the transactional
+outbox. Casework owns review decisions; BREG accepts only the approved result
+for the exact submitted proposal. OpenFn orchestrates these contracts. The Evidence source performs an
 exact authorized Farm lookup and returns only `localIdentifier`; Evidence signs
 one registration Boolean. The assertion makes no claim about eligibility,
 ownership, activity or land rights. An unresolved holding produces unavailable
@@ -69,7 +70,7 @@ The released Lightning and worker images are `linux/amd64`; Apple Silicon uses
 Docker emulation. Native arm64 adaptor loading does not establish arm64 worker
 support. See the [container packaging notes](deployment/README.md) for the glibc
 worker packaging and architecture limits. Allow enough Docker disk and memory
-for two databases and the upstream application builds.
+for three databases and the upstream application builds.
 
 From this repository:
 
@@ -81,32 +82,33 @@ python3 pilot/smoke.py
 
 Setup builds the pinned images, runs the real native adaptor worker check,
 generates private inputs and Evidence fixtures, initializes the isolated BREG
-package, applies upstream Lightning migrations, and provisions the pilot
+and Casework packages, initializes the persistent Transit signer, applies
+upstream Lightning migrations, and provisions the pilot
 operator and credentials. Webhooks stay disabled until authentication is
 attached and checked. A failed step stops setup and identifies the step without
 printing raw service output. Setup is not a statement that the full acceptance
 journey passed; the smoke must finish with `"status": "passed"`.
 
-Open [Lightning](http://localhost:4000). The generated login is in
-`pilot/agriculture/.runtime/openfn/secrets/operator.json`; open that private file
+Open [Lightning](http://localhost:4010). The generated login is in
+`pilot/agriculture/.runtime-0.38/openfn/secrets/operator.json`; open that private file
 locally to read the email and password. Do not paste its contents into tickets,
 job expressions or logs. There is no shared default password. The same private
 folder contains the operator API token, separate intake and committed-event keys,
 credential associations and
 `project.json`, which records the actual project and trigger UUIDs.
 
 The smoke creates a fresh synthetic identity once and stores it in owner-only
-`pilot/agriculture/.runtime/smoke-state.json`. It exercises registration,
-unchanged and conflicting retries, correction submission, separate reviewer
-approval/application, event-driven Evidence and destination effects. Repeating
+`pilot/agriculture/.runtime-0.38/smoke-state.json`. It exercises registration,
+unchanged and conflicting retries, correction submission, separate Casework reviewer
+approval/BREG application, event-driven Evidence and destination effects. Repeating
 it reuses its original keys and records. It does not clear data or make a new
 registration on every run. Direct native HTTP checks are also available in the
 [agriculture guide](pilot/agriculture/README.md).
 
 ## Forms and manual review
 
 The generated project has registration and correction webhook endpoints at
-`http://localhost:4000/i/TRIGGER_UUID`. Read their UUIDs from private
+`http://localhost:4010/i/TRIGGER_UUID`. Read their UUIDs from private
 `project.json`, and send the intake `webhook-api-key` in `x-api-key`. Keep that key
 in the form integration's credential store. Never give form clients the separate
 `committed-api-key`; it belongs to the authenticated event bridge. Do not place
@@ -139,9 +141,11 @@ request UUID, inspect the displayed synthetic proposal, then decide:
   python3 /opt/pilot/agriculture/review.py apply REQUEST_UUID
 ```
 
-The reviewer helper binds the action to its exact inspected proposal version,
-effect digest and lifecycle ETag. A stale snapshot is refused. A record ETag is
-not a lifecycle ETag. The smoke invokes these explicit steps only for the
+The reviewer helper binds the Casework task to its inspected source context and
+revision, and the BREG application to its exact proposal version, effect digest
+and lifecycle ETag. A stale snapshot is refused. A record ETag is
+not a lifecycle ETag. Approval leaves BREG in `submitted` with an approved
+external review result; only application moves it to `applied`. The smoke invokes these explicit steps only for the
 synthetic correction it created and verified.
 
 ## Stop, resume and restart recovery
@@ -159,15 +163,15 @@ Stop retains named volumes, Registry history and private configuration. Start
 checks the retained package and reapplies the supported upstream startup path.
 Restart recovery restarts existing application services and verifies the same
 records, correction and destination effects. It does not recreate databases or
-Mint's shared network namespace.
+ThunderID's shared network namespace.
 
 Always use `deployment/compose.sh` for lower-level commands. It fixes the Compose
-project to `registry-openfn-pilot` and preserves the operator UID/GID in ignored
-`deployment/.env`. If the pilot moves to another operator, transfer ownership
+project to `registry-openfn-pilot-038` and preserves the operator UID/GID in ignored
+`deployment/.env-0.38`. If the pilot moves to another operator, transfer ownership
 of its retained private files explicitly and update that file. Do not solve a
 permission failure by making keys world-readable.
 
-Reset is separate and permanently destroys this synthetic pilot's containers,
+Reset is separate and permanently destroys only the current 0.38 pilot's containers,
 volumes, records, history and generated private keys:
 
 ```sh
@@ -187,7 +191,7 @@ remove a partially activated database to make readiness pass.
 | Idempotency or ETag conflict | Confirm whether the payload changed or the baseline is stale. Reuse the original payload for an exact replay. For a deliberate new correction, review current state and create a new submission; do not force a write with a stale ETag. |
 | Already-submitted correction | The workflow rereads the request and verifies its original content before reporting `already_submitted`. A missing action alone is not proof of success. |
 | Rejected correction | Read the review decision. A repeat of the original submission reports the existing request ID; inspect its current BREG state and decision. It does not resubmit it. Submit a revised proposal with a new submission ID after review, or use a currently advertised revise action through the BREG adaptor. The pilot form does not automatically revise or approve. |
-| Authentication failure or expired credentials | Private-key JWT obtains fresh short-lived access tokens through Mint. Check Mint readiness, host clock, client registration and private-key permissions. Correct the OpenFn credential/profile when keys change. Do not paste a temporary access token into job source or rotate unrelated keys during an exact retry. |
+| Authentication failure or expired credentials | Private-key JWT obtains fresh short-lived access tokens through ThunderID. Check issuer readiness, host clock, client registration and private-key permissions. Correct the OpenFn credential/profile when keys change. Do not paste a temporary access token into job source or rotate unrelated keys during an exact retry. |
 | Evidence unavailable, denied or unverifiable | Leave the destination unchanged. Check the authorized exact source lookup, selected requirement, requester credentials and reviewed trust/profile configuration. Unavailable is not a signed negative. Retry the original failed committed-event work order only after the cause is resolved. |
 | Failed committed-event workflow after webhook acknowledgement | Use Lightning's failed work order and original input for recovery. BREG's successful HTTP handoff does not imply that the asynchronous workflow succeeded. Destination deduplication makes an exact successful effect replay safe. |
 | Pending or dead-letter BREG delivery | Restore bridge/OpenFn availability, inspect delivery status, then explicitly replay the eligible retained dead letter. Preserve event identity and use the listed generation. Do not edit the outbox or manufacture a new event ID. |
@@ -218,16 +222,16 @@ This deliberately interrupts selected services and then restores them. It checks
 BREG retry/dead-letter replay before OpenFn accepts a delivery, retry of a failed
 destination step after acceptance, and retained queued work across application
 restarts. Each scenario keeps a separate synthetic identity in private
-`.runtime/recovery-state.json`; it removes no records or volumes. Select
+`.runtime-0.38/recovery-state.json`; it removes no records or volumes. Select
 `before-accept`, `after-accept`, or `pending-restart` instead of `all` for one case.
 
 For value-free BREG delivery status and an explicit eligible replay:
 
 ```sh
 ./deployment/compose.sh run --rm --no-deps tools bregctl webhook list \
-  --runtime-config /config/breg/runtime.json
+  --runtime-config /config/breg/runtime.yaml
 ./deployment/compose.sh run --rm --no-deps tools bregctl webhook replay \
-  --runtime-config /config/breg/runtime.json \
+  --runtime-config /config/breg/runtime.yaml \
   --event-id EVENT_UUID --delivery-id DELIVERY_ID --expected-generation GENERATION
 ```
 
@@ -239,7 +243,7 @@ uses the stable event ID, so that change does not duplicate an applied effect.
 ## Private data and diagnostics
 
 This is a loopback-only, supervised local deployment, not a production internet
-configuration. Host ports 4000, 4001 and 4002 bind only to loopback. The tools
+configuration. Host ports 4010, 4011, 4012 and 4013 bind only to loopback. The tools
 service has privileged package, database and reviewer material and is an
 operator surface. Retain access controls on the Docker daemon and host files.
 

deployment/Lightning.Dockerfile

@@ -1,6 +1,6 @@
 # syntax=docker/dockerfile:1
 ARG LIGHTNING_IMAGE=openfn/lightning:v2.18.2@sha256:a2327173ece7b5ac4904744acf63f928dd2cbe496788d31266060e85bd359203
-ARG ADAPTORS_IMAGE=registry-openfn-worker:pilot
+ARG ADAPTORS_IMAGE=registry-openfn-worker:pilot-038
 FROM ${ADAPTORS_IMAGE} AS adaptors
 FROM ${LIGHTNING_IMAGE}
 COPY --from=adaptors --chown=lightning:root /opt/registry-adaptors /opt/registry-adaptors