Skip to content

fix: recover E2EE opens that outrun the ledger and cut cold dispatch reads - #2762

Merged
zxch3n merged 4 commits into
mainfrom
fix/e2ee-convex-skew-prepare-dispatch
Nov 14, 2026
Merged

zxch3n merged 4 commits into
mainfrom
fix/e2ee-convex-skew-prepare-dispatch

Conversation

@zxch3n

@zxch3n zxch3n commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Related issue

Problem / pressure

  • skew-convex-slow (P0): when the Convex ledger feed is slower than Streams (e.g. 900 ms vs 20 ms), a frame from a rotated epoch + 1, or from a newly admitted device at the current epoch, reaches openContent before the local verified ledger has that record. Core authentication fails with ContentError('unauthorized'), which both hosts surfaced as terminal e2ee_invalid-ledger (a missing-read-key at a future epoch was also refused by keyPending). The room then never recovers.
  • Cold prepareDispatch (P1): a cold machine-sharing latest() does ledger refresh → readHead → readVersion in series (~1.2 s at 0 ms RTT, >10 s at 5 s RTT), and concurrent cold dispatches each repeat it. Warm (15-minute RAM grant) is 18–37 ms.

Summary

Ledger lag on content opens

  • Desktop (e2ee-runtime.mts) and portable (e2ee-client session/runtime) openContent now also route unauthorized through keyPending. For an active member device, a frame is retryable key-pending when its unverified epoch is at most 4 ahead of the verified epoch, or at the current epoch with an unknown author. Frames further ahead (forged 999999), older unknown authors, malformed frames and access loss keep their original failure; no network is touched by the classification.
  • The shared renderer adapter (desktop-e2ee-content.ts) already turns key-pending into one shared refresh({ installKeys: true, keyMode: 'receive' }) (ledger history, then this device's key) plus one retry, so the open returns plaintext transparently once the ledger catches up. Receptions are now spaced by KEY_PENDING_REFRESH_INTERVAL_MS = 3000 per content instance: waiting reads join the next reception (still inside the 30 s wait), so a peer streaming unverifiable frames costs at most ~0.33 refresh/s per workspace. Refresh runs outside openContent, so no host lock or semaphore is held across it.
  • @loro-dev/streams-crdt (external, 0.20.0) projects any provider open failure as payload_protection_error, retryable: false, then retries the live read with a ≥1 s floor until maxAttempts and disconnects. Resolving lag inside one adapter open keeps that path from being reached in the normal skew case; the adapter creates a fresh deadline per reception and never reuses an aborted signal.

Cold dispatch reads (e2ee-machine-sharing)

  • Concurrent cold latest(force = false) reads of one binding share one forked authenticated read; a read started before an invalidation is never joined after it, and forced/denial refreshes are unchanged.
  • readHead replies may carry an optional inlined version (cloud-api request includeVersion?: true, reply type and zod schema extended). It is used only for the exact head revision/digest, verified exactly like a readVersion reply, and stripped before the head is returned or pinned.
  • Desktop and CLI transports send includeVersion through makeInlineVersionReadHead; if the backend's validator refuses the field (ArgumentValidationError … includeVersion), the call is repeated once without it and that client instance stops asking. Other transports (private Web) are untouched since core never adds the field itself. No duplicate refresh existed: desktop passes a no-op options.refreshLedger and refreshes via the remote; CLI refreshes via options only.
  • After a reader opens a workspace, main warms other active members' machine grants once per runtime entry (≤16 machines, concurrency 2, errors ignored, outside the account lock, not awaited by open).

Docs: binding rules added to packages/e2ee-machine-sharing/AGENTS.md and apps/electron/src/main/services/AGENTS.md. The latter's ceiling is raised 1240 → 1290: the lag rule spans both hosts, e2ee-client/AGENTS.md is already at its 1400 ceiling and components/src/providers/AGENTS.md is at 1250/1250.

Visual explanation

sequenceDiagram
  participant S as Streams (20 ms)
  participant R as Renderer adapter
  participant H as Host openContent
  participant C as Convex ledger (900 ms)
  S->>R: frame epoch e+1 / new device
  R->>H: open
  H-->>R: key-pending (epoch ≤ e+4 or unknown author at e)
  Note over H: epoch > e+4 → original terminal failure
  R->>C: refresh(installKeys, receive) — shared, ≥3 s apart
  C-->>H: ingest ledger + install key
  R->>H: open (one retry)
  H-->>R: plaintext
Loading

Cold dispatch: N × (refresh → readHead → readVersion) becomes 1 × (refresh → readHead[+version]), shared by all concurrent callers; warm after workspace open.

Before / after

Before After
Lagging frame → e2ee_invalid-ledger, room stuck key-pending → refresh → plaintext
Cold dispatch: 3 serial RTTs per caller 2 RTTs once per binding (1 fewer once backend inlines), often pre-warmed

Test plan

  • apps/electron: node --import tsx --test e2ee-runtime / e2ee-account-runtime / e2ee-cloud-client — 299/299. New: owner opens a new device's frame before ingesting its admission → key-pending, epoch +4 → key-pending, +5 and 999999 → terminal, plaintext after ingest (fails with invalid-ledger without the fix); pre-warm reads another member's machine once per entry; includeVersion sent, and after a refusal sent once then dropped.
  • packages/e2ee-machine-sharing: vitest 33/33. New: three concurrent cold authorizes → one readHead, then warm; inlined original skips readVersion, mismatched inline falls back, restart from the pin works (both new tests fail with singleflight/inline disabled).
  • packages/components: desktop-e2ee-content + runtime-provider 126/126; new spacing test with fake timers.
  • packages/e2ee-client: unit 84/84; browser runtime.browser.test.ts 38/40 — updated case: lagging epoch is now key-pending, forged far-future epoch is not. The 2 failures (control route reopens after confirmed close before stale … , chromium+webkit, 120 s timeout) fail identically on main without this change.
  • apps/cli: cloud-cli-port + e2ee-runtime 210 passed. Typecheck: electron node+web, e2ee-machine-sharing, e2ee-client, cloud-api, components clean; CLI only reports the unbuilt acp-extension-claude submodule.
  • oxlint, oxfmt --check on changed files clean; pnpm run docs check passes. Not run: full pnpm check, macOS E2E, a real skewed Convex deployment.

Limits: the backend does not implement includeVersion yet (private), so the inline saving lands only after it deploys; the "immediate catch-up for a strictly higher epoch inside the cooldown" variant was not implemented — the plain 3 s spacing is the rate cap.

🤖 Generated with Claude Code

…reads

Streams can deliver a frame from a rotated epoch or a newly admitted device
before the slower Convex ledger feed. Desktop and portable hosts now classify
such frames (at most four epochs ahead, or an unknown author at the current
epoch, for an active member device) as retryable key-pending instead of a
terminal invalid-ledger; frames further ahead keep their original failure.
The shared content adapter's existing refresh+retry then catches up, and its
receptions are spaced to one per 3 s so unverifiable frames cannot drive a
refresh per frame.

Machine sharing: concurrent cold latest() reads of one binding share a single
authenticated read (never across an invalidation); readHead may inline the
active original, verified like readVersion and never persisted in the pin;
desktop and CLI transports ask for it and fall back once per client if the
backend refuses the field. Opening a workspace warms other members'
machine-sharing grants in the background.

Model: claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Model: claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zxch3n
zxch3n marked this pull request as ready for review November 14, 2026 06:41
@zxch3n
zxch3n merged commit 34b2a8c into main Nov 14, 2026
12 checks passed
@zxch3n
zxch3n deleted the fix/e2ee-convex-skew-prepare-dispatch branch November 14, 2026 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants