Skip to content

Test redacted Durable Object failure reports and flags through the router - #37

Draft
RhysSullivan wants to merge 6 commits into
d040/do-failure-reportsfrom
d040/do-failure-reports-tests
Draft

RhysSullivan wants to merge 6 commits into
d040/do-failure-reportsfrom
d040/do-failure-reports-tests

Conversation

@RhysSullivan

@RhysSullivan RhysSullivan commented Oct 8, 2026 •

Copy link
Copy Markdown

Tests for #36.

workerd runtime probe (runtime.test.ts). This test bundles the real Worker and Durable Object and runs them in Miniflare/workerd with a tail consumer. It records every tail event (the source of Workers Logs and Issues, uncaught exception events included), workerd's raw stdout and stderr, each response, and the Analytics Engine data points the code writes. The probe's FAILURES dataset turns each point into a tagged log line. Seven faults are injected, each with a synthetic secret in the message, the error name and the instance URL:

  • a flagged and a plain storage read failure;
  • a storage write failure under /_emulate/seed and /_emulate/credentials;
  • idFromName throwing;
  • an overloaded stub;
  • a Worker config read throwing.

The test asserts:

  • each fault's status and report;
  • each fault's exact tail events, awaited before anything is checked: one Worker event, plus one object event when the fault is inside the object. That makes 11 events in all, each carrying its own fault's header;
  • every event's outcome is ok;
  • there are no exception entries;
  • the code writes nothing to the console: every log line is one of the probe's data points, one per failure;
  • the secret appears nowhere in the complete tail events (nothing stripped), the runtime output, the data points or the responses;
  • the instance name appears in exactly three event fields: the client's event.request.url, and the x-emulator-instance / x-emulator-base-url headers our Worker sets on its request to the object. Cloudflare can store an invocation's request with telemetry, which is why observability is off. Any new field carrying it fails the test.

It adds miniflare and esbuild as dev dependencies of @emulators/cloudflare. They are pinned to versions already in the store, because of minimumReleaseAge.

Telemetry settings. wrangler.jsonc sets observability, logs, traces and issues each to false, has no tail_consumers or logpush, and binds FAILURES to emulate_hosts_failures.

Worker. Every failure test also asserts that the console stays silent and that the stored data points carry no secret.

  • A retryable reset is answered once, with the full report, and recorded as exactly one data point without the instance hash. No replay.
  • A retryable WorkOS GET /oauth2/authorize is not replayed.
  • A synthetic token, sign-in code, address and instance name in the error message, path, query and authorization header do not appear in the response or the data point. The data point does not hold the instance hash either.
  • An unknown service, an unmatched path and a malformed cf-ray are reported as unknown, unmatched and null.
  • An overloaded failure gets a 503; a remote failure gets a 500.
  • A body that fails to read, or an idFromName that throws, is reported without reaching the object.
  • Any other Worker failure becomes worker_error.
  • An error named SYNTHETICtoken482913 is reported as other, a TypeError as TypeError, and a method SYNTHETICTOKEN as OTHER.

Durable Object. These tests use a real object over in-memory storage; each injected storage failure is one-shot.

  • A failure inside the object becomes the redacted emulator_error report, and a secret-bearing error thrown inside the router stays out of the response and the data point.
  • A flagged storage failure becomes a 503 emulator_unavailable report with its flags. This holds while the object loads, through the service router during reset, and through seed and credentials. It used to be rethrown.
  • A plain storage failure under seed or credentials becomes a 500 report with no secret, not a 400 that quotes the raw message.
  • An unsupported credential type is still a 400 with the emulator's message.
  • A secret-bearing error name is reported as other.
  • End to end through the Worker, the object's flagged report arrives as a 503 and is recorded once, by the object.

Core

  • routePattern returns the matched template.
  • createServer with rethrowUnexpectedErrors rethrows errors without a status and keeps API errors.
  • Seed and credential requests answer a ControlPlaneRejection with a 400 and send any other error to the error handler (500, or a rethrow with rethrowUnexpectedErrors).

Mutation checks. Each of these reverts makes the listed tests fail:

revert failing tests
DO rethrows flagged errors (the r2 code) 3 DO tests + runtime probe
Worker top-level catch removed worker_error test + runtime probe
body read and idFromName moved outside the catch body/addressing test + runtime probe
errorClass accepts any alphanumeric name 2 class tests + runtime probe
method accepts any uppercase word allowlist test
control plane answers any error with 400 seed/credential DO test + core test
console.error(report) put back runtime probe + 7 Worker/DO tests
instance hash written to the data point 4 Worker/DO tests
Worker adds the instance name in another header to the object runtime probe (new carrier field)
observability.enabled set to true telemetry settings test
Worker calls the object once more after a failure runtime probe (the earlier wait-then-200 ms check missed it)

Results: @emulators/cloudflare 40 passed and @emulators/core 88 passed. With the exact-event wait, the runtime probe passed 32 of 32 reruns: 20 one at a time and 12 with four running at once. One run of the cloudflare suite failed once, and its output wasn't captured. It did not recur in 85 reruns: 30 of the runtime probe alone, 31 of the full suite, and 24 with four suites running at once. pnpm build, format:check, type-check, lint and test pass across the repo.

@RhysSullivan
RhysSullivan added this pull request to stack #38 October 8, 2026 04:35
@RhysSullivan
RhysSullivan force-pushed the d040/do-failure-reports-tests branch 2 times, most recently from 3d0bb86 to 0315e92 Compare October 8, 2026 05:46
@RhysSullivan RhysSullivan changed the title Test Durable Object failure reports and the reset retry Test redacted Durable Object failure reports and flags through the router Oct 8, 2026
@RhysSullivan
RhysSullivan force-pushed the d040/do-failure-reports-tests branch from 0315e92 to d19ac97 Compare October 8, 2026 07:32
Replaces the retry tests: a retryable reset and a WorkOS authorize are
answered once. Synthetic tokens, codes, addresses and the instance name in
an error, path or header must not reach the response or the log. A one-shot
storage failure inside the router, the control plane's catches and the
object's load keeps its flags all the way to the Worker's 503.
A Miniflare probe runs the real Worker and Durable Object with injected
storage, addressing and stub failures, and checks every tail event,
workerd's output and each response: no exception events and no secret.
Unit tests cover secret-bearing error names and methods, body and
addressing failures, and plain seed and credential failures.
@RhysSullivan
RhysSullivan force-pushed the d040/do-failure-reports-tests branch from d19ac97 to 16d21da Compare October 8, 2026 09:44
@RhysSullivan
RhysSullivan force-pushed the d040/do-failure-reports-tests branch from d4e5cff to 5347c6b Compare October 8, 2026 10:15

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant