Report Durable Object failures without secrets instead of an opaque 500

[RhysSullivan]

Problem

Executor's CI loses about one e2e job in 20 on busy days to an HTTP 500 from emulators.dev on /_emulate/credentials, /_emulate/reset or /_emulate/seed. From 30 Sep to 8 Oct there were 11 such failures; 9 of them were on 7 Oct, and 3 of 250 main CI runs went red from them. The client gets Cloudflare's opaque "Worker threw exception" page, so neither side could say what failed. emulate-hosts has no Workers Logs.

Cause

Cloudflare analytics for emulate-hosts over the same window show two kinds of failure:

1. The Durable Object was never reached (42 exceptions in 22 minutes). The stub.fetch from the Worker to the instance's Durable Object throws after about 12 s, and the Durable Object records no invocation at all. Every one of the 11 CI failures falls in one of these minutes (chart below). They come from LAX, SJC and PHX; 7 Oct 20:38 to 23:26 was all PHX, which is where GitHub's runners enter Cloudflare. The rest of the Worker's traffic in that window succeeded (PHX: about 10k requests). This is a transient fault on Cloudflare's side between the Worker and the object. Without logs its exact message is not knowable yet.
2. The emulator threw (55 exceptions; at least some are Cloudflare relocating the object mid-request, see below). These are Durable Object scriptThrewException (52, including 36 in one minute on one Autumn instance on 6 Oct) and exceededMemory (3, one Resend instance). None of them coincides with a CI failure.

The Worker didn't catch either kind, so both reached the client as a bare 500.

Solution

• No error escapes the Worker or the Durable Object. Cloudflare records an uncaught exception with its message, stack and the request URL. So both boundaries answer every failure with a JSON report (diagnostics.ts) and never throw:
  • Durable Object (durable-object.ts). Its own exceptions become emulator_error (500). Cloudflare's flagged failures, such as cannot access storage because object has moved to a different machine, become emulator_unavailable with retryable/overloaded set and a 503. They used to be rethrown so the Worker could see their flags, and that rethrow is what Cloudflare logged. The Worker now passes the object's report through unchanged.
  • Worker (worker.ts). Reading the body, idFromName and stub.fetch are all inside one catch. A failure there becomes emulator_unavailable: 503 when flagged, 500 otherwise (for example a remote memory kill). Any other failure in the Worker's router becomes worker_error (500).
• No retry. A retryable flag does not prove the object never ran the request. Replaying a reset changes Google's signing key and PostHog's token and erases a seed written in between. Replaying WorkOS's GET /oauth2/authorize issues a second code. Each failure is answered once. A retry would need request-id dedupe inside the object first.
• Report fields come from fixed lists. An instance URL is the only access control for its emulator, and paths, messages, error names and stacks can quote tokens, codes and addresses. No field copies text from the request or the error:
  • error: emulator_unavailable, emulator_error or worker_error;
  • service: a key of the service registry, else unknown;
  • instanceId: the first 12 hex digits of SHA-256(service:instance), or null for an unknown service. Generated instance names end in 96 random bits, so their hashes can't be enumerated. The hash does not hide a predictable or low-entropy name, such as one a caller chose or a legacy fixed name: hashing guesses confirms it.
  • method: one of the 7 standard methods, else OTHER;
  • route: a template the service's router declares (such as /domains/:id) or one of the object's own control paths, else unmatched or unknown;
  • errorClass: an allowlisted name (the built-in JS error classes and the runtime's DOMException names), else other;
  • the retryable, overloaded and remote flags;
  • ray, when the cf-ray header has a ray's format.
• Seed and credential failures (core). The control plane answers 400 invalid_seed / 400 unsupported only for a typed ControlPlaneRejection, which the emulators throw for their own refusals (an unsupported credential type, an invalid PlanetScale client). Any other error goes to the app's error handler. On Cloudflare that means the redacted 500 report; before, a storage failure came back as a 400 with its raw message.
• createServer gains rethrowUnexpectedErrors. The Durable Object sets it, so a route error without an HTTP status reaches the object's report instead of a 500 that carries the raw message. API errors such as 404s are answered as before.
• Local emulate. The core router no longer has a flagged-error passthrough, so local routing is unchanged. One local change remains: an unexpected (untyped) error from a seed or credential request is now a 500 from the error handler, not a 400. Its message is still shown locally.
• Observability stays off, and reports go to Analytics Engine. Workers Issues keeps every 5xx response and every console.error with the invocation's URL, whether or not Workers Logs is on (Cloudflare: "Issues detection does not require Workers Logs or tracing"). For an instance, that URL contains the instance name. These settings are not versioned, so wrangler rollback leaves them as they are. So:

  • wrangler.jsonc sets observability.enabled, logs, traces and issues all to false. That makes each deploy also turn off anything enabled from the dashboard. Production has observability: null today, so this deploy changes no setting.

  • The failure path in the Worker and the Durable Object (worker.ts, durable-object.ts, diagnostics.ts) writes nothing to the console. Bundled provider handlers still can: the GitHub and Vercel OAuth authorize routes console.warn the caller's redirect_uri on a mismatch (github/src/routes/oauth.ts:112, vercel/src/routes/oauth.ts:69), and core's debug() logs when DEBUG is set, which it isn't in the Worker. With telemetry off, Cloudflare stores none of these lines. Those handlers are local-dev diagnostics shared with the CLI, so this PR leaves them alone.

  • Each failure is written once, by whichever boundary answered it, to the emulate_hosts_failures Analytics Engine dataset (FAILURES binding). A data point stores only what is written to it. Every row reads back with all 20 blob and 20 double slots, plus index1, timestamp, dataset and _sample_interval. The slots a point doesn't write read back as '' and 0. The written slots are:

    • index: service;
    • blobs: error, service, method, route, errorClass, ray;
    • doubles: status, retryable, overloaded, remote.

    The instance hash is left out, and the ray joins a data point to the client's copy of the report.

  • Analytics Engine samples writes, and samples again at query time, so SUM(_sample_interval) is an estimate, and no single record is guaranteed. Rows are kept for 3 months. Reading them through SQL needs an account-scoped token with Account Analytics Read.

A Worker-side emulator_unavailable report says the call to the object failed. It does not prove the object never ran the request, which is also why there is no retry.

Not changed: why Cloudflare sometimes cannot reach the object. That is outside this code; the logs will show its flags the next time it happens. Deploying this stack is a production change to emulators.dev and has not been done.

Runtime probe (workerd)

src/__tests__/runtime.test.ts in #37 bundles the real Worker and Durable Object, runs them in Miniflare/workerd, and captures every tail event (the source of Workers Logs, exception events included) plus workerd's raw stdout and stderr. It injects 7 faults, each with a synthetic secret in the error message, the error name and the instance URL:

• a flagged and a plain storage failure while the object loads;
• a plain storage write failure under /_emulate/seed and /_emulate/credentials;
• idFromName throwing;
• an overloaded stub;
• a Worker config read throwing.

	DO events with outcome: exception	exception entries	secret or instance suffix in exceptions, logs, stdio or responses
r2 head behaviour (DO rethrows flagged errors)	1	1: name: "SYNTHETICtoken482913", message with the full instance URL, full stack	yes
this head	0	0	no

The probe waits for each fault's exact set of tail events before checking: a Worker event, plus an object event when the fault is inside the object (11 in all). It checks the complete events, with nothing stripped. The instance name appears in exactly three fields of the invocations' requests: the client's event.request.url, and the x-emulator-instance / x-emulator-base-url headers. Our Worker sets those headers on its request to the object; they are not platform metadata. Cloudflare can store an invocation's request with each telemetry record. The URL and these headers stay out of Cloudflare's stores only because telemetry is off.

Saved-record proof (isolated deployment, synthetic data)

A throwaway Worker, d040-probe-r3 on workers.dev, ran this code with the same fault injection. Everything was read back through the Telemetry API (real-time-issues and cloudflare-workers datasets), the Issues API (/workers/observability/issues and each issue's occurrences) and Analytics Engine SQL. The Worker has since been deleted. Each run used a fresh synthetic instance and sent 7 faults plus 2 ordinary requests.

run	settings	Workers Logs events	Issues occurrences	instance name kept	secret kept	AE points
r2 code + Issues on	logs on, invocation logs off, issues on	0	11 (7 error-log, 4 http-status)	yes: invocation.url/path in every Worker occurrence	no	n/a
this code (2cedc9b), deployed over settings turned on from the dashboard	deploy read back observability: null	0	0	no	no	7 of 7: written slots allowlisted, every other slot '' or 0

• Issues keeps the URL. Every Worker-side occurrence kept invocation.url with the instance path, for both error-log (console.error) and http-status (a handled 5xx). Query strings and x-emulator-* headers were not kept; the request headers kept were cf-ipcountry, cf-ray, content-type, content-length and user-agent.
• Propagation. One extra occurrence came from the previous version about 5 s after wrangler deploy returned, while that version was still rolling out.

Post-deploy checks

These checks roll back only on evidence of disclosure. Ingestion is reported as advice. They claim only what the data can decide:

• Decidable: the telemetry settings; every stored row against the allowlist; every failure report the procedure receives; whether a given ray has a row.
• Not decidable: completeness. Analytics Engine samples at write time and again at query time. A row's _sample_interval is a statistical weight that names no request (sampling). So a ray with a row is recorded, and a ray without one is not found: compatible with sampling, completeness unverified. Nearby rows never confirm or excuse it. SUM(_sample_interval) is reported as an estimate.

The scripts are in the D-040 findings data, outside this repo: checkpoint.py, positive_control.sh, report_body.py, harness_failures.py, ae_check_sql.py and ae_query.sh. Cloudflare reads need a token with Workers Scripts Read and Account Analytics Read.

Deploy.

1. Before deploying, GET /accounts/{account}/workers/scripts/emulate-hosts/settings reads observability: null, and 87d64cf7 is at 100%.
2. Deploy from packages/@emulators/cloudflare with npx wrangler@4.148.0 deploy. Wrangler 4.134 or later is needed, because older versions don't send the issues key.
3. T0 is when the deployments list shows the new version at 100%. Record the commit it was built from. Start 60 s after T0: on the probe, the old version still answered about 5 s after wrangler deploy returned.

At each checkpoint (T0, 24 h, 7 days):

1. Settings. GET …/settings, saved as S.json. It gives the observability settings and the bindings.

2. Positive control. positive_control.sh COMMIT C.json 3 120 (see below).

3. Smoke checks, at T0 only. Use a fresh instance with a synthetic name and synthetic secrets. checkpoint.py requires one result for each of these names (SMOKE_CHECKS):

   • create-instance;
   • credentials-api-key and credentials-oauth-client (/_emulate/credentials);
   • seed (/_emulate/seed);
   • resend-domain-create, then resend-domain-read: POST /domains with a synthetic name, then GET /domains/{id}, which returns 200;
   • reset (/_emulate/reset), then reset-cleared: the same GET /domains/{id} returns 404. Reset reapplies the instance's seed, so only state created through the API shows that reset ran;
   • github-provider and google-provider: a provider route on each;
   • oauth-authorize, oauth-consent and oauth-token;
   • workos-authorize-ledger: after one WorkOS GET /oauth2/authorize, the instance ledger shows exactly one entry. The ledger skips /_emulate requests and reset clears it, so it isn't checked for reset;
   • openid-configuration (/.well-known/openid-configuration);
   • unsupported-credential: 400 with its message;
   • unknown-service and unmatched-path: 404.

   Record each result in SMOKE.json: the expected status, then the first run's status, whether it was readable, and whether the body was right (bodyOk). Opaque means a 5xx without emulate's report, or no response. If a readable run is wrong, repeat that check on a second fresh instance and record it as rerun; otherwise set rerun to null.

   • Pass every non-2xx body through report_body.py check BODY --routes routes-2cedc9b.json --forbid SECRETS, where SECRETS lists the synthetic names and secrets. Save each result as B<n>.json. (2xx bodies return the instance's own data by design.)
   • Then run one executor-next e2e all job.

4. Harness, at 24 h and 7 days. Run harness_failures.py ZIPDIR T0 H.json over the CI, Cloud tests on main and Deploy job logs since T0. It reads the executor-next#2199 lines.

5. Queries. Run these at least W = 15 min after the last control:

   • ae_check_sql.py sql writes violations.sql (the allowlist query, below) and routes.sql.

   • checkpoint.py sql --dataset emulate_hosts_failures --deploy T0 Q/ C*.json H.json, given every control file so far, writes:

     • lookup.sql: one export of the rows whose ray is an observed ray;
     • natural.sql: the rows without any control's ray, with SUM(_sample_interval).

     Rays are compared by their 16-hex id on both sides (substring(blob6, 1, 16)). The Worker stores cf-ray as it arrives, with or without the colo suffix.

   • Run each query with ae_query.sh.

6. Decide. Run checkpoint.py check --name T0|24h|7d --now … --settings S.json --violations V.out --routes routes-2cedc9b.json --routes-result R.out --lookup L.out --natural N.out --control C.json [--harness H.json] [--body-check B<n>.json …] [--smoke SMOKE.json] (--smoke is required at T0). It refuses to run before the last control is 15 minutes old, so a checkpoint is never left pending.

Rollback triggers (only these)

1. Telemetry on. The settings read-back has any true under observability: logs, invocation logs, traces or Issues.

2. A stored row outside the allowlist. The violations query returns a row, or ae_check_sql.py routes rejects a (service, route) pair or a ray. This query must return 0 rows:

   SELECT * FROM emulate_hosts_failures
   WHERE NOT (
     blob1 IN ('emulator_unavailable', 'emulator_error', 'worker_error')
     AND blob2 IN ('apple', 'autumn', 'aws', 'clerk', 'context', 'github', 'gitlab', 'google', 'mcp', 'microsoft', 'mongoatlas', 'okta', 'planetscale', 'posthog', 'resend', 'slack', 'spotify', 'stripe', 'vercel', 'workos', 'x', 'unknown')
     AND index1 = blob2
     AND blob3 IN ('GET', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE', 'OPTIONS', 'OTHER')
     AND (blob4 IN ('unmatched', 'unknown') OR startsWith(blob4, '/'))
     AND blob5 IN ('Error', 'TypeError', 'RangeError', 'SyntaxError', 'ReferenceError', 'EvalError', 'URIError', 'AggregateError', 'AbortError', 'TimeoutError', 'DataCloneError', 'QuotaExceededError', 'InvalidStateError', 'NetworkError', 'OperationError', 'other')
     AND length(blob6) IN (0, 16, 20)
     AND double1 IN (500.0, 503.0)
     AND double2 IN (0.0, 1.0)
     AND double3 IN (0.0, 1.0)
     AND double4 IN (0.0, 1.0)
     AND (double1 = 503.0) = (double2 = 1.0 OR double3 = 1.0)
     AND blob7 = '' AND blob8 = '' AND blob9 = '' AND blob10 = '' AND blob11 = '' AND blob12 = '' AND blob13 = ''
     AND blob14 = '' AND blob15 = '' AND blob16 = '' AND blob17 = '' AND blob18 = '' AND blob19 = '' AND blob20 = ''
     AND double5 = 0.0 AND double6 = 0.0 AND double7 = 0.0 AND double8 = 0.0 AND double9 = 0.0 AND double10 = 0.0
     AND double11 = 0.0 AND double12 = 0.0 AND double13 = 0.0 AND double14 = 0.0 AND double15 = 0.0 AND double16 = 0.0
     AND double17 = 0.0 AND double18 = 0.0 AND double19 = 0.0 AND double20 = 0.0
   )
   ORDER BY timestamp
   FORMAT JSON

   AE SQL has no regular expressions and rejects queries over 10,000 characters. So the exact route (one of the 1,051 templates the routers declare at 2cedc9b, or /__seed, /__reset, /__token, unmatched or unknown) and the ray format (^[0-9a-f]{16}(-[A-Z]{3})?$) are checked by ae_check_sql.py routes. Regenerate the template list if the deployed commit isn't 2cedc9b.

3. A failure report with a field it shouldn't have. A report received by a control or a smoke check has an error outside its three values, a key outside its 10 fields, a value outside its list or format, or a synthetic secret or instance name. A body counts as a report by its structure: a JSON object whose error is one of the three, or that has at least 5 of the 10 report keys. So a changed error can't hide a report. Secrets are searched in the raw bytes and in every decoded JSON key and string, so a JSON escape can't hide one. Reports are this PR's surface. A secret in any other body, such as an API error unchanged by this PR, is marked investigate, not a trigger.

4. A reproducible smoke-check regression. A T0 smoke check returns the wrong status or body with a readable (non-opaque) response, and the same check is wrong again, readably, on a second fresh instance. Emulators broken for every e2e job is a regression, not an ingestion question. checkpoint.py decides it from SMOKE.json. A failure that passes on the rerun, a failure that wasn't rerun, an opaque answer on either run, and a missing or malformed result all make the checkpoint unverified.

Never a trigger:

• Ingestion problems: a missing binding, rows not found, or the control unavailable.
• Opaque 5xx: from a control or the harness. Cloudflare can answer before this code runs.
• A query or read that didn't run.

The CI harness (#2199) logs only values it vouches for, so it can't show trigger 3. A logged value outside the lists would be a #2199 bug.

The positive control

• Temporary, pinned to 2cedc9b. It creates one fresh Resend instance (prefix d040-control, generated suffix) and posts {"domains":1} to /_emulate/seed three times, 2 minutes apart.
• Why it fails safely. Resend's seedFromConfig iterates domains and throws a TypeError before any insert. That gives a 500 emulator_error and one row, with nothing stored. In workerd, the unmodified Worker and Durable Object gave exactly that response and row. On production today (87d64cf7) the same request is a 400.
• It relies on a bug. A malformed seed should be a 400; this is reported separately and is not fixed here. Once the bug is fixed, the control answers 400 and is unavailable. That is not a deploy failure. The script refuses to run against any other commit.
• Each response is classified:
  • ok: 500 emulator_error, resend, POST, /_emulate/seed, TypeError, no flags, the instance's own hash, and a cf-ray header and a report ray with the same 16-hex id;
  • unavailable: 400 invalid_seed;
  • opaque: a 5xx without a report, or no response within the 30 s curl deadline. This is inconclusive: the script retries it at most twice, 30 s apart. If it persists, check Workers analytics for that minute (exceptions and Durable Object invocations).
  • invalid: anything else;
  • disclosure: the report failed the body check.
• A complete batch has all three slots ending ok.
• Load and data. Up to 10 requests (one create, then 3 slots of up to 3 attempts) and 9 point writes per checkpoint before sampling: one per seed attempt that reaches the code, so 3 for a clean batch. All on a throwaway instance. The instance name is never written to the output: a record whose body fails the check keeps no report values.

Decision table

At each checkpoint:

outcome	when	action
rollback-trigger	any trigger above	Roll back now (below). The final verdict is "rolled back".
unverified	No trigger, but at least one of these: • a safety read or query didn't run; • the control batch isn't complete (unavailable, opaque after retries, invalid, or a slot missing); • a control has no row with its exact ray, or its row has other fields; • the lookup failed; • the binding isn't listed; • a body is marked investigate; • a smoke check is missing, failed once but passed on a second fresh instance, failed without a rerun, or had an opaque response.	Keep the deploy and record the reasons. • A read that didn't run: rerun it. • Binding missing: redeploy this commit. • Opaque control: check Workers analytics for that minute. • Control unavailable: check the deployments list and commit. • Smoke-check failure that didn't reproduce: record it and escalate to Rhys.
pass	Every safety check ran clean, the batch is complete, every control has its row, and at T0 every smoke check passed.	Continue.

At 7 days, checkpoint.py final gives the verdict. Pass it every checkpoint result, reruns included. A checkpoint given more than once keeps its worst outcome and the reasons from every run, so a later pass can't erase an earlier trigger or unverified result.

verdict	when
rolled back	any checkpoint was a rollback trigger
pass	T0, 24 h and 7 days all passed
unverified	anything else. Every unverified or missing checkpoint stays listed with its reasons, even if a later one passed.

It also reports these estimates:

• Natural failures since T0: rows and SUM(_sample_interval) without the control rays.
• Harness 5xx: recorded, not found, no ray, or opaque.

Compare with the baseline:

• CI emulator 5xx: 11, and 3 of 250 main runs;
• "never reached" Worker exceptions: 42 in 22 minutes;
• Durable Object scriptThrewException: 52;
• exceededMemory: 3.

The rows' retryable/overloaded flags decide whether a dedupe-backed retry is worth building.

Tested (synthetic data and read-only queries only):

• Self-test. 96 cases, including each of the reviewers' reproductions:
  • a sampled row 30 s away;
  • one ray returned with weights 2 and 4;
  • a recorded control alongside an opaque or a young one;
  • unenforced control fields;
  • an opaque control 5xx;
  • -SJC rays on every side;
  • ray-less failures in the same second;
  • two readable 400s where 200 was expected, on both instances (rollback-trigger);
  • a report-shaped body with an out-of-list error, with and without an extra key;
  • SYNTHETIC-secret-1 hidden by JSON escapes;
  • T0 trigger, T0 pass, 24 h pass and 7 d pass (rolled back).
• Mutations. 23 mutants of the scripts each fail it.
• Probe dataset. lookup.sql and natural.sql ran against the probe's dataset. 17 of its 17 rows match by exact ray. The 4 failures with no row are not found: run B's rollout-window miss, run B2's two sampled points and a fake ray. The earlier reconciler counted B2's two points as sampled.
• Mock server. positive_control.sh ran against a local mock. It retried opaque slots, applied the 30 s deadline, stopped on a 400 and on each disclosure (an instance name in a field, the same name fully JSON-escaped, an out-of-list error), refused another commit, and wrote no instance name.

Why not change the code.

• A writeDataPoint throw can't be reached. A point has 1 index of at most 11 bytes, 6 blobs from fixed lists (the longest route is 87 bytes) and 4 doubles. The documented limits are 1 index of 96 bytes, 20 blobs totalling 16 KB, 20 doubles and 250 points per invocation.
• What can fail happens after the call: sampling, ingestion, or a missing binding. writeDataPoint returns nothing, so no counter in the Worker can see these. The binding is read directly in step 1, and Test redacted Durable Object failure reports and flags through the router #37 pins it in wrangler.jsonc.

Rollback (drilled on the probe)

Observability settings are not part of a version:

• After observability was turned on outside wrangler, wrangler rollback to a version deployed with it off left logs and Issues on.
• Turning the settings off first, then rolling back to a version deployed with them on, left them off. Faults sent afterwards left 0 Logs and 0 Issues records.

So a rollback is two steps:

1. PATCH /accounts/{account}/workers/scripts/emulate-hosts/script-settings with {"observability":{"enabled":false,"logs":{"enabled":false,"invocation_logs":false},"traces":{"enabled":false},"issues":{"enabled":false}}}. Read it back and confirm observability is null or disabled.
2. wrangler rollback 87d64cf7-aee9-4ba4-b2ee-b20d0a0d2d26. This also removes the FAILURES binding, since bindings are part of a version.

Before / after

Before: the 11 CI failures carry no body; the harness shows External emulator failed: /_emulate/reset (HTTP 500).

After:

{"error":"emulator_unavailable","service":"resend","instanceId":"781f9f1e9db7","method":"GET","route":"/emails","errorClass":"Error","retryable":true,"overloaded":false,"remote":false,"ray":null}

executor-next#2199 prints these fields. Tests are in #37.

How measured

Cloudflare GraphQL Analytics (workersInvocationsAdaptive and durableObjectsInvocationsAdaptiveGroups, filtered to emulate-hosts, by minute, colo and object name), 30 Sep 00:00 to 8 Oct 05:00 UTC, compared with the failure timestamps in 748 executor-next CI job logs. The only deployment in the window is 87d64cf7 (29 Sep, this repo's main).