Skip to content

fix(runtime): retry stalled CDN fetches when verifying protocol offloads - #9092

Open
bokelley wants to merge 1 commit into
mainfrom
fix/runtime-offload-fetch-retry
Open

bokelley wants to merge 1 commit into
mainfrom
fix/runtime-offload-fetch-retry

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Why

Production deploy run 44353722716 (head 859ea919be) failed during the image build, not the rollout:

#24 [runtime-artifacts 1/1] RUN node scripts/prune-runtime-artifacts.cjs --offload-protocol && ...
#24 64.06 The operation was aborted due to timeout
error releasing builder: deadline_exceeded

Before it removes any local protocol archive, prune-runtime-artifacts.cjs --offload-protocol fetches CDN discovery and then each archive, about 120 of them, six at a time. Every request has a 60 s AbortSignal.timeout and no retry, so one slow response from the Fly builder fails the deploy. The code hasn't changed since #8814, and discovery answers in about 2.5 s now, so this was transient.

What

  • Discovery and each archive check run inside withFetchRetries: up to 3 attempts, waiting 2 s and then 10 s. Each attempt keeps its own 60 s timeout. Every retry logs a warning line, so the build log shows it.
  • Retried: a fetch that throws, a timeout or abort, a body stream that breaks partway, and HTTP 5xx or 429.
  • Fails at once (OffloadRejected): any other 4xx, a Fly-fallback response (fly-request-id), an archive without an R2 strong ETag, or bytes that differ from the committed ones. Those are answers, not stalls.
  • No change to safety: nothing is pruned until every body has been verified.

Tests

npm run test:runtime-artifacts: 14/14 pass. This needs the 3.3.0-rc.0 docs snapshot checked out.

  • A new test covers both recovery paths: discovery that times out once, and an archive that answers 503 once. It also checks that a byte mismatch fails after a single request, and that a stall that never clears gives up after 3 attempts with the archive still on disk.
  • The existing rejection loop now also covers 503 and passes zero delays.

🤖 Generated with Claude Code

The image build verifies ~120 protocol archives against the CDN, each with
a 60 s timeout and no retry, so one slow response fails the whole deploy
(run 44353722716: "The operation was aborted due to timeout" at 64 s).

Retry discovery and each archive up to twice (2 s, then 10 s) on a thrown
fetch, an aborted or interrupted body, or a 5xx/429. A definitive answer,
such as a 4xx, a Fly fallback, a missing R2 ETag or different bytes, still
fails at once, and nothing is pruned until every body verifies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — bounded retry hardening on CDN verification in a build-time script.

Checked:

  • Only two files touched: scripts/prune-runtime-artifacts.cjs and its test. No static/schemas/source/**, docs/reference/**, dist/**, or package.json version changes — no changeset owed, no schema↔docs coherence concern, no oneOf discriminator impact, no released-artifact mutation.
  • Retry logic (3 attempts, per-attempt 60s timeout) correctly fail-closes on definitive negatives (OffloadRejected, Fly fallback, bad ETag, byte mismatch) and only retries transient failures. Timeout rebuilt per attempt so it resets; file read/hash baseline stays outside the wrapper.
  • Tests cover retry-then-succeed, no-retry-on-definitive, and exhaustion.
  • No Critical/High/Medium findings. No high-risk paths, no gated paths, no no-auto-approve team match, no deletions.

Decision table: rows 1–8 do not fire (no blocking/medium findings, gated_paths false, high_risk false, no team gate). Note review_decision is REVIEW_REQUIRED, but gated_paths is false so row 2 does not apply. Falls through to row 9 → approve.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant