You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci(schema): add bounded retries for immutable release asset fetches #1194
The required Validate schemas are up-to-date check performs network reads from the immutable, pinned AdCP release during model generation. A transient CDN reset currently fails the entire gate without a bounded retry.
Observed on Python rc.4 adoption PR #1193, exact head 71c2bc0f0d1a2bf783f001fdbba29801cefd9825:
syntax/import/codegen/drift checks were skipped because generation aborted
The exact signed rc.4 inputs and local generation already passed on the same head. This incident is a CI transport-resilience failure, not evidence of schema drift.
Desired behavior
Retry immutable release-asset reads with a small, bounded policy for transient network failures.
Preserve the exact version and URL; never fall back to latest or another release.
Preserve signature/hash verification and fail closed after the retry budget.
Distinguish transport exhaustion from schema drift in logs and check summaries.
Add deterministic tests for transient success, permanent failure, hash/signature failure, and prohibition of version fallback.
Acceptance criteria
One transient reset/5xx can recover without changing the selected artifact.
Exhausted retries still fail the required check.
Integrity/authenticity failures are never retried into acceptance.
Logs record attempts and terminal classification without leaking credentials.
Existing schema generation and drift tests remain green.
Classification: Bug Bucket(s): ci-scripts (no ci or infrastructure label exists in this repo — flagged in run summary) Status: ready-to-implement
What the experts said:
code-reviewer: Bounded retries on _http_get (URLError / 5xx) are safe and non-breaking; SHA256 and cosign checks run post-fetch so integrity is never weakened by a retry. Recommends pre-fetching canonical $ref targets into the local temp schema tree and dropping --allow-remote-refs entirely, rather than parsing codegen subprocess stderr — the codegen error format is a third-party implementation detail that can shift without notice.
dx-expert: Clear DX win — transient CDN resets are pure contributor tax. Permanent/transient split is correct; SHA256 mismatch must be permanent (never retried). Treating 404 as permanent is also safer than retrying it.
My take: Both experts converge: the fix is sound, non-breaking, and in-scope. The recommended implementation path (pre-fetch + drop --allow-remote-refs) is cleaner and more durable than a subprocess stderr wrapper.
Implementation scope:
scripts/sync_schemas.py — Add _http_get_with_retry(url, *, max_attempts=3, backoff_base=1.0): retry on URLError and HTTPError 5xx only; raise immediately on 4xx and any other HTTPError; never retry inside the SHA256 or cosign verification branches (those already follow _http_get returns, so the check is naturally outside the retry loop). Log attempt number and terminal classification without leaking URLs containing credentials.
scripts/generate_types.py — In flatten_schemas (or a new _prefetch_canonical_refs helper), detect preserved canonical URLs in the temp schema tree, fetch them with _http_get_with_retry (importing or inlining the helper from sync_schemas.py), write them into the temp directory, and rewrite the remaining canonical $ref values to local relative paths. Drop --allow-remote-refs from _run_datamodel_codegen args once pre-fetching is in place.
tests/test_sync_schemas_retry.py (new) — Deterministic unit tests: (1) transient success — mock urlopen to [Errno 104] once then succeed, assert final content returned; (2) budget exhausted — mock to always fail, assert URLError raised after max_attempts; (3) hash failure — mock to return wrong SHA256, assert integrity error is not retried; (4) cosign failure — mock cosign verify-blob to return non-zero, assert it is not retried; (5) 5xx retry — mock HTTPError(code=503), assert retry fires; (6) 4xx no-retry — mock HTTPError(code=404), assert single attempt only; (7) version fallback prohibition — after codegen path change, assert --allow-remote-refs is absent from generated subprocess args.
Required checks:pytest tests/test_sync_schemas_retry.py -v, ruff check scripts/, black --check scripts/.
Non-breaking: all changes are in scripts/ (CI tooling); no public SDK API, Pydantic model, or exported type surface is touched.
Activity
bokelley commentedon Sep 20, 2026
Triage
Classification: Bug
Bucket(s): ci-scripts (no
ciorinfrastructurelabel exists in this repo — flagged in run summary)Status: ready-to-implement
What the experts said:
_http_get(URLError / 5xx) are safe and non-breaking; SHA256 and cosign checks run post-fetch so integrity is never weakened by a retry. Recommends pre-fetching canonical$reftargets into the local temp schema tree and dropping--allow-remote-refsentirely, rather than parsing codegen subprocess stderr — the codegen error format is a third-party implementation detail that can shift without notice.My take: Both experts converge: the fix is sound, non-breaking, and in-scope. The recommended implementation path (pre-fetch + drop
--allow-remote-refs) is cleaner and more durable than a subprocess stderr wrapper.Implementation scope:
scripts/sync_schemas.py— Add_http_get_with_retry(url, *, max_attempts=3, backoff_base=1.0): retry onURLErrorandHTTPError5xx only; raise immediately on 4xx and any otherHTTPError; never retry inside the SHA256 or cosign verification branches (those already follow_http_getreturns, so the check is naturally outside the retry loop). Log attempt number and terminal classification without leaking URLs containing credentials.scripts/generate_types.py— Inflatten_schemas(or a new_prefetch_canonical_refshelper), detect preserved canonical URLs in the temp schema tree, fetch them with_http_get_with_retry(importing or inlining the helper fromsync_schemas.py), write them into the temp directory, and rewrite the remaining canonical$refvalues to local relative paths. Drop--allow-remote-refsfrom_run_datamodel_codegenargs once pre-fetching is in place.tests/test_sync_schemas_retry.py(new) — Deterministic unit tests: (1) transient success — mockurlopento[Errno 104]once then succeed, assert final content returned; (2) budget exhausted — mock to always fail, assertURLErrorraised aftermax_attempts; (3) hash failure — mock to return wrong SHA256, assert integrity error is not retried; (4) cosign failure — mockcosign verify-blobto return non-zero, assert it is not retried; (5) 5xx retry — mockHTTPError(code=503), assert retry fires; (6) 4xx no-retry — mockHTTPError(code=404), assert single attempt only; (7) version fallback prohibition — after codegen path change, assert--allow-remote-refsis absent from generated subprocess args.pytest tests/test_sync_schemas_retry.py -v,ruff check scripts/,black --check scripts/.scripts/(CI tooling); no public SDK API, Pydantic model, or exported type surface is touched.Triaged by Claude Code. Session: https://claude.ai/code/session_01Cm1HDj5TKDDLbS6bHCRtgc
Generated by Claude Code
bokelley commentedon Sep 29, 2026
CI network-flake protection only; reopen if flakes return.