Skip to content

ci(schema): add bounded retries for immutable release asset fetches #1194

Closed as not planned

Description

@bokelley

Problem

The required Validate schemas are up-to-date check performs network reads from the immutable, pinned AdCP release during model generation. A transient CDN reset currently fails the entire gate without a bounded retry.

Observed on Python rc.4 adoption PR #1193, exact head 71c2bc0f0d1a2bf783f001fdbba29801cefd9825:

The exact signed rc.4 inputs and local generation already passed on the same head. This incident is a CI transport-resilience failure, not evidence of schema drift.

Desired behavior

  • Retry immutable release-asset reads with a small, bounded policy for transient network failures.
  • Preserve the exact version and URL; never fall back to latest or another release.
  • Preserve signature/hash verification and fail closed after the retry budget.
  • Distinguish transport exhaustion from schema drift in logs and check summaries.
  • Add deterministic tests for transient success, permanent failure, hash/signature failure, and prohibition of version fallback.

Acceptance criteria

  • One transient reset/5xx can recover without changing the selected artifact.
  • Exhausted retries still fail the required check.
  • Integrity/authenticity failures are never retried into acceptance.
  • Logs record attempts and terminal classification without leaking credentials.
  • Existing schema generation and drift tests remain green.

Activity

  1. added
    bugSomething isn't working
    pythonPull requests that update python code
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 20, 2026
  2. bokelley commented on Sep 20, 2026

    @bokelley
    ContributorAuthor

    Triage

    Classification: Bug
    Bucket(s): ci-scripts (no ci or infrastructure label exists in this repo — flagged in run summary)
    Status: ready-to-implement

    What the experts said:

    • code-reviewer: Bounded retries on _http_get (URLError / 5xx) are safe and non-breaking; SHA256 and cosign checks run post-fetch so integrity is never weakened by a retry. Recommends pre-fetching canonical $ref targets into the local temp schema tree and dropping --allow-remote-refs entirely, rather than parsing codegen subprocess stderr — the codegen error format is a third-party implementation detail that can shift without notice.
    • dx-expert: Clear DX win — transient CDN resets are pure contributor tax. Permanent/transient split is correct; SHA256 mismatch must be permanent (never retried). Treating 404 as permanent is also safer than retrying it.

    My take: Both experts converge: the fix is sound, non-breaking, and in-scope. The recommended implementation path (pre-fetch + drop --allow-remote-refs) is cleaner and more durable than a subprocess stderr wrapper.

    Implementation scope:

    • scripts/sync_schemas.py — Add _http_get_with_retry(url, *, max_attempts=3, backoff_base=1.0): retry on URLError and HTTPError 5xx only; raise immediately on 4xx and any other HTTPError; never retry inside the SHA256 or cosign verification branches (those already follow _http_get returns, so the check is naturally outside the retry loop). Log attempt number and terminal classification without leaking URLs containing credentials.
    • scripts/generate_types.py — In flatten_schemas (or a new _prefetch_canonical_refs helper), detect preserved canonical URLs in the temp schema tree, fetch them with _http_get_with_retry (importing or inlining the helper from sync_schemas.py), write them into the temp directory, and rewrite the remaining canonical $ref values to local relative paths. Drop --allow-remote-refs from _run_datamodel_codegen args once pre-fetching is in place.
    • tests/test_sync_schemas_retry.py (new) — Deterministic unit tests: (1) transient success — mock urlopen to [Errno 104] once then succeed, assert final content returned; (2) budget exhausted — mock to always fail, assert URLError raised after max_attempts; (3) hash failure — mock to return wrong SHA256, assert integrity error is not retried; (4) cosign failure — mock cosign verify-blob to return non-zero, assert it is not retried; (5) 5xx retry — mock HTTPError(code=503), assert retry fires; (6) 4xx no-retry — mock HTTPError(code=404), assert single attempt only; (7) version fallback prohibition — after codegen path change, assert --allow-remote-refs is absent from generated subprocess args.
    • Required checks: pytest tests/test_sync_schemas_retry.py -v, ruff check scripts/, black --check scripts/.
    • Non-breaking: all changes are in scripts/ (CI tooling); no public SDK API, Pydantic model, or exported type surface is touched.

    Triaged by Claude Code. Session: https://claude.ai/code/session_01Cm1HDj5TKDDLbS6bHCRtgc


    Generated by Claude Code

  3. added and removed
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 20, 2026
  4. bokelley commented on Sep 29, 2026

    @bokelley
    ContributorAuthor

    CI network-flake protection only; reopen if flakes return.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingclaude-triagedpythonPull requests that update python code

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions