Skip to content

CI: v3 reference seller (translator) storyboard job fails on main — seller never boots within 30s #981

Closed

Description

@KonstantinMirin

Summary

AdCP storyboard runner — v3 reference seller (translator) fails on unmodified main. The seller process never becomes ready inside the 30s boot window:

INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
Seller failed to start within 30s
##[error]Process completed with exit code 1.

The failure is in the Boot v3 reference seller (translator) step. The upstream-still-alive probe never runs — the readiness loop times out first — so the Node mock server is not implicated.

Evidence that it is main, not a PR

Reproduced on a79e29a2 (current main) with no changes applied, by pushing main unmodified to a fork and letting the same workflow run:

It also failed three consecutive times on #980, a test-only PR that touches nothing outside tests/conformance/, including on an isolated job re-run.

Why it looks like a regression rather than a flake

That leaves a change in the seller's own boot path, a transitive Python dependency resolved at install time, or the runner image, as the likely causes.

Suggested next step

The 30s budget also has no diagnostic on timeout — the step reports only "failed to start" and discards the seller's stdout/stderr, which is why the cause is not visible in the log. Dumping the captured seller log on timeout would make this self-diagnosing, independent of the underlying regression.

Activity

  1. added
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Jul 28, 2026
  2. bokelley commented on Jul 28, 2026

    @bokelley
    Contributor

    Triage

    Classification: Bug
    Bucket(s): examples (no matching label exists — gap noted)
    Status: ready-to-implement

    What the experts said:

    • code-reviewer: Boot step is missing log capture (blocker — why CI shows only the two Alembic lines) and the kill -0 dead-process probe present in both sibling storyboard jobs (storyboard-multi-platform-seller line 769, storyboard-sales-proposal-mode line 868); inline pip install on line 448 has no upper bounds on asyncpg, sqlalchemy, or respx.
    • dx-expert: "Will assume transactional DDL" is emitted immediately before the first SQL roundtrip (SELECT version_num FROM alembic_version) — the hang is precisely there, consistent with an asyncpg event-loop attachment change in the 2026-07-27 19:24Z–2026-07-28 20:35Z window; wrapping the to_thread call with asyncio.wait_for converts the silent 30s burn to a clear TimeoutError traceback.

    My take: Four-for-four consecutive failures on unmodified main — not a flake. Two independent fixes are needed: a startup timeout in the seller app (in-scope for a PR) and workflow diagnostics in .github/workflows/ci.yml (human-authored; that file is off-limits for automated PRs).

    • examples/v3_reference_seller/src/app.py, _bootstrap_schema_and_load_tokens: Wrap await asyncio.to_thread(_run_alembic_upgrade_head, db_url) with asyncio.wait_for(..., timeout=25.0) so a hang surfaces as a TimeoutError traceback inside the 30s CI budget. Add logger.info before/after each phase (alembic upgrade, token load). Non-breaking; run pytest examples/v3_reference_seller/tests/ + ruff check + mypy src/adcp/.
    • .github/workflows/ci.yml boot step (lines 566–589) — human-authored: Add PYTHONUNBUFFERED=1, redirect python -m src.app > /tmp/seller.log 2>&1 &, add cat /tmp/seller.log in the timeout branch before exit 1, add kill -0 "$SELLER_PID" inside the poll loop, add /tmp/seller.log to the if: always() artifact. Once logs are visible, check asyncpg changelog for the breakage window and add a ceiling in the inline pip install on line 448.

    Triaged by Claude Code. Session: https://claude.ai/code/session_01RUBuSETHCuU6yr3Ryo2sns


    Generated by Claude Code

  3. added
    bugSomething isn't working
    and removed
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Jul 28, 2026
  4. KonstantinMirin commented on Jul 29, 2026

    @KonstantinMirin
    CollaboratorAuthor

    Draft — one message to bokelley (plan step 1.4)

    Intended as a comment on #981, with the second and third points cross-posted to
    #976 and #980 respectively. Not yet sent — awaiting the user's go-ahead.


    1. #981 can be closed — #984 already fixed it

    AdCP storyboard runner — v3 reference seller (translator) is a required
    check, so while it was red every open PR in the repo was unmergeable — #980,
    #982, #983 included.

    It was never a change in the seller's boot path. main did not move between
    2026-07-02 (a79e29a2) and the first red run on 2026-07-28 20:35Z, so
    there was no repo-side change in the window to blame.

    What did change is a dependency. The only package that published inside the
    green→red window (last green 2026-07-27 19:24Z) is mcp 1.29.0, published
    2026-07-28 13:41Z
    , which the then-current pin mcp>=1.27.0,<2.0 resolves.
    asyncpg / SQLAlchemy / alembic all published outside the window, so the
    asyncpg-event-loop theory in the triage comment can be ruled out.

    #984 fixed it by moving the pin to mcp>=2.0.0,<3.0 and porting to the v2
    MCPServer API. The translator job passed on #984's head, and main has
    carried the fix since 2026-07-28 22:10Z.

    Two things remain worth doing, both independent of the regression:

    • The boot step discards the seller's stdout, which is why four consecutive
      failures produced no diagnostic beyond two Alembic lines. Redirecting to
      /tmp/seller.log, cat-ing it in the timeout branch, and adding the
      kill -0 dead-process probe the two sibling storyboard jobs already have
      would make this class of failure self-diagnosing.
    • The inline pip install in that job has no upper bounds. The pin that
      actually broke was in pyproject.toml rather than that line, but the same
      exposure exists there.

    2. #976 needs your enforcement audit before it can be implemented

    Your triage assigns you the required-component enforcement audit
    (_check_params_present vs. the attacker-controlled parsed label) and routes
    #976 to human review rather than a bot PR. That audit is the gate on the work —
    happy to build the rest of the stack against it once you've had a look.

    3. Release shape — a decision I shouldn't make in a PR title

    #978 makes previously-accepted URLs raise; #976 changes a code that goes out on
    the wire. Both are behaviour-visible to adopters, but as fix: commits
    release-please cuts a patch — a rejection-set expansion arriving in a patch
    release is how integrators get surprised in production.

    Suggested: title them feat(signing): … so release-please cuts a minor, with
    the behaviour change spelled out in the changelog body. BREAKING CHANGE: would
    force a major, which seems too heavy for 6.x. Your call — I'll title them
    whichever way you prefer.

    Related: landing #982 (6.6.1) first would ship the #969 security fix on a clean
    changelog before this stack churns it.

    4. What's coming

    A stack of three PRs on top of #980, all targeting main (a fork contributor
    can't set a base branch that doesn't exist upstream, so each will show #980's
    files in its diff until #980 lands):

    PR Issues Path
    A #978 + #977 canonical.py — rejection set + IDNA A-label conversion
    C #979 canonical.py — trailing empty query
    B #976 verifier — step-1 structured-field precheck (gated on §2)

    #980 itself has been rebased onto current main and should now go green.

  5. bokelley commented on Jul 29, 2026

    @bokelley
    Contributor

    Triage (updated)

    Classification: Bug — dependency regression (resolved)
    Status: not-actionable

    @KonstantinMirin's analysis changes the picture: the root cause was mcp 1.29.0 (published 2026-07-28 13:41Z), not an asyncpg event-loop change. The prior asyncio.wait_for implementation brief was aimed at the wrong layer and is superseded. #984 fixed it by pinning to mcp>=2.0.0,<3.0 and porting to the v2 MCPServer API; main has carried the fix since 2026-07-28 22:10Z and the translator job is green.

    Two follow-up improvements remain from the comment — both touch .github/workflows/ci.yml (off-limits for automated PRs, human-authored only):

    • Redirect seller stdout to /tmp/seller.log and cat on timeout, plus kill -0 dead-process probe (makes this class of failure self-diagnosing)
    • Add upper bounds to the inline pip install on line 448

    The primary regression is fixed. This issue can be closed; the workflow diagnostics are independent hardening and can be tracked separately if prioritized.


    Triaged by Claude Code. Session: https://claude.ai/code/cse_019TCbWPgMuvvruBcAo6xPxe


    Generated by Claude Code

  6. bokelley commented on Jul 29, 2026

    @bokelley
    Contributor

    Closed as fixed by #984. The MCP v2 migration removed the mcp 1.29.0 drift under the previous <2.0 pin, and the translator storyboard check is green on current main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions