Skip to content

Dashboard: verified agent card stuck on 'not yet checked'; Recheck 503 for all owners since #7452; storyboard picker 404s without compliance_target #7578

Closed

Description

@idemidenko

Summary

Since the 2026-09-17 deploys the dashboard card for https://sales-agent.elementaltv.io/mcp (org org_01KESCR4BWMJSFEHHG00B78GN4) renders permanently in its skeleton state — not yet checked, the "Connect agent" auth form, No badges yet — although the record behind it is intact (/verification → verified: true, /compliance → status: degraded, 31/84, badge media-buy 3.1 Spec issued 2026-09-16). From that skeleton no owner action can start a compliance run any more. Three independent causes, all reproducible today (2026-09-18):

1. GET /api/registry/agents/{url}/compliance takes 22–24 s; the dashboard aborts at 15 s

server/public/dashboard-agents.html fans out /compliance, /compliance/history, /auth-status per agent under one FETCH_TIMEOUT_MS = 15_000 deadline and logs Agent compliance fetch timed out after 15s; in-flight workers cancelled, rendering with partial data. For this agent /compliance answers in 22.2–23.6 s on every attempt (5 samples, anonymous and authenticated alike), /storyboard-status in 21.8 s, while /compliance/history takes 0.4 s. Other agents with large storyboard sets show the same latency (sales-agent.addiscovery.ai/mcp 23.9 s, adcp.advendio.com/mcp/demo 22.4 s); small ones answer in 0.5 s. Both slow routes share complianceDb.getStoryboardStatuses(agentUrl, { requireRowsForLatestRun: true }) (db/compliance-db.ts:1541, LATERAL join over agent_compliance_step_diagnostics per storyboard row). The card therefore never hydrates: the hydrated actions (Update auth, History, Requeue comply, the full-suite Test) are never rendered, only the skeleton's Connect agent form — which reads as "your credential is gone" to the owner.

2. POST /refresh ("Recheck & retest") is fenced for every owner since #7452

server/src/routes/registry-api.ts (refresh_authorization_provenance_required): for any caller that is not a static admin the route returns 503 Refresh is temporarily unavailable for this credential. Please try again later. with Retry-After: 60. Introduced by #7452 (merged 2026-09-17 12:03 UTC), which is exactly when the button stopped working for us (~13:20 UTC). The message names "this credential", so owners re-save their bearer token and get the same answer; the credential itself is fine (/auth-status → has_auth: true, auth_type: bearer, and the token authenticates against the agent). If the fence is intentional until authorization provenance is durable, the message should say the feature is disabled platform-wide, and the button should be hidden or disabled instead of inviting a credential re-save.

3. The storyboard picker's "Preview" and "Run" call the storyboard routes without the agent's compliance target → 404 for every 3.1 storyboard

Test your agent → picker → Run (.storyboard-step-btn) fetches /api/storyboards/{id}/first-step, and Preview (.storyboard-picker-item) fetches /api/storyboards/{id}, both with no compliance_target, so the server resolves the default 3.0.25 catalog and answers 404 Storyboard not found (the panel shows "Storyboard not found.") — for media_buy_seller/billing_finality_delivery, billing_gate_dispatch, sales_non_guaranteed/non_guaranteed_buy, every id we tried. ?compliance_target=3.1 returns the step. The picker already knows the agent's target (applicable-storyboards → requested_compliance_target: "3.1"); pass it through.

Consequences for this agent

  • No owner-triggered run since d2c9f4d0 (2026-09-17 08:09 UTC): refresh fenced, requeue button unreachable, picker Preview and Run both 404 — so the full-storyboard Run (POST .../storyboard/{id}/run) is unreachable too. Not one dashboard control can start a compliance run for this agent; the only way left is calling POST .../storyboard/{id}/run by hand from the owner's browser session, one storyboard at a time, recorded non-authoritative.
  • No scheduled heartbeat run has ever appeared in /compliance/history for this agent (all 10 listed runs are owner_test), although the record advertises check_interval_hours: 6.
  • The org API key (sk_…, /api/me → organizations: []) gets 403 You do not have permission to test this agent on applicable-storyboards and 403 … refresh this agent on /refresh since the API-key authority isolation (fix(auth): isolate API keys to exact credentials and organizations #7461), so there is no scripted fallback either.

Proof the agent and credential are fine

Calling POST /api/registry/agents/{url}/storyboard/media_buy_seller%2Fbilling_finality_delivery/run by hand from the owner's browser session (2026-09-18 08:45 UTC) returned 200 in 25 s: run 6ed6711e, is_authoritative: false, storyboard passing 7/7; the runner authenticated with the stored bearer and our logs show its report_usage call. So the credential works, the agent works, and every dashboard control that should have started that run is broken independently.

Related: #6054 (owner credential context), #7428 (owner requeue produced no run), #7506 (heartbeat completion).

Asks

  1. Either raise the dashboard deadline / render the card from /compliance/history + /auth-status when /compliance is late, or make /compliance fast for agents with many storyboards (the badge, tracks and notices do not need the per-storyboard diagnostics join).
  2. Replace the /refresh 503 text with a platform-wide message and hide/disable the button while the fence is up; keep Requeue comply reachable from the skeleton card.
  3. Pass compliance_target from the picker to /api/storyboards/{id} and /api/storyboards/{id}/first-step.
  4. Please queue a heartbeat for https://sales-agent.elementaltv.io/mcp so the post-deploy state (a new report_usage tool, deployed 2026-09-17 13:03 UTC) gets graded; happy to retest on request.

Activity

  1. added
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 18, 2026
  2. added this to the P0 Bugs milestone on Sep 18, 2026
  3. bokelley commented on Sep 18, 2026

    @bokelley
    Contributor

    Triage

    Classification: Bug (three independent bugs)
    Bucket(s): admin-tool, compliance-suite
    Status: ready-to-implement
    Milestone: P0 Bugs

    What the experts said:

    • internal-tools-strategist: All three bugs confirmed. Bug 1 is the critical path — card never exits skeleton until /compliance returns within 15s. Bug 2 fence is intentional per fix(auth): authorize web admins by authenticated credential #7452; the fix is message text + button UX, not the fence itself. Bug 3 is a pure frontend passthrough gap, independent of the others.
    • code-reviewer: Raise-timeout and graceful-degradation options for Bug 1 are wrong — only the SQL opt-out is correct, and the includeDiagnostics flag does not yet exist in getStoryboardStatuses. Bug 3 fix must distinguish orphaned vs. active storyboard buttons (different compliance_target source). Confirmed: "Requeue comply" (/monitoring/requeue) and "Recheck & retest" (/refresh) are distinct routes; the fence only blocks /refresh — requeue is already reachable once the card hydrates.

    My take: Three P0 bugs blocking every dashboard compliance control for agents with large storyboard sets, introduced/exposed by the 2026-09-17 deploys. No protocol or schema changes; all fixes are server/UI only, no changeset.

    Implementation brief:

    • Bug 1 — server/src/db/compliance-db.ts:1538: add includeDiagnostics?: boolean to getStoryboardStatuses options; gate the LATERAL join over agent_compliance_step_diagnostics on that flag (emit NULL literals when false). server/src/routes/registry-api.ts:6980: call { requireRowsForLatestRun: true, includeDiagnostics: false } from the card-level /compliance handler. The drill-down call site (line 7909) keeps its existing call unchanged.

    • Bug 2 — server/src/routes/registry-api.ts:8764: replace 'Refresh is temporarily unavailable for this credential.' with a platform-wide message that names the right action, e.g. 'Recheck & retest is temporarily paused platform-wide. Use Requeue comply to schedule the next run.'. Optionally: detect code === 'refresh_authorization_provenance_required' in the dashboard 503 handler (dashboard-agents.html:3631) to disable the "Recheck & retest" button for the session.

    • Bug 3 — server/public/dashboard-agents.html in renderStoryboardPicker: stamp data-compliance-target on each button. Active storyboard buttons (lines 4042–4043): use data.requested_compliance_target. Orphaned storyboard buttons (lines 4072–4073): use statusInfo.requested_compliance_target || data.requested_compliance_target (orphaned storyboards may target a different version than current discovery). Click handlers (lines 4188, 4264): read dataset.complianceTarget and conditionally append ?compliance_target=<value> — guard against empty/null.

    Ask #4 (queue heartbeat for https://sales-agent.elementaltv.io/mcp): operational request — please route via admin console or ops channel; not tracked here.


    Bug 1 follow-up — 2026-09-21: @idemidenko confirmed Bugs 2 and 3 resolved by #7594. Bug 1 is still open. Expert code review of the #7594 diff against the codebase found two distinct misses:

    Miss 1 (confirmed, mechanically explains the 29.4s /storyboard-status sample): registry-api.ts:7933 was not updated. The /storyboard-status handler still calls getStoryboardStatuses(agentUrl, { requireRowsForLatestRun: true }) with no includeDiagnostics: false — the full 84× LATERAL join over agent_compliance_step_diagnostics still executes on every /storyboard-status request. Minimum fix:

    // registry-api.ts:7933 — mirror the fix already applied at line 7000
    statuses = await complianceDb.getStoryboardStatuses(agentUrl, {
      requireRowsForLatestRun: true,
      includeDiagnostics: false,
    });
    

    Miss 2 (residual /compliance latency): With includeDiagnostics: false applied, the getStoryboardStatuses call at the card level should be near-instantaneous. The 22–30s on /compliance (including the 22.1s anonymous path) is from other serial queries in the /compliance handler — candidates are getComplianceStatusWithStoryboardCounts and getPublicSelectedGradingStatuses (the latter uses a non-indexable split_part + regex filter). Needs handler-level query timing or pg_stat_statements to localize before fixing.

    Deeper fix (removes the need for the includeDiagnostics gate entirely): Add a composite index (run_id, storyboard_id, step_id, captured_at DESC) on agent_compliance_step_diagnostics. Currently only (run_id) is indexed (migration 489_compliance_step_diagnostics.sql:66–72); each LATERAL iteration filters storyboard_id, failed_validations_jsonb, and step_id without index support. The composite index makes each LATERAL iteration O(1) and fixes the performance root cause for all callers.

    To the reporter (@idemidenko): this is the same Bug 1, not a new issue. Please reopen this issue (or ask a maintainer to). Filing a new issue is fine if reopening isn't available to you, but the full context (2026-09-17 regression, proof-of-working agent, partial fix history) is valuable and lives here.


    Triaged by Claude Code. Session: https://claude.ai/code/session_019Q3y9AGzybZdH1UYCSVjJ9


    Generated by Claude Code

  4. added
    bugSomething isn't working
    admin-toolInternal tools for AAO staff
    claude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.
    priority:P0Immediate priority: user/revenue/security/reliability bug to work now
    and removed
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 18, 2026
  5. idemidenko commented on Sep 21, 2026

    @idemidenko
    Author

    Follow-up after #7594 deployed (the refresh_availability block with refresh_authorization_provenance_required is present on /compliance, so this is the fixed build): bugs 2 and 3 are addressed, but bug 1 still reproduces for https://sales-agent.elementaltv.io/mcp (84 storyboards).

    Samples taken 2026-09-21 ~10:30Z, all HTTP 200:

    endpoint time
    /compliance (org API key) 30.1 s, 22.4 s, 28.9 s
    /compliance (anonymous) 22.1 s
    /storyboard-status 29.4 s
    /compliance/history?limit=10 0.6 s

    So the endpoint is still 1.5–2× over the dashboard's FETCH_TIMEOUT_MS = 15_000, and the card still renders the skeleton (not yet checked, the OAuth connect form, No badges yet, "Test your agent") for a verified agent. Because the hydrated action row never appears, the "Requeue comply" control that the new fence notice points to as the alternative action is not reachable from the UI either. Only the fixed-text disabled "Recheck paused" button and the notice show up, after clicking it once.

    If includeDiagnostics: false did land on the card-level /compliance call, the remaining cost is elsewhere in getStoryboardStatuses(..., { requireRowsForLatestRun: true }) for large storyboard sets; history on the same agent is sub-second. Could this be reopened, or should I file it as a new issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    admin-toolInternal tools for AAO staffbugSomething isn't workingclaude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.compliance-suitepriority:P0Immediate priority: user/revenue/security/reliability bug to work nowseverity:critical

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions