Skip to content

Dashboard card still stuck on 'not yet checked' after #7594: /compliance takes 22–30 s for an 84-storyboard agent, client aborts at 15 s #7630

Closed

Description

@idemidenko

Summary

Follow-up to #7578 (closed by #7594). Bugs 2 and 3 from that issue are fixed on the deployed build, but bug 1 still reproduces: GET /api/registry/agents/{url}/compliance takes 22–30 s for https://sales-agent.elementaltv.io/mcp (84 storyboards), the dashboard's FETCH_TIMEOUT_MS = 15_000 fires first, and the card for a verified agent stays in its skeleton state (not yet checked, the OAuth connect form, No badges yet, "Test your agent"). Reported as a comment on #7578 first (#7578 (comment)); filing separately since that issue is closed.

Reproduction (2026-09-21 ~10:30Z, all HTTP 200)

endpoint time
/compliance (org API key) 30.1 s, 22.4 s, 28.9 s
/compliance (anonymous) 22.1 s
/storyboard-status 29.4 s
/compliance/history?limit=10 0.6 s

The deployed build is post-#7594: the /compliance response carries refresh_availability with code: refresh_authorization_provenance_required.

Why the #7594 change did not move the number

#7594 added includeDiagnostics: false to the card-level getStoryboardStatuses call (registry-api.ts /compliance handler) and gates the agent_compliance_step_diagnostics LATERAL join on it (compliance-db.ts getStoryboardStatuses). But:

  • /compliance (join removed) and /storyboard-status (join still present, default includeDiagnostics) take the same 22–30 s, so the diagnostics join was not the dominant cost.
  • /compliance/history on the same agent is sub-second.

What the two slow routes still share is the latest_run CTE (agent_compliance_runs … ORDER BY tested_at DESC LIMIT 1) plus the correlated EXISTS (SELECT 1 FROM agent_storyboard_status latest WHERE latest.agent_url = $1 AND latest.run_id = …) predicates, evaluated per storyboard row, and getComplianceStatusWithStoryboardCounts runs the same shape once more. I have no access to EXPLAIN on the hosted DB, so this is a pointer, not a diagnosis; an index on agent_storyboard_status (agent_url, run_id) and agent_compliance_runs (agent_url, tested_at DESC) WHERE dry_run = false AND is_authoritative = true would be the first things to check.

Impact

For agents with large storyboard sets the card never hydrates, so the hydrated-only action row never renders. That includes "Requeue comply", which the new fence notice names as the alternative action, so the notice points at a control the affected owners cannot reach. The only thing that renders is the disabled "Recheck paused" button plus the notice, after one click.

Ask

Bring /compliance under the 15 s client budget for agents with ~80+ storyboards (or let the card render its action row from the fast /compliance/history + /auth-status data while /compliance is still in flight).

Activity

  1. added
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 21, 2026
  2. added this to the P0 Bugs milestone on Sep 21, 2026
  3. bokelley commented on Sep 21, 2026

    @bokelley
    Contributor

    Triage

    Classification: Bug
    Bucket(s): admin-tool, compliance-suite
    Status: ready-to-implement
    Milestone: P0 Bugs

    What the experts said:

    • internal-tools-strategist: Confirmed P0; progressive-render fallback recommended as resilience complement.
    • code-reviewer: Primary cause is 8 sequential await calls in the /compliance handler (registry-api.ts ~6900–7000) — badges, specialisms, notices, observations, and storyboard statuses all fire one at a time; at ~3–4 s each, 22–30 s is fully explained. The agent_compliance_runs partial index already exists (migration 589); the missing agent_storyboard_status (agent_url, run_id) index is secondary, not primary, at 84 rows.

    My take: The fix is parallelizing the 5+ independent calls after the early-exit guards clear in both /compliance and /storyboard-status; the missing (agent_url, run_id) index and a progressive-render fallback are worthwhile complements.

    Implementation scope:

    • server/src/routes/registry-api.ts: After getComplianceStatusWithStoryboardCounts resolves (early-exit checks cleared), fan out getBadgesForAgent, getPublicSelectedGradingStatuses, getLatestDeclaredSpecialisms, getLatestNotices, getLatestObservations, and getStoryboardStatuses via Promise.allSettled. Apply the same parallelization to the /storyboard-status handler.
    • New migration: CREATE INDEX CONCURRENTLY IF NOT EXISTS ON agent_storyboard_status (agent_url, run_id).
    • server/public/dashboard-agents.html: Render action row (including "Requeue comply") from the fast /compliance/history + /auth-status responses while /compliance is still in flight.
    • No protocol/schema changes; no changeset; server deploy only.

    Triaged by Claude Code. Session: https://claude.ai/code/session_01QCPMVnrT2SntKHz3APdL7w


    Generated by Claude Code

  4. added
    bugSomething isn't working
    admin-toolInternal tools for AAO staff
    claude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.
    priority:P0Immediate priority: user/revenue/security/reliability bug to work now
    and removed
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 21, 2026
  5. EvgenyAndroid commented on Sep 24, 2026

    @EvgenyAndroid
    Contributor

    Still reproducing after #7664, from a second operator's agents.

    #7664 merged at 12:45:40Z and did not run a Deploy workflow of its own, but its commit df855ad0ab is an ancestor of 2e7193bfdf (#7648), whose Deploy succeeded at 12:54Z — so the fix has been live for roughly 90 minutes.

    Measured at ~14:15Z, all HTTP 200:

    endpoint agent storyboards time
    /compliance sell.nofluffadvisory.com/mcp 82 23.6s
    /compliance adcp.signal-stack.io/mcp 38 8.7s, 10.0s, 9.3s
    /compliance/history?limit=10 sell.nofluffadvisory.com/mcp — 0.24s

    That is essentially unchanged from the 22–30s this issue reported for an 84-storyboard agent, and it reproduces the same shape: the compliance projection is slow while history is fast (0.24s here against the 0.6s reported).

    The scaling with storyboard count is fairly clean — roughly 0.29s per storyboard on the 82-storyboard agent and 0.24s on the 38-storyboard one — which is consistent with the per-card fan-out being the cost rather than a fixed overhead.

    One caveat on the numbers: these are anonymous reads from a single client, so they include network path and any Cloudflare variance. I also saw one 60s timeout on the signals agent that did not reproduce across three immediate retries, so treat that as noise rather than a data point.

    Not asking for a reopen — you may already be tracking this as expected post-fix behaviour, and I can't see the server-side budget metrics that would say whether the fan-out landed as intended.

  6. idemidenko commented on Sep 25, 2026

    @idemidenko
    Author

    Another data point after #7664, from our agent (https://sales-agent.elementaltv.io/mcp), anonymous reads on 2026-09-25, all HTTP 200:

    endpoint time
    /compliance 27.5 s, 22.5 s, 26.7 s
    /compliance/history?limit=10 0.42 s

    That matches what this issue originally reported (22–30 s) and is still above the dashboard's 15 s client abort, so the card still can't render for us. It also matches @EvgenyAndroid's numbers above. It looks like the fan-out either didn't take effect or isn't where the time goes. Could this be reopened?

  7. added
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    and removed
    claude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.
    on Sep 25, 2026
  8. bokelley commented on Sep 25, 2026

    @bokelley
    Contributor

    Triage

    Classification: Bug (partial fix — API latency unchanged)
    Bucket(s): admin-tool, compliance-suite
    Status: ready-for-human

    Post-#7664 data from two reporters:

    • @EvgenyAndroid (82 storyboards → 23.6 s, 0.29 s/storyboard; 38 storyboards → 9.1 s, 0.24 s/storyboard)
    • @idemidenko (84 storyboards → 22.5–27.5 s)

    Linear per-storyboard scaling confirms the bottleneck is inside the DB query, not the top-level fan-out.

    What #7664 resolved vs. what remains:

    • ✅ Progressive render: history + auth now load first (~0.4 s); "Requeue comply" appears before /compliance resolves. The acute "owner can't reach controls" issue is fixed.
    • ❌ /compliance API latency: still 22–27 s. The 8 s withDatabaseDeadline wraps only the supplemental fan-out batch — but getComplianceStatusWithStoryboardCounts runs before that section and is the likely home of the per-storyboard correlated EXISTS predicates described in the issue body.

    Revised target: getComplianceStatusWithStoryboardCounts in compliance-db.ts — batch the per-storyboard EXISTS into a single WHERE run_id = $runId predicate or CTE join so it stops scaling linearly with storyboard count. The new index (migration 609) reduces row scans but doesn't eliminate the N-loop.

    Ask: @bokelley — reopen for the latency follow-up, or file a fresh issue? Severity may have dropped now that owner controls are reachable via the progressive render.


    Triaged by Claude Code. Session: https://claude.ai/code/session_01H3xRwF4Zyiihrk3itYamRN


    Generated by Claude Code

  9. added
    claude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.
    and removed
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 25, 2026
  10. idemidenko commented on Oct 26, 2026

    @idemidenko
    Author

    A month-later data point, since the remaining half of this issue (the /compliance latency itself) was never reopened or moved to a new issue.

    Anonymous reads for https://sales-agent.elementaltv.io/mcp (84 storyboards) on 2026-10-26, all HTTP 200:

    endpoint time
    /compliance 26.1 s, 21.2 s, 20.3 s

    That matches the 22–30 s from September, so the per-storyboard cost inside getComplianceStatusWithStoryboardCounts that the triage above identified still appears to be there. #7664's progressive render keeps the owner controls usable, but anything that waits on this endpoint (the card body, API consumers) still pays the full ~20–26 s, and it grows with the suite.

    Could this be reopened, or tracked in a new issue if that's preferred?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    admin-toolInternal tools for AAO staffbugSomething isn't workingclaude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.compliance-suitepriority:P0Immediate priority: user/revenue/security/reliability bug to work nowseverity:critical

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions