Skip to content

Owners cannot obtain an authoritative compliance run: human refresh fenced with no ETA, and requeue cannot reach agents with a recent last_checked_at (heartbeat orders by last_checked_at ASC) #7632

Closed

Description

@idemidenko

Summary

Since the 2026-09-17 auth sweep (#7452, provenance follow-up #7457) an agent owner has no way to obtain an authoritative compliance run:

  1. POST /api/registry/agents/{url}/refresh returns 503 refresh_authorization_provenance_required for every non-static-admin caller. The code comment says this is deliberate and has no environment switch, but there is no tracking issue and no ETA for restoring human refresh.
  2. The alternative the fence notice points to, "Requeue comply" (/monitoring/requeue), cannot help an agent that is already due: requeueForHeartbeat only sets next_compliance_check_at = NULL, and the heartbeat batch is ordered by last_checked_at ASC NULLS FIRST LIMIT 10. An owner who used "Recheck & retest" while it still worked has a recent last_checked_at and is therefore sorted behind every agent checked earlier.

Net effect for https://sales-agent.elementaltv.io/mcp (org org_01KESCR4BWMJSFEHHG00B78GN4, media-buy 3.1 badge): we ship compliance fixes (e.g. #7406's canonical_format_validate_input, deploying today) and cannot get them graded.

Evidence

Heartbeat is running for other agents but has not reached ours since the sweep (from /compliance/history, 2026-09-21):

agent last heartbeat run last run of any kind
sales-agent.elementaltv.io/mcp 2026-09-14T03:17Z 2026-09-17T08:09Z (owner_test)
adcp.advendio.com/mcp/demo 2026-09-18T00:11Z same
sales-agent.addiscovery.ai/mcp 2026-09-15T07:03Z same

Our check_interval_hours is 6, compliance_opt_out false, lifecycle_stage production; 35 of our last 50 runs were heartbeats, so the agent was in rotation until the week of the sweep.

Selection query (compliance-db.ts getAgentsDueForCheck):

WHERE … AND (m.next_compliance_check_at IS NULL OR m.next_compliance_check_at < NOW())
ORDER BY last_checked_at ASC NULLS FIRST, agent_url ASC
LIMIT $1   -- 10 per tick (compliance-heartbeat.ts)

requeueForHeartbeat (compliance-db.ts) writes only next_compliance_check_at = NULL, which does not change the ORDER BY position.

Owner controls today: "Recheck & retest" → 503; "Requeue comply" → 200 but no effect on ordering (and the button itself only renders on a hydrated card, see #7630); per-storyboard /storyboard/{id}/run works but is recorded is_authoritative: false, so it cannot update the aggregate status or badge.

Asks

  1. A tracking issue (or a note here) for restoring human-initiated refresh, with a rough ETA, so owners know whether to wait or route around it.
  2. An interim path to an authoritative run while the fence is up: either an admin-triggered run on request through a documented channel, or a heartbeat priority for explicitly requeued agents.
  3. Make requeue actually move the agent forward: e.g. requeueForHeartbeat also sets a requeued_at that the batch orders by first (ORDER BY requeued_at ASC NULLS LAST, last_checked_at ASC NULLS FIRST), or clears last_checked_at for ordering purposes. Otherwise the fence notice recommends an action that cannot reach the agents it is shown to.

Related: #7578, #7630, #7452, #7457.

Activity

  1. added
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 21, 2026
  2. added this to the P0 Bugs milestone on Sep 21, 2026
  3. bokelley commented on Sep 21, 2026

    @bokelley
    Contributor

    Thanks for filing!

    Triage

    Classification: Bug (bugs 1 & 2 confirmed; bug 3 hypothesis retracted — 2026-09-22)
    Bucket(s): compliance-suite, admin-tool
    Status: deferred — draft PR #7648 open for bugs 1 & 2
    Milestone: P0 Bugs

    What the experts said:

    My take: Immediate unblock: request an admin-triggered refresh via a documented channel (no code needed — static admin key path is live). Permanent fix is the requeued_at column — draft PR #7648 open. Dependency: #7630 must also land so the Requeue button is reachable before the ordering fix helps anyone.

    Implementation scope (covered by #7648):

    • Immediate (no code): Establish a documented admin-request channel (Slack/email) for owners to request a refresh run while the fence is up; post it in the 503 response and/or the fence notice
    • registry-api.ts:HUMAN_REFRESH_AVAILABILITY — add tracking_issue: 7457 (or docs_url) so the 503 body carries a reference owners can follow
    • Migration: ALTER TABLE agent_registry_metadata ADD COLUMN requeued_at TIMESTAMPTZ NULL — no backfill
    • compliance-db.ts:requeueForHeartbeat — also write requeued_at = NOW()
    • compliance-db.ts:getAgentsDueForCheck — ORDER BY requeued_at ASC NULLS LAST, last_checked_at ASC NULLS FIRST, agent_url ASC
    • Clear requeued_at in recordComplianceRun / compliance-heartbeat.ts on successful authoritative pickup
    • getAuthInfoByOrgAndUrl ORDER BY fix — retracted 2026-09-22: agent_contexts has had UNIQUE (organization_id, agent_url) since migration 109; migration 525 collapsed URL variants; rows[0] cannot flip between duplicate rows. The 02:30Z–07:57Z bearer change on 2026-09-21 still needs a separate explanation and is not addressed by fix(registry): serve explicit heartbeat requeues first and reference the refresh fence tracking issue #7648.
    • No changeset (app-layer only; no protocol/schema surface touched)
    • Dependency: Dashboard card still stuck on 'not yet checked' after #7594: /compliance takes 22–30 s for an 84-storyboard agent, client aborts at 15 s #7630

    Sibling-repo-fix-needed (adcp-client, surfaced 2026-09-24):

    The 2026-09-24 comment reports a successful local run (242 passed / 0 failed with bundle 3.1.24 grafted from protocol/3.1.24.tgz), but surfaces three blockers for moving hosted grading onto the stable SDK line. These are adcp-client bugs and require separate issues there:

    1. @adcp/sdk@13.1.1 bundles 3.1.20, not 3.1.24 — compliance/cache/3.1.24 is absent; hosted grading must graft the bundle externally or a 13.1.x release must include it. File in adcontextprotocol/adcp-client.
    2. Capabilities probe validates against closed pre-3.1 compliance_testing.scenarios enum — schemas-data/3.0 and 3.1.0-beta.7 have a closed enum; the 3.1 set is open, but the probe appears to negotiate against the pre-3.1 schema first, emitting capabilities-probe-failed and skipping domain bundles unless --compliance-version is given explicitly. This is the Fix for #6725 (compliance_testing.scenarios closed-enum check) ships only on adcp-client 14.x, but graded compliance cache 3.1.20 ships only on 13.x — no release carries both #7368 symptom surviving on the stable line. File in adcontextprotocol/adcp-client.
    3. 13.x flat 120 s budget — runs stop after 71 of 76 storyboards; the 14.x max(120, 10 × n) scaled-budget rule is needed (or a backport) for full assessments on large agent surfaces. File in adcontextprotocol/adcp-client.

    The admin-triggered refresh ask (org org_01KESCR4BWMJSFEHHG00B78GN4) remains pending for a maintainer with the static admin key. Once #7648 lands and the above SDK issues are addressed, hosted grading should be able to produce an authoritative 3.1.24 run without grafting.


    Triaged by Claude Code. Session: https://claude.ai/code/session_01MVcnutrTNFh6JMxzcnMnV2 · Updated: https://claude.ai/code/session_01Q7v9tG2vLhoLK7kSqhWnw2

  4. added
    bugSomething isn't working
    admin-toolInternal tools for AAO staff
    claude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.
    priority:P0Immediate priority: user/revenue/security/reliability bug to work now
    and removed
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 21, 2026
  5. idemidenko commented on Sep 21, 2026

    @idemidenko
    Author

    Taking the triage's immediate unblock literally, since no request channel exists yet: could someone with the static admin key run POST /api/registry/agents/https%3A%2F%2Fsales-agent.elementaltv.io%2Fmcp/refresh (org org_01KESCR4BWMJSFEHHG00B78GN4)?

    The build we want graded is live as of 2026-09-21 ~11:00Z: anonymous tools/list now returns 15 tools including validate_input (previously 14), so canonical_format_validate_input can be exercised for the first time on the hosted runner. Our last authoritative run is still 2026-09-17T08:09Z.

    Happy to use whatever channel you designate for this going forward.

  6. idemidenko commented on Sep 21, 2026

    @idemidenko
    Author

    Update, and one more finding for the same cluster.

    Owner storyboard runs work again after re-saving the bearer; the saved credential silently changed on the registry side. Timeline from /monitoring/requests for https://sales-agent.elementaltv.io/mcp (org org_01KESCR4BWMJSFEHHG00B78GN4):

    • every health_check / discovery call succeeded through 2026-09-21T02:30:19Z
    • from 07:57:36Z every call failed with MCP error 0: [AUTH_401] [unknown-bearer-token] Unknown bearer token.
    • auth-status reported has_auth: true, auth_type: bearer throughout; POST /storyboard/{id}/run returned 422 needs_oauth (the SDK's auth_required path)

    Nothing changed on the agent side in that window: no deploy or pod restart until 10:45Z, no writes to the principal store, and the token registered in the dashboard still hashes to the agent's sandbox principal (verified against the agent's own DB; an authenticated call with it succeeds). Re-entering the same bearer through the connect form at ~11:20Z fixed it immediately: canonical_format_validate_input then ran and graded passing 16/16 (run 960c9ecd-4c32-4106-b103-05f1676df3d3, is_authoritative: false).

    So between 02:30Z and 07:57Z the registry started presenting a different bearer for this agent. getAuthInfoByOrgAndUrl selects FROM agent_contexts WHERE organization_id = $1 AND agent_url = $2 and takes rows[0] with no ordering; if more than one row exists for that pair, a change in physical row order (e.g. the migration in #7457 or #7622's deploy) would flip which token is sent. Worth checking how many agent_contexts rows that pair has on production.

    The admin-triggered authoritative refresh requested above is still the ask: the owner run is non-authoritative and does not update the storyboard status rows, aggregate, or badge.

  7. idemidenko commented on Sep 22, 2026

    @idemidenko
    Author

    Opened #7648 (draft) for the permanent fix from the triage: requeued_at column, requeueForHeartbeat sets it, getAgentsDueForCheck orders by it first, cleared when an authoritative run is published; plus tracking_issue (#7457) on the 503 body, refresh_availability, OpenAPI, and the dashboard notice. New integration case covers the ordering flip and the reset.

    One correction to bug 3 in the triage: agent_contexts has had UNIQUE (organization_id, agent_url) since migration 109, and migration 525 collapsed URL variants, so rows[0] in getAuthInfoByOrgAndUrl cannot flip between duplicate rows. I left that change out of #7648; the bearer change between 02:30Z and 07:57Z on 2026-09-21 still needs another explanation.

    The interim ask stands: could someone with the static admin key run POST /api/registry/agents/https%3A%2F%2Fsales-agent.elementaltv.io%2Fmcp/refresh (org org_01KESCR4BWMJSFEHHG00B78GN4)? Last authoritative run is still 2026-09-17T08:09Z.

  8. idemidenko commented on Sep 24, 2026

    @idemidenko
    Author

    Regrade request now that 3.1.24 and @adcp/sdk@13.1.1 are published.

    I ran a local full assessment against prod (https://sales-agent.elementaltv.io/mcp, build 24ebc9a) with @adcp/sdk@13.1.1 and bundle 3.1.24. The bundle was grafted from protocol/3.1.24.tgz, sha256 f9e451ab…25eb4c3.

    The ask is unchanged: could someone with the admin key run POST /api/registry/agents/https%3A%2F%2Fsales-agent.elementaltv.io%2Fmcp/refresh (org org_01KESCR4BWMJSFEHHG00B78GN4) once hosted grading uses 3.1.24? Our last authoritative run is 2026-09-17T08:09Z, on 3.1.20 with 14.0.0-rc.35.

    Three things I hit that matter for moving hosted grading onto the stable line:

    1. @adcp/sdk@13.1.1 does not ship compliance/cache/3.1.24. It ships 3.1.20. Hosted grading has to graft the bundle, or a 13.1.x release has to include it.
    2. The capabilities probe on 13.1.1 still validates compliance_testing.scenarios against the closed pre-3.1 enum. It reports must be one of: "force_creative_status", …, "simulate_budget_spend" and emits the capabilities-probe-failed error ("domain and specialism bundles were skipped"). In this run the media-buy bundles still executed because --compliance-version was given. The closed enum exists only in the SDK's schemas-data/3.0 and 3.1.0-beta.7 sets; the 3.1 set is open. So the probe appears to validate against a pre-3.1 schema before version negotiation. This is the Fix for #6725 (compliance_testing.scenarios closed-enum check) ships only on adcp-client 14.x, but graded compliance cache 3.1.20 ships only on 13.x — no release carries both #7368 symptom surviving on the stable line.
    3. 13.x defaults to a flat 120 s budget. With it, the run stopped after 71 of 76 selected storyboards. It needs a scaled budget (or the 14.x max(120, 10 × n) rule) to cover a full assessment.
  9. added a commit that references this issue on Sep 24, 2026
    2e7193b
  10. idemidenko commented on Sep 25, 2026

    @idemidenko
    Author

    Thanks for merging #7648. One follow-up, since the regrade ask above is now on a closed issue:

    Our agent (https://sales-agent.elementaltv.io/mcp) still has no authoritative run after 2026-09-17T08:09Z (owner_test, 14.0.0-rc.35, bundle 3.1.20). The requeue fix only changes queue order, so it doesn't start a new run by itself.

    Could someone with the admin key run POST /api/registry/agents/https%3A%2F%2Fsales-agent.elementaltv.io%2Fmcp/refresh (org org_01KESCR4BWMJSFEHHG00B78GN4), preferably once hosted grading is on 3.1.24? The local 3.1.24 + @adcp/sdk@13.1.1 result is in the comment above: 242 passed, 0 failed. If there's a better place to track the hosted move to 3.1.24, I'm happy to open an issue for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    admin-toolInternal tools for AAO staffbugSomething isn't workingclaude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.compliance-suitepriority:P0Immediate priority: user/revenue/security/reliability bug to work nowseverity:critical

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions