Skip to content

Heartbeat scheduler has effectively stopped registry-wide — ~26 runs/day to 1, no heartbeat anywhere in 14 hours #7546

Closed

Description

@EvgenyAndroid

Registry-wide heartbeat volume has collapsed over the last two days and has now effectively stopped. This is reproducible from the public API with no owner access, so it should be cheap to confirm or dismiss.

What the public feeds show

Pulling /compliance/history for all 35 registered agents (31 return rows; 886 rows visible in total, each feed capped at 30) and bucketing every run by day and trigger:

date (UTC) heartbeat runs owner_test
2026-09-08 34 2
2026-09-09 23 2
2026-09-10 27 0
2026-09-11 29 1
2026-09-12 24 2
2026-09-13 26 10
2026-09-14 5 3
2026-09-15 1 8

The newest heartbeat row anywhere in the registry is 2026-09-15T07:03:04.819Z on sales-agent.addiscovery.ai/mcp — 14.3 hours ago at the time of writing. Of the 30 agents that have ever been heartbeated, 29 have a newest heartbeat older than 24 hours and 18 older than 48. The only grading activity since 07:03Z is two owner_test runs on seller.purrsonality.rocketscience.pl/mcp at 16:20:29.957Z and 16:57:38.324Z — owner-initiated, not the scheduler.

The truncation cuts the other way, for the avoidance of doubt: a 30-row cap drops old rows, so it cannot manufacture a decline in the most recent days.

Second signal: cards are not advancing either

I snapshotted every agent's /compliance card at 11:05Z and again at 21:14Z today. Of the 33 agents present in both snapshots, zero had last_checked_at advance in those 10.2 hours. (A 34th, seller.purrsonality, was missing from the first snapshot so I have no baseline for it; its card did advance, on the two owner tests above.)

Against each agent's own declared check_interval_hours, the fleet median is now 5.2× its interval, with 30 of 32 past 2×, 21 past 4×, and 7 past 8×.

Why I think this is worth a look rather than a quiet backlog

This failure mode is already written down in this repo, as a known follow-up to my own earlier report. #4253 was closed on the strength of #4265, which fixed three real causes — the pre-stamp TTL, dry_run always true, and manual evals not reaching the dashboard. The closing comment then carved out what #4265 explicitly did not fix:

Scheduler MAX_CONCURRENCY = 5 with unbounded waitQueue — 5 hung jobs can block the heartbeat indefinitely. Needs production diagnostic data from /api/jobs/status lastError field before a fix can be designed.

A bounded worker pool with an unbounded queue, where a handful of stuck jobs starve everything behind them, produces exactly this shape: not an error, not a gap for one agent, but throughput decaying to near-zero fleet-wide while the odd run still slips through. I can't see /api/jobs/status from outside, so I'm not asserting that is the cause — but it was named as the thing to check when production data showed up, and this is production data.

What I am not claiming

  • Not that any particular agent is being singled out. Mine (sell.nofluffadvisory.com/mcp) is in this set at 4.5× its interval, which is better than the fleet median; that is why I am filing this as a fleet observation rather than about my own card.
  • Not that this is the same defect as Compliance heartbeat scheduling can silently stall across agents #7355, which concerns one agent and a declaration-shape hypothesis.
  • Not a diagnosis of the mechanism. Only that the public numbers show heartbeat throughput falling from ~26/day to 1/day across 2026-09-14 and 09-15, and that a matching mechanism is already documented as unfixed.

Two curl calls against any agent's /compliance and /compliance/history will confirm or refute the above in a minute. Happy to re-run the fleet sweep at any point, or to hand over the raw per-agent numbers.

Activity

  1. added
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 15, 2026
  2. added this to the P0 Bugs milestone on Sep 15, 2026
  3. bokelley commented on Sep 15, 2026

    @bokelley
    Contributor

    Triage

    Classification: Bug
    Bucket(s): admin / ops tools, compliance-suite
    Status: ready-to-implement
    Milestone: P0 Bugs

    What the experts said:

    • code-reviewer: Mechanism confirmed from code — scheduler.ts sets job.executing = true before await this.acquireSlot(), so a hung job holds its slot permanently; releaseSlot() is in finally and never fires until the promise settles. Five hung jobs = five permanently occupied slots = everything else waits forever. Load-bearing fix: Promise.race timeout in runJob; queue depth cap is secondary hardening.
    • internal-tools-strategist: Same reading. Recommends pulling /internal/jobs first (minutes of work) to identify the stuck slots and calibrate the default timeout, then applying executionTimeoutMs + Promise.race fix.

    My take: Both experts confirm the reporter's hypothesis directly from code — the #4265 "needs production diagnostic data" caveat is resolved by reading scheduler.ts. Pure server-side fix, no protocol or schema impact.

    Implementation scope:

    • server/src/addie/jobs/scheduler.ts — add executionTimeoutMs to JobConfig; wrap config.runner(...) in Promise.race timeout in runJob; surface activeJobs/queuedJobs in getStatus()
    • server/src/addie/jobs/job-definitions.ts — set executionTimeoutMs on the heartbeat job (and any other long-running job)
    • No changeset (server-only, not AdCP protocol surface); no wire changes
    • Stage deploy → verify /internal/jobs slot counts drain; monitor heartbeat recovery to ~26/day

    Triaged by Claude Code. Session: https://claude.ai/code/session_01TcEqTfUv2WYbaFoyi3CBZN


    Generated by Claude Code

  4. added
    bugSomething isn't working
    claude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.
    priority:P0Immediate priority: user/revenue/security/reliability bug to work now
    and removed
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 15, 2026
  5. EvgenyAndroid commented on Sep 17, 2026

    @EvgenyAndroid
    ContributorAuthor

    Thanks for the quick fix. One follow-up from the public data, as of 2026-09-17T11:01Z.

    Public compliance-history rows with triggered_by: heartbeat across the 35 listed agents, per UTC day from 2026-09-11 to 2026-09-16: 30, 25, 27, 6, 1, 2. The Deploy to Fly.io job for the #7556 merge commit (fd398e1e) completed successfully at 2026-09-16T17:38:54Z. After that there are two heartbeat rows (2026-09-16T18:19Z thewrap-com.prebidselleragentgateway.optimera.nyc, 2026-09-16T18:23Z signals.purrsonality.rocketscience.pl/mcp) and none since.

    Caveat: this is only what the public history can show. Since #7453, runs that time out are kept as audit evidence but withheld from the public grade and history. Heartbeat errors, skipped agents and batches aborted at the new two-hour cap also leave no public row, and the heartbeat can select agents outside the public listing. So heartbeats may be running without leaving public rows.

    Question: right now, does /api/admin/jobs show activeJobs at maxConcurrency with queuedJobs above zero?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingclaude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.community-reportedcompliance-suitepriority:P0Immediate priority: user/revenue/security/reliability bug to work now

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions