Repository navigation
Description
Registry-wide heartbeat volume has collapsed over the last two days and has now effectively stopped. This is reproducible from the public API with no owner access, so it should be cheap to confirm or dismiss.
What the public feeds show
Pulling /compliance/history for all 35 registered agents (31 return rows; 886 rows visible in total, each feed capped at 30) and bucketing every run by day and trigger:
| date (UTC) | heartbeat runs | owner_test |
|---|---|---|
| 2026-09-08 | 34 | 2 |
| 2026-09-09 | 23 | 2 |
| 2026-09-10 | 27 | 0 |
| 2026-09-11 | 29 | 1 |
| 2026-09-12 | 24 | 2 |
| 2026-09-13 | 26 | 10 |
| 2026-09-14 | 5 | 3 |
| 2026-09-15 | 1 | 8 |
The newest heartbeat row anywhere in the registry is 2026-09-15T07:03:04.819Z on sales-agent.addiscovery.ai/mcp — 14.3 hours ago at the time of writing. Of the 30 agents that have ever been heartbeated, 29 have a newest heartbeat older than 24 hours and 18 older than 48. The only grading activity since 07:03Z is two owner_test runs on seller.purrsonality.rocketscience.pl/mcp at 16:20:29.957Z and 16:57:38.324Z — owner-initiated, not the scheduler.
The truncation cuts the other way, for the avoidance of doubt: a 30-row cap drops old rows, so it cannot manufacture a decline in the most recent days.
Second signal: cards are not advancing either
I snapshotted every agent's /compliance card at 11:05Z and again at 21:14Z today. Of the 33 agents present in both snapshots, zero had last_checked_at advance in those 10.2 hours. (A 34th, seller.purrsonality, was missing from the first snapshot so I have no baseline for it; its card did advance, on the two owner tests above.)
Against each agent's own declared check_interval_hours, the fleet median is now 5.2× its interval, with 30 of 32 past 2×, 21 past 4×, and 7 past 8×.
Why I think this is worth a look rather than a quiet backlog
This failure mode is already written down in this repo, as a known follow-up to my own earlier report. #4253 was closed on the strength of #4265, which fixed three real causes — the pre-stamp TTL, dry_run always true, and manual evals not reaching the dashboard. The closing comment then carved out what #4265 explicitly did not fix:
Scheduler
MAX_CONCURRENCY = 5with unboundedwaitQueue— 5 hung jobs can block the heartbeat indefinitely. Needs production diagnostic data from/api/jobs/statuslastErrorfield before a fix can be designed.
A bounded worker pool with an unbounded queue, where a handful of stuck jobs starve everything behind them, produces exactly this shape: not an error, not a gap for one agent, but throughput decaying to near-zero fleet-wide while the odd run still slips through. I can't see /api/jobs/status from outside, so I'm not asserting that is the cause — but it was named as the thing to check when production data showed up, and this is production data.
What I am not claiming
- Not that any particular agent is being singled out. Mine (
sell.nofluffadvisory.com/mcp) is in this set at 4.5× its interval, which is better than the fleet median; that is why I am filing this as a fleet observation rather than about my own card. - Not that this is the same defect as Compliance heartbeat scheduling can silently stall across agents #7355, which concerns one agent and a declaration-shape hypothesis.
- Not a diagnosis of the mechanism. Only that the public numbers show heartbeat throughput falling from ~26/day to 1/day across 2026-09-14 and 09-15, and that a matching mechanism is already documented as unfixed.
Two curl calls against any agent's /compliance and /compliance/history will confirm or refute the above in a minute. Happy to re-run the fleet sweep at any point, or to hand over the raw per-agent numbers.
Activity
bokelley commentedon Sep 15, 2026
Triage
Classification: Bug
Bucket(s): admin / ops tools, compliance-suite
Status: ready-to-implement
Milestone: P0 Bugs
What the experts said:
scheduler.tssetsjob.executing = truebeforeawait this.acquireSlot(), so a hung job holds its slot permanently;releaseSlot()is infinallyand never fires until the promise settles. Five hung jobs = five permanently occupied slots = everything else waits forever. Load-bearing fix:Promise.racetimeout inrunJob; queue depth cap is secondary hardening./internal/jobsfirst (minutes of work) to identify the stuck slots and calibrate the default timeout, then applyingexecutionTimeoutMs+Promise.racefix.My take: Both experts confirm the reporter's hypothesis directly from code — the #4265 "needs production diagnostic data" caveat is resolved by reading
scheduler.ts. Pure server-side fix, no protocol or schema impact.Implementation scope:
server/src/addie/jobs/scheduler.ts— addexecutionTimeoutMstoJobConfig; wrapconfig.runner(...)inPromise.racetimeout inrunJob; surfaceactiveJobs/queuedJobsingetStatus()server/src/addie/jobs/job-definitions.ts— setexecutionTimeoutMson the heartbeat job (and any other long-running job)/internal/jobsslot counts drain; monitor heartbeat recovery to ~26/dayTriaged by Claude Code. Session: https://claude.ai/code/session_01TcEqTfUv2WYbaFoyi3CBZN
Generated by Claude Code
EvgenyAndroid commentedon Sep 17, 2026
Thanks for the quick fix. One follow-up from the public data, as of 2026-09-17T11:01Z.
Public compliance-history rows with
triggered_by: heartbeatacross the 35 listed agents, per UTC day from 2026-09-11 to 2026-09-16: 30, 25, 27, 6, 1, 2. TheDeploy to Fly.iojob for the #7556 merge commit (fd398e1e) completed successfully at 2026-09-16T17:38:54Z. After that there are two heartbeat rows (2026-09-16T18:19Zthewrap-com.prebidselleragentgateway.optimera.nyc, 2026-09-16T18:23Zsignals.purrsonality.rocketscience.pl/mcp) and none since.Caveat: this is only what the public history can show. Since #7453, runs that time out are kept as audit evidence but withheld from the public grade and history. Heartbeat errors, skipped agents and batches aborted at the new two-hour cap also leave no public row, and the heartbeat can select agents outside the public listing. So heartbeats may be running without leaving public rows.
Question: right now, does
/api/admin/jobsshowactiveJobsatmaxConcurrencywithqueuedJobsabove zero?