Feature/stale job monitor#5558
Open
brendankowitz wants to merge 13 commits into
Open
Conversation
brendankowitz
added
New Feature
Spec for a StaleJobWatchdog that emits fhir_oldest_queued_job_age_seconds Prometheus gauge per queue type when no jobs are running. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
6-task plan: notification, metric handler, watchdog, WatchdogsBackgroundService wiring, DI registration, integration test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Previously, a single running job in any queue masked staleness in every other queue, defeating per-queue-type alerting. ComputeQueueAges now evaluates the running check per queue. - StaleJobMetricHandler swapped from a per-key-updated ConcurrentDictionary to a volatile reference swap so ObservableGauge scrapes never observe a partial multi-queue update. - Added logic test asserting a running job in one queue does not suppress another queue's staleness. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add an ObservableGauge<long> named Jobs.QueueDepth to the existing StaleJobMetricHandler, using the same per-tick SQL result set already fetched by StaleJobWatchdog. Reports pending (Created) and running job counts per QueueType via queue_type and state tags, complementing the existing Jobs.OldestQueuedAgeSeconds metric for full active-queue observability. ADR 2605 amended with the depth metric decision. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The watchdog reports both stale-queue age and queue depth metrics, so JobMonitorWatchdog more accurately describes its broader monitoring role. Renames the watchdog and its companion notification and metric handler, moves StaleJobMetricHandler into the Logging/Metrics/Handlers folder to match the post-rebase main layout (PR #5555 moved metric handlers there), and updates the Features/Operations/StaleJob folder to Features/Operations/JobMonitor. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
brendankowitz
force-pushed
the
feature/stale-job-monitor
branch
from
b3a52ff to
98da39a
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5558 +/- ##
==========================================
- Coverage 77.36% 77.33% -0.03%
==========================================
Files 993 997 +4
Lines 36418 36500 +82
Branches 5518 5529 +11
==========================================
+ Hits 28175 28228 +53
- Misses 6884 6911 +27
- Partials 1359 1361 +2 🚀 New features to boost your workflow:
|
- Mark _now as readonly in JobMonitorWatchdogLogicTests - Replace ContainsKey+indexer with TryGetValue in SqlServerWatchdogTests to avoid double dictionary lookups Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
brendankowitz
force-pushed
the
feature/stale-job-monitor
branch
from
4487a78 to
230fbb1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request introduces a stale job monitor for SQL-backed async job queues. The monitor reports the age of the oldest queued job per queue type so stalled queues can be detected before customers observe delayed operations.
Key changes
Stale job monitor
StaleJobWatchdogto query active jobs for eachQueueType, compute the oldest queued job age per queue, log stale queues, and publishStaleJobMetricsNotification.StaleJobMetricsNotificationandStaleJobMetricHandlerto expose the latest queue-age snapshot through theFhirServermeter asJobs.OldestQueuedAgeSecondswith aqueue_typetag.Dependency injection and background service integration
StaleJobWatchdogas a singleton in SQL Server service registration.StaleJobMetricHandleras a singleton MediatR notification handler so the observable gauge reads a stable metric snapshot.WatchdogsBackgroundServiceto startStaleJobWatchdogwith the existing SQL watchdogs.Testing and documentation
docs/arch/adr-2605-stale-job-monitor.md.Related issues
Addresses AB#164461.
Testing
dotnet build .\src\Microsoft.Health.Fhir.Core\Microsoft.Health.Fhir.Core.csproj -c Release -f net8.0 --no-restoredotnet build .\src\Microsoft.Health.Fhir.Core\Microsoft.Health.Fhir.Core.csproj -c Release -f net9.0 --no-restoredotnet build .\src\Microsoft.Health.Fhir.SqlServer\Microsoft.Health.Fhir.SqlServer.csproj -c Release -f net8.0 --no-restoredotnet build .\src\Microsoft.Health.Fhir.SqlServer\Microsoft.Health.Fhir.SqlServer.csproj -c Release -f net9.0 --no-restoreFHIR Team Checklist
New Feature.Azure Healthcare APIs.No-PaaS-breaking-change.docs/arch/adr-2605-stale-job-monitor.md.Semver Change
Feature