Repository navigation
Conversation
getPreviousAlertHistories scanned a fixed 7-day window of AlertHistory per
alert on every tick and reduced it with a $sort + $group aggregation that
MongoDB does not short-circuit per group, so cost grew with history volume.
- Look up each alert's latest history row first and skip the history scan
for alerts already evaluated in the current window.
- Seed non-grouped alerts' previous state from that row; they never scan.
- Grouped alerts scan a window of 52 intervals clamped to [1d, 7d] with a
streamed find on {alert, createdAt}, reducing to the latest row per group
in code.
- getConsecutiveWindowHistories uses find + in-app sort for the same reason.
- A failing evaluator lookup during the skip check marks the alert as due
instead of failing the tick.
Ported from the enterprise repo.
Fixes #2434
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🦋 Changeset detectedLatest commit: 3511be9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🔴 Tier 4 — CriticalTouches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI. Why this tier:
Review process: Deep review from a domain expert. Synchronous walkthrough may be required. Stats
|
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
| const latest = await AlertHistory.findOne( | ||
| { | ||
| alert: new mongoose.Types.ObjectId(alertId), | ||
| createdAt: { $lte: now }, |
There was a problem hiding this comment.
🔵 minor — Non-grouped alerts now look back without any limit; before this PR the limit was 7 days
getLatestAlertHistories filters only on createdAt <= now, and the provider now uses that row as the previous state for non-grouped alerts. Before this PR, getPreviousAlertHistories dropped rows older than 7 days. Example: a non-grouped alert's source fails to load for 2 weeks, so resolveAlertDetails throws and no history rows are written. Its last row is an ALERT with fired=true. Once the source is fixed, getAlertEvaluationDateRange (index.ts:730) starts from that old row and backfills up to the calcAlertDateRange cap. The alert can also send a resolve notification for an incident from 2 weeks ago, which the old code never sent. Grouped alerts are limited to 1–7 days, so the two paths now disagree. Fix: add $gte: now - getAlertHistoryLookbackMs(alert.interval) (or a fixed 7d) to the findOne filter. That needs the interval passed in. The query stays a single index seek.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
PR Review
2 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 2 minor 1 posted as inline comment(s) on the changed lines. 1 listed below. Findings outside the changed lines1 minor
Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
E2E Test Results✅ All tests passed • 464 passed • 1 skipped • 1027s
Tests ran across 4 shards in parallel. |
Deep Review✅ No critical issues found. This PR bounds the check-alerts history lookup by alert interval: a cheap per-alert due-ness gate, interval-bounded streamed per-group scans, a catch-up path after outages, and removal of the now-unused 🟡 P2 — recommended
🔵 P3 nitpicks (4)
Reviewers (9): correctness (independent orchestrator verification), performance, reliability (independent orchestrator verification), testing, maintainability, project-standards, kieran-typescript, data-migrations, previous-comments, agent-native, learnings-researcher. Testing gaps:
No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
…onflicts
Use the enterprise build's text for the alert-history lookback, the latest-
and previous-history fetches, the provider's due-ness gate (now a loadAlerts()
helper), and their tests, so the upstream sync into the enterprise repo
conflicts only where that repo has its own code.
- Drop the unused { alert, group, createdAt } index declaration, as the
enterprise build did. Existing deployments keep the index; it is no longer
created on new ones.
- Keep the guard that treats an alert as due when its window start cannot
be computed, now as a helper at the end of providers/default.ts, and test
it by stubbing getAlertWindowStart instead of clearing the evaluators.
- Move the extra ERROR-row and index-plan tests into their own block.
Refs #2434
getLatestAlertHistories had no lower createdAt bound, and its
state != ERROR filter is not in the {alert, createdAt} index. For an
alert that keeps failing, every tick walked back through every ERROR row
until it found a non-ERROR one, up to the full 30-day retention. It also
let a non-grouped alert pick up a weeks-old row as its previous state.
Bound it by getAlertHistoryLookbackMs(interval), the same window
getPreviousAlertHistories uses, so the skip check, the grouped scan and
the non-grouped path agree. The function now takes alerts instead of IDs.
Also drop an `as any` in a test that pushed the API package over its
lint warning cap.
Refs #2434
… to any The escape-hatch ratchet caps 'as any' in the api package; the stub added one over the baseline. Refs #2434
| // The lower bound caps the walk past consecutive ERROR rows (the | ||
| // state filter is not in the index) for an alert that keeps | ||
| // failing, and matches getPreviousAlertHistories' window. | ||
| createdAt: { |
There was a problem hiding this comment.
🟠 major — Non-grouped alerts with ≤15m intervals now forget their previous state after 1 day instead of 7
The gate findOne in getLatestAlertHistories uses getAlertHistoryLookbackMs (1d for 1m–15m alerts), and loadAlerts now takes non-grouped previous state only from this row (providers/default.ts seeds previousAlerts from latest). So a 5m alert in ALERT state whose evaluations write only ERROR rows for more than 24h (for example, a ClickHouse outage) comes back with an empty previousMap. On recovery, sendNotificationIfResolved sees no previous ALERT and never sends the resolve notification. If the alert is still breaching, it fires a duplicate ALERT notification. The PR's trade-offs only mention this for groups. Fix: give the gate lookup the old 7d bound (or ALERT_HISTORY_MAX_LOOKBACK_MS). It is a one-row seek. Keep the interval-derived window for the grouped range scan only.
Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond
…terval The interval-derived bound (1 day for 1m to 15m alerts) made non-grouped alerts forget their previous state after 1 day. An alert in ALERT whose evaluations only wrote ERROR rows for longer than that would recover without sending its resolve notification. main kept that state for 7 days. Use ALERT_HISTORY_MAX_LOOKBACK_MS instead. It still caps the walk past ERROR rows, at the 7 days main already scanned, and restores the enterprise build's signature, so the call site and tests match it again. Refs #2434
| } | ||
| } | ||
|
|
||
| const [previousAlerts, recentHistoryMap] = await Promise.all([ |
There was a problem hiding this comment.
🟠 major — Grouped alerts lose their catch-up after an outage longer than the shorter lookback
Suppose a grouped alert's latest history row is older than getAlertHistoryLookbackMs(interval) but newer than 7 days. Example: a 1m alert after 30h of downtime, or a 1h alert after 60h. The 7-day check marks the alert due, but the shorter per-group scan returns nothing for it. getAlertEvaluationDateRange (index.ts:734) then falls back to now - 1 window. On main, the start was the old createdAt, which calcAlertDateRange caps at 50 windows (or 6h/24h), so the missed windows were evaluated. Now they are skipped. The comment at index.ts:663 says the lookback only has to cover the catch-up horizon. That holds only if an empty result still anchors the start, and it doesn't. Fix: after getPreviousAlertHistories, for each due grouped alert with no entries in previousAlerts, add the row from the 7-day check under computeHistoryMapKey(alert.id, latest.group || ''). This also restores that group's previous state.
Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond
…the scan The latest-row lookup covers 7 days, but the per-group scan covers only the alert's interval-derived window (1 day for fast alerts). After an outage longer than that window the scan returned nothing, so the evaluation started from now - 1 window and skipped the missed windows that main would have caught up on (up to 50). When the scan has no row for a due grouped alert, seed the row from the latest-row lookup. The evaluation starts from it again, and that group's previous state is restored. Refs #2434
…utage When a due grouped alert's latest row is older than its scan window, the provider seeded only that one row, without groupValues. A group that then resolved was notified with empty group attributes, and the other groups from that evaluation lost their previous state. Read all rows that share the latest row's createdAt instead: one per group, with groupValues. It is one extra query per affected alert, on the first tick after such an outage. Tests: the last evaluation is restored per group with its values; a grouped alert already evaluated this window is skipped from the loaded sentinel alone; the window-start failure test now tells "due" from "skipped". Refs #2434
| }); | ||
| } | ||
|
|
||
| // A grouped alert whose latest row is older than its scan window gets no |
There was a problem hiding this comment.
🔵 minor — Outage catch-up for grouped alerts sits in the provider, outside getPreviousAlertHistories, and repeats its row mapping
Since this change, getPreviousAlertHistories returns nothing for a grouped alert whose last evaluation is older than its scan window. Its docstring still says it returns the latest history per group. The fix-up (an exact-createdAt find) exists only in loadAlerts, so any other caller, including the tests that call the function directly, silently loses the catch-up. The same filter, projection and row-to-AggregatedAlertHistory mapping now also appear in getPreviousAlertHistories, getConsecutiveWindowHistories, the non-grouped seeding and this block. Move the fallback into getPreviousAlertHistories, for example by accepting an optional latestHistories map, and share one row-mapping helper.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Summary
Every check-alerts tick ran
getPreviousAlertHistoriesfor every alert. It scanned a fixed 7-day window ofAlertHistoryand reduced it with$sort+$group/$first. MongoDB runs that as a plainIXSCANover the whole window, not a per-group short-circuit. The cost grows with history volume, and the slow-query log fills up (see #2434).This PR ports the fix that already runs in the enterprise build. The code and comments are copied from it nearly verbatim, so the upstream sync into that repo conflicts only where it has its own code.
Skip check. One
findOneper alert on{alert: 1, createdAt: -1}fetches the latest non-ERROR history row from the last 7 days. If that row is in the current window,processAlertwould skip the alert anyway, so no history scan runs. A sentinel entry keeps the existingshouldSkipAlertCheckpath and its metric. The 7-day bound caps how far the lookup walks past ERROR rows for an alert that keeps failing, and matches how longmainremembered a non-grouped alert's state.Non-grouped alerts take their previous state from that row and never scan.
Grouped alerts scan a window of
MAX_NUM_WINDOWS + 2(52) intervals, clamped to between 1 and 7 days:The scan is a streamed
find()on{alert, createdAt}, reduced to the latest row per group in code, so memory stays at one batch plus one entry per group.Catch-up after a long outage. If a due grouped alert's latest row is older than its scan window, the scan returns nothing. The provider then reads every row from that last evaluation (one per group, with its group-by values), so the evaluation still starts from it and backfills the missed windows, as on
main, and each of those groups keeps its previous state.getConsecutiveWindowHistoriesusesfind().lean()with an in-code sort instead of an aggregation with a group-major$sort.The provider now resolves all alerts first, then runs the skip check and the history fetches, in a
loadAlerts()helper laid out like the enterprise one.Evaluator failures. If an alert's window start cannot be computed during the skip check (its evaluator lookup throws), the alert is treated as due instead of failing the whole tick.
processAlertthen records the error on that alert, as it does today. The enterprise build does not have this guard yet.Index cleanup. The
{alert, group, createdAt}index declaration is removed, because no query uses it now. OSS does not drop indexes at startup, so existing deployments keep the index; new ones no longer create it.Trade-offs
statefilter is not in the index, so for an alert that only writes ERROR rows the skip-check lookup reads up to 7 days of them per tick (about 10,000 for a 1m alert), asmainalready does. Healthy alerts read one index key.How to reproduce
Starting point:
yarn devwith at least one alert, ormake dev-int.On
main:alerthistoriesaggregate. Every tick runs one aggregate per alert, andkeysExaminedgrows with the full 7-day window.On this branch:
Run the same setup. Non-grouped alerts issue only a one-key
findOneper tick. Grouped alerts issue one boundedfindonly when they are due.Run the integration tests:
make dev-int FILE=src/tasks/checkAlerts/__tests__/checkAlerts.int.test.tsmake dev-int FILE=src/tasks/checkAlerts/providers/__tests__/default.int.test.tsA new test runs
explainon the grouped scan and checks that it usesalert_1_createdAt_-1with tight bounds.Testing
packages/apiunit tests pass.checkAlerts,providers/default,singleInvocationAlertandmultiChannelAlerts.findper alert, with no aggregationprocessAlertfrom the sentinel aloneReferences
getPreviousAlertHistoriesreads every history row of the last 7 days, so check-alerts slows down and runs out of memory as history grows #3508 (partial). The grouped scan is capped at 1 day and streamed, so it no longer grows for a week and memory stays bounded. A 1m alert with ~1,300 groups still reads about 1 day of rows per tick, so the scan cost that issue reports is reduced, not removed.getPreviousAlertHistoriesreads every history row of the last 7 days, so check-alerts slows down and runs out of memory as history grows #3508 with a{alert, group, createdAt, state}index and a per-groupDISTINCT_SCAN. This PR removes the{alert, group, createdAt}index declaration and the aggregation, so the two conflict.