Skip to content

perf(api): bound check-alerts history lookup by alert interval - #3702

Open
wrn14897 wants to merge 7 commits into
mainfrom
warren/issue-2434
Open

wrn14897 wants to merge 7 commits into
mainfrom
warren/issue-2434

Conversation

@wrn14897

@wrn14897 wrn14897 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Every check-alerts tick ran getPreviousAlertHistories for every alert. It scanned a fixed 7-day window of AlertHistory and reduced it with $sort + $group/$first. MongoDB runs that as a plain IXSCAN over the whole window, not a per-group short-circuit. The cost grows with history volume, and the slow-query log fills up (see #2434).

This PR ports the fix that already runs in the enterprise build. The code and comments are copied from it nearly verbatim, so the upstream sync into that repo conflicts only where it has its own code.

  • Skip check. One findOne per alert on {alert: 1, createdAt: -1} fetches the latest non-ERROR history row from the last 7 days. If that row is in the current window, processAlert would skip the alert anyway, so no history scan runs. A sentinel entry keeps the existing shouldSkipAlertCheck path and its metric. The 7-day bound caps how far the lookup walks past ERROR rows for an alert that keeps failing, and matches how long main remembered a non-grouped alert's state.

  • Non-grouped alerts take their previous state from that row and never scan.

  • Grouped alerts scan a window of MAX_NUM_WINDOWS + 2 (52) intervals, clamped to between 1 and 7 days:

    • 1m to 15m: 1 day
    • 30m: 26 hours
    • 1h: 52 hours
    • 6h and longer: 7 days

    The scan is a streamed find() on {alert, createdAt}, reduced to the latest row per group in code, so memory stays at one batch plus one entry per group.

  • Catch-up after a long outage. If a due grouped alert's latest row is older than its scan window, the scan returns nothing. The provider then reads every row from that last evaluation (one per group, with its group-by values), so the evaluation still starts from it and backfills the missed windows, as on main, and each of those groups keeps its previous state.

  • getConsecutiveWindowHistories uses find().lean() with an in-code sort instead of an aggregation with a group-major $sort.

  • The provider now resolves all alerts first, then runs the skip check and the history fetches, in a loadAlerts() helper laid out like the enterprise one.

  • Evaluator failures. If an alert's window start cannot be computed during the skip check (its evaluator lookup throws), the alert is treated as due instead of failing the whole tick. processAlert then records the error on that alert, as it does today. The enterprise build does not have this guard yet.

  • Index cleanup. The {alert, group, createdAt} index declaration is removed, because no query uses it now. OSS does not drop indexes at startup, so existing deployments keep the index; new ones no longer create it.

Trade-offs

  • Fast grouped alerts still scan about 1 day. The 1-day minimum lets groups whose last row predates a task outage still auto-resolve. For a 1m grouped alert this cuts keys examined about 7×, not down to a handful.
  • A failing alert's lookup still walks its ERROR rows. The state filter is not in the index, so for an alert that only writes ERROR rows the skip-check lookup reads up to 7 days of them per tick (about 10,000 for a 1m alert), as main already does. Healthy alerts read one index key.
  • Some groups are forgotten sooner. A group stuck in ALERT that writes no new rows (threshold met at 0) now drops out of the previous-state map after the shorter window, not after 7 days. This is the case raised in the issue thread.
  • Some comments describe DocumentDB and cite internal ticket IDs. They were kept verbatim to avoid sync conflicts.

How to reproduce

Starting point: yarn dev with at least one alert, or make dev-int.

On main:

  1. Create a 1m alert and let it run for a while.
  2. Profile the alerthistories aggregate. Every tick runs one aggregate per alert, and keysExamined grows with the full 7-day window.

On this branch:

  1. Run the same setup. Non-grouped alerts issue only a one-key findOne per tick. Grouped alerts issue one bounded find only when they are due.

  2. Run the integration tests:

    • make dev-int FILE=src/tasks/checkAlerts/__tests__/checkAlerts.int.test.ts
    • make dev-int FILE=src/tasks/checkAlerts/providers/__tests__/default.int.test.ts

    A new test runs explain on the grouped scan and checks that it uses alert_1_createdAt_-1 with tight bounds.

Testing

  • packages/api unit tests pass.
  • These integration suites pass locally: checkAlerts, providers/default, singleInvocationAlert and multiChannelAlerts.
  • New tests cover:
    • the lookback calculation
    • per-group results
    • ERROR rows being ignored
    • the window bound
    • one streamed find per alert, with no aggregation
    • the index plan
    • the 7-day bound on the skip-check lookup
    • a grouped alert whose last evaluation is older than its scan window: every group is restored with its values
    • a grouped alert already evaluated this window is skipped by processAlert from the sentinel alone
    • the skip check: already evaluated, grouped and due, non-grouped and due, no history, and a window start that cannot be computed

References

getPreviousAlertHistories scanned a fixed 7-day window of AlertHistory per
alert on every tick and reduced it with a $sort + $group aggregation that
MongoDB does not short-circuit per group, so cost grew with history volume.

- Look up each alert's latest history row first and skip the history scan
  for alerts already evaluated in the current window.
- Seed non-grouped alerts' previous state from that row; they never scan.
- Grouped alerts scan a window of 52 intervals clamped to [1d, 7d] with a
  streamed find on {alert, createdAt}, reducing to the latest row per group
  in code.
- getConsecutiveWindowHistories uses find + in-app sort for the same reason.
- A failing evaluator lookup during the skip check marks the alert as due
  instead of failing the tick.

Ported from the enterprise repo.

Fixes #2434
@wrn14897
wrn14897 requested a review from a team as a code owner November 17, 2026 21:47
@vercel

vercel Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 18, 2026 12:28am UTC
hyperdx-storybook Ready Ready Preview Nov 18, 2026 12:28am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3511be9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Background tasks or delivery pipeline substantially modified — 567 lines (bar: 30):
    • packages/api/src/tasks/checkAlerts/index.ts
    • packages/api/src/tasks/checkAlerts/providers/default.ts
    • packages/api/src/tasks/util.ts

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 4
  • Production lines changed: 576 (+ 718 in test files, excluded from tier calculation)
  • Critical-path lines changed: 567
  • Branch: warren/issue-2434
  • Author: wrn14897

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.17%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

const latest = await AlertHistory.findOne(
{
alert: new mongoose.Types.ObjectId(alertId),
createdAt: { $lte: now },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Non-grouped alerts now look back without any limit; before this PR the limit was 7 days

getLatestAlertHistories filters only on createdAt <= now, and the provider now uses that row as the previous state for non-grouped alerts. Before this PR, getPreviousAlertHistories dropped rows older than 7 days. Example: a non-grouped alert's source fails to load for 2 weeks, so resolveAlertDetails throws and no history rows are written. Its last row is an ALERT with fired=true. Once the source is fixed, getAlertEvaluationDateRange (index.ts:730) starts from that old row and backfills up to the calcAlertDateRange cap. The alert can also send a resolve notification for an incident from 2 weeks ago, which the old code never sent. Grouped alerts are limited to 1–7 days, so the two paths now disagree. Fix: add $gte: now - getAlertHistoryLookbackMs(alert.interval) (or a fixed 7d) to the findOne filter. That needs the interval passed in. The query stays a single index seek.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

2 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 2 minor

1 posted as inline comment(s) on the changed lines. 1 listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/api/src/tasks/checkAlerts/__tests__/checkAlerts.int.test.ts:602 — The index-plan test explains a query written in the test, not the query getPreviousAlertHistories runs → The test builds its own AlertHistory.find({alert, createdAt: {$lte, $gte}, state}) and runs explain on that, so the production code is never checked. If getPreviousAlertHistories dropped its $gte bound, or the provider's catch-up find stopped using the index, this test would still pass. Run explain on the query the function actually issues instead: export a query builder that both use, or spy on AlertHistory.find, capture its filter, and explain that.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 464 passed • 1 skipped • 1027s

Status Count
✅ Passed 464
❌ Failed 0
⚠️ Flaky 4
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

This PR bounds the check-alerts history lookup by alert interval: a cheap per-alert due-ness gate, interval-bounded streamed per-group scans, a catch-up path after outages, and removal of the now-unused {alert, group, createdAt} index declaration. Independent verification of the gating, catch-up, and error-handling logic found no correctness or reliability regression with a concrete failure mode. The remaining findings are maintainability and operational follow-ups.

🟡 P2 — recommended

  • packages/api/src/tasks/checkAlerts/providers/default.ts:487 — The "build an AggregatedAlertHistory from a row, then previousAlerts.set(computeHistoryMapKey(...))" shape is duplicated across four sites (streamed reduce, non-grouped seed, catch-up scan, sentinel seed), each carrying a different subset of {group, fired, groupValues}, so a future field addition can silently be dropped from one path and corrupt per-group state restoration.
    • Fix: Extract one toAggregatedAlertHistory(alertObjectId, row) factory (plus a setPrevious helper) so the field set is defined in a single place.
  • packages/api/src/models/alertHistory.ts:177 — Removing the index declaration is correct, but since the worker runs with autoIndex: false and OSS never drops indexes, existing clusters keep the orphaned {alert, group, createdAt} index forever with no consumer, incurring ongoing write/storage overhead and diverging from fresh deployments.
    • Fix: Add an upgrade/changelog note that operators may manually drop the orphaned index to reclaim it.
🔵 P3 nitpicks (4)
  • packages/api/src/tasks/checkAlerts/providers/default.ts:449 — The inline type NonNullable<ReturnType<(typeof latestHistories)['get']>> (repeated at the gatedAlerts declaration) reduces to the already-imported LatestAlertHistory.
    • Fix: Declare a small named type and use latest: LatestAlertHistory at both sites.
    • maintainability, kieran-typescript
  • packages/api/src/tasks/checkAlerts/providers/default.ts:400 — loadAlerts() concentrates detail resolution, the gate, four-way bucketing, two follow-up scans, seeding, and connection grouping in one long function.
    • Fix: Split the partition/gate step and the seeding step into named helpers so each phase is independently readable and testable.
  • packages/api/src/tasks/checkAlerts/providers/default.ts:433 — Comments reference an internal managed datastore and private tracker IDs that are meaningless to external OSS readers.
    • Fix: Keep the technical rationale but genericize or drop the internal datastore name and ticket IDs.
  • packages/api/src/tasks/checkAlerts/providers/default.ts:545 — The new logger.info/logger.error calls use the two-arg (obj, msg) form while the rest of this file uses the single-object { message, ...fields } form.
    • Fix: Pick one logger signature for the file to keep structured-log fields consistent.

Reviewers (9): correctness (independent orchestrator verification), performance, reliability (independent orchestrator verification), testing, maintainability, project-standards, kieran-typescript, data-migrations, previous-comments, agent-native, learnings-researcher.

Testing gaps:

  • Equal-createdAt tie-break in the latest-per-group reduce is documented as "first row wins" but untested; groupValues round-tripping through the streamed cursor reduce is only asserted via the separate catch-up path, not the primary scan.
  • No end-to-end test runs a multi-window (numConsecutiveWindows > 1) alert through getAlertTasks/loadAlerts to confirm recentHistoryMap is still populated now that getConsecutiveWindowHistories receives only due-with-history alerts.
  • No explain() assertion that the gate findOne and the catch-up exact-createdAt query are served by alert_1_createdAt_-1 with tight bounds; the ERROR-heavy-alert gate-walk cost is described only in a comment.
  • Two ce-project-standards-reviewer findings were suppressed at confidence 50; the two highest-stakes personas (correctness, adversarial) and reliability were dispatched but had not returned at synthesis time — their dimensions were covered by independent orchestrator verification, which found no P0/P1.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

…onflicts

Use the enterprise build's text for the alert-history lookback, the latest-
and previous-history fetches, the provider's due-ness gate (now a loadAlerts()
helper), and their tests, so the upstream sync into the enterprise repo
conflicts only where that repo has its own code.

- Drop the unused { alert, group, createdAt } index declaration, as the
  enterprise build did. Existing deployments keep the index; it is no longer
  created on new ones.
- Keep the guard that treats an alert as due when its window start cannot
  be computed, now as a helper at the end of providers/default.ts, and test
  it by stubbing getAlertWindowStart instead of clearing the evaluators.
- Move the extra ERROR-row and index-plan tests into their own block.

Refs #2434
getLatestAlertHistories had no lower createdAt bound, and its
state != ERROR filter is not in the {alert, createdAt} index. For an
alert that keeps failing, every tick walked back through every ERROR row
until it found a non-ERROR one, up to the full 30-day retention. It also
let a non-grouped alert pick up a weeks-old row as its previous state.

Bound it by getAlertHistoryLookbackMs(interval), the same window
getPreviousAlertHistories uses, so the skip check, the grouped scan and
the non-grouped path agree. The function now takes alerts instead of IDs.

Also drop an `as any` in a test that pushed the API package over its
lint warning cap.

Refs #2434
… to any

The escape-hatch ratchet caps 'as any' in the api package; the stub added
one over the baseline.

Refs #2434
// The lower bound caps the walk past consecutive ERROR rows (the
// state filter is not in the index) for an alert that keeps
// failing, and matches getPreviousAlertHistories' window.
createdAt: {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — Non-grouped alerts with ≤15m intervals now forget their previous state after 1 day instead of 7

The gate findOne in getLatestAlertHistories uses getAlertHistoryLookbackMs (1d for 1m–15m alerts), and loadAlerts now takes non-grouped previous state only from this row (providers/default.ts seeds previousAlerts from latest). So a 5m alert in ALERT state whose evaluations write only ERROR rows for more than 24h (for example, a ClickHouse outage) comes back with an empty previousMap. On recovery, sendNotificationIfResolved sees no previous ALERT and never sends the resolve notification. If the alert is still breaching, it fires a duplicate ALERT notification. The PR's trade-offs only mention this for groups. Fix: give the gate lookup the old 7d bound (or ALERT_HISTORY_MAX_LOOKBACK_MS). It is a one-row seek. Keep the interval-derived window for the grouped range scan only.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

…terval

The interval-derived bound (1 day for 1m to 15m alerts) made non-grouped
alerts forget their previous state after 1 day. An alert in ALERT whose
evaluations only wrote ERROR rows for longer than that would recover
without sending its resolve notification. main kept that state for 7 days.

Use ALERT_HISTORY_MAX_LOOKBACK_MS instead. It still caps the walk past
ERROR rows, at the 7 days main already scanned, and restores the
enterprise build's signature, so the call site and tests match it again.

Refs #2434
}
}

const [previousAlerts, recentHistoryMap] = await Promise.all([

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — Grouped alerts lose their catch-up after an outage longer than the shorter lookback

Suppose a grouped alert's latest history row is older than getAlertHistoryLookbackMs(interval) but newer than 7 days. Example: a 1m alert after 30h of downtime, or a 1h alert after 60h. The 7-day check marks the alert due, but the shorter per-group scan returns nothing for it. getAlertEvaluationDateRange (index.ts:734) then falls back to now - 1 window. On main, the start was the old createdAt, which calcAlertDateRange caps at 50 windows (or 6h/24h), so the missed windows were evaluated. Now they are skipped. The comment at index.ts:663 says the lookback only has to cover the catch-up horizon. That holds only if an empty result still anchors the start, and it doesn't. Fix: after getPreviousAlertHistories, for each due grouped alert with no entries in previousAlerts, add the row from the 7-day check under computeHistoryMapKey(alert.id, latest.group || ''). This also restores that group's previous state.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

…the scan

The latest-row lookup covers 7 days, but the per-group scan covers only
the alert's interval-derived window (1 day for fast alerts). After an
outage longer than that window the scan returned nothing, so the
evaluation started from now - 1 window and skipped the missed windows
that main would have caught up on (up to 50).

When the scan has no row for a due grouped alert, seed the row from the
latest-row lookup. The evaluation starts from it again, and that group's
previous state is restored.

Refs #2434
…utage

When a due grouped alert's latest row is older than its scan window, the
provider seeded only that one row, without groupValues. A group that
then resolved was notified with empty group attributes, and the other
groups from that evaluation lost their previous state.

Read all rows that share the latest row's createdAt instead: one per
group, with groupValues. It is one extra query per affected alert, on
the first tick after such an outage.

Tests: the last evaluation is restored per group with its values; a
grouped alert already evaluated this window is skipped from the loaded
sentinel alone; the window-start failure test now tells "due" from
"skipped".

Refs #2434
});
}

// A grouped alert whose latest row is older than its scan window gets no

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Outage catch-up for grouped alerts sits in the provider, outside getPreviousAlertHistories, and repeats its row mapping

Since this change, getPreviousAlertHistories returns nothing for a grouped alert whose last evaluation is older than its scan window. Its docstring still says it returns the latest history per group. The fix-up (an exact-createdAt find) exists only in loadAlerts, so any other caller, including the tests that call the function directly, silently loses the catch-up. The same filter, projection and row-to-AggregatedAlertHistory mapping now also appear in getPreviousAlertHistories, getConsecutiveWindowHistories, the non-grouped seeding and this block. Move the fallback into getPreviousAlertHistories, for example by accepting an optional latestHistories map, and share one row-mapping helper.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 3511be90 Deployed Nov 18, 2026 by vercel[bot]
Preview – hyperdx-storybook — 3511be90 Deployed Nov 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

1 participant