Skip to content

fix(errata): fail daily and weekly runs when v4 engine experiences an outage (#2789) - #2804

Open
isihussain wants to merge 1 commit into
danielmiessler:mainfrom
isihussain:fix/errata-v4-engine-outage-exit-code
Open

isihussain wants to merge 1 commit into
danielmiessler:mainfrom
isihussain:fix/errata-v4-engine-outage-exit-code

Conversation

@isihussain

Copy link
Copy Markdown

Summary

Fixes #2789.

In LIFEOS/ERRATA/Errata.ts, the v4 pass ran in shadow mode at the end of daily() and weekly(). Its execution was wrapped in a ry/catch block that never incremented ailures. While occasional single errors in a shadow stage should not derail an entire run, a complete engine outage (such as timeouts across all candidate calls due to network disruption or API downtime) resulted in Errata.ts daily exiting with code . As a result, the errata-daily workflow recorded a success and failed to raise a Vigil alert.

Solution

  1. Added V4Outage.ts with isV4Outage() to distinguish single non-fatal errors from engine outages:
    • Tolerates single/isolated errors (< 50% of candidates).
    • Treats the stage as failed when detect or diagnose errors on $\ge 50%$ of candidates, when an explicit error entry is present, or when the execution throws.
  2. Updated Errata.ts (daily and weekly) to mark ailures++ and log the failure reason when isV4Outage() returns true, ensuring the process exits non-zero and triggers the normal Vigil alarm.
  3. Added unit tests in V4Outage.test.ts verifying clean runs, non-fatal thresholds, 100% detect outages, and $\ge 50%$ diagnose failure conditions.

Signed-off-by: isihussain isihussain@users.noreply.github.com

… outage (danielmiessler#2789)

Errata daily previously caught v4 errors without incrementing failures,
meaning that if every call to the v4 engine timed out or failed (such as an
api.cloudflare.com network outage), errata daily still exited with status 0.
This caused the scheduled run to be recorded as successful and silenced
the normal Vigil alert.

Single occasional errors remain non-fatal for shadow stages, but when
detect or diagnose errors on at least 50% of candidates or when the stage
throws, mark the stage as failed and exit non-zero so Vigil raises an alert.

Add V4Outage.ts and V4Outage.test.ts covering non-fatal vs outage thresholds.

Closes danielmiessler#2789

Signed-off-by: isihussain <isihussain@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

errata daily exits 0 when every v4 engine call fails, so a night with no engine answers reads as a success

1 participant