Repository navigation
fix(errata): fail daily and weekly runs when v4 engine experiences an outage (#2789) - #2804
Open
isihussain wants to merge 1 commit into
Open
isihussain wants to merge 1 commit into
isihussain wants to merge 1 commit into
Conversation
… outage (danielmiessler#2789) Errata daily previously caught v4 errors without incrementing failures, meaning that if every call to the v4 engine timed out or failed (such as an api.cloudflare.com network outage), errata daily still exited with status 0. This caused the scheduled run to be recorded as successful and silenced the normal Vigil alert. Single occasional errors remain non-fatal for shadow stages, but when detect or diagnose errors on at least 50% of candidates or when the stage throws, mark the stage as failed and exit non-zero so Vigil raises an alert. Add V4Outage.ts and V4Outage.test.ts covering non-fatal vs outage thresholds. Closes danielmiessler#2789 Signed-off-by: isihussain <isihussain@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2789.
In LIFEOS/ERRATA/Errata.ts, the v4 pass ran in shadow mode at the end of daily() and weekly(). Its execution was wrapped in a ry/catch block that never incremented ailures. While occasional single errors in a shadow stage should not derail an entire run, a complete engine outage (such as timeouts across all candidate calls due to network disruption or API downtime) resulted in Errata.ts daily exiting with code . As a result, the errata-daily workflow recorded a success and failed to raise a Vigil alert.
Solution
Signed-off-by: isihussain isihussain@users.noreply.github.com