Skip to content

ci: Track release pipeline failures and stuck releases in an issue - #138

Merged
BYK merged 1 commit into
mainfrom
ci/release-pipeline-alerts
Nov 9, 2026
Merged

BYK merged 1 commit into
mainfrom
ci/release-pipeline-alerts

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

From 11-07 to 11-09, @sentry/api didn't release for 2 days and nobody noticed. Build was red on main, and every Release run failed on the stale release/1.52.0 branch, but nothing reported it. Releases can also get stuck with no failing run at all. 83a2b74, for example, landed while 1.52.0 was pending and sat unreleased.

This PR adds a Release pipeline health workflow. It keeps one issue, "Release pipeline failing on main", open while the pipeline is broken.

Failure alerts.

  • When Build (push builds only) or Release fails on main, the workflow opens the issue, or comments on it if it's already open.
  • Each message links the run and the commit.
  • The issue body mentions @BYK, which subscribes him to later comments.

Recovery.

  • The next successful Release run closes the issue and links that run.
  • Skipped and cancelled runs don't count.

Daily staleness check. At 08:23 UTC, and on manual dispatch, the workflow reports to the same issue in either of these cases:

  • The oldest "Generated" commit on main that isn't in the newest npm release is more than 24h old. The newest npm release comes from npm view @sentry/api version, matched to its git tag. A missing tag is reported too.
  • An open getsentry/publish request for this repo has had ci-failed for more than 24h, counted from when the label was added.

The first check measures how long a schema commit has been waiting. A commit that landed 1h after a long quiet stretch doesn't trigger it. A commit stranded a minute after a release, like 83a2b74, still does.

The workflow uses the default GITHUB_TOKEN with contents: read and issues: write. getsentry/publish is public, so that token can read its issues and their events. No new secrets, no Slack.

Testing

I ran both scripts under dash.

Staleness check on real data:

  • Today's main is reported as up to date.
  • With the clock moved 2 days ahead, it reports 83a2b74.
  • At 8234586, with npm at 1.51.0 and the clock at 11-09 15:00Z, it reports 6 unreleased "Generated" commits, the oldest being dc25434 from 44h earlier. The 11-07 incident would have alerted about a day in.

Stubbed runs:

  • A ci-failed request older than 24h is reported. A newer one, and requests for other repos, are not.
  • When npm, the issue list, or the events fetch fails, the run fails.
  • The issue script opens, comments on, or closes the issue, or does nothing when no issue is open. It exits non-zero when listing or writing issues fails.

actionlint and shellcheck --shell=sh --enable=check-extra-masked-returns pass.

No UI changes, so no screenshots.

Notes

via @BYK.

--

View Session in Sentry

Co-Authored-By: BYK <126780+BYK@users.noreply.github.com>
@BYK
BYK merged commit 7162117 into main Nov 9, 2026
11 checks passed
@BYK
BYK deleted the ci/release-pipeline-alerts branch November 9, 2026 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant