Repository navigation
ci(release): Release only green builds, release stranded commits, and drop failed releases - #137
Open
sentry-junior[bot] wants to merge 7 commits into
Open
sentry-junior[bot] wants to merge 7 commits into
sentry-junior[bot] wants to merge 7 commits into
Conversation
Co-Authored-By: BYK <126780+BYK@users.noreply.github.com>
Co-Authored-By: BYK <126780+BYK@users.noreply.github.com>
sentry-junior
Bot
requested review from
MathurAditya724 and
betegon
as code owners
November 9, 2026 16:43
BYK
added a commit
that referenced
this pull request
Nov 9, 2026
) From 11-07 to 11-09, `@sentry/api` didn't release for 2 days and nobody noticed. Build was red on `main`, and every Release run failed on the stale `release/1.52.0` branch, but nothing reported it. Releases can also get stuck with no failing run at all. 83a2b74, for example, landed while 1.52.0 was pending and sat unreleased. This PR adds a `Release pipeline health` workflow. It keeps one issue, **"Release pipeline failing on main"**, open while the pipeline is broken. **Failure alerts.** - When Build (push builds only) or Release fails on `main`, the workflow opens the issue, or comments on it if it's already open. - Each message links the run and the commit. - The issue body mentions @BYK, which subscribes him to later comments. **Recovery.** - The next successful Release run closes the issue and links that run. - Skipped and cancelled runs don't count. **Daily staleness check.** At 08:23 UTC, and on manual dispatch, the workflow reports to the same issue in either of these cases: - The oldest "Generated" commit on `main` that isn't in the newest npm release is more than 24h old. The newest npm release comes from `npm view @sentry/api version`, matched to its git tag. A missing tag is reported too. - An open getsentry/publish request for this repo has had `ci-failed` for more than 24h, counted from when the label was added. The first check measures how long a schema commit has been waiting. A commit that landed 1h after a long quiet stretch doesn't trigger it. A commit stranded a minute after a release, like 83a2b74, still does. The workflow uses the default `GITHUB_TOKEN` with `contents: read` and `issues: write`. getsentry/publish is public, so that token can read its issues and their events. No new secrets, no Slack. ## Testing I ran both scripts under `dash`. **Staleness check on real data:** - Today's `main` is reported as up to date. - With the clock moved 2 days ahead, it reports 83a2b74. - At 8234586, with npm at 1.51.0 and the clock at 11-09 15:00Z, it reports 6 unreleased "Generated" commits, the oldest being dc25434 from 44h earlier. The 11-07 incident would have alerted about a day in. **Stubbed runs:** - A `ci-failed` request older than 24h is reported. A newer one, and requests for other repos, are not. - When `npm`, the issue list, or the events fetch fails, the run fails. - The issue script opens, comments on, or closes the issue, or does nothing when no issue is open. It exits non-zero when listing or writing issues fails. `actionlint` and `shellcheck --shell=sh --enable=check-extra-masked-returns` pass. No UI changes, so no screenshots. ## Notes - This works with or without the companion Release PR (#137), and it assumes #136, which is merged. - If the staleness check itself fails, for example because npm is down, GitHub's usual failed-run email for scheduled workflows is the only signal. - No admin setup is needed. <!-- junior-request-attribution:start --> via @BYK. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:agent%3A3469a89d1b152f2e4ef70bbb981e697e --> -- [View Session in Sentry](https://sentry.sentry.io/explore/conversations/agent%3A3469a89d1b152f2e4ef70bbb981e697e/?project=4510944073809921) <!-- junior-session-footer:end --> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: BYK <126780+BYK@users.noreply.github.com>
Move the stale-branch cleanup out of Release preflight and into the release pipeline health workflow: when Build fails on a release branch, close its publish request and delete the branch. Preflight now only checks the release tag, unreleased commits, and whether a release is pending.
A Release run also succeeds when preflight skips, so treating any success as recovery closed alerts while nothing was releasing.
Contributor
Author
|
Yes, that's simpler. Done in c24715f:
Trade-off: the drop happens on the first failure, and with a green |
BYK
requested changes
Nov 9, 2026
BYK
left a comment
Member
There was a problem hiding this comment.
I feel like this PR is still unnecessarily complex?
Preflight keeps only the three cases where craft prepare would fail. The drop deletes the branch first and treats closing the publish request as best effort, since the release bot may not be allowed to close issues there. The tag check goes: craft merges and deletes a release branch once the release is out, so a published version never has a branch left to drop.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4f930a2. Configure here.
A rerun, or a branch that advanced after the failed Build, made the lease fail and the job go red.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

@sentry/apireleases have gotten stuck in two ways. This PR changes Release, and theRelease pipeline healthworkflow from #138, so neither needs a person to fix it.Incident 1: commits stranded during a pending release. The old job guard skipped every release merge. A commit that landed while a release was pending was never released until someone pushed again. In the last 40 release merges, 9 brought in commits like this. 83a2b74 is the latest.
Incident 2: a failed release blocked all later ones (11-07 to 11-09). Release ran even when Build failed. It cut
release/1.52.0, Build failed on that branch, and every later Release run failed with craft'sRelease branch "release/1.52.0" already existsuntil someone cleaned up by hand.Changes
Release (
release.yml):workflow_runafter Build succeeds onmain(push builds from this repo only). It checks out that commit and puts a localmainon it, because craft prepares from the default branch.concurrency: { group: release, cancel-in-progress: false }.workflow_dispatchworks as before and skips preflight.release-preflight.shskips the three cases wherecraft preparewould fail:release/*branch exists: "release branch already exists". The merge of that release releases what's left.Release pipeline health (
release-pipeline-health.yml):release/**,drop-failed-release.shdeletes the branch, using--force-with-leaseon the failed commit. It then tries to close the getsentry/publish request with a "Superseded" comment. If the bot can't list or close that request, the job logs a warning and still succeeds, because the branch is already gone. If the branch is already gone or has moved since the failed Build (a rerun, or a new push), the job logs "nothing to drop" and passes (f62189b). The next push tomainreleases again.actions: read.Testing
dash: I used throwaway git remotes that mirror craft and publish (release branch, tag, merge), withghstubbed.mainskips becauserelease/1.56.0is pending.craft prepare --dry-run(3.9.0) from this checkout shape would have cutrelease/1.53.0with 83a2b74.actionlintandshellcheck --shell=sh --enable=check-extra-masked-returnspass.No UI changes, so no screenshots.
For admins
drop-failed-releaseuses the release bot app (SENTRY_RELEASE_BOT_*). It needs to delete branches here, which it can already do since it pushes them. Closing getsentry/publish issues is optional: without triage access there, the job warns and the request stays open.via @BYK.
--
View Session in Sentry