Skip to content

ci(release): Release only green builds, release stranded commits, and drop failed releases - #137

Open
sentry-junior[bot] wants to merge 7 commits into
mainfrom
ci/self-healing-release
Open

sentry-junior[bot] wants to merge 7 commits into
mainfrom
ci/self-healing-release

Conversation

@sentry-junior

@sentry-junior sentry-junior Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

@sentry/api releases have gotten stuck in two ways. This PR changes Release, and the Release pipeline health workflow from #138, so neither needs a person to fix it.

Incident 1: commits stranded during a pending release. The old job guard skipped every release merge. A commit that landed while a release was pending was never released until someone pushed again. In the last 40 release merges, 9 brought in commits like this. 83a2b74 is the latest.

Incident 2: a failed release blocked all later ones (11-07 to 11-09). Release ran even when Build failed. It cut release/1.52.0, Build failed on that branch, and every later Release run failed with craft's Release branch "release/1.52.0" already exists until someone cleaned up by hand.

Changes

Release (release.yml):

  • Runs on workflow_run after Build succeeds on main (push builds from this repo only). It checks out that commit and puts a local main on it, because craft prepares from the default branch.
  • Drops the commit-message guard, so release merges run too. That run is what releases commits that landed while the previous release was pending.
  • concurrency: { group: release, cancel-in-progress: false }.
  • workflow_dispatch works as before and skips preflight.

release-preflight.sh skips the three cases where craft prepare would fail:

  • A release/* branch exists: "release branch already exists". The merge of that release releases what's left.
  • The commit predates the newest tag, because Builds finished out of order: craft would cut that version again.
  • No commits since the last tag: "no commits found since the last release".

Release pipeline health (release-pipeline-health.yml):

  • When Build fails on release/**, drop-failed-release.sh deletes the branch, using --force-with-lease on the failed commit. It then tries to close the getsentry/publish request with a "Superseded" comment. If the bot can't list or close that request, the job logs a warning and still succeeds, because the branch is already gone. If the branch is already gone or has moved since the failed Build (a rerun, or a new push), the job logs "nothing to drop" and passes (f62189b). The next push to main releases again.
  • Release-branch Build failures are reported to the pipeline issue too.
  • The issue closes only when a Release run's Prepare release step succeeds. A skipped run also finishes as a success, so it no longer closes an open alert. This needs actions: read.
  • Concurrency moved to the two jobs that write the issue, so a waiting branch drop can't be cancelled.

Testing

  • Preflight and drop script, under dash: I used throwaway git remotes that mirror craft and publish (release branch, tag, merge), with gh stubbed.
    • The stranded commit (83a2b74 model) skips while 1.2.0 is pending, and is released by the merge commit's run.
    • Two syncs in order: A releases, B waits, and the merge releases B.
    • B's Build first: B covers both, A skips, and a late run for A skips.
    • A release merge with nothing new skips.
    • Incident 2: the fix's Release skips while the failed branch exists. After the drop, it releases both commits.
    • Drop edge cases: the branch moved, or it's already gone (the job logs "nothing to drop", passes, and leaves the branch and the request alone); no open request; the issue lookup or close fails (warning, branch still deleted).
  • Pipeline issue: with real Release run 42666230595, which prepared a release, the issue closes. With Prepare release skipped, it stays open.
  • Live, read-only: preflight on today's main skips because release/1.56.0 is pending.
  • craft: earlier, craft prepare --dry-run (3.9.0) from this checkout shape would have cut release/1.53.0 with 83a2b74.
  • actionlint and shellcheck --shell=sh --enable=check-extra-masked-returns pass.

No UI changes, so no screenshots.

For admins

  • drop-failed-release uses the release bot app (SENTRY_RELEASE_BOT_*). It needs to delete branches here, which it can already do since it pushes them. Closing getsentry/publish issues is optional: without triage access there, the job warns and the request stays open.

via @BYK.

--

View Session in Sentry

sentry-junior Bot and others added 2 commits November 9, 2026 16:22
Co-Authored-By: BYK <126780+BYK@users.noreply.github.com>
Co-Authored-By: BYK <126780+BYK@users.noreply.github.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread .github/scripts/release-preflight.sh Resolved
BYK added a commit that referenced this pull request Nov 9, 2026
)

From 11-07 to 11-09, `@sentry/api` didn't release for 2 days and nobody
noticed. Build was red on `main`, and every Release run failed on the
stale `release/1.52.0` branch, but nothing reported it. Releases can
also get stuck with no failing run at all. 83a2b74, for example, landed
while 1.52.0 was pending and sat unreleased.

This PR adds a `Release pipeline health` workflow. It keeps one issue,
**"Release pipeline failing on main"**, open while the pipeline is
broken.

**Failure alerts.**
- When Build (push builds only) or Release fails on `main`, the workflow
opens the issue, or comments on it if it's already open.
- Each message links the run and the commit.
- The issue body mentions @BYK, which subscribes him to later comments.

**Recovery.**
- The next successful Release run closes the issue and links that run.
- Skipped and cancelled runs don't count.

**Daily staleness check.** At 08:23 UTC, and on manual dispatch, the
workflow reports to the same issue in either of these cases:
- The oldest "Generated" commit on `main` that isn't in the newest npm
release is more than 24h old. The newest npm release comes from `npm
view @sentry/api version`, matched to its git tag. A missing tag is
reported too.
- An open getsentry/publish request for this repo has had `ci-failed`
for more than 24h, counted from when the label was added.

The first check measures how long a schema commit has been waiting. A
commit that landed 1h after a long quiet stretch doesn't trigger it. A
commit stranded a minute after a release, like 83a2b74, still does.

The workflow uses the default `GITHUB_TOKEN` with `contents: read` and
`issues: write`. getsentry/publish is public, so that token can read its
issues and their events. No new secrets, no Slack.

## Testing

I ran both scripts under `dash`.

**Staleness check on real data:**
- Today's `main` is reported as up to date.
- With the clock moved 2 days ahead, it reports 83a2b74.
- At 8234586, with npm at 1.51.0 and the clock at 11-09 15:00Z, it
reports 6 unreleased "Generated" commits, the oldest being dc25434 from
44h earlier. The 11-07 incident would have alerted about a day in.

**Stubbed runs:**
- A `ci-failed` request older than 24h is reported. A newer one, and
requests for other repos, are not.
- When `npm`, the issue list, or the events fetch fails, the run fails.
- The issue script opens, comments on, or closes the issue, or does
nothing when no issue is open. It exits non-zero when listing or writing
issues fails.

`actionlint` and `shellcheck --shell=sh
--enable=check-extra-masked-returns` pass.

No UI changes, so no screenshots.

## Notes

- This works with or without the companion Release PR (#137), and it
assumes #136, which is merged.
- If the staleness check itself fails, for example because npm is down,
GitHub's usual failed-run email for scheduled workflows is the only
signal.
- No admin setup is needed.

<!-- junior-request-attribution:start -->
via @BYK.
<!-- junior-request-attribution:end -->

<!-- junior-session-footer:start -->
<!-- junior-conversation-id:agent%3A3469a89d1b152f2e4ef70bbb981e697e -->

--

[View Session in
Sentry](https://sentry.sentry.io/explore/conversations/agent%3A3469a89d1b152f2e4ef70bbb981e697e/?project=4510944073809921)

<!-- junior-session-footer:end -->

Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com>
Co-authored-by: BYK <126780+BYK@users.noreply.github.com>

@BYK BYK left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is an auto-release repo can we not merge the behaviors with #138 and this and automatically delete the release branch when it fails instead? Looks simpler to me?

Move the stale-branch cleanup out of Release preflight and into the release
pipeline health workflow: when Build fails on a release branch, close its
publish request and delete the branch. Preflight now only checks the release
tag, unreleased commits, and whether a release is pending.
@sentry-junior sentry-junior Bot changed the title ci(release): Release only green builds, clear stale release branches, and stop stranding commits ci(release): Release only green builds, release stranded commits, and drop failed releases Nov 9, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread .github/workflows/release.yml Resolved
A Release run also succeeds when preflight skips, so treating any success as
recovery closed alerts while nothing was releasing.
@sentry-junior

sentry-junior Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Yes, that's simpler. Done in c24715f:

  • When Build fails on release/**, release-pipeline-health.yml now closes the getsentry/publish request with a "Superseded" comment and deletes the branch. The delete is guarded by --force-with-lease on the failed commit and skips versions that are already tagged. The next push to main re-releases, and the failure goes to the pipeline issue.
  • Preflight no longer scans branches or searches publish issues. It only checks the newest tag, unreleased commits and any pending release/* branch.
  • I moved concurrency to the jobs that write the issue, so a queued cleanup can't be dropped.
  • Bugbot's follow-up is fixed in 3c8e84d: the pipeline issue now closes only when a Release run's Prepare release step succeeds, so a run where preflight skips no longer closes an open alert.

Trade-off: the drop happens on the first failure, and with a green main that's most likely a flake. If nothing new lands afterwards, the 24h staleness check catches it.

@BYK BYK left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like this PR is still unnecessarily complex?

Comment thread .github/scripts/drop-failed-release.sh Resolved
Comment thread .github/scripts/drop-failed-release.sh Resolved
Comment thread .github/scripts/release-preflight.sh Resolved
Preflight keeps only the three cases where craft prepare would fail. The drop
deletes the branch first and treats closing the publish request as best
effort, since the release bot may not be allowed to close issues there. The
tag check goes: craft merges and deletes a release branch once the release is
out, so a published version never has a branch left to drop.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4f930a2. Configure here.

Comment thread .github/scripts/drop-failed-release.sh Resolved
A rerun, or a branch that advanced after the failed Build, made the lease
fail and the job go red.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant