Skip to content

Evaluate a merge queue or required status checks for v2/main #5588

Closed as not planned

Description

@cliffhall

Problem

Nothing in v2/main's ruleset tests a PR against the tip it merges into. The live ruleset v2/main: PRs only has three rules (no deletion, no force-push, changes by pull request with zero required approvals) and no required status checks at all: a PR's CI is green at merge by convention, not by enforcement, and it is never required to have run on the combined tree.

On 2026-11-18 two PRs that were each green merged minutes apart and left v2/main red (#5569). The procedural answer is #5589. This issue is the question of whether to enforce it.

Options to evaluate

  • A merge queue on v2/main. GitHub builds the combined tree and runs the required checks on it before merging, so no PR needs a catch-up push. It needs: a merge_group trigger on every workflow whose jobs become required (none has one today), the required checks named in the ruleset, a decision on the queue's merge method (this repo merges with merge commits), and a check that the DCO job, the interface diff and the dispatched-run reporter jobs behave on a merge_group event. pr-flow step 10, AGENTS.md and docs/quality-gate.md change to match, as does how the Version Packages and Prepare Python Release PRs merge.
  • Required status checks with "require branches to be up to date". Simpler to switch on, but every merge makes every other open PR stale, and each catch-up is another gate run and CI run. It costs the most exactly when several PRs are open at once.
  • Required status checks without the up-to-date requirement. Does not close this gap, but would turn "CI is green at merge" from a convention into a rule. Worth deciding on its own.
  • Leave the ruleset as it is and rely on pr-flow: catch up and re-test before merging when v2/main moved under the PR #5589 plus CI on the push to v2/main, on the grounds that v2/main is a develop branch and nothing publishes from it without a milestone merge PR that runs CI again.

Done when

A decision is recorded here with its reasoning. If it is a change, the ruleset, the workflows, AGENTS.md (Project status and direction describes this ruleset), pr-flow and docs/quality-gate.md change together, in a follow-up issue if the work is more than one PR.

Note

A ruleset change is a maintainer's act in the repository settings; it is not something a PR or an agent applies.

Activity

  1. added this to the v2.6.0 milestone on Nov 18, 2026
  2. added
    choreMaintenance: deps, build tooling, CI, cleanup — no user-facing behavior change
    on Nov 18, 2026
  3. cliffhall commented on Nov 18, 2026

    @cliffhall
    MemberAuthor

    Research for the decision (nothing was changed)

    Everything below was read from the API on 2026-11-18, at v2/main = 7f383a8e. No ruleset, setting or workflow was touched.

    Current state

    Rulesets that apply

    Branch Ruleset Rules Required checks Bypass
    v2/main v2/main: PRs only (24968529) no deletion, no force-push, PR required (0 approvals; merge, squash and rebase allowed by the rule) none none
    main main: milestone merges only (24866866) PR required, merge commits only Source is v2/main (GitHub Actions), not strict none
    main L3 - full protection (org 2749948 and repo 3187904), Copilot review for default branch (20596567) default branch only: 1 approval, code-owner review, Copilot review none org admins, repo admins, deploy keys (the L3 pair)

    Repository settings: merge commits and rebase allowed, squash off, auto-merge on, "update branch" on, head branches deleted on merge. The org is on the Enterprise plan and the repository is public. No classic branch protection exists on either branch.

    What runs on a PR. typescript.yml and python.yml trigger on push (any branch) and workflow_dispatch, not pull_request; dco.yml and everything-mcp-diff.yml trigger on pull_request. No workflow has a merge_group trigger. Every PR runs every server: 34 to 51 check runs per head (37 typical), and the job names are matrix-generated (Validate memory (Node 26), Test git (Windows), ...). The automated version PRs get CI through workflow_dispatch, mirrored as four commit statuses (TypeScript (dispatched), Python (dispatched), DCO (dispatched), Interface diff (dispatched)).

    How long it takes. Over the last 45 maintainer PRs merged into v2/main, wall time from first job start to last job end: median 238 s, p90 344 s, max 696 s. Test git (Windows) is the usual long pole at 4 to 5 minutes. One full run is about 43 job-minutes (2,581 s over 37 jobs on the sampled head). Standard runners are free on a public repository, so the cost is latency and runner concurrency, not money.

    Is "green at merge" already true? Yes. Of those 45 PRs, none merged with a failed or unfinished check on its head. (Three show a cancelled DCO signoff, each a duplicate superseded by a later run that passed.)

    How often v2/main moves under a PR. Since the ruleset was created (2026-10-12), 200 PRs merged in 37 days:

    • 151 (75%) merged on a base that had moved since the branch last caught up, by up to 48 merges.
    • 86 (43%) merged after a commit touching the same server directory had landed under them.
    • Merges come in bursts: 8 in 50 seconds on 11-18 (07:12:15 to 07:13:04), 7 filesystem PRs in 53 seconds on 11-05.

    Did that produce a red v2/main? Once that the record shows: aef8fd72 (this issue's incident, #5569), red from 05:48 to 06:04 UTC, 16 minutes, repaired in #5571. The two other failed push runs on v2/main in the window were not combined-tree failures: d1b59ba8 (a Windows filesystem test on a PR that was up to date; a timeout later addressed by #5331) and one Version Packages workflow failure at b6bb2f65.

    A caveat the issue's fourth option depends on. "CI on the push to v2/main" does not run to completion on most commits. typescript.yml and python.yml cancel a superseded run on the same ref everywhere except main, so in a burst each merge cancels the previous one's run. Of 203 v2/main tips in the window, 98 had both workflows finish, 20 had one, and 85 had neither. Only the last commit of a burst is reliably tested. A break persists into that last commit, so it is still found, but later and without saying which merge caused it.

    What a red v2/main costs here. Nothing publishes from it. The cost is that the local gate runs every server, so every worktree cut from a red tip has a red gate that is not its own doing, for every agent working at the time.

    The options

    A. Merge queue on v2/main

    • Availability: GitHub documents merge queue for public organization repositories and for Enterprise Cloud; both hold here. The ruleset targets the exact ref, so the "no wildcard patterns" restriction does not apply. Not tested live.
    • What changes:
      • Ruleset (repo admin only): add required status checks and the merge queue rule, merge method merge, so history keeps its Merge pull request #N commits.
      • Workflows: merge_group: on typescript.yml and python.yml (GitHub's docs say a required check that does not report on the merge group fails the merge). Required checks are matched by name and the matrix names change whenever a server or Node leg does, so each workflow needs one aggregate job (if: always(), fails unless every needed job passed or was skipped) to be the required context. If DCO signoff or the interface diff were made required they would need a merge_group path too; both read github.event.pull_request.*, so the simplest course is to leave them unrequired.
      • Docs and skills: AGENTS.md (Project status and direction, and the merge steps under Issue-driven work style), pr-flow step 10, release (version PRs), security-advisory (the public fix PR), docs/quality-gate.md. The pr-flow: catch up and re-test before merging when v2/main moved under the PR #5589 procedure becomes redundant.
    • main and the milestone merge PR: unaffected. The queue rule lives in the v2/main ruleset only; the merge PR into main still merges directly with a merge commit, and Source is v2/main still holds because queue-made merge commits are ordinary v2/main commits.
    • Agent-driven merges: "merge" becomes "enqueue". An agent must wait for the merged state (one backgrounded loop) before closing the issue and moving the card, and must handle ejection: a failed merge-group run removes the PR from the queue. In exchange the fan-out problem goes away with no catch-up pushes and no race.
    • Version Packages PR: the sharp edge. Any merge to v2/main regenerates it (new head, rewritten body). If it sits in the queue behind another PR, that PR's merge regenerates it and it is ejected. It has to be enqueued when the queue is empty, and "Part of #N immediately before merging" becomes "immediately before enqueueing, re-checked after merge". The Prepare Python Release PR has no such problem.
    • Unverified: whether the dispatched runs on the version PRs satisfy a required check (the workflow comments record that the PR rollup ignores dispatched check runs, which is why the statuses exist); whether the unfiltered push trigger also fires on gh-readonly-queue/** branches, which would double every queue run unless excluded.
    • Cost: one extra full CI run per merge on the combined tree (about 43 job-minutes, free) and 4 to 6 minutes of latency per merge, with a burst tested in parallel rather than serially. The post-merge push run on v2/main would re-test a tree the queue already tested.
    • Prevents: a red v2/main from any combination of PRs, including bursts seconds apart. Does not prevent: flaky tests (they eject PRs instead), or anything CI does not check.

    B. Required checks with "require branches to be up to date"

    • What changes: the ruleset, plus the same aggregate jobs. No merge_group.
    • Cost: on this window's numbers 151 of 200 merges would have needed a catch-up, and every merge re-stales every other open PR. The 7-PR burst becomes 7 serial rounds of catch-up plus about 4 minutes of CI. AGENTS.md requires the local gate before every push, so each catch-up is also a gate run under the machine-wide lease, unless the rule gains an exception for the server-side "update branch" merge.
    • Prevents: the same failure as A. Does not prevent: its own cost, which is quadratic in open PRs and lands exactly on the fan-out pattern this repository uses.

    C. Required checks without it

    • What changes: the ruleset and the aggregate jobs. This is also the first half of A.
    • Cost: near zero at merge time. The risk is at the version PRs (the unverified dispatched-check point above). Rulesets can run in Evaluate mode on this plan, which would show what the rule would have blocked without blocking anything.
    • Prevents: merging a PR whose own CI is red or unfinished. That did not happen in the 45 PRs sampled. Does not prevent: the incident behind this issue.

    D. Status quo plus the #5589 procedure

    Recommendation

    1. Rule out B. It prevents the same failure as the queue at a cost that grows with the number of open PRs.
    2. Land pr-flow: catch up and re-test before merging when v2/main moved under the PR #5589 regardless. It is cheap, and it is the fallback while anything else is built.
    3. Do C as groundwork, built queue-ready: aggregate jobs in typescript.yml and python.yml, the merge_group triggers added but inert, the ruleset's required checks trialed in Evaluate mode first, and the version-PR question settled on a real version PR.
    4. Then turn the queue on (A), once step 3 has shown the version PRs pass.

    The case for stopping at step 2 is real: one combined-tree break in 200 merges, repaired in 16 minutes, on a branch nothing publishes from. The case for going to step 4 is that 75% of merges here already land on a moved base, they land in bursts that a check-then-merge procedure cannot serialize, a red tip fails every concurrent agent's gate, and the queue's latency is paid by machines. I recommend going to step 4, in stages.

    Questions for the maintainer

    1. Where does this end: the merge queue (A), required checks only (C), or the procedure alone (D)?
    2. If A or C: is the required set just the two aggregate CI results (TypeScript, Python), or also DCO signoff and the interface diff? Each extra required check needs a merge_group path under A and a dispatched-run equivalent on the version PRs.
    3. If A: is it acceptable that the Version Packages PR can only be enqueued on an empty queue, with the Part of #N step moving to just before enqueueing?
  4. cliffhall commented on Nov 18, 2026

    @cliffhall
    MemberAuthor

    Decision (maintainer, 2026-11-18): option D, the procedure alone.

    v2/main keeps its current ruleset: no merge queue and no required status checks. The protection against a combined-tree break is the catch-up-and-re-test rule and the pr-flow step 10 procedure from #5589 (PR #5616).

    The analysis above stays as the record of what was measured and what the other options would cost, should this be revisited. A (merge queue), B (required checks with up-to-date branches) and C (required checks only) are not being adopted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    choreMaintenance: deps, build tooling, CI, cleanup — no user-facing behavior changev2

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions