ci(studio): re-run an edit accuracy shard on a fresh runner when its runner is too slow

[miguel-heygen]

What

A merge queue edit accuracy shard that lands on a slow runner now stops after its first four cases and runs once more on a fresh runner, instead of running into its timeout and dropping the PR from the queue.

Why

In merge queue run 38754829357, Studio: edit accuracy (7/20) was cancelled at its 45 minute timeout and the PR left the queue. No test hung. The runner was about 5x slow before any test ran:

	shard 7	shards 6 and 8, same run
bun run --cwd packages/core build	44 s	9 s
workspace install	41 s	11-17 s
median case	92 s	14 s
load at the end	~10	~6.7

The slow cases failed their time-based checks (15 previews never held still within 15 s), so 18 of 38 cases differed from the base branch, and their two confirmation re-runs started on the same slow machine. The other 19 shards took 6-9 min; shard 7 took 5-8 min in the 14 queue runs before it.

The blank-page control does not show this: its rAF p95 median was 17.1 ms on the slow runner and 16.9 / 17.0 ms on shards 6 and 8. Case wall time does.

Related work

Refs #5646 (dropped from the queue by this run).

How

• slowRunner (report.mjs): the first 4 cases' median wall time over 60 s marks a slow runner. Across all 20 shards of a healthy queue run (38730689165) the slowest first-four median was 22 s; shard 7's was 95.5 s.
• run.mjs: once marked, no new case starts; the browsers close as usual, slow-runner.json is written and the bench exits 75.
• scripts/ci/edit-accuracy-shard.sh: the shard step's script, unchanged, moved out of the workflow so the re-run job runs the same thing.
• ci.yml:
  • a shard whose bench stopped on a slow runner, or that was cancelled, marks itself (retry) in its evidence;
  • studio-edit-accuracy-retry-plan lists the marked shards;
  • studio-edit-accuracy-retry runs each of them once on a fresh runner and replaces that shard's evidence artifact (overwrite: true);
  • the gate now waits for the re-run.
• The shard timeout goes from 45 to 25 min, so a cancelled shard plus its re-run and the gate fit inside the merge queue's 60 minute check wait. Healthy shards take 6-9 min.
• The time-based checks themselves are unchanged.

Test plan

• Unit tests added/updated:
  • report.test.mjs: shard 7's first cases (95.5, 133.7, 57.9, 107.5 s) stop the shard; a healthy first four (22 s median) and one with a single 92.7 s case pass; fewer than 4 cases never judge.
  • scripts/edit-accuracy-retry.test.mjs runs the mark and plan steps' own shell. It checks that a cancelled or slow shard is marked and a successful one is not, and that the plan lists exactly the marked shards ([7,12], []). It also checks that the re-run replaces the evidence, that the gate waits for it, and that the timeouts fit in 60 min.
  • check-edit-accuracy-filter.test.mjs now follows shell scripts a step runs, so the moved bench command stays covered by the path filter.
• Failing first:
  • the new workflow test fails 4/4 on main's ci.yml;
  • the new report tests fail 3/3 on main's report.mjs.
• Mutants each fail a test: an unmarked cancelled shard, no overwrite, a gate that does not wait for the re-run, and a limit of 200 s or 20 s.
• Passes 3 runs in a row: the 12 edit accuracy unit files (87 tests), the new workflow test, the filter test and the merge queue workflow test.
• Manual testing performed: a real pr grid shard with the limit lowered to 1 s stopped after 4 cases with exit 75 and slow-runner.json, closed its browsers and left no orphan process. oxfmt, oxlint and actionlint are clean.
• Comments follow CONTRIBUTING.md "Comments".

[github-actions]

Edit accuracy: accurate 271 (base branch 271), smooth 129 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (1)

• move-tween-px-r0-root-z100: tracking 0.04, pressJump 0, drop 0, reload 0, render 0.02, renderKey -, undo true, teleport true / tracking 0.04, pressJump 0, drop 0, reload 0, render 0.02, renderKey -, undo true, teleport true / tracking 0.04, pressJump 0, drop 0, reload 0, render 0.02, renderKey -, undo true, teleport true

[jrusso1020]

Approved at 6b4a3e846c72e7af9796274990d21165196bec46. I read the workflow jobs in full, ratchet.mjs gate, run.mjs, report.mjs and the new tests. I checked whether the retry can hide a real failure, and I found no way for it to.

• Only slow or cancelled shards are retried. The mark step writes retry only when job.status is cancelled or a slow-runner.json exists. A bench that finishes with failing cases exits normally, so it is never marked. The plan job then lists no shards, the retry job is skipped, and the gate judges the original evidence. run.mjs writes slow-runner.json only once slowRunner trips on the first four cases' median (> 60 s).
• One retry at most. The retry job runs edit-accuracy-shard.sh but has no mark step, and the plan job depends only on the original matrix. Nothing can queue a second retry.
• It fails closed. A stopped bench exits 75 before writeReport, so neither the slow first run nor a slow retry leaves a results.json. If the retry is slow again, hangs again, or never uploads (with if-no-files-found: error, the original marker-only artifact stays), the gate counts 19 of 20 shards reported results and fails. If a retry runner stops during the --rerun passes, the cases keep only their first run. Under passes(), a single failing run still fails (1×2 < 1 is false), so losing the confirmations can't turn a failure into a pass.
• Evidence is replaced, not merged. The artifact keeps the same edit-accuracy-shard-N name with overwrite: true, so the gate never mixes the slow runner's partial evidence with the fresh run's.
• Steps really run after a timeout. In run 38754829357, shard 7/20 ended cancelled, and its if: always() "Upload shard evidence" step still succeeded. So the mark step, which runs between the bench and the upload, will see cancelled in the case this PR targets.
• Merge queue. Only Studio: edit accuracy gate is required, and it now waits for the retry with !cancelled(). A skipped retry (no marked shards) doesn't block it. The retry job takes SHARDS and EDIT_ACCURACY_GRID from the same merge_group vs pull_request switch as the matrix (20/full or 5/pr), so a retried shard runs the same slice. Timeouts: 25 + 5 + 25 + 5 = 60. That leaves out Build (3.3 min in run 38754829357) and runner pickup, so a worst case of shard timeout plus retry timeout could run past the 60 min check wait. The case this fixes stops after four cases, though, so it finishes well inside.
• Path filter. scripts/ci/edit-accuracy-shard.sh is in the edit_accuracy filter. check-edit-accuracy-filter.test.mjs now follows .sh files that a step runs, and it includes the retry job, so the moved bench command stays covered.

Nits, not blocking:

• When a real perf regression makes every case slow, the gate reports a missing shard rather than naming the slow cases. It still fails, and slow-runner.json in the retry artifact explains why, but the PR comment won't say so.
• A skipped retry job shows up as Studio: edit accuracy retry (${{ matrix.shard }}). That's cosmetic.

All required CI is green at this head; the retry path was skipped on this PR since no shard was slow. I didn't trigger a retry myself.

— Rames