Skip to content

Give the validation runner contract step headroom over its measured run time - #825

Merged
coneilen merged 1 commit into
mainfrom
coneilen-microsoft-windows-preview-rebuild
Oct 15, 2026
Merged

coneilen merged 1 commit into
mainfrom
coneilen-microsoft-windows-preview-rebuild

Conversation

@coneilen

Copy link
Copy Markdown
Collaborator

Summary

The windows-spikes Swift job's "Verify validation runner contract" step had a 5-minute (300s) timeout. That contract has grown with each new set of runner test cases: about 75s a day ago, 180–276s across recent PRs, and 302s on run 38663273264, which failed CI on a 1–2s overrun. The same contract measured up to 341s in deterministic hardening (run 38659424901). Timeouts that close to normal run time make CI flaky, so this raises the step's timeout to 15 minutes and guards it.

Changes

  • .github/workflows/windows-port-validation.yml: raise the "Verify validation runner contract" step timeout from 5 to 15 minutes. The job's 45-minute timeout is unchanged: the job measured 21–25 minutes, so even a 15-minute contract run stays inside it.
  • Tools/windows/Tests/ValidationRunner.Tests.ps1: a new contract fails if that step's timeout is missing or below 12 minutes. That's about twice the measured 341s worst case, the same measured-headroom rule the file already applies to other timeouts. A comment records the measured runs.

Test plan

RED: guard extracted from Tools/windows/Tests/ValidationRunner.Tests.ps1 run against origin/main windows-port-validation.yml -> RED: cap=5, exit 1
GREEN: same guard against the edited workflow -> GREEN: cap=15, exit 0; negative with the step timeout removed -> RED: cap empty, exit 1
REGRESSION: PowerShell Parser.ParseFile on Tools/windows/Tests/ValidationRunner.Tests.ps1 -> 0 errors; git diff --check -> clean; bash Tools/ci/tests/classify-changes.test.sh -> 62 passed, 0 failed; exact-head CI runs the full contract in windows-spikes Swift and both deterministic hardening jobs

Measurements (from gh run view step timings, 23 recent port runs and 24 hardening jobs):

  • port contract step: average 178s, maximum 302s;
  • hardening contract step: maximum 341s;
  • windows-spikes Swift job: 21–25 minutes.

Not verified locally: I didn't run the full contract locally, because it opens hidden native windows. Exact-head CI is the regression gate. macOS make test and make check were not run, because no Swift changed.

Checklist

  • I have read the Contributing Guidelines
  • I have signed off my commits (git commit -s) per the DCO
  • Tests pass locally (make test)
  • Code follows the existing style (make check)
  • I added the test/contract before the implementation and observed the intended RED failure

…un time

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: coneilen <coneilen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant