Repository navigation
fix(weekly-mutation): check the tool shards before a sharded run, and run one shard with any tool - #120
Merged
Conversation
… run one shard with any tool With more than one shard, each shard now runs the mutate command with --help right after the install and fails within seconds unless it names COSYTE_MUTATE_SHARD, naming @cosyte/vitest-config 0.1.5 or later, instead of running the whole suite on every shard first. With shard-count 1, the mutation.json a tool that does not shard writes during the run is read as the report of shard 1 of 1, so a small library can adopt with 0.1.4. The save step's comment no longer says a shard stopped at its time limit saves its results: a cancelled job does not run it. Self-check adds a one-shard run with a stand-in of 0.1.4, and checks that each shard of its first weekly run restored an entry an earlier run saved. The README drops a planning path and id, says sharding landed in config for the next patch release, and a new test keeps planning ids out of tracked files. The gate test's commit email takes no domain, which the PHI scan accepts, and CHANGELOG moves the published references' lines under their headings.
NSchatz
added a commit
that referenced
this pull request
Nov 10, 2026
… public surface scan (#121) ## What Three follow-ups from the review of #120, the sharded weekly mutation run: 1. **The public surface scan is wider.** `test/public-surface.test.mjs` now flags the planning folder after a slash, a dot or a hyphen too (a path into another checkout, a relative path), and decision ids of one to four digits, not only two or three. No tracked file trips the wider patterns. 2. **The shard check never runs a full mutation for a compound `mutate-cmd`.** "Check that cosyte-mutate can shard" no longer appends `--help` to the end of the line. It runs `mutate-cmd` as written, once, with `COSYTE_MUTATE_SHARD` set to `8589934592/8589934592`: the tool puts a file in shard k of n when its 32-bit FNV-1a hash mod n is k - 1, and with n = 2^33 no hash reaches n - 1, so the shard holds no file. Every part of `a && b` sees the variable. `COSYTE_MUTATE_INCREMENTAL_FILE` is unset for the check, and the empty shard reports it leaves are deleted. 3. **A tool that names the variable but ignores it is caught before the full run.** The check fails, saying sharding needs `@cosyte/vitest-config` 0.1.5 or later, when the command wrote a `mutation.json` (it ran everything), is still running after 180 seconds (`timeout` stops it and its process group), exits non-zero, or wrote no shard report. README.md ("The sharded weekly run"), the workflow comments and CHANGELOG.md say the same. CHANGELOG.md also moves #120's lines under the reference its merge published, `workflows-2026-11-10-27aa79969b26`. Inputs and their defaults, job ids, outputs, secrets and permissions are unchanged. The one-shard path, the shard step and the merge are untouched, so the result the release gate reads is unchanged. ## Why The review of #120 found three gaps. The public surface test skipped a planning path after a slash, dot or hyphen, and missed one and four digit decision ids. The shard check put `--help` only at the end of `mutate-cmd`, so in a compound command the first part ran a full mutation inside the check. And a tool whose help names `COSYTE_MUTATE_SHARD` but ignores it passed the check and was caught only after hours of mutating on every shard. ## Proof Red on the base (27aa799), with the new tests and the old code: - `test/public-surface.test.mjs`: "the planning folder pattern catches the folder wherever a path puts it" failed on the `cosyte/` prefix; "the decision id pattern catches ids of one to four digits" failed on a one digit id. - `test/testing-standard-jobs.test.mjs`: "the probe never runs a full mutation for a compound mutate command" failed with "a full run happened inside the probe" (the first command wrote `mutation.json`); "the probe fails a tool whose help names the shard variable but which ignores it" failed because the old check passed it ("cosyte-mutate can shard"). The other new probe tests failed on the old messages. Green after the change: `node --test "test/*.test.mjs"` passes, and `harness gate github-profile` (actionlint 1.7.12, the tests, the dash check, the package lists) passes. ## Choices - **Probe safely rather than refuse a compound command.** Refusing would break any caller whose `mutate-cmd` runs several commands; the empty shard probe needs no parsing of the command line. To undo, restore the `--help` step from #120. - **The time bound is 180 seconds, in the step's `PROBE_SECONDS`.** A tool that shards loads its config and finishes an empty shard in seconds, also across several packages; the bound caps what a tool that ignores the variable costs each shard. Not an input, so no caller-facing change. To change it, edit `PROBE_SECONDS` and the README line. - **The help check is gone, not kept beside the new one.** Appending `--help` is what ran a full mutation in a compound command, and the behavioural check covers every tool the help check caught. To undo, restore it from #120. Kind: code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Changes that make the sharded weekly mutation run safe for a library to adopt. Each change and the test that covers it:
<mutate-cmd> --helpwithCI=falseand fails within seconds unless every help it printed namesCOSYTE_MUTATE_SHARD. The error says sharding needs@cosyte/vitest-config0.1.5 or later, orshard-count: 1. Before this, a tool that does not shard ran the whole suite on every shard and only then failed. Tests:test/testing-standard-jobs.test.mjs, "the probe asks the mutate command for its help...", "the probe passes the stand-in that shards and refuses the stand-in of 0.1.4..." (this one also covers several packages where one of them is old), and "with one shard the probe asks nothing".shard-count: 1, amutation.jsonwritten during the run (newer than a marker touched before the command) is read as the report of shard 1 of 1. Amutation.jsonleft from before the run is never read. With more than one shard,mutation.jsonis still refused. Tests: "with one shard, the mutation.json a tool that does not shard writes is the report of shard 1 of 1", "with one shard, a mutation.json the run did not write is not read...", "with several shards, a mutation.json is never read as a shard report". On a real runner, self-check has a newweekly-mutation-one-shardjob that runs the stand-in as 0.1.4, andquality-artifactschecks its merged report and note.!cancelled()does not run in a cancelled job. Test: "the save step says what it does not save...".quality-artifactshas a new step, "The first weekly run started from an earlier run's results". For each shard of the first weekly call, it reads the run id in the restored cache key and requires that an earlier run saved it (an earlier attempt of the same run also counts). Test: "self-check proves each shard of its first weekly run restored an earlier run's results", which runs the step's script against made-up notes.test/public-surface.test.mjsreads every tracked file for the planning folder and its id shapes.test/gate-workflows.test.mjsnow usesgate-tester. The shared PHI scan flags any address with a domain (example.comincluded; it refused that on the first push), and a domainless value passes. Proof: the pre-push scan passed on this branch.@cosyte/vitest-config, 0.1.5, which is not on npm yet. The sections "The sharded weekly run" and "The testing standard's jobs" describe the check, the one-shard path and the cross-run proof.The stand-in
cosyte-mutateintest/fixtures/quality-stubs/now answers--helpthe way the real one does (stderr and exit 2 under CI). WithCOSYTE_MUTATE_STUB_AS=0.1.4it plays the release that does not shard. Tests are intest/quality-stubs.test.mjs. The new self-check job is added to the two pinned context lists. In CHANGELOG, the lines for the six references published since the last heading are moved under their own headings, and a new Unreleased entry is added.Why
Before any library adopts the sharded weekly run, a few problems needed fixing. A red weekly run makes the release mutation gate, which is on by default, refuse every release of that library. The old setup could go red without a useful message: an old tool ran for hours on every shard and then failed. The fix also gives a small library a way to adopt with today's released tool, and makes the comments and README match what the workflow actually does.
Proof
harness gate(actionlint 1.7.12 with shellcheck,node --test "test/*.test.mjs",check-no-emdash,package-lists): passed.test/testing-standard-jobs.test.mjsandtest/quality-stubs.test.mjsfailed (13 failures). With the README from the base,test/public-surface.test.mjsfailed onREADME.md:3303(a planning path and a standards id). All of them pass now.gate@example.comand passedgate-tester.Choices
mutate-cmdwith--helpadded, and adds no new input. This works forpnpm exec cosyte-mutate fullandpnpm -r exec cosyte-mutate full. With more than one shard, a command that cannot take--helpat its end fails the check. To undo or widen this, add an optional probe-command input.@cosyte/vitest-configversion against 0.1.5 instead.!cancelled()step, and we have not measured whether StrykerJS leaves a usable incremental file when it is stopped partway. To change this, bound the command inside the step withtimeoutso the job is not cancelled.quality-artifactsis red once, and the next run is green.quality-artifactsis not a required context. To undo, make that step warn instead of fail.Kind: code