Repository navigation
feat(release): gate publishes on the weekly mutation result, shard the weekly run - #119
Merged
Merged
Conversation
…e weekly run The weekly mutation run is split across shard-count parallel jobs, each keeping its StrykerJS incremental results in the Actions cache, and its mutation job merges the shards into one report per package before the floor is applied as before. release.yml runs pre-publish gates in its version job: the latest weekly mutation result (read through the API, never re-run), and, when a caller turns them on, the benchmarks with the scaling gate, the package checks on the packed tarball and the docs samples against the installed tarball. Every GitHub release carries a CycloneDX SBOM. release.yml runs no full mutation pass.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
NSchatz
added a commit
to cosyte/config
that referenced
this pull request
Nov 10, 2026
…tal results (#206) ## What `cosyte-mutate full` can now run in shards, with Stryker's incremental results kept between runs. - `COSYTE_MUTATE_SHARD=<k>/<n>`: the run lists every file the config's `mutate` patterns name under the working directory (never inside `node_modules`, `.git` or Stryker's temp directory; picked by the same rule as `selectChangedFiles`), keeps the files whose 32-bit FNV-1a hash of the path (relative, forward slashes) mod `n` is `k - 1`, runs Stryker on just those, and writes `reports/mutation/shard-<k>-of-<n>.json` and `.html` beside the full report, never `mutation.json`. - A shard with no file writes the empty report at its shard path and exits 0 without starting Stryker. - A value that is not two whole numbers with `1 <= k <= n` exits 2 before anything is written or started. - `COSYTE_MUTATE_INCREMENTAL_FILE=<file>` passes `incremental: true` and that file (resolved against the working directory) as `incrementalFile` to Stryker, sharded or not. - With neither variable set (or both empty), a full run is exactly as before: same options to Stryker, same output. A changed run reads neither. - New exports from `@cosyte/vitest-config/mutate`: `parseShard`, `fnv1a32`, `filesInShard`, `expandMutate`, `shardReportPath`, `SHARD_VARIABLE`, `INCREMENTAL_VARIABLE`; `runMutate` takes an optional `env` (default `process.env`). Types in `mutate.d.ts`. - Docs: the package README has a "A full run in shards" section, the bin's `--help` names both variables, and the testing standard (decision 0004, section 4) describes the sharded full run. A patch changeset for `@cosyte/vitest-config`. ## Why The weekly full mutation run in cosyte/.github (cosyte/.github#119) splits one full run across parallel jobs and caches Stryker's incremental results, because an unsharded run of a large library takes hours on a 4-core hosted runner (hl7: 11,422 mutants). That workflow sets `COSYTE_MUTATE_SHARD` and `COSYTE_MUTATE_INCREMENTAL_FILE`, collects `*/reports/mutation/shard-<k>-of-<n>.json`, and merges them per package with its `scripts/mutation-shards.mjs`, which refuses a missing shard or a file in two shards. It was proved against a stand-in for this tool; the released `cosyte-mutate` (0.1.4) does not read the variables, so every shard fails. This makes the real tool match the stand-in's contract: the same variable format and refusals, the same hash over the same path form, the same report names. ## Proof Red against the base: the new tests were added first and run against the unchanged `mutate.js`: 16 failed, 27 passed (the passing ones are the existing suite plus the "unset variables unchanged" and "a changed run reads neither variable" guards, which must pass on both sides). Green after the change: `packages/vitest-config` 50 tests pass, coverage 100% statements, branches, functions and lines. `cosyte-mutate changed --base origin/main` then `cosyte-ratchet check --skip coverage --base origin/main`: 297 mutants in `mutate.js`, none surviving in changed code, ratchet OK. The shard partition test (`filesInShard`, and again through `runMutate` with a fake Stryker over a real fixture tree): ```ts it("puts every file in exactly one shard, for 1, 2, 8 and 16 shards", () => { for (const n of [1, 2, 8, 16]) { const shards = shardsOf(MANY, n); const all = shards.flat(); expect(all.length, `n=${n}`).toBe(MANY.length); expect(new Set(all).size, `n=${n}`).toBe(MANY.length); expect([...all].sort(), `n=${n}`).toEqual([...MANY].sort()); shards.forEach((files, i) => { for (const file of files) expect(fnv1a32(file) % n, file).toBe(i); }); ... ``` The stub-parity tests: the stand-in's `hashOf` and `sourceFiles` are copied unchanged from `test/fixtures/quality-stubs/stubs.mjs` into the test file, and ```ts it("matches the stand-in's hash on the same paths, non-ASCII ones included", () => { ... for (const path of paths) expect(fnv1a32(path), path).toBe(stubHashOf(path)); }); it("picks each shard's files exactly as the stand-in does", () => { ... const ours = expandMutate({ cwd: d, patterns: ["src/**/*.mjs"] }); expect(ours).toEqual(stubSourceFiles(d)); for (const n of [1, 2, 3, 8, 16]) { for (let k = 1; k <= n; k += 1) { expect(filesInShard(ours, { k, n }), `${k}/${n}`).toEqual( stubSourceFiles(d).filter((file) => stubHashOf(file) % n === k - 1), ); } } }); ``` Also covered: the FNV-1a reference values (`""`, `"a"`, `"foobar"`); 19 malformed shard values refused, and exit 2 through `runMutate` with nothing written; the empty shard; the incremental options with and without a shard; and one run of real StrykerJS over a fixture with `1/1` and an incremental file, which writes `shard-1-of-1.json` and the incremental file and no `mutation.json`. The repository gate (`harness gate config`) passed before push. ## Choices - **Shard value format.** Accepted exactly what the stand-in accepts (`^\d+/\d+$`, then `1 <= k <= n`), so `08/16` is shard 8 of 16. Undo: tighten the regular expression in `parseShard`. - **Unset and empty are the same.** An empty `COSYTE_MUTATE_SHARD` or `COSYTE_MUTATE_INCREMENTAL_FILE` is read as unset, as the stand-in does. Undo: compare with `undefined` only. - **Shard report beside the full report.** `shard-<k>-of-<n>.json` goes in the directory of the config's `jsonReporter.fileName`, as `changed.json` does, so a config run from the repository root that reports under `packages/<name>/reports/mutation/` gets its shards there, which is where the merge writes `mutation.json`. Undo: write under `reports/mutation/` always in `shardReportPath`. - **Also skip `.git`.** The listing skips `.git` besides `node_modules` and the temp directory (from the config's `tempDirName`), and never follows a symbolic link. Undo: drop the name check in `expandMutate`. - **The incremental file is resolved against the working directory** before it reaches Stryker, as the config file is. Undo: pass the value as given. - **`runMutate` reads `process.env` by default**, so a caller of the entry point shards the same way the bin does; the suite passes `env: {}` so a test run inside a sharded run is not affected. Kind: code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
weekly-mutation.ymlis sharded and incremental. A newshardjob runs the caller'smutate-cmdon each ofshard-countshards (new input, default 8, 1 to 16) in parallel;mutationkeeps its id, merges the shard reports into one report per package and applies the floor exactly as before.release.ymlruns four pre-publish gates in itsversionjob, afterVerifyand before anything can publish, and every GitHub release carries a CycloneDX SBOM.release.ymlruns no full mutation pass anywhere: the mutation gate reads the latest weekly result and starts nothing.scripts/mutation-shards.mjs(the merge),scripts/mutation-gate.mjs,scripts/tarball-samples.mjs,scripts/sbom.mjs;scripts/github-release.mjstakes--sbom.self-check.ymlruns a sharded weekly run (2 shards) and then an incremental one (weekly-mutation-incremental, which restores what the first saved), and a newrelease-gatesjob runs the mutation gate, the samples gate and the SBOM for real on this repository and the fixture, with no write permission.ci / mutationis meant to become a required check when a library adopts it. CHANGELOG.md "Unreleased" says what a caller must act on.The sharding and cache design
mutate-cmdwithCOSYTE_MUTATE_SHARD=<k>/<n>andCOSYTE_MUTATE_INCREMENTAL_FILE=reports/mutation/incremental.json;cosyte-mutate fullis to mutate only its share and writereports/mutation/shard-<k>-of-<n>.jsonper package. Variables rather than arguments, so they reach everycosyte-mutatea multi-package command starts.shard-count,fail-fast: false, each shard bounded at 340 minutes, inside GitHub's 6 hour job limit. A count outside 1 to 16 fails before any shard starts.cosyte-mutation-v1-<working-directory>-<count>-<shard>-(this branch's first, then the default branch's, so a scheduled run on main reuses main's latest results), and saves its incremental files under a key unique to the run, even when its run failed. Each shard's summary and note says whether it started incremental or full.mutationruns with!cancelled()after the shards and fails at once if any shard did not succeed (a skip would satisfy a required context). It merges withscripts/mutation-shards.mjsfrom the tooling checkout atjob.workflow_sha, refusing a package missing a shard or a file in two shards, then runscosyte-ratchet check --skip coverageas before: red below the floor, never a commit.The release gates
mutation-workflow(weekly-mutation.yml),mutation-max-age-days(8).github/workflows/<mutation-workflow>in the checkoutpnpm run test:perf, thencosyte-bench runandcheckper suite)run-benchmarks,bench-suites,bench-targetrun-benchmarksoff; with no suite the scaling gate runs aloneprepublish-check.mjsmanifest and pack layers; types per exports entry staypnpm attwinVerify)run-package-checksrun-package-checksoffnpm packtarball, installed in a clean directoryrun-tarball-samples,docs-dir(docs-content)sbom.cdx.json), attached in the call that creates the releaseThe mutation gate reads through the API with the automatic token at
actions: read, whichversionalready holds. A refused token, a 404 or 5xx, a network fault or an unreadable body fails closed; a latest run that did not succeed, or finished more than the age limit ago, or no completed run at all, refuses. Whether the caller has the workflow is read from the checkout, not the API, so a token without the grant cannot read as "no workflow". The gates are inversion, which holds noid-token: write;releaseneeds it and never opts back in after a failure, so a refusal stops the publish before anyone is asked to approve it.Why
A full mutation run takes hours: hl7's first full run counted 11,422 mutants, estimated at 45 minutes to 3.5 hours on 16 cores, so roughly 3 to 14 hours on a 4-core hosted runner, past the weekly run's 180 minute bound. So a full run never sits in front of a release. The release reads the latest weekly result instead, and the weekly run is split across shards and made incremental so it finishes. The other gates move checks a reader depends on (the scaling gate, the packed tarball, the docs samples) in front of the publish, off until each library adopts them.
Proof
release-gates,github-release(--sbom),testing-standard-jobs(shards, cache, merge job, self-check) andquality-stubs(the shard contract), and the four new script suites failing to load their scripts.harness gate github-profilepassed all 4 commands: actionlint 1.7.12 with shellcheck 0.11.0,node --test "test/*.test.mjs"(1359 pass, 0 fail),node scripts/check-no-emdash.mjs,node test/package-lists.mjs.self-check.ymljob, among themweekly-mutation / shard (1)andshard (2),weekly-mutation-incremental(whosequality-artifactsproof requires both shards to say they started from the first run's cached results) andrelease-gates. Code scanning first raised three warnings in the new tests and the stub (an unanchored URL pattern, a check-then-read, an incomplete escape); a later commit answers them.scripts/tarball-samples.mjsagainst a copy of the fixture found a real defect shape first:dist/is in the fixture's.gitignore, sonpm packleft it out and the sample failed withERR_MODULE_NOT_FOUND. The fixture now declaresfiles: ["dist"]and the sample passes.Choices
COSYTE_MUTATE_SHARD,COSYTE_MUTATE_INCREMENTAL_FILE) thatcosyte-mutatereads, not a wrapper config the workflow writes: a wrapper cannot follow a caller whose command runscosyte-mutatein several directories or with--config.cosyte-mutate0.1.4 does not read them yet, so each shard fails with a message naming that; the workflow side is proved against the stand-in. Undo: revertweekly-mutation.ymlto the single job.version, not a new job: no new check context in any caller, and no publish credential where they run. Undo: move the steps into a jobreleaseneeds.pnpm list --prod, offline, so a registry fault cannot hold a release. Undo: put the SBOM step behind an input.pnpm attw. Undo: add a pinned publint step.version's bound moves from 30 to 60 minutes for the gates.Kind: code