Skip to content

fix(weekly-mutation): check the tool shards before a sharded run, and run one shard with any tool - #120

Merged
NSchatz merged 1 commit into
mainfrom
fix/sharded-weekly-adoption
Nov 10, 2026
Merged

NSchatz merged 1 commit into
mainfrom
fix/sharded-weekly-adoption

Conversation

@NSchatz

@NSchatz NSchatz commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

What

Changes that make the sharded weekly mutation run safe for a library to adopt. Each change and the test that covers it:

  1. The tool is checked before a sharded run. With more than one shard, each shard has a new step right after the install, "Check that cosyte-mutate can shard". It runs <mutate-cmd> --help with CI=false and fails within seconds unless every help it printed names COSYTE_MUTATE_SHARD. The error says sharding needs @cosyte/vitest-config 0.1.5 or later, or shard-count: 1. Before this, a tool that does not shard ran the whole suite on every shard and only then failed. Tests: test/testing-standard-jobs.test.mjs, "the probe asks the mutate command for its help...", "the probe passes the stand-in that shards and refuses the stand-in of 0.1.4..." (this one also covers several packages where one of them is old), and "with one shard the probe asks nothing".
  2. One shard works with any tool. With shard-count: 1, a mutation.json written during the run (newer than a marker touched before the command) is read as the report of shard 1 of 1. A mutation.json left from before the run is never read. With more than one shard, mutation.json is still refused. Tests: "with one shard, the mutation.json a tool that does not shard writes is the report of shard 1 of 1", "with one shard, a mutation.json the run did not write is not read...", "with several shards, a mutation.json is never read as a shard report". On a real runner, self-check has a new weekly-mutation-one-shard job that runs the stand-in as 0.1.4, and quality-artifacts checks its merged report and note.
  3. The cancel comment is corrected. The save step's comment (and the README) no longer say that a shard stopped at its time limit saves its results. GitHub cancels that job, and a step with !cancelled() does not run in a cancelled job. Test: "the save step says what it does not save...".
  4. Restore across two runs. quality-artifacts has a new step, "The first weekly run started from an earlier run's results". For each shard of the first weekly call, it reads the run id in the restored cache key and requires that an earlier run saved it (an earlier attempt of the same run also counts). Test: "self-check proves each shard of its first weekly run restored an earlier run's results", which runs the step's script against made-up notes.
  5. The README no longer names an internal path or id. The dash gate section's last paragraph is rewritten. A new test/public-surface.test.mjs reads every tracked file for the planning folder and its id shapes.
  6. The gate test's commit email has no domain. test/gate-workflows.test.mjs now uses gate-tester. The shared PHI scan flags any address with a domain (example.com included; it refused that on the first push), and a domainless value passes. Proof: the pre-push scan passed on this branch.
  7. The README's "What the real tool still needs" section is updated. It says sharding has landed in cosyte/config and ships in the next patch release of @cosyte/vitest-config, 0.1.5, which is not on npm yet. The sections "The sharded weekly run" and "The testing standard's jobs" describe the check, the one-shard path and the cross-run proof.

The stand-in cosyte-mutate in test/fixtures/quality-stubs/ now answers --help the way the real one does (stderr and exit 2 under CI). With COSYTE_MUTATE_STUB_AS=0.1.4 it plays the release that does not shard. Tests are in test/quality-stubs.test.mjs. The new self-check job is added to the two pinned context lists. In CHANGELOG, the lines for the six references published since the last heading are moved under their own headings, and a new Unreleased entry is added.

Why

Before any library adopts the sharded weekly run, a few problems needed fixing. A red weekly run makes the release mutation gate, which is on by default, refuse every release of that library. The old setup could go red without a useful message: an old tool ran for hours on every shard and then failed. The fix also gives a small library a way to adopt with today's released tool, and makes the comments and README match what the workflow actually does.

Proof

  • harness gate (actionlint 1.7.12 with shellcheck, node --test "test/*.test.mjs", check-no-emdash, package-lists): passed.
  • Red first: before the workflow and stub changes, the new tests in test/testing-standard-jobs.test.mjs and test/quality-stubs.test.mjs failed (13 failures). With the README from the base, test/public-surface.test.mjs failed on README.md:3303 (a planning path and a standards id). All of them pass now.
  • The pre-push PHI scan refused gate@example.com and passed gate-tester.

Choices

  • The probe uses the caller's mutate-cmd with --help added, and adds no new input. This works for pnpm exec cosyte-mutate full and pnpm -r exec cosyte-mutate full. With more than one shard, a command that cannot take --help at its end fails the check. To undo or widen this, add an optional probe-command input.
  • The probe reads the help text, not a version number. That way it checks what the tool does, not its package version. To undo, compare the installed @cosyte/vitest-config version against 0.1.5 instead.
  • The cancel comment is corrected, and saving on cancel is not added. A job cancelled at its limit runs no !cancelled() step, and we have not measured whether StrykerJS leaves a usable incremental file when it is stopped partway. To change this, bound the command inside the step with timeout so the job is not cancelled.
  • The cross-run check is strict. If no earlier entry exists (a fresh fork, or entries GitHub evicted), quality-artifacts is red once, and the next run is green. quality-artifacts is not a required context. To undo, make that step warn instead of fail.

Kind: code

… run one shard with any tool

With more than one shard, each shard now runs the mutate command with --help right after the
install and fails within seconds unless it names COSYTE_MUTATE_SHARD, naming
@cosyte/vitest-config 0.1.5 or later, instead of running the whole suite on every shard first.
With shard-count 1, the mutation.json a tool that does not shard writes during the run is read as
the report of shard 1 of 1, so a small library can adopt with 0.1.4.

The save step's comment no longer says a shard stopped at its time limit saves its results: a
cancelled job does not run it. Self-check adds a one-shard run with a stand-in of 0.1.4, and checks
that each shard of its first weekly run restored an entry an earlier run saved. The README drops a
planning path and id, says sharding landed in config for the next patch release, and a new test
keeps planning ids out of tracked files. The gate test's commit email takes no domain, which the PHI scan
accepts, and CHANGELOG moves the published references' lines under their headings.
@NSchatz
NSchatz merged commit 27aa799 into main Nov 10, 2026
40 checks passed
@NSchatz
NSchatz deleted the fix/sharded-weekly-adoption branch November 10, 2026 18:01
NSchatz added a commit that referenced this pull request Nov 10, 2026
… public surface scan (#121)

## What

Three follow-ups from the review of #120, the sharded weekly mutation
run:

1. **The public surface scan is wider.** `test/public-surface.test.mjs`
now flags the planning folder after a slash, a dot or a hyphen too (a
path into another checkout, a relative path), and decision ids of one to
four digits, not only two or three. No tracked file trips the wider
patterns.
2. **The shard check never runs a full mutation for a compound
`mutate-cmd`.** "Check that cosyte-mutate can shard" no longer appends
`--help` to the end of the line. It runs `mutate-cmd` as written, once,
with `COSYTE_MUTATE_SHARD` set to `8589934592/8589934592`: the tool puts
a file in shard k of n when its 32-bit FNV-1a hash mod n is k - 1, and
with n = 2^33 no hash reaches n - 1, so the shard holds no file. Every
part of `a && b` sees the variable. `COSYTE_MUTATE_INCREMENTAL_FILE` is
unset for the check, and the empty shard reports it leaves are deleted.
3. **A tool that names the variable but ignores it is caught before the
full run.** The check fails, saying sharding needs
`@cosyte/vitest-config` 0.1.5 or later, when the command wrote a
`mutation.json` (it ran everything), is still running after 180 seconds
(`timeout` stops it and its process group), exits non-zero, or wrote no
shard report.

README.md ("The sharded weekly run"), the workflow comments and
CHANGELOG.md say the same. CHANGELOG.md also moves #120's lines under
the reference its merge published, `workflows-2026-11-10-27aa79969b26`.

Inputs and their defaults, job ids, outputs, secrets and permissions are
unchanged. The one-shard path, the shard step and the merge are
untouched, so the result the release gate reads is unchanged.

## Why

The review of #120 found three gaps. The public surface test skipped a
planning path after a slash, dot or hyphen, and missed one and four
digit decision ids. The shard check put `--help` only at the end of
`mutate-cmd`, so in a compound command the first part ran a full
mutation inside the check. And a tool whose help names
`COSYTE_MUTATE_SHARD` but ignores it passed the check and was caught
only after hours of mutating on every shard.

## Proof

Red on the base (27aa799), with the new tests and the old code:

- `test/public-surface.test.mjs`: "the planning folder pattern catches
the folder wherever a path puts it" failed on the `cosyte/` prefix; "the
decision id pattern catches ids of one to four digits" failed on a one
digit id.
- `test/testing-standard-jobs.test.mjs`: "the probe never runs a full
mutation for a compound mutate command" failed with "a full run happened
inside the probe" (the first command wrote `mutation.json`); "the probe
fails a tool whose help names the shard variable but which ignores it"
failed because the old check passed it ("cosyte-mutate can shard"). The
other new probe tests failed on the old messages.

Green after the change: `node --test "test/*.test.mjs"` passes, and
`harness gate github-profile` (actionlint 1.7.12, the tests, the dash
check, the package lists) passes.

## Choices

- **Probe safely rather than refuse a compound command.** Refusing would
break any caller whose `mutate-cmd` runs several commands; the empty
shard probe needs no parsing of the command line. To undo, restore the
`--help` step from #120.
- **The time bound is 180 seconds, in the step's `PROBE_SECONDS`.** A
tool that shards loads its config and finishes an empty shard in
seconds, also across several packages; the bound caps what a tool that
ignores the variable costs each shard. Not an input, so no caller-facing
change. To change it, edit `PROBE_SECONDS` and the README line.
- **The help check is gone, not kept beside the new one.** Appending
`--help` is what ran a full mutation in a compound command, and the
behavioural check covers every tool the help check caught. To undo,
restore it from #120.

Kind: code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant