Skip to content

docs(testing): test the critical path and its failures, fewer tests, and keep them fast - #5

Merged
thatsrue merged 7 commits into
mainfrom
docs/testing-method
Nov 19, 2026
Merged

thatsrue merged 7 commits into
mainfrom
docs/testing-method

Conversation

@thatsrue

@thatsrue thatsrue commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

Rewrites docs/testing.md into four groups: what to test, how many, how a test checks, and speed. The four current rules stay (two gain a sentence); seven are new. Each rule cites its source.

Why

The current rules say what a test may assert, but not what to test, how many tests are enough, or how fast they must be. Agents writing tests have produced many near-duplicate tests and slow suites.

  • One table, not many tests (rule 3): seriouslysean/occasional-wotd's tests/utils/url-utils.spec.ts holds 75 tests for an 85-line module.
  • No test only for coverage (rule 4): several repos gate on 80 to 90 % coverage.
  • Small, hermetic, parallel, no sleep (rules 8 to 10): the slowest suite measured ran one file at a time and shelled out to a host tool; parallel files and a stub cut it from about 10 minutes to about 2.
  • 60 s per file, 300 s for a file that starts child processes or a local server (rule 11): measured on that suite after a pass over its slow files, every file fits; the only files over 60 s (65 to 126 s) start a daemon or a command per test, and trimming did not bring them under 60.
  • Removing or loosening needs a mutation check, and documented text stays pinned ("Removing a test"): in 7 of 9 pull requests that trimmed tests under these rules, a reviewer who broke the source found a removal or loosening that reading had passed, most often an assertion on text the repository's own docs promise.

Left out: a per-case limit (no source sets one for every runner), a whole-gate limit (each repo's gate owns that), and coverage thresholds (a call per repo).

No issue: docs

How you checked it

  • Every link returns 200; the coverage quote is on the cited page; ISTQB §4.2 and §5.2 are equivalence partitioning and risk management.

  • A second review checked each rule against its cited page; rules the sources don't state are marked as these repos' choice, and rule 1 now caps tests per change.

  • The repository has no check to run on docs.

  • A bug fix comes with a test that fails without it

  • Docs match how it works now

  • No secrets or private information

@thatsrue
thatsrue merged commit 16cd768 into main Nov 19, 2026
1 check passed
@thatsrue
thatsrue deleted the docs/testing-method branch November 19, 2026 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant