Skip to content

Latest commit

 

History

History
1084 lines (886 loc) · 55.3 KB

testing-spec.md

File metadata and controls

1084 lines (886 loc) · 55.3 KB

Executable.md Testing

Motivation

Executable documents can contain probabilistic behavior. Tests ground that behavior in observable results and give authors confidence in their documents.

Testing Mode

Tests run only in testing mode. During regular execution, <Test> and its entire body are skipped: it renders nothing, binds nothing, and runs no side effect.

<Testing> enables testing mode for its expanded subtree:

<Testing>
  <Example />
</Testing>

Installing the package registers <Testing>, <AssertThrows> and the value assertions as ordinary non-reserved defaults (executable-mdx-spec.md §5.3): a repository component of any of those names is chosen ahead of the registered one, exactly as it would be ahead of any other package's default, and then decides for itself what a testing boundary means.

<Test> is not among them. Canonical core registers and owns the default <Test> construct, because core owns what an invocation of one means for the run (see Checked command failures inside a test below). Installing this package supplies that construct's testing behavior — activation, isolated bindings and scope, the timeout, assertion and unexpected-failure classification, TestResult staging and recording, reporting, and the testing completion failure — through the TestBehavior operation core calls from inside the invocation. <Test> is still an ordinary non-reserved default, so a repository component of that name is chosen ahead of core's and decides for itself what a test means.

A test result is journaled once the invocation that produced it has been dismantled, so a teardown failure is part of the outcome recorded for it. That is still while expansion runs and before the root close, in discovery order and identified by source position. A run that ends between a test finishing and its result being written journals nothing for that test, and re-runs it on resume.

A boundary reports how many tests ran and how many failed once its body has finished. A body that stopped — expansion ended by throwing rather than by collecting a diagnostic — never counted the tests it did not reach, so it reports no outcome and journals none, and the failure travels on unchanged. An error the body merely collected leaves it complete: that diagnostic is part of what the boundary rendered, and the tests beside it still ran.

The CLI command is equivalent to wrapping the entrypoint in <Testing> for activating testing mode and for journaling the results of the tests it runs. It is not equivalent for boundary reporting: an explicit <Testing> reports and journals a boundary outcome, and a root run reports none.

xmd test <entrypoint>

Test discovery follows normal component expansion. A test runs only when the expanded component tree reaches it, including through imports, components, and conditional rendering. Imported but unrendered tests do not run.

Ordinary content outside a <Test> expands normally. An error in that content aborts the run as an infrastructure error.

<Testing> emits the naturally expanded report without adding a summary. It fails after expansion when any test failed or when no tests were discovered. xmd test exits with status 0 when every test passes and status 1 otherwise.

Testing uses the standard journal and replay behavior. Each completed test and each <Testing> boundary outcome is journaled as a testing-owned durable operation (test_result, testing_boundary) while expansion runs, before the root close event, identified deterministically by source position. On a full replay of a completed journal the recorded results and boundary outcomes are restored from the stream into the current collectors — test bodies and their effects are not rerun — so the testing outcome and recorded results are preserved. On live and partial runs nothing is restored: re-expansion records each result exactly once, in discovery order, with completed records replaying in place.

Choose a reporter

The default --reporter=pretty prints the rendered report on stdout and progress on stderr. Use --reporter=failures for a compact report in CI or a Sidekick Round.

xmd test ./tests --reporter=failures

The failures reporter omits the rendered report, directory headings, test starts, and successful outcomes. It writes failure and cancellation diagnostics and journal paths to stderr. A completed invocation ends with one summary on stderr.

[xmd test] 11 passed | 1 failed (324ms) · 1 of 3 documents failed

The counts cover tests recorded by the root documents. Nested executions can add diagnostics and journals. Exit status and journal retention follow the same rules for either reporter. Output that commands write directly to the terminal keeps its normal routing.

Progress and failure journals

The pretty reporter writes live progress to stderr. A pipe or CI log shows which test is running and which operation a timeout interrupted.

[xmd test] start test "held" at suite.test.md:1:1
[xmd test] fail test "held" at suite.test.md:1:1 (timeout; interrupted sh exec at suite.test.md:3:1)
[xmd test] journal saved: /work/.xmd/test-artifacts/2026-…-6f1c…/root-1.jsonl (root 1 suite.test.md)

A test's start is reported when its behavior begins and its outcome once its result is recorded, which follows the test's teardown — a component that expanded successfully is not a passing test. An <Execution> invocation reports its start only after resolution and validation accept it, then reports how that invocation settles. These milestones do not claim that a child launched or passed. An invocation stopped before activation reports no milestone. Ordinary components and command blocks produce no start or terminal lines, including on failure or cancellation; their observations still identify the operation a test timeout interrupted. The CLI's terminal failure summary is unchanged.

When command cancellation unwinds a test before a result is recorded, progress reports cancelled test with its name, location and interrupted operation when one exists. This notice is not a pass/fail verdict and creates no test result. It follows the delegated test work's unwind.

Work inside a nested <Execution> child is labelled with that child, for example child 1 ./child.md:. Absolute source paths inside the command's captured working directory are displayed relative to it; authored relative paths and absolute paths outside that directory retain their spelling. Canonical result and journal locations keep their original identities. The report names tests, elements, command languages and modifier words, locations and states. It never carries a captured value, a prop, command or prompt text, an assertion's operands or anything a document rendered, and the rendered report on stdout is unchanged by it.

Each root document and each nested run child keeps its journal in memory. When that execution fails, times out or is cancelled, the command saves the recorded events as JSONL after the execution's assertions and teardown have finished, and reports the file's absolute path. A child's journal is saved when the child fails, even when the test binds that failure with as. A passing execution writes no file.

Files go to a fresh directory for each invocation, beneath XMD_TEST_ARTIFACTS when it is set and .xmd/test-artifacts in the working directory otherwise. The directory is created only when a journal is saved. An explicit --journal names the root's journal file instead, and no automatic root copy is made. A saved file selects no authored journal, grants nothing to <CollectJournal>, and is not continuation input. A process killed without running its cleanup saves nothing.

Registration, mode, and activation

Three things are distinct, and only the last one makes a test run.

Registration is installTestingComponents(). It registers the components above and supplies what core's <Test> does. It activates nothing: after it, testing is still false, <Test> is still skipped, and the assertion components are usable in ordinary content.

Mode is the public TestApi.testing operation. It answers whether testing mode is on, and it is replaceable policy: middleware may narrow it to false, observe it, refuse, and compose behavior around recording. An answer of true says a test should run. It cannot say that anything exists to collect the result, flush the last one, or decide the run's outcome.

Complete activation is what does own those. There are two, and no third:

  • useTesting(), for one root document execution; and
  • a <Testing> element, for its lexical subtree.

Each owns a result collector, the final flush of the last staged result, and the completion policy that settles the run — and each is package-owned, private to @executablemd/testing, and expires with the scope or boundary that established it. Registration plus a testing override is not a substitute: neither the boolean nor a contextual name is accepted as activation, so a separately loaded copy composing policy around the public surface composes policy and nothing else.

Canonical core enforces the requirement, at the invocation boundary. Installing the testing package adds an activation guard, and core's own <Test> consults it before it mints a test harness and before it dispatches the public TestBehavior chain that supplies what a test does. The guard reads the public mode: inactive leaves <Test> invisible exactly as before, and active requires a complete activation before any harness or body work.

The behavior surface keeps its stable name, its argument, its result, its loaded-copy composition and its ordinary delegating behavior, and a handler composed around it may observe, wrap, delegate or refuse what a test does. What it may not do is answer for the decision that preceded it. Answering without delegating is a legitimate use of that surface, and it reaches a decision core has already taken.

Any refusal raised while that decision is being taken is a configuration failure of the composition, and never a test outcome — whether the activation could not be proved, a handler on the seam refused with an error of its own, or the chain was answered rather than delegated. Refusing by throwing is a supported thing for a handler there to do, and what it throws is its own business, so what marks a refusal is where it was raised rather than what type it is.

A test whose activation was refused never ran. Absorbing that as a contained test failure would leave a composition with no session — which has no completion policy to convert a contained failure into a document failure — reporting a run that ran no test as a success. Canonical expansion therefore withholds a refused decision from every component failure handler, including one a package or a document installs, rather than relying on a handler to decline it.

So under any composition, a refused activation:

  • records, journals and reports no test result;
  • renders no assertion diagnostic;
  • causes no side effect the body would have caused; and
  • fails the document execution, so a caller holding an execution handle observes Err.

Inactive tests are unaffected. Testing mode off still means <Test> renders nothing, binds nothing, and runs nothing, with no result of any kind.

Directory Targets

xmd test accepts a directory as well as a document, and defaults to the current directory. These are the same command:

xmd test
xmd test .

A directory target discovers the documents beneath it recursively. The default pattern is **/*.test.md, so ordinary Markdown such as README.md or notes.md does not run. --pattern chooses a different glob:

xmd test workflows --pattern "**/*.spec.md"
xmd test workflows --pattern "**/*.spec.md" --pattern "**/*.check.md"

Explicit patterns replace the default rather than adding to it, and repeating the option forms an inclusive union. Patterns are relative to the target root. A document that several patterns match runs once, and the documents run in the code-point order of their normalized relative paths, whatever order the patterns were written in. A pattern chosen deliberately to be broad, such as --pattern "**/*.md", selects and runs ordinary Markdown.

Under a directory target each document searches for components in three places, in order: its own directory, the target root, then the configured component directories. A repeated directory keeps its first position. A component therefore resolves beside the test that invokes it however deep in the suite it sits, and two directories may hold same-named components without either reaching the other. No --include is needed, and the working directory does not change.

A single-document target keeps the configured component directories exactly.

Each document runs in its own execution and scope: bindings, component registrations, and testing sessions belong to one document and are torn down before the next one starts. A document that declares no tests fails on its own rather than inheriting an earlier document's results.

The pretty reporter precedes each document's report with a heading naming its relative path. Both reporters identify a failed document by that path. A failing document does not stop the run; the remaining documents still execute, and a closing line names how many of them failed. The run exits 0 only when every discovered document passes, and 1 when any document fails or when no document matches.

--pattern applies to a directory, so supplying it with a single document is rejected. A --pattern with no value, or one followed by another option, is rejected rather than read as a glob; --pattern=<glob> expresses a glob that begins with -. Options after -- are not inspected. --journal writes one trace for one document, so a directory target rejects it before any document runs.

Package test plugins

Declare the plugins a package's Markdown tests use in its xmd.json:

{
  "test": {
    "plugins": ["./tests/support/test-plugin.mjs"]
  }
}

xmd test starts at the test document's directory and searches upward for the nearest xmd.json. It checks the directory containing the nearest package.json and stops there. Without a package manifest the search ends at the filesystem root. An explicit symlink to a test uses the resolved document's directory. A nearer config replaces the whole selection, including when it declares no plugins. The invocation directory and directory target do not affect which config is selected.

The CLI reads and validates the selected config before loading any Plugin for that document. Paths beginning with ./ or ../, and bare installed package specifiers, resolve from the config directory. Absolute local paths and file: URLs use the same rules as --plugin. Explicit flags append in argv order and resolve from the invocation directory. Duplicate Plugin names refuse the document before installation. A malformed config or a configured module load failure names the config path. Plugin modules are trusted code that execute with the host's permissions.

Each test document owns a fresh Plugin installation scope, including a directory run spanning several packages. Install-time resources, registrations and middleware end before the next document starts. Module imports follow the host's ordinary module cache. A selection or installation failure is reported for that document, and directory execution continues. This lookup belongs to xmd test. Other commands and REPL.Test use their own Plugin selections.

Tests requested from the REPL

REPL.Test uses the same directory discovery convention and real per-document useTesting sessions. Both REPL target forms retain the finalized configured-first REPL profile; CLI directory promotion above remains specific to xmd test. The REPL testing contract owns its structured report, admission, feedback, cancellation and retained-history rules.

Atomic Tests

Atomic tests use <Test>:

<Test name="Renders hello world">
  <Let as="result">
    Hello World
  </Let>
  <AssertEquals actual={result}>
    Hello World
  </AssertEquals>
</Test>

The name prop is optional metadata. An unnamed test is identified by its source location; headings in the body remain ordinary output and are not inferred as names.

The timeout prop is optional and declares this test's own timeout as a duration (500ms, 30s, 5min), replacing the 20-second default. A malformed duration is refused where it was written and fails only that test.

Checked command failures inside a test

A foreground command that exits nonzero is a checked failure, and outside a test it fails the run (executable-mdx-spec §3.6). Inside a <Test> it is contained: it becomes that test's failed result, the run's own fatal record stays clear, the tests after it still run, and the testing session's existing completion policy is what fails the overall run. A contained outcome is decided once — replay restores the failed result and the failing testing outcome from the journal without running the command again.

Containment belongs to the <Test> construct, and canonical core owns it. Core registers <Test> as one of its own defaults, and grants containment only to that exact definition, only while it is expanding it. Nothing else confers it: not a name, an origin, a context, a marker, an option, the request, or an installation. TestBehavior supplies what a test does and decides nothing about which element is one — it accepts no function, component name, marker or container, and cannot make any other component contain a failure.

A repository component named Test, or another package that registers the name, still wins ordinary default resolution and is chosen ahead of core's. Core's definition was therefore not expanded, and the one that was receives no containment: a checked failure inside it is the run's, exactly as it would be anywhere else the document did not authorize one.

useTesting() with execute() needs nothing else. The host attaches no installation, names no component, and receives no identity.

A test body behaves like any regular component body. Tests run sequentially in expansion order. Each test runs in a child Effection scope and an isolated binding environment. It inherits ambient context and bindings, but its context changes, bindings, and ongoing effects do not escape. Its scope is fully torn down before the next test starts.

Each test has a 20-second timeout by default; a timeout=<duration> prop on the <Test> element declares that one test's own. A timed-out test is halted, reported as failed, and fully torn down before execution continues.

An assertion failure, unexpected error, or teardown error fails only the current test. Later tests still run. Unexpected errors remain distinct from assertion failures. Output produced before an assertion failure or an unexpected error remains in the report and is followed by the failure diagnostic. Two failures report the diagnostic alone: a test that timed out was halted mid-body, and a test whose teardown failed had already returned, so neither has body output to keep. A teardown failure is reported as a diagnostic rather than as report text, and carries the original error as its cause.

Nested <Test> elements are invalid. Skip, focus, and retry behavior is not supported.

Nested Executions

A test can run another document as a real root execution, without translating it into TypeScript and without launching another xmd process:

<Test name="bootstraps a package">
  <Execution host="run" target="./scripts/bootstrap.md" props={{ name: "x" }} as="run">
    <CollectOutput as="output" />

    <AssertEquals actual={run.kind} expected="settled" />
    <AssertEquals actual={run.result.ok} expected={true} />
    <AssertStringIncludes actual={output} expected="created x" />
  </Execution>
</Test>

This is not component composition. <Execution> creates a child document execution with its own root import, target selection, journal, output stream, scope and teardown. host names a host profile the trusted host supplies by reusing the production assembly its command line already produces; a host that offers no such profile refuses before the child's root is imported, and so does an <Execution> written anywhere but inside a canonical <Test>.

host="run" takes exactly one of target or source. A target is an ordinary document reference — a path, optionally followed by # and one target selector — resolved by the same loader xmd run uses, so a kebab-named document in an arbitrary directory is addressable without being a component. A source is markdown supplied directly and follows the production run -e path: it reports the <eval> identity and writes no file. props are the child root's props.

A child runs in the working directory the <Execution> was written in, not the one the process was launched from. A <Dir> or <TempDir> around an <Execution> therefore scopes the child as it scopes everything else in it: a relative target resolves there, and a host profile that discovers an ambient repository discovers it from there. The child's scope does not descend from the document's, so this is the trusted host installing what the invocation was standing in rather than the child inheriting it.

A host="run" child gets the production run assembly, which includes the Plugins the invocation selected. It runs in an isolated scope and inherits no middleware, no registration and no installation-scoped resource, so the host installs the same Plugin values again inside that scope, with command run — never the parent's live middleware and never a second reading of the command line. What it reinstalls is the run profile: the bundled Git Plugin, which the test root itself does not carry, plus whatever the invocation selected. A child of an invocation that named no --plugin therefore receives Git's thirteen names and nothing else, and one that selected a Plugin also receives whatever that Plugin contributes for command run.

host="workflow", and the <WorkflowRun> scope it requires, are specified in issue #454 and are not built: a host that provides no workflow profile refuses them, naming that.

Outcome and failure

With as, the binding receives one outcome, and it is readable from the assertion body:

type ExecutionOutcome =
  | { kind: "settled"; result: Result<Json> }
  | { kind: "suspended"; runId: string; suspensionId: string };

A settled Ok or Err is test data: the assertion body runs and can assert about either. Without as there is nothing to assert about, so a settled Err fails the owning test rather than passing vacuously. A host refusal before a child exists raises directly and binds nothing.

Declarations

<DiagnosticJournal>, <CollectOutput> and <CollectJournal> are declarations, valid only as direct children of <Execution> and only before assertion content. They are installed for the whole child lifetime, before the child's root is imported; malformed or conflicting declarations fail before it too.

The harness reads its children in two passes. First it expands the declaration prefix in a scan that stops where assertion content begins. After the child execution is over, the ordinary pass runs: declarations answer with what they observed, and the assertions run. An ordinary repository component that shadows a declaration ends the scan where it is written.

<CollectOutput as="…"> accumulates the child's rendered output for assertions. It is passive: the child's output is displayed progressively either way, and collection changes neither routing nor completion. When the child fails or is cancelled, the binding holds the partial prefix. Ordinary <Let> stays lexical — it captures the rendered content it wraps and never the child's stream.

<DiagnosticJournal> selects an isolated diagnostic journal for authored collection from a run child, equivalent to the process-output retention selected by xmd run --journal. Without it the child selects no authored diagnostic journal and retains no process channels because output was displayed. <CollectJournal as="…"> binds a read-only snapshot of the authored journal the host selected; it grants no retention and is malformed without that selection. A trusted test host may separately save already-admitted execution events as host-private failure artifacts after joined cleanup. Those files confer no authored collection or continuation rights and do not change retainProcessOutput.

A component declaration is recognized by the definition it resolves to, never by its name. A repository CollectOutput.md is chosen ahead of this package's, so it is an ordinary component: it renders where it is written and configures nothing. A structural declaration is recognized only where core's grammar owns it; <Answers> below is reserved syntax, so a repository file never supplies that declaration.

Deterministic Agent and elicitation configuration

A run child can receive deterministic Agent behavior and elicitation answers without changing the document it executes:

<Test name="revises one generated document">
  <Execution host="run" target="./prompt.md" as="run">
    <TestAgent>
      <TestAgent.Scenario
        session="review"
        src="./agents/review.md"
        models={{selected:"fixture-review",options:[
          {id:"fixture-review",efforts:{selected:"deep",options:["brief","deep"]}}
        ]}}
      />
    </TestAgent>

    <Answers>
      <Answer
        template="Review this program:{?source}"
        value={{ decision: "approve" }}
      />
    </Answers>

    <AssertEquals actual={run.result.ok} expected={true} />
  </Execution>
</Test>

In this exact position, <TestAgent> and <Answers> are declarations rather than wrappers around assertion content. Each is a direct child of <Execution> in its declaration prefix and may appear at most once. The TestAgent declaration accepts direct <TestAgent.Scenario> children and requires one unless it declares inspection. Scenario and inspection models share nullable catalog validation. The inspection object, catalogs, models, effort catalogs and options arrays are detached and frozen before host middleware receives the request. Only data crosses the boundary; configuration state is created inside the child. The Answers declaration contains one or more direct <Answer> matchers and no other content. Their props, scenario documents, matcher templates, declaration order and validation keep the contracts in test-agent-spec.md and executable MDX §6.16.2. Both declarations belong only to host="run"; another host profile refuses them before creating a child.

<Answers delegate={true}> is invalid as child configuration. An unmatched elicitation fails through the Answers matcher provider and never falls through to the run profile's WebForm. A malformed declaration, an unreadable behavior document, a duplicate scenario mapping or malformed answer data fails before the child root is imported. A well-formed answer that does not satisfy the child's eventual Elicit schema fails at that Elicit through ordinary final response validation.

The configuration belongs to one exact <Execution>. Its TestAgent provider, session state, scenario journals and answer matchers are created inside that child's isolated scope and are torn down before its outcome is published. A sibling execution receives fresh state even when it repeats the same declarations under the same <Test>; a named session cannot continue across the boundary. The behavior-document path resolves from the outer test document under the TestAgent package's existing containment rule.

The Agent context a Plan is written with

A configured child can also give <Plan> its Agent context, which is the one thing about it a document cannot arrange for itself.

Three surfaces, and they stay distinct. A production run supplies the production ACPX context from the Agent stack the command settled. A configured <TestAgent> child supplies the deterministic one from the controlled provider its own declaration produced. A direct xmd test root and an unconfigured child supply none: the root does not resolve <Plan> at all, because the run profile's vocabulary belongs to a run, and an unconfigured child resolves the same protected bytes every run resolves and is refused before a directory, a provider, an Agent turn or a review exists.

Availability is the only thing that differs. What a Plan then runs under is the Plan writer frame's fixed policy: the Plan system instruction, the deny-all permission mode, the empty MCP-server and native-tool sets, the prompt-failure policy and the document-capability refusals, installed in one place rather than restated by each provider, so a second implementation cannot bring a weaker one. The controlled provider is installed inside the Plan invocation rather than inherited from what the child registered around itself.

A configured child is given a Plan writer root of its own, created outside that frame and removed when the child settles — the Plan sessions underneath it included, a named one among them, which production keeps and a test may not. The Plan invocation still creates and proves its own empty session directory under that root.

Only the canonical declaration this harness recognizes grants any of it. A repository component named TestAgent ends the scan and configures nothing, so a child that reached one is refused exactly as an unconfigured child is.

A scenario reaches the Plan's own turn by its exact authored label. The trusted child host privately connects that label to the opaque conversation identity <Plan> derives for the invocation. Scenario selection uses the label; provider and scenario state remain keyed by the opaque identity and working directory. This mapping belongs only to the sealed Plan-writer path, so ordinary Agent use keeps exact-session matching and siblings share no state.

Only detached declaration data enters the host-profile request. A test cannot send an Agent provider, Elicitation middleware, context, Api handler, execution installation, controller or scope to the child. The trusted host consumes the immutable data and installs the controlled providers before root import. Host middleware may observe or refuse that request and cannot alter the declarations or supply an outcome.

With TestAgent configuration, the child receives the controlled provider, controlled native launcher and all seven ordinary Agent defaults, including <Session>. Normal run-profile component resolution still decides whether those defaults are selected: a repository Prompt or Session shadows the first-party default exactly as it does under xmd run. With no TestAgent declaration, no outer TestAgent provider crosses into the child. With no Answers declaration, the run profile retains its ordinary WebForm provider.

Scenario configuration, provider-private behavior journals and declared answers are not child journal records. Prompts and elicitations the child actually performs retain their ordinary production results and replay without contacting either controlled provider.

What may run a child

Canonical core mints one opaque harness per <Test> invocation and expires it with that invocation; each <Execution> spends a single-use authorization from it.

The harness is delivered rather than published: a trusted host attaches an installer to the execution, canonical <Test> calls it inside the invocation with that invocation's harness, and the installer registers the definitions that can run a child with the harness in their closure — shadowing a refusing default for exactly that test's body. There is nothing to read, so a repository component named Test, Execution or WorkflowRun, a package registering those names, printErrors(fn), public middleware on any surface including the <Test> behavior hook, a separately loaded package copy, and replaceable context all acquire none of it. A document whose host attached no installer recognizes <Execution> and refuses it, inside a canonical <Test> as everywhere else.

Host-profile middleware is policy. It receives a request rather than a child, so it may observe, refuse and delegate the exact immutable request — and cannot create a child, replace target, source, props, action or journal policy, substitute a completion, or publish an outcome. A handler that returns without delegating produces no child and is reported as the protocol violation it is.

The trusted provider is attached to the execution as a value before untrusted installation, middleware or document code begins. The authorized definitions close over that provider; public host middleware never receives it.

<Execution> receives an invocation-owned binding channel directly from the engine. It reports only whether this exact invocation has as, and publishes the exact child outcome once before assertions expand. The channel exposes no binding name or environment and does not travel through the public Component Api, Context, props, component metadata or stable names.

Test API

TestApi controls testing mode. The testing operation returns false by default and true beneath <Testing>:

import { testing } from "@executablemd/testing";

const active = yield* testing;

TestApi records completed tests in discovery order. Each result contains its pass or fail status, optional name, source location, and structured error details when it failed. Rendered test output is not duplicated in the result.

useTesting composes testing around the core execution entrypoint. It installs the testing components and collectors, activates testing mode for the execution, and returns a session whose results operation snapshots completed tests in discovery order:

import { execute } from "@executablemd/core";
import { useTesting } from "@executablemd/testing";

const tests = yield* useTesting();
const execution = yield* execute(options);
const outcome = yield* execution;
const results = yield* tests.results;

Execution completion is an Effection Result<string>: Ok(output) on success, Err(error) on document, infrastructure, or testing failure — completion never throws once the execution handle exists. Under useTesting, an otherwise successful execution completes as Err(TestFailureError) after the output stream closes when any test failed or no tests were discovered. A failure produced by the document itself passes through unchanged, and the session's results remain available after failure. One useTesting session applies per execution scope; its middleware is removed with that scope. xmd test composes useTesting around the same execute call the run command uses.

Registering the testing components without useTesting leaves testing mode inactive: <Test> is skipped, assertion components stay usable, and an explicit <Testing> boundary still activates its subtree and turns its failures — or an empty boundary — into an Err outcome for the execution.

TestApi middleware is policy rather than activation. Overriding testing to true over that registration activates nothing, and a <Test> reached under it refuses before its body expands, as Registration, mode, and activation describes.

Several documents share fixtures, never sessions. Fixture and provider state that spans a suite is installed in an enclosing scope, and each document runs in a child scope holding one useTesting() session and one execution. A session's results are cumulative and its completion policy is the whole session's, so one session around several documents would let a zero-test document pass on the strength of an earlier one's tests.

Assertions

Assertion components follow the conventional assertion function names and parameter names. The initial components are:

  • <Assert> and <AssertFalse>
  • <AssertEquals> and <AssertNotEquals>
  • <AssertStrictEquals> and <AssertNotStrictEquals>
  • <AssertExists>
  • <AssertStringIncludes>
  • <AssertMatch> and <AssertNotMatch>
  • <AssertGreater> and <AssertGreaterOrEqual>
  • <AssertLess> and <AssertLessOrEqual>
  • <AssertThrows>

Props map directly to the corresponding function parameters: expr for truthiness assertions, actual and expected for comparisons, and optional msg where supported. The operands are captures (executable-mdx-spec.md §6.5): the assertion evaluates them itself, so what it compares is the value the author wrote rather than a JSON projection of it — which is what lets <AssertStrictEquals> compare by reference, <AssertExists> tell undefined from absence, and <AssertMatch> take a real RegExp. msg is an ordinary validated string prop.

as and slot belong to the engine, which consumes them before validation, so an assertion accepts them like any other component: as binds the assertion's diagnostic text.

Equality assertions and <AssertStringIncludes> accept either an expected prop or rendered children as the expected string. The two forms are mutually exclusive. Expected children expand in the current scope and environment, use the same trailing-whitespace trimming as <Let>, and do not render separately.

Numeric comparisons require an expected prop. Match assertions require a RegExp through the expected prop. Unary assertions do not accept expected children.

Comparisons are those of node:assert/strict, a runtime builtin rather than a dependency, so a published package needs no registry configuration to install. <AssertEquals> and <AssertNotEquals> are deep strict equality; <AssertStrictEquals> and <AssertNotStrictEquals> are Object.is. Existence, falsity, string inclusion and the ordering comparisons have no builtin equivalent and are defined here: existence rejects only null and undefined, string inclusion is String.prototype.includes, and the ordering comparisons apply the relational operators — which coerce object operands, so a value whose toString throws fails the comparison itself.

Deep equality is stricter than the @std/assert implementation used before 0.5.2 in two respects: 0 and -0 are no longer equal, and an object with a null prototype is no longer equal to an otherwise-identical plain object. Two distinct Error values with the same name and message now compare equal, where previously their stacks made them differ. Pathological operands — an invalid Date, a WeakMap — are not specified and may differ between runtimes.

Ordinary REPL entries carry the existing assertion defaults without installing testing behavior or a TestAgent. Their discovery and execution use the same assertion forms and validation as ordinary xmd run; REPL.Test remains the explicit Markdown-test boundary.

Assertion components work inside and outside tests. A failed assertion throws an AssertionError. Outside a test, that error aborts document expansion. Inside a test, <Test> contains and records it.

Assertions emit Markdown diagnostics in testing mode. During regular execution, diagnostics are hidden unless --verbose is enabled. Failed assertions still throw when diagnostics are hidden.

Diagnostics identify the assertion component and outcome. They include the optional message and relevant actual and expected values. Failure diagnostics include the underlying assertion detail when available. Their exact Markdown layout is not prescribed, and formatting arbitrary values must not change the assertion outcome or introduce a new failure.

<AssertThrows> is the one assertion that is not a value comparison. It wraps children rather than taking actual/expected, and it requires a message prop: a literal string is matched as a substring of the raised error's message, or an expression evaluating to a RegExp is tested against it. The assertion passes when expanding the body raises an error whose message matches, and fails when the body raises nothing or raises a non-matching error. It accepts an optional literal as binding that captures the complete caught error segment — including its cause — into the environment for later assertions. Unlike the other assertions it emits no pass diagnostic: the return channel carries the caught segment for as to bind, and no other channel preserves a durable rendered segment. A failure aborts the document when it occurs outside a test (inside a <Test> it is contained and recorded), unchanged.

Additional assertion components use the same rules: they state their comparison here, they raise an AssertionError on failure, and they use the shared diagnostic behavior. A component must decide its outcome before anything formats its operands, so that a value with a throwing or mutating toJSON, toString or getter cannot change whether an assertion passes.

Mocking

Testing has no separate mocking DSL. Tests install mocks through existing context API middleware or helpers from @executablemd/runtime/test. Middleware installed within a test applies to subsequent expansion in that test and is removed with its scope.

Unsupported Syntax

BDD syntax such as describe, it, beforeEach, and beforeAll is not supported. Gherkin syntax such as Given, When, and Then is not supported.

Repository Runtime Suites

The sections above specify the Executable Markdown testing language. This one specifies something different: how this repository's own test corpus — its TypeScript test files and its Markdown suites — is discovered, divided, and run under each supported runtime. It changes nothing about <Test>, <Testing>, assertions, or mocking.

One corpus

listTestFiles(root) is the only discovery source. It returns sorted repository-relative POSIX paths for every *.test.ts beneath tests/ in each root workspace member plus scripts/tests/, and the Markdown suites, and nothing else. Walking those concrete directories rather than the repository root is what keeps nested worktrees, the site, and the deliberately malformed fixtures out without naming any of them.

Its TypeScript paths are the same set Deno's own test-file pattern finds in the repository. A test file discovery cannot see would run under no runtime at all, so the equivalence is a regression rather than a convention.

A Markdown suite is a regular *.test.md file directly inside examples/adversarial-implementation/tests/, or a regular file at one of these registered paths:

  • packages/cli/tests/document-suites/evaluate/Evaluate.test.md
  • packages/cli/tests/document-suites/file-copy/FileCopy.test.md
  • packages/cli/tests/document-suites/inline/Inline.test.md
  • packages/cli/tests/document-suites/plan/Plan.test.md
  • packages/cli/tests/document-suites/props/Props.test.md
  • packages/cli/tests/document-suites/syntax/Syntax.test.md
  • packages/cli/tests/document-suites/targets/Targets.test.md
  • packages/cli/tests/document-suites/verbose/Verbose.test.md
  • packages/test-agent/src/NativeSessionLaunch.test.md
  • packages/test-agent/src/SessionConfiguration.test.md

Discovery skips symlinks and directories. Adversarial discovery does not descend. The individually registered paths admit only the listed documents, so helpers and intentionally failing fixtures stay outside the corpus.

Applicable corpus

applicableTestFiles(runtime, root) is discovery minus the entries scripts/runtime-test-exclusions.ts records for that runtime, in the same sorted order. Every supported runtime is keyed there, including the one that excludes nothing: an absent key reads as an unmeasured runtime, which is the one thing weights and partitions must never see.

A file leaves a runtime only through that manifest, and each entry carries a reason and an issue. The manifest is validated structurally; it cannot show that an excluded test has become portable, so removing a stale entry stays a manual act.

Versioned weights

test-weights.json records how long each applicable file takes under each runtime, in milliseconds. It is version 1 and carries provenance — the commit, the run URL, the attempt, the runner label, and all three runtime versions — because a weight measured on another machine, image, or commit describes a corpus that no longer exists.

The reader checks the version, provenance, runtime names, test paths, and positive integer weights. A test path is either a repository-relative TypeScript test path under tests/ or a Markdown suite path matching the discovery rules.

deno task weights:measure is the only command that writes it, in two steps.

weights:measure <runtime> <directory> measures one runtime. It runs each applicable file alone, in sorted order, through that runtime's exact one-file command, and times the child with a monotonic clock. It keeps <directory>/test-weights-<runtime>.json current: the record carries the runtime, a status, the full provenance, the applicable corpus it derived, every file that passed with its milliseconds in the order it ran, and the failure if there was one. Its status is partial before the first file and after each one that passes, failed with the file and exit status when a file fails — which stops the measurement — and complete once every file passed. A run that is cut off leaves partial, so an unfinished measurement is never mistaken for a finished one. The record has one current shape, and a reader refuses a missing, extra or malformed field.

weights:measure assemble <directory> writes test-weights.json from the three records, and only when every one is complete; agrees with the commit, run URL, attempt and runner the environment names; records the same three runtime versions as the others; derived the same applicable corpus as the checkout; and measured each applicable file exactly once, and no other file. Otherwise it refuses with every problem named and writes nothing — a file assembled from a runtime that failed halfway, or from two runs, would be indistinguishable from one measured on a healthy corpus.

Both writes go through a staged copy of the invocation's own, so each is atomic and a cancelled run removes only its own staging. Ordinary verification never rewrites either.

Fallback

A file with a recorded weight uses it. A file without one is charged the maximum recorded weight among the applicable files that do have one: conservative, non-zero, and named in the shard's log. Weights recorded for files that have since left the corpus are ignored entirely, so a deleted outlier cannot inflate that fallback forever.

When no applicable file has a recorded weight there is no conservative number to pick — a zero would pack the corpus onto one shard and a constant would be a timing assumption in disguise — so the partition refuses.

Partition

Each runtime's applicable corpus is partitioned independently, by longest-processing-time-first assignment:

  1. order the files by descending effective weight, breaking equal weights by ascending repository path;
  2. create the requested number of indexed empty shards;
  3. assign each file in turn to the shard with the lowest total weight, breaking equal totals by the lowest shard index;
  4. sort each shard's assigned paths; and
  5. retain shard indices rather than re-sorting the shards.

This minimizes the longest shard rather than equalizing file counts, because the wall clock a run waits on is the slowest shard. A count below one or above the applicable file count is refused, as is an index outside 1..count.

The result is a pure function of the applicable paths, the weights, the count, and the index: repeated calls return byte-identical lists, which is what lets a matrix job be told only its index and still be certain it runs exactly the files no sibling runs. The shards are pairwise disjoint and their union is exactly the applicable corpus — every applicable file runs exactly once per runtime.

Execution

runtime-tests.ts <deno|node|bun> [<index>/<count>] runs a runtime's tests. With no selection it runs the applicable TypeScript files in one native invocation, then each applicable Markdown suite through its one-file command, and exits with the first failure of either portion. With a selection it resolves that shard and invokes its files serially, in sorted order, each through that runtime's one-file command:

  • Deno — deno test --allow-all --frozen --reporter=dot <file>
  • Node — tsx --tsconfig tsconfig.node.json --test --test-concurrency=1 <file>
  • Bun — bun test --timeout=300000 <file>

A Markdown suite runs as xmd test <file> through the same runtime's CLI source entrypoint, resolved from the runner's checkout:

  • Deno — deno run --quiet --allow-all --node-modules-dir=manual --cached-only --frozen packages/cli/src/deno.ts
  • Node — tsx --tsconfig tsconfig.node.json packages/cli/src/node.ts
  • Bun — bun packages/cli/src/bun.ts

The CLI runs from the repository root. A shard names each Markdown suite when it starts and passes test <file> --reporter=failures. Weight measurement passes the same arguments. Local unsharded runs use the default reporter. Deno weight measurement also uses the same dot reporter as Deno shards. The CLI inherits the environment (PATH, the Deno cache and install variables, XDG_CACHE_HOME, TMPDIR and XMD_TEST_ARTIFACTS). Individually registered suites use the CLI's default includes and file permissions. Deno launches of Plan, NativeSessionLaunch and SessionConfiguration select the test-only worker plugin. It carries the prepared Deno flags into TestAgent workers. Syntax and Verbose receive a temporary executable that invokes the same runtime's source CLI with its arguments intact. Each suite owns and removes its executable. Verbose also receives the absolute path to its fixture components.

The adversarial suites append the following arguments to the source entrypoint: test <file> --plugin packages/cli/tests/fixtures/workflow-test-permissions.mjs --include <root>/examples/adversarial-implementation/tests/components --include <root>/examples/adversarial-implementation/components, with the include paths absolute so that a nested <Dir> cannot change where components are found. Preparation, ProductionRole and Submission include tests/production-components in place of tests/components. The child's environment also receives XMD_EXAMPLE_COMMAND and XMD_EXAMPLE_TEST_COMMAND. Each is the prepared Deno CLI above followed by run --allow-absolute or test, respectively, as quoted shell words. Fixture documents run their real workflow through those commands under every outer runtime, because the run profile they exercise is the Deno host's. The artifact-return suites also receive XMD_ARTIFACT_RETURN_EVIDENCE when it is set. The fixture plugin adds --node-modules-dir=manual --cached-only --frozen when the Deno source host relaunches itself as a TestAgent worker, including workers for isolated run-profile children.

Each adversarial suite but Implementation, whose named groups carry their own deadlines, runs under a whole-launch deadline. Preparation has 420 seconds, Friction has 180, DecisionResolution and Delivery have 300, and the other adversarial suites have 120. A suite that reaches it is stopped and its cleanup joined, its path and deadline are reported, and it fails with status 1. The files after it still run. SessionConfiguration has a 120-second whole-launch deadline under each runtime. Other individually registered suites have a 300-second whole-launch deadline under Bun and use their authored test deadlines under Deno and Node.

The same operation reads the prepared node_modules/effection link before and after a Markdown suite. A link that changed or disappeared fails the file with a message naming both targets; a file that already failed keeps its own status.

A malformed runtime, index, or count is refused before any test process starts.

A numeric failure does not prevent the files assigned after it: the shard continues, retains the first failed file and its status, and exits with that status once every assigned file has had its chance. A status without a numeric code — a signalled child — is failure. The shard streams the output its selected reporter emits. Deno's dot reporter keeps failure details and omits test console logs.

Cancellation is distinct from failure. Halting a shard halts the process group of the file it is running, releases that file's scratch state and starts no later file.

deno task test is the local Deno suite. With no arguments it runs Deno's own whole-repository deno test --allow-all --frozen, including the files the CI corpus leaves to smoke jobs, then each applicable Markdown suite through its one-file command. Otherwise its arguments reach deno test verbatim unless a positional path selects an applicable Markdown suite. Native Markdown doctests outside that corpus, focused paths, --changed and --related keep their Deno meaning. Select an applicable Markdown suite by its repository-relative path, optionally prefixed with ./. That selection may include native test paths and must contain only paths. Each portion runs only when it is nonempty, and the first failure is the status. pnpm test:node and bun run test:bun are the no-selection runtime-tests.ts run of their runtime.

CI topology

test-deno, test-node, and test-bun are fail-fast: false matrices under those same job IDs. Each shows its selection in its name, passes that selection only to its Test step, and keeps that runtime's own preparation unchanged and repeated per shard. green requires all three job IDs; GitHub collapses a matrix into one result behind its ID, so a shard that fails, is cancelled, times out, or unexpectedly skips makes that dependency non-success.

Concurrency inside a shard is not enabled.

The required repl-bench job checks the bench tools on Ubuntu for pull requests. Pushes to main run the tools, the driver exit and teardown tests, and the Markdown terminal journeys on macOS and Ubuntu. Those terminal tests exercise the host's pseudo-terminal and retain screen and terminal evidence on failure. green requires the job on both events.

Remeasurement and calibration

Remeasure after any change to the corpus, the exclusions, or a runner command. A hosted workflow_dispatch produces the measurement on the runner CI uses. Each runtime is measured in its own job, serially within it, and uploads its record as test-weights-<runtime>-attempt-<attempt> however the job ended. An assemble job, which runs only when all three succeeded, assembles that attempt's records and uploads test-weights.json as the test-weights artifact. Each measuring job has 165 minutes, its measuring step 150 of them, and assembly 10, so the run's execution fits in 175 minutes; time queued for a runner is outside it. Committing that artifact is a deliberate act, which is what keeps the provenance true of the run that produced it. Milliseconds are never edited by hand.

Each measuring job sets XMD_TEST_ARTIFACTS to the absolute workspace path ${{ github.workspace }}/.xmd/test-artifacts. On failure or cancellation it uploads the complete saved journal trees, including hidden files, as measure-test-journals-<runtime>-attempt-<attempt>; missing files are ignored. These artifacts follow Progress and failure journals and remain separate from the measurement records selected for assembly.

Counts are measured rather than chosen. For each runtime the lower bound is floor(sum of applicable weights / 420000) + 1, and the installed count is the smallest for which five consecutive runs on one fixed head keep every shard Test step, and that runtime's whole execution window, under 420 seconds. A miss increments only that runtime and begins a fresh sequence of five. Narrowing the corpus, dropping a runtime, or enabling in-shard concurrency is not a response to a miss. A single file whose own weight reaches 420 seconds cannot be solved by any count, and is reported rather than absorbed.