Executable documents can contain probabilistic behavior. Tests ground that behavior in observable results and give authors confidence in their documents.
Tests run only in testing mode. During regular execution, <Test> and its
entire body are skipped: it renders nothing, binds nothing, and runs no side
effect.
<Testing> enables testing mode for its expanded subtree:
<Testing>
<Example />
</Testing>Installing the package registers <Testing>, <AssertThrows> and the value
assertions as ordinary non-reserved defaults (executable-mdx-spec.md §5.3):
a repository component of any of those names is chosen ahead of the registered
one, exactly as it would be ahead of any other package's default, and then
decides for itself what a testing boundary means.
<Test> is not among them. Canonical core registers and owns the default
<Test> construct, because core owns what an invocation of one means for the
run (see Checked command failures inside a test below). Installing this
package supplies that construct's testing behavior — activation, isolated
bindings and scope, the timeout, assertion and unexpected-failure
classification, TestResult staging and recording, reporting, and the testing
completion failure — through the TestBehavior operation core calls from
inside the invocation. <Test> is still an ordinary non-reserved default, so a
repository component of that name is chosen ahead of core's and decides for
itself what a test means.
A test result is journaled once the invocation that produced it has been dismantled, so a teardown failure is part of the outcome recorded for it. That is still while expansion runs and before the root close, in discovery order and identified by source position. A run that ends between a test finishing and its result being written journals nothing for that test, and re-runs it on resume.
A boundary reports how many tests ran and how many failed once its body has finished. A body that stopped — expansion ended by throwing rather than by collecting a diagnostic — never counted the tests it did not reach, so it reports no outcome and journals none, and the failure travels on unchanged. An error the body merely collected leaves it complete: that diagnostic is part of what the boundary rendered, and the tests beside it still ran.
The CLI command is equivalent to wrapping the entrypoint in <Testing> for
activating testing mode and for journaling the results of the tests it runs. It
is not equivalent for boundary reporting: an explicit <Testing> reports and
journals a boundary outcome, and a root run reports none.
xmd test <entrypoint>Test discovery follows normal component expansion. A test runs only when the expanded component tree reaches it, including through imports, components, and conditional rendering. Imported but unrendered tests do not run.
Ordinary content outside a <Test> expands normally. An error in that content
aborts the run as an infrastructure error.
<Testing> emits the naturally expanded report without adding a summary. It
fails after expansion when any test failed or when no tests were discovered.
xmd test exits with status 0 when every test passes and status 1
otherwise.
Testing uses the standard journal and replay behavior. Each completed test
and each <Testing> boundary outcome is journaled as a testing-owned
durable operation (test_result, testing_boundary) while expansion runs,
before the root close event, identified deterministically by source
position. On a full replay of a completed journal the recorded results and
boundary outcomes are restored from the stream into the current collectors
— test bodies and their effects are not rerun — so the testing outcome and
recorded results are preserved. On live and partial runs nothing is
restored: re-expansion records each result exactly once, in discovery
order, with completed records replaying in place.
The default --reporter=pretty prints the rendered report on stdout and
progress on stderr. Use --reporter=failures for a compact report in CI or a
Sidekick Round.
xmd test ./tests --reporter=failuresThe failures reporter omits the rendered report, directory headings, test starts, and successful outcomes. It writes failure and cancellation diagnostics and journal paths to stderr. A completed invocation ends with one summary on stderr.
[xmd test] 11 passed | 1 failed (324ms) · 1 of 3 documents failed
The counts cover tests recorded by the root documents. Nested executions can add diagnostics and journals. Exit status and journal retention follow the same rules for either reporter. Output that commands write directly to the terminal keeps its normal routing.
The pretty reporter writes live progress to stderr. A pipe or CI log shows which test is running and which operation a timeout interrupted.
[xmd test] start test "held" at suite.test.md:1:1
[xmd test] fail test "held" at suite.test.md:1:1 (timeout; interrupted sh exec at suite.test.md:3:1)
[xmd test] journal saved: /work/.xmd/test-artifacts/2026-…-6f1c…/root-1.jsonl (root 1 suite.test.md)
A test's start is reported when its behavior begins and its outcome once its
result is recorded, which follows the test's teardown — a component that
expanded successfully is not a passing test. An <Execution> invocation reports
its start only after resolution and validation accept it, then reports how that
invocation settles. These milestones do not claim that a child launched or
passed. An invocation stopped before activation reports no milestone. Ordinary
components and command blocks produce no start or terminal lines, including on
failure or cancellation; their observations still identify the operation a test
timeout interrupted. The CLI's terminal failure summary is unchanged.
When command cancellation unwinds a test before a result is recorded, progress
reports cancelled test with its name, location and interrupted operation when
one exists. This notice is not a pass/fail verdict and creates no test result.
It follows the delegated test work's unwind.
Work inside a nested <Execution> child is labelled with that child, for example
child 1 ./child.md:. Absolute source paths inside the command's captured working
directory are displayed relative to it; authored relative paths and absolute
paths outside that directory retain their spelling. Canonical result and
journal locations keep their original identities. The report names
tests, elements, command languages and modifier words, locations and states. It
never carries a captured value, a prop, command or prompt text, an assertion's
operands or anything a document rendered, and the rendered report on stdout is
unchanged by it.
Each root document and each nested run child keeps its journal in memory.
When that execution fails, times out or is cancelled, the command saves the
recorded events as JSONL after the execution's assertions and teardown have
finished, and reports the file's absolute path. A child's journal is saved when
the child fails, even when the test binds that failure with as. A passing
execution writes no file.
Files go to a fresh directory for each invocation, beneath XMD_TEST_ARTIFACTS
when it is set and .xmd/test-artifacts in the working directory otherwise.
The directory is created only when a journal is saved. An explicit --journal
names the root's journal file instead, and no automatic root copy is made. A
saved file selects no authored journal, grants nothing to <CollectJournal>,
and is not continuation input. A process killed without running its cleanup
saves nothing.
Three things are distinct, and only the last one makes a test run.
Registration is installTestingComponents(). It registers the components
above and supplies what core's <Test> does. It activates nothing: after it,
testing is still false, <Test> is still skipped, and the assertion
components are usable in ordinary content.
Mode is the public TestApi.testing operation. It answers whether testing
mode is on, and it is replaceable policy: middleware may narrow it to false,
observe it, refuse, and compose behavior around recording. An answer of true
says a test should run. It cannot say that anything exists to collect the
result, flush the last one, or decide the run's outcome.
Complete activation is what does own those. There are two, and no third:
useTesting(), for one root document execution; and- a
<Testing>element, for its lexical subtree.
Each owns a result collector, the final flush of the last staged result, and
the completion policy that settles the run — and each is package-owned, private
to @executablemd/testing, and expires with the scope or boundary that
established it. Registration plus a testing override is not a substitute:
neither the boolean nor a contextual name is accepted as activation,
so a separately loaded copy composing policy around the public surface
composes policy and nothing else.
Canonical core enforces the requirement, at the invocation boundary. Installing
the testing package adds an activation guard, and core's own <Test> consults
it before it mints a test harness and before it dispatches the public
TestBehavior chain that supplies what a test does. The guard reads the public
mode: inactive leaves <Test> invisible exactly as before, and active requires
a complete activation before any harness or body work.
The behavior surface keeps its stable name, its argument, its result, its loaded-copy composition and its ordinary delegating behavior, and a handler composed around it may observe, wrap, delegate or refuse what a test does. What it may not do is answer for the decision that preceded it. Answering without delegating is a legitimate use of that surface, and it reaches a decision core has already taken.
Any refusal raised while that decision is being taken is a configuration failure of the composition, and never a test outcome — whether the activation could not be proved, a handler on the seam refused with an error of its own, or the chain was answered rather than delegated. Refusing by throwing is a supported thing for a handler there to do, and what it throws is its own business, so what marks a refusal is where it was raised rather than what type it is.
A test whose activation was refused never ran. Absorbing that as a contained test failure would leave a composition with no session — which has no completion policy to convert a contained failure into a document failure — reporting a run that ran no test as a success. Canonical expansion therefore withholds a refused decision from every component failure handler, including one a package or a document installs, rather than relying on a handler to decline it.
So under any composition, a refused activation:
- records, journals and reports no test result;
- renders no assertion diagnostic;
- causes no side effect the body would have caused; and
- fails the document execution, so a caller holding an execution handle observes
Err.
Inactive tests are unaffected. Testing mode off still means <Test> renders
nothing, binds nothing, and runs nothing, with no result of any kind.
xmd test accepts a directory as well as a document, and defaults to the
current directory. These are the same command:
xmd test
xmd test .A directory target discovers the documents beneath it recursively. The default
pattern is **/*.test.md, so ordinary Markdown such as README.md or
notes.md does not run. --pattern chooses a different glob:
xmd test workflows --pattern "**/*.spec.md"
xmd test workflows --pattern "**/*.spec.md" --pattern "**/*.check.md"Explicit patterns replace the default rather than adding to it, and repeating
the option forms an inclusive union. Patterns are relative to the target root.
A document that several patterns match runs once, and the documents run in the
code-point order of their normalized relative paths, whatever order the
patterns were written in. A pattern chosen deliberately to be broad, such as
--pattern "**/*.md", selects and runs ordinary Markdown.
Under a directory target each document searches for components in three places,
in order: its own directory, the target root, then the configured component
directories. A repeated directory keeps its first position. A component
therefore resolves beside the test that invokes it however deep in the suite it
sits, and two directories may hold same-named components without either
reaching the other. No --include is needed, and the working directory
does not change.
A single-document target keeps the configured component directories exactly.
Each document runs in its own execution and scope: bindings, component registrations, and testing sessions belong to one document and are torn down before the next one starts. A document that declares no tests fails on its own rather than inheriting an earlier document's results.
The pretty reporter precedes each document's report with a heading naming its
relative path. Both reporters identify a failed document by that path. A failing document does not stop the
run; the remaining documents still execute, and a closing line names how many
of them failed. The run exits 0 only when every discovered document passes,
and 1 when any document fails or when no document matches.
--pattern applies to a directory, so supplying it with a single document is
rejected. A --pattern with no value, or one followed by another option, is
rejected rather than read as a glob; --pattern=<glob> expresses a glob that
begins with -. Options after -- are not inspected. --journal writes one
trace for one document, so a directory target rejects it before any document
runs.
Declare the plugins a package's Markdown tests use in its xmd.json:
{
"test": {
"plugins": ["./tests/support/test-plugin.mjs"]
}
}xmd test starts at the test document's directory and searches upward for
the nearest xmd.json. It checks the directory containing the nearest
package.json and stops there. Without a package manifest the search ends
at the filesystem root. An explicit symlink to a test uses the resolved
document's directory. A nearer config replaces the whole selection, including
when it declares no plugins. The invocation directory and directory target
do not affect which config is selected.
The CLI reads and validates the selected config before loading any Plugin for
that document. Paths beginning with ./ or ../, and bare installed package
specifiers, resolve from the config directory. Absolute local paths and
file: URLs use the same rules as --plugin. Explicit flags append in argv
order and resolve from the invocation directory. Duplicate Plugin names
refuse the document before installation. A malformed config or a configured
module load failure names the config path. Plugin modules are trusted code
that execute with the host's permissions.
Each test document owns a fresh Plugin installation scope, including a
directory run spanning several packages. Install-time resources,
registrations and middleware end before the next document starts. Module
imports follow the host's ordinary module cache. A selection or installation
failure is reported for that document, and directory execution continues.
This lookup belongs to xmd test. Other commands and REPL.Test use their
own Plugin selections.
REPL.Test uses the same directory discovery convention and real per-document
useTesting sessions. Both REPL target forms retain the finalized configured-first
REPL profile; CLI directory promotion above remains specific to xmd test.
The REPL testing contract owns its
structured report, admission, feedback, cancellation and retained-history rules.
Atomic tests use <Test>:
<Test name="Renders hello world">
<Let as="result">
Hello World
</Let>
<AssertEquals actual={result}>
Hello World
</AssertEquals>
</Test>The name prop is optional metadata. An unnamed test is identified by its
source location; headings in the body remain ordinary output and are not
inferred as names.
The timeout prop is optional and declares this test's own timeout as a
duration (500ms, 30s, 5min), replacing the 20-second default. A malformed
duration is refused where it was written and fails only that test.
A foreground command that exits nonzero is a checked failure, and outside a test
it fails the run (executable-mdx-spec §3.6). Inside a <Test> it is
contained: it becomes that test's failed result, the run's own fatal record
stays clear, the tests after it still run, and the testing session's existing
completion policy is what fails the overall run. A contained outcome is decided
once — replay restores the failed result and the failing testing outcome from
the journal without running the command again.
Containment belongs to the <Test> construct, and canonical core owns it.
Core registers <Test> as one of its own defaults, and grants containment only
to that exact definition, only while it is expanding it. Nothing else confers
it: not a name, an origin, a context, a marker, an option, the request, or an
installation. TestBehavior supplies what a test does and decides nothing
about which element is one — it accepts no function, component name, marker or
container, and cannot make any other component contain a failure.
A repository component named Test, or another package that registers the name,
still wins ordinary default resolution and is chosen ahead of core's. Core's
definition was therefore not expanded, and the one that was receives no
containment: a checked failure inside it is the run's, exactly as it would be
anywhere else the document did not authorize one.
useTesting() with execute() needs nothing else. The host attaches no
installation, names no component, and receives no identity.
A test body behaves like any regular component body. Tests run sequentially in expansion order. Each test runs in a child Effection scope and an isolated binding environment. It inherits ambient context and bindings, but its context changes, bindings, and ongoing effects do not escape. Its scope is fully torn down before the next test starts.
Each test has a 20-second timeout by default; a timeout=<duration> prop on the
<Test> element declares that one test's own. A timed-out test is halted,
reported as failed, and fully torn down before execution continues.
An assertion failure, unexpected error, or teardown error fails only the current test. Later tests still run. Unexpected errors remain distinct from assertion failures. Output produced before an assertion failure or an unexpected error remains in the report and is followed by the failure diagnostic. Two failures report the diagnostic alone: a test that timed out was halted mid-body, and a test whose teardown failed had already returned, so neither has body output to keep. A teardown failure is reported as a diagnostic rather than as report text, and carries the original error as its cause.
Nested <Test> elements are invalid. Skip, focus, and retry behavior is not
supported.
A test can run another document as a real root execution, without translating it
into TypeScript and without launching another xmd process:
<Test name="bootstraps a package">
<Execution host="run" target="./scripts/bootstrap.md" props={{ name: "x" }} as="run">
<CollectOutput as="output" />
<AssertEquals actual={run.kind} expected="settled" />
<AssertEquals actual={run.result.ok} expected={true} />
<AssertStringIncludes actual={output} expected="created x" />
</Execution>
</Test>This is not component composition. <Execution> creates a child document
execution with its own root import, target selection, journal, output stream,
scope and teardown. host names a host profile the trusted host supplies by
reusing the production assembly its command line already produces; a host that
offers no such profile refuses before the child's root is imported, and so does
an <Execution> written anywhere but inside a canonical <Test>.
host="run" takes exactly one of target or source. A target is an ordinary
document reference — a path, optionally followed by # and one target selector —
resolved by the same loader xmd run uses, so a kebab-named document in an
arbitrary directory is addressable without being a component. A source is
markdown supplied directly and follows the production run -e path: it reports
the <eval> identity and writes no file. props are the child root's props.
A child runs in the working directory the <Execution> was written in, not the
one the process was launched from. A <Dir> or <TempDir> around an
<Execution> therefore scopes the child as it scopes everything else in it: a
relative target resolves there, and a host profile that discovers an ambient
repository discovers it from there. The child's scope does not descend from the
document's, so this is the trusted host installing what the invocation was
standing in rather than the child inheriting it.
A host="run" child gets the production run assembly, which includes the
Plugins the invocation selected. It runs in an isolated scope and inherits no
middleware, no registration and no installation-scoped resource, so the host
installs the same Plugin values again inside that scope, with command run
— never the parent's live middleware and never a second reading of the command
line. What it reinstalls is the run profile: the bundled Git Plugin, which the test
root itself does not carry, plus whatever the invocation selected. A child of an
invocation that named no --plugin therefore receives Git's thirteen names and
nothing else, and one that selected a Plugin also receives whatever that Plugin
contributes for command run.
host="workflow", and the <WorkflowRun> scope it requires, are specified in
issue #454 and are not built: a host that provides no workflow profile refuses
them, naming that.
With as, the binding receives one outcome, and it is readable from the
assertion body:
type ExecutionOutcome =
| { kind: "settled"; result: Result<Json> }
| { kind: "suspended"; runId: string; suspensionId: string };A settled Ok or Err is test data: the assertion body runs and can assert
about either. Without as there is nothing to assert about, so a settled Err
fails the owning test rather than passing vacuously. A host refusal before a
child exists raises directly and binds nothing.
<DiagnosticJournal>, <CollectOutput> and <CollectJournal> are declarations,
valid only as direct children of <Execution> and only before assertion content.
They are installed for the whole child lifetime, before the child's root is
imported; malformed or conflicting declarations fail before it too.
The harness reads its children in two passes. First it expands the declaration prefix in a scan that stops where assertion content begins. After the child execution is over, the ordinary pass runs: declarations answer with what they observed, and the assertions run. An ordinary repository component that shadows a declaration ends the scan where it is written.
<CollectOutput as="…"> accumulates the child's rendered output for assertions.
It is passive: the child's output is displayed progressively either way, and
collection changes neither routing nor completion. When the child fails or is
cancelled, the binding holds the partial prefix. Ordinary <Let> stays
lexical — it captures the rendered content it wraps and never the child's stream.
<DiagnosticJournal> selects an isolated diagnostic journal for authored
collection from a run child, equivalent to the process-output retention
selected by xmd run --journal. Without it the child selects no authored
diagnostic journal and retains no process channels because output was
displayed. <CollectJournal as="…"> binds a read-only snapshot of the authored
journal the host selected; it grants no retention and is malformed without that
selection. A trusted test host may separately save already-admitted execution
events as host-private failure artifacts after joined cleanup. Those files
confer no authored collection or continuation rights and do not change
retainProcessOutput.
A component declaration is recognized by the definition it resolves to, never
by its name. A repository CollectOutput.md is chosen ahead of this package's,
so it is an ordinary component: it renders where it is written and configures
nothing. A structural declaration is recognized only where core's grammar owns
it; <Answers> below is reserved syntax, so a repository file never supplies
that declaration.
A run child can receive deterministic Agent behavior and elicitation answers
without changing the document it executes:
<Test name="revises one generated document">
<Execution host="run" target="./prompt.md" as="run">
<TestAgent>
<TestAgent.Scenario
session="review"
src="./agents/review.md"
models={{selected:"fixture-review",options:[
{id:"fixture-review",efforts:{selected:"deep",options:["brief","deep"]}}
]}}
/>
</TestAgent>
<Answers>
<Answer
template="Review this program:{?source}"
value={{ decision: "approve" }}
/>
</Answers>
<AssertEquals actual={run.result.ok} expected={true} />
</Execution>
</Test>In this exact position, <TestAgent> and <Answers> are declarations rather
than wrappers around assertion content. Each is a direct child of <Execution>
in its declaration prefix and may appear at most once. The TestAgent declaration
accepts direct <TestAgent.Scenario> children and requires one unless it
declares inspection.
Scenario and inspection models share nullable catalog validation. The
inspection object, catalogs, models, effort catalogs and options arrays are
detached and frozen before host middleware receives the request. Only data crosses the
boundary; configuration state is created inside the child.
The Answers declaration contains one or more direct <Answer> matchers and no
other content. Their props, scenario documents, matcher templates, declaration
order and validation keep the contracts in test-agent-spec.md and executable
MDX §6.16.2. Both declarations belong only to host="run"; another host profile
refuses them before creating a child.
<Answers delegate={true}> is invalid as child configuration. An unmatched
elicitation fails through the Answers matcher provider and never falls through
to the run profile's WebForm. A malformed declaration, an unreadable behavior
document, a duplicate scenario mapping or malformed answer data fails before
the child root is imported. A well-formed answer that does not satisfy the
child's eventual Elicit schema fails at that Elicit through ordinary final
response validation.
The configuration belongs to one exact <Execution>. Its TestAgent provider,
session state, scenario journals and answer matchers are created inside that
child's isolated scope and are torn down before its outcome is published. A
sibling execution receives fresh state even when it repeats the same
declarations under the same <Test>; a named session cannot continue across
the boundary. The behavior-document path resolves from the outer test document
under the TestAgent package's existing containment rule.
A configured child can also give <Plan> its Agent context, which is the one
thing about it a document cannot arrange for itself.
Three surfaces, and they stay distinct. A production run supplies the
production ACPX context from the Agent stack the command settled. A configured
<TestAgent> child supplies the deterministic one from the controlled provider
its own declaration produced. A direct xmd test root and an unconfigured
child supply none: the root does not resolve <Plan> at all, because the run
profile's vocabulary belongs to a run, and an unconfigured child resolves the
same protected bytes every run resolves and is refused before a directory, a
provider, an Agent turn or a review exists.
Availability is the only thing that differs. What a Plan then runs under is the Plan writer frame's fixed policy: the Plan system instruction, the deny-all permission mode, the empty MCP-server and native-tool sets, the prompt-failure policy and the document-capability refusals, installed in one place rather than restated by each provider, so a second implementation cannot bring a weaker one. The controlled provider is installed inside the Plan invocation rather than inherited from what the child registered around itself.
A configured child is given a Plan writer root of its own, created outside that frame and removed when the child settles — the Plan sessions underneath it included, a named one among them, which production keeps and a test may not. The Plan invocation still creates and proves its own empty session directory under that root.
Only the canonical declaration this harness recognizes grants any of it. A
repository component named TestAgent ends the scan and configures nothing, so
a child that reached one is refused exactly as an unconfigured child is.
A scenario reaches the Plan's own turn by its exact authored label. The trusted
child host privately connects that label to the opaque conversation identity
<Plan> derives for the invocation. Scenario selection uses the label; provider
and scenario state remain keyed by the opaque identity and working directory.
This mapping belongs only to the sealed Plan-writer path, so ordinary Agent
use keeps exact-session matching and siblings share no state.
Only detached declaration data enters the host-profile request. A test cannot send an Agent provider, Elicitation middleware, context, Api handler, execution installation, controller or scope to the child. The trusted host consumes the immutable data and installs the controlled providers before root import. Host middleware may observe or refuse that request and cannot alter the declarations or supply an outcome.
With TestAgent configuration, the child receives the controlled provider,
controlled native launcher and all seven ordinary Agent defaults, including
<Session>. Normal run-profile component resolution still decides whether those
defaults are selected: a repository Prompt or Session shadows the first-party
default exactly as it does under xmd run. With no TestAgent declaration, no
outer TestAgent provider crosses into the child. With no Answers declaration,
the run profile retains its ordinary WebForm provider.
Scenario configuration, provider-private behavior journals and declared answers are not child journal records. Prompts and elicitations the child actually performs retain their ordinary production results and replay without contacting either controlled provider.
Canonical core mints one opaque harness per <Test> invocation and expires it
with that invocation; each <Execution> spends a single-use authorization from
it.
The harness is delivered rather than published: a trusted host attaches an
installer to the execution, canonical <Test> calls it inside the invocation
with that invocation's harness, and the installer registers the definitions that
can run a child with the harness in their closure — shadowing a refusing default
for exactly that test's body. There is nothing to read, so a repository component
named Test, Execution or WorkflowRun, a package registering those names,
printErrors(fn), public middleware on any surface including the <Test>
behavior hook, a separately loaded package copy, and replaceable context all
acquire none of it. A document whose host attached no installer recognizes
<Execution> and refuses it, inside a canonical <Test> as everywhere else.
Host-profile middleware is policy. It receives a request rather than a child, so it may observe, refuse and delegate the exact immutable request — and cannot create a child, replace target, source, props, action or journal policy, substitute a completion, or publish an outcome. A handler that returns without delegating produces no child and is reported as the protocol violation it is.
The trusted provider is attached to the execution as a value before untrusted installation, middleware or document code begins. The authorized definitions close over that provider; public host middleware never receives it.
<Execution> receives an invocation-owned binding channel directly from the
engine. It reports only whether this exact invocation has as, and publishes
the exact child outcome once before assertions expand. The channel exposes no
binding name or environment and does not travel through the public Component
Api, Context, props, component metadata or stable names.
TestApi controls testing mode. The testing operation returns false by
default and true beneath <Testing>:
import { testing } from "@executablemd/testing";
const active = yield* testing;TestApi records completed tests in discovery order. Each result contains its
pass or fail status, optional name, source location, and structured error
details when it failed. Rendered test output is not duplicated in the result.
useTesting composes testing around the core execution entrypoint. It
installs the testing components and collectors, activates testing mode for
the execution, and returns a session whose results operation snapshots
completed tests in discovery order:
import { execute } from "@executablemd/core";
import { useTesting } from "@executablemd/testing";
const tests = yield* useTesting();
const execution = yield* execute(options);
const outcome = yield* execution;
const results = yield* tests.results;Execution completion is an Effection Result<string>: Ok(output) on
success, Err(error) on document, infrastructure, or testing failure —
completion never throws once the execution handle exists. Under
useTesting, an otherwise successful execution completes as
Err(TestFailureError) after the output stream closes when any test failed
or no tests were discovered. A failure produced by the document itself
passes through unchanged, and the session's results remain available after
failure. One useTesting session applies per execution scope; its
middleware is removed with that scope. xmd test composes useTesting
around the same execute call the run command uses.
Registering the testing components without useTesting leaves testing mode
inactive: <Test> is skipped, assertion components stay usable, and an
explicit <Testing> boundary still activates its subtree and turns its
failures — or an empty boundary — into an Err outcome for the execution.
TestApi middleware is policy rather than activation. Overriding
testing to true over that registration activates nothing, and a <Test>
reached under it refuses before its body expands, as Registration, mode, and
activation describes.
Several documents share fixtures, never sessions. Fixture and provider state
that spans a suite is installed in an enclosing scope, and each document runs
in a child scope holding one useTesting() session and one execution. A
session's results are cumulative and its completion policy is the whole
session's, so one session around several documents would let a zero-test
document pass on the strength of an earlier one's tests.
Assertion components follow the conventional assertion function names and parameter names. The initial components are:
<Assert>and<AssertFalse><AssertEquals>and<AssertNotEquals><AssertStrictEquals>and<AssertNotStrictEquals><AssertExists><AssertStringIncludes><AssertMatch>and<AssertNotMatch><AssertGreater>and<AssertGreaterOrEqual><AssertLess>and<AssertLessOrEqual><AssertThrows>
Props map directly to the corresponding function parameters: expr for
truthiness assertions, actual and expected for comparisons, and optional
msg where supported. The operands are captures
(executable-mdx-spec.md §6.5): the assertion evaluates them itself, so what it
compares is the value the author wrote rather than a JSON projection of it —
which is what lets <AssertStrictEquals> compare by reference,
<AssertExists> tell undefined from absence, and <AssertMatch> take a real
RegExp. msg is an ordinary validated string prop.
as and slot belong to the engine, which consumes them before validation, so
an assertion accepts them like any other component: as binds the assertion's
diagnostic text.
Equality assertions and <AssertStringIncludes> accept either an expected
prop or rendered children as the expected string. The two forms are mutually
exclusive. Expected children expand in the current scope and environment, use
the same trailing-whitespace trimming as <Let>, and do not render
separately.
Numeric comparisons require an expected prop. Match assertions require a
RegExp through the expected prop. Unary assertions do not accept expected
children.
Comparisons are those of node:assert/strict, a runtime builtin rather than a
dependency, so a published package needs no registry configuration to install.
<AssertEquals> and <AssertNotEquals> are deep strict equality;
<AssertStrictEquals> and <AssertNotStrictEquals> are Object.is. Existence,
falsity, string inclusion and the ordering comparisons have no builtin
equivalent and are defined here: existence rejects only null and undefined,
string inclusion is String.prototype.includes, and the ordering comparisons
apply the relational operators — which coerce object operands, so a value whose
toString throws fails the comparison itself.
Deep equality is stricter than the @std/assert implementation used before
0.5.2 in two respects: 0 and -0 are no longer equal, and an object with a
null prototype is no longer equal to an otherwise-identical plain object. Two
distinct Error values with the same name and message now compare equal, where
previously their stacks made them differ. Pathological operands — an invalid
Date, a WeakMap — are not specified and may differ between runtimes.
Ordinary REPL entries carry the existing assertion defaults without installing testing behavior or a TestAgent. Their discovery and execution use the same assertion forms and validation as ordinary xmd run; REPL.Test remains the explicit Markdown-test boundary.
Assertion components work inside and outside tests. A failed assertion throws an
AssertionError. Outside a test, that error aborts document expansion. Inside a
test, <Test> contains and records it.
Assertions emit Markdown diagnostics in testing mode. During regular execution,
diagnostics are hidden unless --verbose is enabled. Failed assertions still
throw when diagnostics are hidden.
Diagnostics identify the assertion component and outcome. They include the optional message and relevant actual and expected values. Failure diagnostics include the underlying assertion detail when available. Their exact Markdown layout is not prescribed, and formatting arbitrary values must not change the assertion outcome or introduce a new failure.
<AssertThrows> is the one assertion that is not a value comparison. It wraps
children rather than taking actual/expected, and it
requires a message prop: a literal string is matched as a substring of the
raised error's message, or an expression evaluating to a RegExp is tested
against it. The assertion passes when expanding the body raises an error whose
message matches, and fails when the body raises nothing or raises a non-matching
error. It accepts an optional literal as binding that captures the complete
caught error segment — including its cause — into the environment for later
assertions. Unlike the other assertions it emits no pass diagnostic: the return channel
carries the caught segment for as to bind, and no other channel preserves a
durable rendered segment. A failure aborts the document when it occurs outside a
test (inside a <Test> it is contained and recorded), unchanged.
Additional assertion components use the same rules: they state their comparison
here, they raise an AssertionError on failure, and they use the shared
diagnostic behavior. A component must decide its outcome before anything formats
its operands, so that a value with a throwing or mutating toJSON, toString
or getter cannot change whether an assertion passes.
Testing has no separate mocking DSL. Tests install mocks through existing
context API middleware or helpers from @executablemd/runtime/test. Middleware
installed within a test applies to subsequent expansion in that test and is
removed with its scope.
BDD syntax such as describe, it, beforeEach, and beforeAll is not
supported. Gherkin syntax such as Given, When, and Then is not supported.
The sections above specify the Executable Markdown testing language. This one
specifies something different: how this repository's own test corpus — its
TypeScript test files and its Markdown suites — is discovered, divided, and
run under each supported runtime. It changes nothing about <Test>,
<Testing>, assertions, or mocking.
listTestFiles(root) is the only discovery source. It returns sorted
repository-relative POSIX paths for every *.test.ts beneath tests/ in each
root workspace member plus scripts/tests/, and the Markdown suites, and
nothing else. Walking those concrete directories rather than the repository
root is what keeps nested worktrees, the site, and the deliberately malformed
fixtures out without naming any of them.
Its TypeScript paths are the same set Deno's own test-file pattern finds in the repository. A test file discovery cannot see would run under no runtime at all, so the equivalence is a regression rather than a convention.
A Markdown suite is a regular *.test.md file directly inside
examples/adversarial-implementation/tests/, or a regular file at one of
these registered paths:
packages/cli/tests/document-suites/evaluate/Evaluate.test.mdpackages/cli/tests/document-suites/file-copy/FileCopy.test.mdpackages/cli/tests/document-suites/inline/Inline.test.mdpackages/cli/tests/document-suites/plan/Plan.test.mdpackages/cli/tests/document-suites/props/Props.test.mdpackages/cli/tests/document-suites/syntax/Syntax.test.mdpackages/cli/tests/document-suites/targets/Targets.test.mdpackages/cli/tests/document-suites/verbose/Verbose.test.mdpackages/test-agent/src/NativeSessionLaunch.test.mdpackages/test-agent/src/SessionConfiguration.test.md
Discovery skips symlinks and directories. Adversarial discovery does not descend. The individually registered paths admit only the listed documents, so helpers and intentionally failing fixtures stay outside the corpus.
applicableTestFiles(runtime, root) is discovery minus the entries
scripts/runtime-test-exclusions.ts records for that runtime, in the same
sorted order. Every supported runtime is keyed there, including the one that
excludes nothing: an absent key reads as an unmeasured runtime, which is the one
thing weights and partitions must never see.
A file leaves a runtime only through that manifest, and each entry carries a reason and an issue. The manifest is validated structurally; it cannot show that an excluded test has become portable, so removing a stale entry stays a manual act.
test-weights.json records how long each applicable file takes under each
runtime, in milliseconds. It is version 1 and carries provenance — the commit,
the run URL, the attempt, the runner label, and all three runtime versions —
because a weight measured on another machine, image, or commit describes a
corpus that no longer exists.
The reader checks the version, provenance, runtime names, test paths, and
positive integer weights. A test path is either a repository-relative
TypeScript test path under tests/ or a Markdown suite path matching
the discovery rules.
deno task weights:measure is the only command that writes it, in two steps.
weights:measure <runtime> <directory> measures one runtime. It runs each
applicable file alone, in sorted order, through that runtime's exact one-file
command, and times the child with a monotonic clock. It keeps
<directory>/test-weights-<runtime>.json current: the record carries the
runtime, a status, the full provenance, the applicable corpus it derived, every
file that passed with its milliseconds in the order it ran, and the failure if
there was one. Its status is partial before the first file and after each one
that passes, failed with the file and exit status when a file fails — which
stops the measurement — and complete once every file passed. A run that is cut
off leaves partial, so an unfinished measurement is never mistaken for a
finished one. The record has one current shape, and a reader refuses a missing,
extra or malformed field.
weights:measure assemble <directory> writes test-weights.json from the
three records, and only when every one is complete; agrees with the commit,
run URL, attempt and runner the environment names; records the same three
runtime versions as the others; derived the same applicable corpus as the
checkout; and measured each applicable file exactly once, and no other file.
Otherwise it refuses with every problem named and writes nothing — a file
assembled from a runtime that failed halfway, or from two runs, would be
indistinguishable from one measured on a healthy corpus.
Both writes go through a staged copy of the invocation's own, so each is atomic and a cancelled run removes only its own staging. Ordinary verification never rewrites either.
A file with a recorded weight uses it. A file without one is charged the maximum recorded weight among the applicable files that do have one: conservative, non-zero, and named in the shard's log. Weights recorded for files that have since left the corpus are ignored entirely, so a deleted outlier cannot inflate that fallback forever.
When no applicable file has a recorded weight there is no conservative number to pick — a zero would pack the corpus onto one shard and a constant would be a timing assumption in disguise — so the partition refuses.
Each runtime's applicable corpus is partitioned independently, by longest-processing-time-first assignment:
- order the files by descending effective weight, breaking equal weights by ascending repository path;
- create the requested number of indexed empty shards;
- assign each file in turn to the shard with the lowest total weight, breaking equal totals by the lowest shard index;
- sort each shard's assigned paths; and
- retain shard indices rather than re-sorting the shards.
This minimizes the longest shard rather than equalizing file counts, because the
wall clock a run waits on is the slowest shard. A count below one or above the
applicable file count is refused, as is an index outside 1..count.
The result is a pure function of the applicable paths, the weights, the count, and the index: repeated calls return byte-identical lists, which is what lets a matrix job be told only its index and still be certain it runs exactly the files no sibling runs. The shards are pairwise disjoint and their union is exactly the applicable corpus — every applicable file runs exactly once per runtime.
runtime-tests.ts <deno|node|bun> [<index>/<count>] runs a runtime's tests. With
no selection it runs the applicable TypeScript files in one native invocation,
then each applicable Markdown suite through its one-file command, and exits with
the first failure of either portion. With a selection it resolves that shard and
invokes its files serially, in sorted order, each through that runtime's
one-file command:
- Deno —
deno test --allow-all --frozen --reporter=dot <file> - Node —
tsx --tsconfig tsconfig.node.json --test --test-concurrency=1 <file> - Bun —
bun test --timeout=300000 <file>
A Markdown suite runs as xmd test <file> through the same runtime's CLI
source entrypoint, resolved from the runner's checkout:
- Deno —
deno run --quiet --allow-all --node-modules-dir=manual --cached-only --frozen packages/cli/src/deno.ts - Node —
tsx --tsconfig tsconfig.node.json packages/cli/src/node.ts - Bun —
bun packages/cli/src/bun.ts
The CLI runs from the repository root. A shard names each Markdown suite when
it starts and passes test <file> --reporter=failures. Weight measurement
passes the same arguments. Local unsharded runs use the default reporter.
Deno weight measurement also uses the same dot reporter as Deno shards.
The CLI inherits the environment (PATH, the Deno cache and install variables, XDG_CACHE_HOME,
TMPDIR and XMD_TEST_ARTIFACTS). Individually registered suites use the
CLI's default includes and file permissions. Deno launches of Plan,
NativeSessionLaunch and SessionConfiguration select the test-only worker
plugin. It carries the
prepared Deno flags into TestAgent workers. Syntax and Verbose receive a
temporary executable that invokes the same runtime's source CLI with its
arguments intact. Each suite owns and removes its executable. Verbose
also receives the absolute path to its fixture components.
The adversarial suites append the following arguments to the source entrypoint:
test <file> --plugin packages/cli/tests/fixtures/workflow-test-permissions.mjs --include <root>/examples/adversarial-implementation/tests/components --include <root>/examples/adversarial-implementation/components, with the include paths
absolute so that a nested <Dir> cannot change where components are found.
Preparation, ProductionRole and Submission include
tests/production-components in place of tests/components. The child's
environment also receives XMD_EXAMPLE_COMMAND and XMD_EXAMPLE_TEST_COMMAND.
Each is the prepared Deno CLI above followed by run --allow-absolute or
test, respectively, as quoted shell words. Fixture documents
run their real workflow through those commands under every outer runtime,
because the run profile they exercise is the Deno host's. The artifact-return
suites also receive XMD_ARTIFACT_RETURN_EVIDENCE when it is set. The fixture
plugin adds --node-modules-dir=manual --cached-only --frozen when the Deno
source host relaunches itself as a TestAgent worker, including workers for
isolated run-profile children.
Each adversarial suite but Implementation, whose named groups carry their
own deadlines, runs under a whole-launch deadline. Preparation has 420
seconds, Friction has 180, DecisionResolution and Delivery have 300, and
the other adversarial suites have 120. A suite that
reaches it is stopped and its cleanup joined, its path and deadline are
reported, and it fails with status 1. The files after it still run.
SessionConfiguration has a 120-second whole-launch deadline under each
runtime. Other individually registered suites have a 300-second whole-launch
deadline under Bun and use their authored test deadlines under Deno and Node.
The same operation reads the prepared node_modules/effection link before and
after a Markdown suite. A link that changed or disappeared fails the file with
a message naming both targets; a file that already failed keeps its own status.
A malformed runtime, index, or count is refused before any test process starts.
A numeric failure does not prevent the files assigned after it: the shard continues, retains the first failed file and its status, and exits with that status once every assigned file has had its chance. A status without a numeric code — a signalled child — is failure. The shard streams the output its selected reporter emits. Deno's dot reporter keeps failure details and omits test console logs.
Cancellation is distinct from failure. Halting a shard halts the process group of the file it is running, releases that file's scratch state and starts no later file.
deno task test is the local Deno suite. With no arguments it runs Deno's own
whole-repository deno test --allow-all --frozen, including the files the CI
corpus leaves to smoke jobs, then each applicable Markdown suite through its
one-file command. Otherwise its arguments reach deno test verbatim unless a
positional path selects an applicable Markdown suite. Native Markdown
doctests outside that corpus, focused paths, --changed and --related keep
their Deno meaning. Select an applicable Markdown suite by its
repository-relative path, optionally prefixed with ./. That selection may
include native test paths and must contain only paths. Each portion runs only
when it is nonempty, and the first failure is the status. pnpm test:node and
bun run test:bun are the no-selection runtime-tests.ts run of their runtime.
test-deno, test-node, and test-bun are fail-fast: false matrices under
those same job IDs. Each shows its selection in its name, passes that selection
only to its Test step, and keeps that runtime's own preparation unchanged and
repeated per shard. green requires all three job IDs; GitHub collapses a
matrix into one result behind its ID, so a shard that fails, is cancelled, times
out, or unexpectedly skips makes that dependency non-success.
Concurrency inside a shard is not enabled.
The required repl-bench job checks the bench tools on Ubuntu for pull
requests. Pushes to main run the tools, the driver exit and teardown tests,
and the Markdown terminal journeys on macOS and Ubuntu. Those terminal tests
exercise the host's pseudo-terminal and retain screen and terminal evidence
on failure. green requires the job on both events.
Remeasure after any change to the corpus, the exclusions, or a runner command.
A hosted workflow_dispatch produces the measurement on the runner CI uses.
Each runtime is measured in its own job, serially within it, and uploads its
record as test-weights-<runtime>-attempt-<attempt> however the job ended. An
assemble job, which runs only when all three succeeded, assembles that
attempt's records and uploads test-weights.json as the test-weights
artifact. Each measuring job has 165 minutes, its measuring step 150 of them,
and assembly 10, so the run's execution fits in 175 minutes; time queued for a
runner is outside it. Committing that artifact is a deliberate act, which is
what keeps the provenance true of the run that produced it. Milliseconds are
never edited by hand.
Each measuring job sets XMD_TEST_ARTIFACTS to the absolute workspace path
${{ github.workspace }}/.xmd/test-artifacts. On failure or cancellation it
uploads the complete saved journal trees, including hidden files, as
measure-test-journals-<runtime>-attempt-<attempt>; missing files are ignored.
These artifacts follow Progress and failure journals
and remain separate from the measurement records selected for assembly.
Counts are measured rather than chosen. For each runtime the lower bound is
floor(sum of applicable weights / 420000) + 1, and the installed count is the
smallest for which five consecutive runs on one fixed head keep every shard
Test step, and that runtime's whole execution window, under 420 seconds. A
miss increments only that runtime and begins a fresh sequence of five. Narrowing
the corpus, dropping a runtime, or enabling in-shard concurrency is not a
response to a miss. A single file whose own weight reaches 420 seconds cannot be
solved by any count, and is reported rather than absorbed.