diff --git a/.github/versions.env b/.github/versions.env index f2f6c0e..d44f49a 100644 --- a/.github/versions.env +++ b/.github/versions.env @@ -1,4 +1,4 @@ # The versions CI builds and tests with; .github/actions/setup-mcpp exports them. -MCPP_VERSION=2026.11.8.1 +MCPP_VERSION=2026.11.10.1 # The xlings that mcpp release bundles. XLINGS_VERSION=v2026.10.26.1 diff --git a/CHANGELOG.md b/CHANGELOG.md index 645a6c9..bda0118 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,34 @@ # Changelog +## [0.6.0] - 2026-11-10 + +The mtest protocol's normative text is the mcpp test spec (mcpp `docs/specs/test.md`, SPEC-017 v2.0), +which keeps its clause numbers; `spec/protocol.md` lists what each part of mtest implements of it. + +- **A test session** (spec O9): `mtest serve` answers `test/plan`, `test/run`, `test/add`, + `test/cancel` and `test/status`, and states `test/build`, `test/case`, `test/program` and + `test/run` as they happen. A run is answered at once and its programs run on the session's + workers beside other runs; a case added joins the run in progress; cancelling ends the process and + everything it started, and the cases that had not begun run in another process. The session builds + nothing: it starts each program from the launch mcpp gives (`mcpp test --no-run`, spec B5). +- **A case's own output** (spec O8). Under a runner the library keeps what each case writes, on each + stream, states it as `output` events and writes it on to the terminal; a case declared inside + another keeps its own. +- **A case's checks** (spec O3): its `end` states the checks that held and failed (`checks`); a + failure states its statement (`source`) and whether it ended the case (`fatal`). +- **A failing case reads as a person reads it.** Under its line, each failure: its place, the + statement, the two sides or that it did not hold, the message the check added, and whether the case + went on; the line counts the checks that failed, and a case that declared cases counts them. +- The library's entry states the capabilities it implements and its session (`test-capabilities`, + `test-session`; mcpp 2026.11.10.1 reads them, an earlier mcpp ignores them). +- The engine and the conformance suite ask for mcpplibs.os `^0.4.1`, as the library does, and build + `-openkal` over openkal-platform 0.25.0. +- **A program only a runner starts is listed through that runner.** The listing pass (P3) starts a + program as its run does: under `--run-under` (and a `[trace.*]` tool's `run_under`), with the + run's `--env` variables. A program of another system -- started on a virtual machine of its system + or under an emulator -- was started directly to list its cases and failed with "did not start"; + its cases could not be run through mtest from the host that built it. + ## [0.5.2] - 2026-11-09 - **A copy of a test program that does something else is an object** (`child`). A test that starts diff --git a/README.md b/README.md index c36317c..c734bba 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ int main() { ```toml [dev-dependencies] -mcpplibs.mtest = "0.5.2" +mcpplibs.mtest = "0.6.0" ``` A failure names its place and both sides (`expected 2, actual 3`). Under a runner the same program diff --git a/README.zh-CN.md b/README.zh-CN.md index 6680412..04bc531 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -37,7 +37,7 @@ int main() { ```toml [dev-dependencies] -mcpplibs.mtest = "0.5.2" +mcpplibs.mtest = "0.6.0" ``` 失败时给出位置与两侧的值（`expected 2, actual 3`）。在运行器之下，同一程序按协议报告；单独运行时每个用例输出一行。 diff --git a/adapters/gtest/mcpp.toml b/adapters/gtest/mcpp.toml index d558a1b..1746fa3 100644 --- a/adapters/gtest/mcpp.toml +++ b/adapters/gtest/mcpp.toml @@ -13,7 +13,7 @@ [package] namespace = "mcpplibs.mtest" name = "gtest" -version = "0.5.2" +version = "0.6.0" description = "GoogleTest speaking the mtest protocol: a test main that lists, selects and reports GoogleTest's cases" license = "Apache-2.0" diff --git a/adapters/gtest/src/protocol.cpp b/adapters/gtest/src/protocol.cpp index d536913..482fbc1 100644 --- a/adapters/gtest/src/protocol.cpp +++ b/adapters/gtest/src/protocol.cpp @@ -128,7 +128,7 @@ int mcpplibs_mtest_gtest_main(int argc, char** argv) { {"protocol", 1}, {"level", 1}, {"framework", "googletest"}, - {"version", "0.5.2"}, + {"version", "0.6.0"}, {"isolation", {"process", "case"}}, {"guide", "https://github.com/mcpp-community/mtest/blob/main/adapters/gtest/docs/" "writing-tests.md"}}); diff --git a/capi/mcpp.toml b/capi/mcpp.toml index 889c00f..c0e566a 100644 --- a/capi/mcpp.toml +++ b/capi/mcpp.toml @@ -4,7 +4,7 @@ [package] namespace = "mcpplibs.mtest" name = "capi" -version = "0.5.2" +version = "0.6.0" description = "mtest's C ABI: the protocol's events for any language with a C FFI" license = "Apache-2.0" diff --git a/capi/src/mtest.cpp b/capi/src/mtest.cpp index 559ec55..b81633b 100644 --- a/capi/src/mtest.cpp +++ b/capi/src/mtest.cpp @@ -116,7 +116,7 @@ void mtest_init(const char* framework, int level) { {"protocol", 1}, {"level", std::clamp(level, 1, 3)}, {"framework", text(framework)}, - {"version", "0.5.2"}, + {"version", "0.6.0"}, {"isolation", {"process", "case"}}}); } diff --git a/conformance/mcpp.toml b/conformance/mcpp.toml index 71509a8..4dac10e 100644 --- a/conformance/mcpp.toml +++ b/conformance/mcpp.toml @@ -8,7 +8,7 @@ [package] namespace = "mcpplibs.mtest" name = "conformance" -version = "0.5.2" +version = "0.6.0" description = "The conformance suite of the mtest protocol, for every implementation of its program side" license = "Apache-2.0" @@ -33,5 +33,5 @@ main = "src/fixture.cpp" [dependencies] mcpplibs.mtest = { path = "../lib" } -mcpplibs.os = "0.4.1" +mcpplibs.os = "^0.4.1" nlohmann.json = "3.12.0" diff --git a/docs/commands.md b/docs/commands.md index be93f64..ae66563 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -36,7 +36,7 @@ nothing; a built program's own listing is the authority (`manifest/src/manifest. | Command | Does | |---|---| -| `run` | runs the selection; `--jobs N`, `--seed N`, `--fail-fast`, `--retries N`, `--isolate` (a process per case), `--timeout 5m`, `--env K=V`, `--run-under CMD`, `--monitor LEVEL` (P10), `--group G` (runs reported together); `--junit`, `--html`, `--report-json FILE`, `--md FILE`; `--coverage` measures the coverage map. A program's cases are divided among workers that would otherwise idle (`[run] split`), and programs run together within the machine's memory (`[run] memory`) | +| `run` | runs the selection; `--jobs N`, `--seed N`, `--fail-fast`, `--retries N`, `--isolate` (a process per case), `--timeout 5m`, `--env K=V`, `--run-under CMD` (every start of a program, its listing included: a runner such as an emulator or a machine of the program's system), `--monitor LEVEL` (P10), `--group G` (runs reported together); `--junit`, `--html`, `--report-json FILE`, `--md FILE`; `--coverage` measures the coverage map. A program's cases are divided among workers that would otherwise idle (`[run] split`), and programs run together within the machine's memory (`[run] memory`) | | `rerun` | the cases that failed in the last run | | `smoke` | the `smoke` set | | `golden` | a run with `MTEST_GOLDEN=update`: golden files are written, not compared | diff --git a/docs/service.md b/docs/service.md index 055e7ba..f8a27a5 100644 --- a/docs/service.md +++ b/docs/service.md @@ -12,3 +12,35 @@ Each request runs on its own thread, so `status` and `cancel` are answered while a run runs. An MCP client is configured with the command `mtest serve` in the project's directory. + +## The test session (0.6.0) + +`mtest serve` is also the test session of the mcpp test spec (mcpp `docs/specs/test.md`, O9): a client +that holds it runs an mcpp project's cases as a person asks for them, adds cases to a run in progress +and cancels a run, a program or a case. The library's entry states it (`test-session = ["mtest", +"serve"]`), and `mcpp test --guide` reports it. + +| Method | Params | Result | +|---|---|---| +| `test/plan` | `project`, optional `filter`, `options` | `programs`: the records of `mcpp test --cases` | +| `test/run` | `project`, `selection` (`[{program, cases}]`; empty: every program), `options` | `run`, at once | +| `test/add` | `run`, `selection` | `run`: the cases join the run in progress | +| `test/cancel` | `run`, optional `program` and `case` | what was cancelled | +| `test/status` | optional `run` | each run's status, counts and cases | +| `shutdown` | | every run cancelled | + +`options`: `mcpp` (a program, or a program and its first arguments; else `MTEST_MCPP`, else `mcpp`), +`args` (more arguments of `mcpp test`: `--profile`, `--target`), `isolation` (`case`: a process per +case). + +- **Notifications.** `test/build` (`begin`, then `end` with the programs built), `test/case` (`begin`, + `output`, `failure`, `end`, with the fields of the program's events), `test/program` (a process's + status, exit and time) and `test/run` (the run's status and counts). +- **Builds are mcpp's.** The session asks mcpp for the programs it runs (`mcpp test --no-run + --message-format json`, mcpp 2026.11.10.1 or later), which builds them when they need it, and starts + each from the launch the record carries. One build runs at a time; the processes share the + session's workers. +- **Cancelling** ends the process and everything it started. A case that had begun ends `cancelled`; + the cases its process was to run and that had not begun run in another process, unless the run or + the program was cancelled. A run of cancelled cases and no failure ends `cancelled`. +- **The end of the input** lets the runs in progress end; `shutdown` cancels them. diff --git a/engine/mcpp.toml b/engine/mcpp.toml index 977cb02..0bf81dd 100644 --- a/engine/mcpp.toml +++ b/engine/mcpp.toml @@ -10,7 +10,7 @@ [package] namespace = "mtest" name = "mtest" -version = "0.5.2" +version = "0.6.0" description = "The mtest engine: run, schedule, store, explain and analyse tests of any language" license = "Apache-2.0" authors = ["mcpp-community"] @@ -30,7 +30,7 @@ main = "src/main.cpp" [dependencies] mcpplibs.mtest.manifest = { path = "../manifest" } # the static case manifest -mcpplibs.os = "0.4.1" +mcpplibs.os = "^0.4.1" mcpplibs.cmdline = "0.0.2" nlohmann.json = "3.12.0" marzer.tomlplusplus = "3.4.0" diff --git a/engine/src/commands.cpp b/engine/src/commands.cpp index a6d9ab2..20fbdf3 100644 --- a/engine/src/commands.cpp +++ b/engine/src/commands.cpp @@ -121,6 +121,11 @@ std::expected selection_of(const context& ctx, const r.exclude_tags = strings(p, "exclude_tag"); r.exclude_programs = strings(p, "exclude_program"); r.failed_only = p.value("failed", false); + r.run_under = strings(p, "run_under"); + if (auto e = p.find("env"); e != p.end() && e->is_object()) + for (auto& item_ : e->items()) + if (item_.value().is_string()) + r.env.emplace_back(item_.key(), item_.value().get()); if (auto since = text_of(p, "since"); !since.empty()) { auto paths = changed_since(ctx.cfg.root, since); if (!paths) @@ -1829,7 +1834,7 @@ json catalog() { plain({{"out", {{"type", "string"}}}, {"argv", {{"type", "array"}, {"items", {{"type", "string"}}}}}}), true); - return {{"schema", "mtest.catalog/1"}, {"version", "0.5.2"}, {"tools", tools}}; + return {{"schema", "mtest.catalog/1"}, {"version", "0.6.0"}, {"tools", tools}}; } } // namespace mcpplibs::mtest::engine diff --git a/engine/src/exec.cpp b/engine/src/exec.cpp index 0e5c8da..23daa44 100644 --- a/engine/src/exec.cpp +++ b/engine/src/exec.cpp @@ -189,7 +189,9 @@ report_reading read_report(std::string_view text, std::string_view program) { std::expected, std::string> list_program(const program_spec& spec, const std::filesystem::path& work, - const std::filesystem::path& root, std::vector* components) { + const std::filesystem::path& root, std::vector* components, + const std::vector& run_under, + const std::vector>& env) { if (detect_level(spec) < 1) { case_info c; c.id = spec.name; @@ -203,7 +205,10 @@ list_program(const program_spec& spec, const std::filesystem::path& work, const auto report = work / "list.ndjson"; std::error_code ec; std::filesystem::remove(report, ec); - const auto argv = with_loader(spec.argv); + // Started as its run starts it: under the run's prefix, with the run's variables. + auto argv = run_under; + for (auto& a : with_loader(spec.argv)) + argv.push_back(std::move(a)); os::process::spawn_options o; o.program = resolve_program(argv.front()); o.args.assign(argv.begin() + 1, argv.end()); @@ -211,6 +216,8 @@ list_program(const program_spec& spec, const std::filesystem::path& work, o.in = os::process::stdio::null; for (const auto& [k, v] : spec.env) o.env.emplace_back(k, v); + for (const auto& [k, v] : env) + o.env.emplace_back(k, v); o.env.emplace_back("MTEST_PROTOCOL", "1"); o.env.emplace_back("MTEST_MODE", "list"); o.env.emplace_back("MTEST_REPORT", os::fs::utf8(report)); @@ -220,7 +227,7 @@ list_program(const program_spec& spec, const std::filesystem::path& work, auto done = os::process::run(std::move(o), limits); if (!done) return std::unexpected( - std::format("{} did not start: {}", spec.argv.front(), done.error().message())); + std::format("{} did not start: {}", argv.front(), done.error().message())); auto text = os::fs::read(report); auto r = read_report(text ? std::string_view(*text) : std::string_view{}, spec.name); if (r.level == 0) { diff --git a/engine/src/exec.cppm b/engine/src/exec.cppm index fa68677..b6cdaab 100644 --- a/engine/src/exec.cppm +++ b/engine/src/exec.cppm @@ -34,7 +34,9 @@ struct run_outcome { // Lists the cases of a program at level 1 or more (P3); an L0 program has one case, its own name. std::expected, std::string> list_program(const program_spec& spec, const std::filesystem::path& work, - const std::filesystem::path& root, std::vector* components = nullptr); + const std::filesystem::path& root, std::vector* components = nullptr, + const std::vector& run_under = {}, + const std::vector>& env = {}); // `mtest measure --out FILE -- ARGV`: runs ARGV with this process's streams, writes what it used // (its children's account) to FILE as JSON, and ends with its status. diff --git a/engine/src/main.cpp b/engine/src/main.cpp index 73a13f4..6675713 100644 --- a/engine/src/main.cpp +++ b/engine/src/main.cpp @@ -98,7 +98,7 @@ const json* tool_named(const json& tools, std::string_view name) { cmdline::App application(const json& tools) { auto app = cmdline::App("mtest") - .version("0.5.2") + .version("0.6.0") .description("run tests of any language, and ask about their runs") .option(cmdline::Option("json").global().help("print the answer's data")); for (const auto& t : tools) @@ -253,7 +253,7 @@ int main(int argc, char** argv) { const std::string name(parsed->subcommand_name()); const auto& sub = parsed->subcommand()->get(); if (name == "version") { - std::println("mtest 0.5.2"); + std::println("mtest 0.6.0"); return 0; } const auto cwd = std::filesystem::current_path(); diff --git a/engine/src/select.cpp b/engine/src/select.cpp index 26cd145..97b2d82 100644 --- a/engine/src/select.cpp +++ b/engine/src/select.cpp @@ -561,7 +561,7 @@ std::expected make_plan(const config& c, const store& s, auto listed = list_program(p, s.dir / "work" / "list" / std::to_string(std::hash{}(p.name)), - c.root, &components); + c.root, &components, r.run_under, r.env); if (!listed) return std::unexpected(listed.error()); for (const auto& ci : components) { diff --git a/engine/src/select.cppm b/engine/src/select.cppm index 22272b6..69442b0 100644 --- a/engine/src/select.cppm +++ b/engine/src/select.cppm @@ -32,6 +32,11 @@ struct selection_request { // The change could not be read (a shallow clone without the base): a set that selects by // change selects every case, and says why. Empty: nothing went wrong. std::string unknown_change; + // How a program is started (`--run-under`, `--env`): its listing (P3) starts it as its run + // does, so that a program only a runner can start (an emulator, a machine of its system) is + // listed through that runner. + std::vector run_under; + std::vector> env; }; struct unit_case { diff --git a/engine/src/service.cpp b/engine/src/service.cpp index 6d61ddc..34d4d82 100644 --- a/engine/src/service.cpp +++ b/engine/src/service.cpp @@ -3,6 +3,7 @@ module mcpplibs.mtest.engine.service; import std; import mcpplibs.mtest.engine.model; import mcpplibs.mtest.engine.commands; +import mcpplibs.mtest.engine.session; namespace mcpplibs::mtest::engine { @@ -25,6 +26,17 @@ json error_reply(const json& id, int code, std::string message) { } // namespace int serve(const std::filesystem::path& cwd) { + // The test session (mcpp test spec O9), made on its first method. + std::unique_ptr tests; + std::mutex tests_lock; + auto the_session = [&]() -> session& { + std::lock_guard g(tests_lock); + if (!tests) + tests = std::make_unique([](std::string_view method, const json& params) { + send({{"jsonrpc", "2.0"}, {"method", std::string(method)}, {"params", params}}); + }); + return *tests; + }; std::vector requests; for (std::string line; std::getline(std::cin, line);) { if (line.empty() || line == "\r") @@ -44,19 +56,45 @@ int serve(const std::filesystem::path& cwd) { {"result", {{"protocolVersion", params.value("protocolVersion", std::string("2025-06-18"))}, {"capabilities", {{"tools", {{"listChanged", false}}}}}, - {"serverInfo", {{"name", "mtest"}, {"version", "0.5.2"}}}, + {"serverInfo", {{"name", "mtest"}, {"version", "0.6.0"}}}, {"instructions", "Run tests and ask about runs: run, plan, explain, history, analyze, graph, " - "gate, ci. " + "gate, ci. A test session (test/plan, test/run, test/add, test/cancel, " + "test/status) runs an mcpp project's cases as they are asked for. " "Each result carries a `conclusion`; a failure's `again` repeats it."}}}}); continue; } if (method == "notifications/initialized" || method == "initialized") continue; if (method == "shutdown") { + { + std::lock_guard g(tests_lock); + if (tests) + tests->shutdown(); + } send({{"jsonrpc", "2.0"}, {"id", id}, {"result", nullptr}}); continue; } + if (method.starts_with("test/")) { + const auto verb = method.substr(5); + requests.emplace_back([&, id, verb, params, notification] { + auto& t = the_session(); + json r = verb == "plan" ? t.plan(params) + : verb == "run" ? t.run(params) + : verb == "add" ? t.add(params) + : verb == "cancel" ? t.cancel(params) + : verb == "status" + ? t.status(params) + : json{{"error", std::format("no method test/{}", verb)}}; + if (notification) + return; + if (r.contains("error")) + send(error_reply(id, -32602, r["error"].get())); + else + send({{"jsonrpc", "2.0"}, {"id", id}, {"result", r}}); + }); + continue; + } if (method == "exit") break; if (method == "tools/list") { @@ -96,6 +134,15 @@ int serve(const std::filesystem::path& cwd) { } }); } + // The input ended: what was asked for runs to its end (`shutdown` is how a client cancels). + requests.clear(); + { + std::lock_guard g(tests_lock); + if (tests) { + tests->drain(); + tests->shutdown(); + } + } return 0; } diff --git a/engine/src/session.cpp b/engine/src/session.cpp new file mode 100644 index 0000000..6b2196e --- /dev/null +++ b/engine/src/session.cpp @@ -0,0 +1,647 @@ +module mcpplibs.mtest.engine.session; + +import std; +import mcpplibs.os; +import mcpplibs.mtest.engine.model; + +namespace mcpplibs::mtest::engine { + +namespace os = mcpplibs::os; + +namespace { + +// How mcpp starts one test program (B5 `launch`), and the protocol it implements. +struct launch { + std::vector argv; + std::vector> env; + std::filesystem::path cwd; + int protocol = 0; +}; + +// One process of a run: a program, the cases it runs (none: all of them), and what it did. +struct job { + std::size_t serial = 0; + std::string run; + std::string program; + std::vector cases; + launch how; + std::set begun, ended, cancelled_cases; + bool cancelled = false; // the whole process is cancelled + bool requeue = false; // its cases that did not begin run in another process + os::process::process* proc = nullptr; +}; + +// A run's verdict: a failure fails it; else a cancelled case cancels it. +std::string verdict(bool cancelled, const std::map& counts) { + if (counts.contains("fail") || counts.contains("build-fail")) + return "fail"; + return cancelled || counts.contains("cancelled") ? "cancelled" : "pass"; +} + +struct run_state { + std::filesystem::path project; + std::vector mcpp; // the program, and arguments before `test` + std::vector args; // more arguments of `mcpp test` (profile, target) + bool isolate = false; // a process per case + std::size_t outstanding = 0; // builds and processes not yet ended + bool cancelled = false, ended = false; + std::map counts; + std::map cases; // "/" -> status +}; + +std::vector records_of(std::string_view text) { + std::vector out; + for (std::size_t at = 0; at < text.size();) { + auto nl = text.find('\n', at); + auto line = + text.substr(at, nl == std::string_view::npos ? std::string_view::npos : nl - at); + at = nl == std::string_view::npos ? text.size() : nl + 1; + if (line.empty() || line.front() != '{') + continue; + if (auto j = json::parse(line, nullptr, false); j.is_object()) + out.push_back(std::move(j)); + } + return out; +} + +} // namespace + +struct session::impl { + notify_fn notify; + std::filesystem::path tmp; + std::mutex m; + std::condition_variable cv; + bool stopping = false; + std::deque> pending; + std::vector> running; + std::map runs; + std::size_t next_run = 1, next_job = 1; + std::mutex build_m; // one build at a time: the build state is mcpp's (B5) + std::vector workers; + std::vector planners; + + // `mcpp test` with `args` in `project`, its records. + std::pair> call_mcpp(const run_state& r, std::vector args) { + os::process::spawn_options o; + o.program = r.mcpp.front(); + o.args.assign(r.mcpp.begin() + 1, r.mcpp.end()); + o.args.insert(o.args.end(), args.begin(), args.end()); + o.args.insert(o.args.end(), r.args.begin(), r.args.end()); + o.cwd = r.project; + o.in = os::process::stdio::null; + auto out = os::process::run(o); + if (!out) + return {-1, + {json{{"error", std::format("{} did not start: {}", r.mcpp.front(), + out.error().message())}}}}; + return {out->status.shell_code(), records_of(out->out)}; + } + + void case_end(const job& j, const std::string& id, std::string_view status, const json& extra) { + json p = {{"run", j.run}, + {"program", j.program}, + {"id", id}, + {"event", "end"}, + {"status", status}}; + for (auto it = extra.begin(); it != extra.end(); ++it) + p[it.key()] = *it; + { + std::lock_guard g(m); + auto& r = runs[j.run]; + r.counts[std::string(status)]++; + r.cases[j.program + "/" + id] = std::string(status); + } + notify("test/case", p); + } + + // The run ends when nothing of it is outstanding; m held. + void maybe_end(const std::string& id) { + auto& r = runs[id]; + if (r.outstanding != 0 || r.ended) + return; + r.ended = true; + cv.notify_all(); + notify("test/run", + {{"run", id}, {"status", verdict(r.cancelled, r.counts)}, {"counts", r.counts}}); + } + + void enqueue(const std::string& run, const std::string& program, const launch& how, + const std::vector& cases) { + auto& r = runs[run]; + auto add = [&](std::vector one) { + auto j = std::make_shared(); + j->serial = next_job++; + j->run = run; + j->program = program; + j->cases = std::move(one); + j->how = how; + pending.push_back(j); + ++r.outstanding; + }; + if (r.isolate && !cases.empty()) + for (const auto& c : cases) + add({c}); + else + add(cases); + cv.notify_all(); + } + + // B5: the selected programs built when they need to be, and their launches; then their jobs. + void + build_and_enqueue(const std::string& run, + std::vector>> selection) { + run_state r; + { + std::lock_guard g(m); + r = runs[run]; + } + std::map launches; + { + std::lock_guard b(build_m); + notify("test/build", {{"run", run}, {"event", "begin"}}); + std::vector args = {"test", "--no-run", "--message-format", "json"}; + if (selection.size() == 1) + args.push_back(selection.front().first); + auto [status, records] = call_mcpp(r, args); + json built = json::array(); + for (const auto& rec : records) { + const auto name = rec.value("test", std::string{}); + if (name.empty()) + continue; + if (rec.value("status", std::string{}) == "built" && rec.contains("launch")) { + const auto& l = rec["launch"]; + launch one; + one.argv = l.value("argv", std::vector{}); + if (l.contains("env") && l["env"].is_object()) + for (auto it = l["env"].begin(); it != l["env"].end(); ++it) + one.env.emplace_back(it.key(), it->get()); + one.cwd = l.value("cwd", std::string{}); + one.protocol = rec.value("protocol", json(nullptr)).is_number() + ? rec["protocol"].get() + : 0; + launches[name] = std::move(one); + built.push_back(name); + } else if (rec.value("status", std::string{}) == "compile_fail") { + std::lock_guard g(m); + runs[run].counts["build-fail"]++; + notify("test/program", + {{"run", run}, + {"program", name}, + {"status", "build-fail"}, + {"output", rec.value("compile_output", std::string{})}}); + } + } + const bool ok = status == 0 || !launches.empty(); + json end = {{"run", run}, {"event", "end"}, {"ok", ok}, {"programs", built}}; + if (!ok && !records.empty() && records.front().contains("error")) + end["error"] = records.front()["error"]; + else if (!ok) + end["error"] = std:: + format("mcpp test --no-run exited {} and named no launch (it gives one from " + "2026.11.10.1, feature `test-launch-records`)", + status); + notify("test/build", end); + } + std::lock_guard g(m); + auto& rs = runs[run]; + if (!rs.cancelled) { + if (selection.empty()) + for (const auto& [name, how] : launches) + enqueue(run, name, how, {}); + for (const auto& [program, cases] : selection) + if (auto it = launches.find(program); it != launches.end()) + enqueue(run, program, it->second, cases); + } + --rs.outstanding; + maybe_end(run); + } + + void execute(job& j) { + const auto dir = tmp / j.run / std::format("{}", j.serial); + std::error_code ec; + std::filesystem::create_directories(dir / "tmp", ec); + const auto report = dir / "report.ndjson"; + os::process::spawn_options o; + if (j.how.argv.empty()) + return; + o.program = j.how.argv.front(); + o.args.assign(j.how.argv.begin() + 1, j.how.argv.end()); + o.cwd = j.how.cwd; + o.in = os::process::stdio::null; + o.out = os::process::stdio::pipe; + o.err = os::process::stdio::pipe; + o.own_unit = true; // cancelling ends it and everything it started + for (const auto& [k, v] : j.how.env) + o.env.emplace_back(k, v); + if (j.how.protocol) { + o.env.emplace_back("MTEST_PROTOCOL", "1"); + o.env.emplace_back("MTEST_MODE", "run"); + o.env.emplace_back("MTEST_REPORT", report.string()); + o.env.emplace_back("MTEST_TMPDIR", (dir / "tmp").string()); + if (!j.cases.empty()) { + std::string lines; + for (const auto& c : j.cases) + lines += c + "\n"; + (void)os::fs::write(dir / "select.txt", lines); + o.env.emplace_back("MTEST_SELECT", (dir / "select.txt").string()); + } + } + const auto started = std::chrono::steady_clock::now(); + auto p = os::process::process::spawn(o); + if (!p) { + notify("test/program", + {{"run", j.run}, + {"program", j.program}, + {"status", "not-run"}, + {"reason", std::format("{} did not start: {}", o.program.string(), + p.error().message())}}); + return; + } + { + std::lock_guard g(m); + j.proc = &*p; + if (j.cancelled) + p->kill(); + } + std::string out_text; + std::jthread out_reader([&] { + for (;;) { + auto chunk = p->read_output(); + if (!chunk || chunk->empty()) + break; + out_text += *chunk; + } + }); + std::jthread err_reader([&] { + for (;;) { + auto chunk = p->read_error(); + if (!chunk || chunk->empty()) + break; + } + }); + std::uintmax_t offset = 0; + auto poll = [&] { + std::ifstream in(report, std::ios::binary); + if (!in) + return; + in.seekg(static_cast(offset)); + std::string text{std::istreambuf_iterator(in), {}}; + const auto last = text.rfind('\n'); + if (last == std::string::npos) + return; + offset += last + 1; + for (const auto& e : records_of(std::string_view(text).substr(0, last + 1))) { + const auto kind = e.value("e", std::string{}); + const auto id = e.value("id", std::string{}); + if (kind == "begin") { + { + std::lock_guard g(m); + j.begun.insert(id); + } + notify("test/case", {{"run", j.run}, + {"program", j.program}, + {"id", id}, + {"event", "begin"}, + {"t_ms", e.value("t", 0.0)}}); + } else if (kind == "end") { + { + std::lock_guard g(m); + j.ended.insert(id); + } + json extra = {{"ms", e.value("ms", 0.0)}}; + for (const char* k : {"reason", "checks"}) + if (e.contains(k)) + extra[k] = e[k]; + case_end(j, id, e.value("status", std::string("fail")), extra); + } else if (kind == "output" || kind == "failure") { + json n = e; + n.erase("e"); + n["run"] = j.run; + n["program"] = j.program; + n["event"] = kind; + notify("test/case", n); + } + } + }; + std::optional st; + for (;;) { + auto w = p->wait_for(std::chrono::milliseconds(50)); + poll(); + if (!w) + break; + if (*w) { + st = **w; + break; + } + } + out_reader.join(); + err_reader.join(); + poll(); + { + std::lock_guard g(m); + j.proc = nullptr; + } + const int code = st ? st->shell_code() : -1; + // P5.4: a case that began and did not end failed, or was cancelled. + std::set begun; + { + std::lock_guard g(m); + begun = j.begun; + } + for (const auto& id : begun) + if (!j.ended.contains(id)) { + const bool cancelled = j.cancelled || j.cancelled_cases.contains(id) || j.requeue; + case_end(j, id, cancelled ? "cancelled" : "fail", + {{"reason", cancelled ? std::string("cancelled") + : std::format("the program ended with status {} " + "before the case ended", + code)}}); + std::lock_guard g(m); + j.ended.insert(id); + } + const double ms = + std::chrono::duration(std::chrono::steady_clock::now() - started) + .count(); + std::string status = j.cancelled ? "cancelled" : code == 0 ? "pass" : "fail"; + json prog = {{"run", j.run}, + {"program", j.program}, + {"status", status}, + {"exit", code}, + {"ms", ms}}; + if (!j.how.protocol || (code != 0 && j.begun.empty())) + prog["output"] = out_text; + if (!j.how.protocol) { + // A program of no protocol is one case, judged by its exit status (P1.1). + case_end(j, j.program, + j.cancelled ? "cancelled" + : code == 0 ? "pass" + : "fail", + {{"ms", ms}}); + } + notify("test/program", prog); + std::filesystem::remove_all(dir, ec); + } + + // After a job: the cases it was to run that did not begin, cancelled with it or run again. + void settle(const std::shared_ptr& j) { + std::lock_guard g(m); + std::vector rest; + for (const auto& c : j->cases) + if (!j->begun.contains(c) && !j->cancelled_cases.contains(c)) + rest.push_back(c); + auto& r = runs[j->run]; + if (j->requeue && !j->cancelled && !r.cancelled && !rest.empty()) + enqueue(j->run, j->program, j->how, rest); + else if (j->cancelled || r.cancelled) + for (const auto& c : rest) { + r.counts["cancelled"]++; + r.cases[j->program + "/" + c] = "cancelled"; + notify("test/case", {{"run", j->run}, + {"program", j->program}, + {"id", c}, + {"event", "end"}, + {"status", "cancelled"}, + {"reason", "cancelled"}}); + } + std::erase(running, j); + --r.outstanding; + maybe_end(j->run); + } + + void work() { + for (;;) { + std::shared_ptr j; + { + std::unique_lock l(m); + cv.wait(l, [&] { return stopping || !pending.empty(); }); + if (pending.empty()) + return; + j = pending.front(); + pending.pop_front(); + running.push_back(j); + } + execute(*j); + settle(j); + } + } +}; + +session::session(notify_fn notify) : d_(std::make_unique()) { + d_->notify = std::move(notify); + const auto self = os::process::self(); + d_->tmp = std::filesystem::temp_directory_path() / + std::format("mtest-session-{}-{}", self ? self->id : 0, self ? self->started_ns : 0); + unsigned n = std::max(2u, std::thread::hardware_concurrency()); + for (unsigned i = 0; i < n; ++i) + d_->workers.emplace_back([this] { d_->work(); }); +} + +session::~session() { + shutdown(); +} + +namespace { +run_state state_of(const json& params) { + run_state r; + r.project = params.value("project", std::filesystem::current_path().string()); + const auto options = params.value("options", json::object()); + // `mcpp`: a program, or a program and the arguments before `test`; else MTEST_MCPP, else mcpp. + if (const auto m = options.value("mcpp", json()); m.is_string()) + r.mcpp = {m.get()}; + else if (m.is_array()) + r.mcpp = m.get>(); + if (r.mcpp.empty() || r.mcpp.front().empty()) + r.mcpp = {os::env::get("MTEST_MCPP").value_or("mcpp")}; + r.args = options.value("args", std::vector{}); + r.isolate = options.value("isolation", std::string{}) == "case"; + return r; +} + +std::vector>> selection_of(const json& params) { + std::vector>> out; + for (const auto& s : params.value("selection", json::array())) + out.emplace_back(s.value("program", std::string{}), + s.value("cases", std::vector{})); + return out; +} +} // namespace + +json session::plan(const json& params) { + auto r = state_of(params); + std::vector args = {"test", "--cases", "--message-format", "json"}; + if (auto f = params.value("filter", std::string{}); !f.empty()) + args.push_back(f); + std::lock_guard b(d_->build_m); + auto [status, records] = d_->call_mcpp(r, args); + json programs = json::array(); + for (const auto& rec : records) + if (rec.contains("test")) + programs.push_back(rec); + if (programs.empty() && status != 0) + return {{"error", std::format("mcpp test --cases exited {}", status)}}; + return {{"programs", programs}}; +} + +json session::run(const json& params) { + auto r = state_of(params); + auto selection = selection_of(params); + std::string id; + { + std::lock_guard g(d_->m); + if (d_->stopping) + return {{"error", "the session is shutting down"}}; + id = std::format("r{}", d_->next_run++); + r.outstanding = 1; // its build + d_->runs[id] = std::move(r); + } + { + std::lock_guard g(d_->m); + d_->planners.emplace_back([this, id, selection] { d_->build_and_enqueue(id, selection); }); + } + return {{"run", id}}; +} + +json session::add(const json& params) { + const auto id = params.value("run", std::string{}); + auto selection = selection_of(params); + { + std::lock_guard g(d_->m); + auto it = d_->runs.find(id); + if (it == d_->runs.end()) + return {{"error", std::format("no run {}", id)}}; + if (it->second.ended || it->second.cancelled) + return {{"error", std::format("run {} has ended", id)}}; + ++it->second.outstanding; // its build + d_->planners.emplace_back([this, id, selection] { d_->build_and_enqueue(id, selection); }); + } + return {{"run", id}}; +} + +json session::cancel(const json& params) { + const auto id = params.value("run", std::string{}); + const auto program = params.value("program", std::string{}); + const auto one = params.value("case", std::string{}); + std::lock_guard g(d_->m); + auto it = d_->runs.find(id); + if (it == d_->runs.end()) + return {{"error", std::format("no run {}", id)}}; + auto& r = it->second; + if (program.empty() && one.empty()) + r.cancelled = true; + auto matches = [&](const job& j) { + return j.run == id && (program.empty() || j.program == program); + }; + // What has not started: removed, or the case taken out of its selection. + for (auto p = d_->pending.begin(); p != d_->pending.end();) { + auto& j = **p; + if (!matches(j)) { + ++p; + continue; + } + std::vector dropped; + if (one.empty()) + dropped = j.cases.empty() ? std::vector{} : j.cases; + else if (std::erase(j.cases, one) > 0) + dropped = {one}; + else { + ++p; + continue; + } + for (const auto& c : dropped) { + r.counts["cancelled"]++; + r.cases[j.program + "/" + c] = "cancelled"; + d_->notify("test/case", {{"run", id}, + {"program", j.program}, + {"id", c}, + {"event", "end"}, + {"status", "cancelled"}, + {"reason", "cancelled"}}); + } + if (one.empty() || j.cases.empty()) { + d_->notify("test/program", + {{"run", id}, {"program", j.program}, {"status", "cancelled"}}); + p = d_->pending.erase(p); + --r.outstanding; + } else { + ++p; + } + } + // What runs: its process tree ended. A case shared with others ends the process; the cases it + // was to run that did not begin run in another. + for (const auto& jp : d_->running) { + auto& j = *jp; + if (!matches(j)) + continue; + if (one.empty()) { + j.cancelled = true; + } else { + const bool selected = + j.cases.empty() || std::ranges::find(j.cases, one) != j.cases.end(); + if (!selected || j.ended.contains(one)) + continue; + j.cancelled_cases.insert(one); + if (j.cases.size() == 1) + j.cancelled = true; + else + j.requeue = true; + } + if (j.proc) + j.proc->kill(); + } + d_->maybe_end(id); + return {{"run", id}, + {"cancelled", one.empty() ? (program.empty() ? "run" : "program") : "case"}}; +} + +json session::status(const json& params) { + std::lock_guard g(d_->m); + json runs = json::object(); + const auto only = params.value("run", std::string{}); + for (const auto& [id, r] : d_->runs) { + if (!only.empty() && id != only) + continue; + runs[id] = {{"status", r.ended ? verdict(r.cancelled, r.counts) : std::string("running")}, + {"counts", r.counts}, + {"cases", r.cases}}; + } + return {{"runs", runs}}; +} + +void session::drain() { + std::unique_lock l(d_->m); + d_->cv.wait(l, [&] { + return d_->stopping || + std::ranges::all_of(d_->runs, [](const auto& r) { return r.second.ended; }); + }); +} + +void session::shutdown() { + if (!d_) + return; + { + std::lock_guard g(d_->m); + if (d_->stopping) + return; + d_->stopping = true; + for (auto& [id, r] : d_->runs) + r.cancelled = true; + for (const auto& j : d_->running) { + j->cancelled = true; + if (j->proc) + j->proc->kill(); + } + d_->pending.clear(); + } + d_->cv.notify_all(); + std::vector planners; + { + std::lock_guard g(d_->m); + planners = std::move(d_->planners); + } + planners.clear(); + d_->workers.clear(); + std::error_code ec; + std::filesystem::remove_all(d_->tmp, ec); +} + +} // namespace mcpplibs::mtest::engine diff --git a/engine/src/session.cppm b/engine/src/session.cppm new file mode 100644 index 0000000..6c00f1f --- /dev/null +++ b/engine/src/session.cppm @@ -0,0 +1,39 @@ +// The test session of the mcpp test spec (docs/specs/test.md O9): a client holds one session and +// runs, adds and cancels cases while it does. The session builds nothing: it asks mcpp for the +// programs' launches (B5, `mcpp test --no-run --message-format json`), starts each program with the +// protocol's variables, and states what the program reports as it happens. `mtest serve` answers +// the session's methods beside its commands. +export module mcpplibs.mtest.engine.session; + +import std; +import mcpplibs.mtest.engine.model; + +export namespace mcpplibs::mtest::engine { + +class session { +public: + // A notification: its method (`test/build`, `test/case`, `test/program`, `test/run`) and its + // parameters. Called from the session's threads. + using notify_fn = std::function; + + explicit session(notify_fn notify); + ~session(); + session(const session&) = delete; + session& operator=(const session&) = delete; + + // Each answers a request's parameters with its result, or an `error` member naming why not. + json plan(const json& params); + json run(const json& params); + json add(const json& params); + json cancel(const json& params); + json status(const json& params); + // Every run ended as it does: what the session's client asked for is done. + void drain(); + // Every run cancelled, the threads ended. + void shutdown(); +private: + struct impl; + std::unique_ptr d_; +}; + +} // namespace mcpplibs::mtest::engine diff --git a/engine/tests/engine.cpp b/engine/tests/engine.cpp index 5da76f6..5f45555 100644 --- a/engine/tests/engine.cpp +++ b/engine/tests/engine.cpp @@ -93,6 +93,15 @@ int main() { expect(eq(only["cases"].get(), 1)); expect(eq(only["status"].get(), std::string("pass"))); }; + "a program only a runner starts is listed and run through it (--run-under)"_test[kind::e2e] = + [] { + // Not executable by itself, as a program of another system is not on this one: both the + // listing (P3) and the run start it under the runner. + auto root = project({{"t.cross", script({"a:pass", "b:pass"})}}); + auto j = json_of(mtest(root, {"run", "--json", "--run-under", "bash", "./t.cross"})); + expect(eq(j["cases"].get(), 2)); + expect(eq(j["status"].get(), std::string("pass"))); + }; "a case a crash ended fails with the program's status (P5.4)"_test[kind::e2e] = [] { auto root = project({{"t.sh", script({"dies:crash"}, "crash() { exit 9; }\n")}}); auto j = json_of(mtest(root, {"run", "--json", "bash t.sh"})); @@ -236,6 +245,58 @@ int main() { expect(!replies[3]["result"]["isError"].get()); expect(replies[4]["result"]["data"].contains("conclusion")); }; + "a test session runs the cases asked for, from the launches mcpp gives (spec O9)"_test + [kind::e2e] = [] { + // mcpp's part, B5: the program built, and how it starts. + auto root = + project({{"t.sh", script({"a:pass", "b:fail"})}, + {"mcpp.sh", "echo '{\"test\":\"t\",\"status\":\"built\",\"protocol\":1," + "\"launch\":{\"argv\":[\"bash\",\"t.sh\"],\"env\":{},\"cwd\":" + "\"\",\"via\":\"\"}}'\n"}}); + const json run = {{"jsonrpc", "2.0"}, + {"id", 1}, + {"method", "test/run"}, + {"params", + {{"project", root.generic_string()}, + {"options", {{"mcpp", {"bash", "mcpp.sh"}}, {"isolation", "case"}}}, + {"selection", {{{"program", "t"}, {"cases", {"a", "b"}}}}}}}}; + auto r = mtest(root, {"serve"}, run.dump() + "\n"); + std::vector notes; + json reply; + std::string_view rest = r.out; + while (!rest.empty()) { + auto nl = rest.find('\n'); + auto j = json::parse(rest.substr(0, nl), nullptr, false); + rest = nl == std::string_view::npos ? std::string_view{} : rest.substr(nl + 1); + if (j.is_discarded()) + continue; + if (j.contains("id")) + reply = j; + else + notes.push_back(j); + } + expect(fatal(reply.contains("result"))) << r.out << r.err; + const auto id = reply["result"]["run"].get(); + auto count = [&](std::string_view method, std::string_view key, + std::string_view value) { + return std::ranges::count_if(notes, [&](const json& n) { + return n.value("method", "") == method && + n["params"].value(std::string(key), "") == value; + }); + }; + expect(eq(count("test/build", "event", "end"), 1)) << r.out; + // A process per case: two programs, each case's end. + expect(eq(count("test/program", "program", "t"), 2)) << r.out; + expect(eq(count("test/case", "event", "end"), 2)) << r.out; + const auto end = std::ranges::find_if(notes, [](const json& n) { + return n.value("method", "") == "test/run"; + }); + expect(fatal(end != notes.end())) << r.out; + expect(eq((*end)["params"]["run"].get(), id)); + expect(eq((*end)["params"]["status"].get(), std::string("fail"))); + expect(eq((*end)["params"]["counts"].value("pass", 0), 1)); + expect(eq((*end)["params"]["counts"].value("fail", 0), 1)); + }; "every answer carries a conclusion"_test[kind::e2e] = [] { auto root = project({{"t.sh", script({"a:pass"})}}); (void)mtest(root, {"run", "bash t.sh"}); diff --git a/lib/docs/writing-tests.md b/lib/docs/writing-tests.md index a4a95d3..f386727 100644 --- a/lib/docs/writing-tests.md +++ b/lib/docs/writing-tests.md @@ -53,7 +53,7 @@ E2E cli{"cli", resource{"serial:home"}, timeout{std::chrono::minutes{2}}, [] { / The entry enters the child before any case; the copy runs no case and reports nothing. The files of one directory are then one program (mcpp `[test] programs = "directory"`) however many of them start copies. -- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.5.2"`. +- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.6.0"`. - Identifiers are stable: a selection, a history and a shard name a case by them. ## Expectations diff --git a/lib/mcpp.toml b/lib/mcpp.toml index 3668ffc..8300a35 100644 --- a/lib/mcpp.toml +++ b/lib/mcpp.toml @@ -17,7 +17,7 @@ [package] namespace = "mcpplibs" name = "mtest" -version = "0.5.2" +version = "0.6.0" description = "The C++ test library of mtest: cases, expectations, fixtures, bench, conformance and end-to-end helpers, with no macro and no header, speaking the mtest protocol" license = "Apache-2.0" authors = ["mcpp-community"] @@ -44,6 +44,10 @@ main = "src/entry/main.cpp" test-protocol = 1 test-main = true test-guide = "docs/writing-tests.md" +# What the library implements of the mcpp test spec's optional capabilities (mcpp +# docs/specs/test.md, B6), and the session a client starts (O9, `mtest serve`). +test-capabilities = ["check-details", "metrics", "scheduling", "fixtures", "usage", "output", "session"] +test-session = ["mtest", "serve"] [dependencies] # A range, not one release: the library is in the test graph of every package diff --git a/lib/src/mtest.cpp b/lib/src/mtest.cpp index 3e9443a..691e4db 100644 --- a/lib/src/mtest.cpp +++ b/lib/src/mtest.cpp @@ -30,9 +30,23 @@ struct running_case { bool case_failed = false; // a case it declared failed std::optional dir; std::map> metrics; // name -> value, unit - std::size_t declared = 0; // the cases it declared that ran + std::size_t declared = 0; // the cases it declared that ran + std::size_t declared_failed = 0; // of those, the ones that failed + // Its failures as a person reads them, printed under its line when it ends. + std::vector failures; + std::uint64_t held_before = 0; // the checks held in the program when it began + bool stopped = false; // a fatal expectation ended it + // SPEC-017 O8: what it writes, kept while it runs under a runner and stated as `output` + // events when it ends; a case it declares keeps its own, and this one's resumes after it. + std::optional out_capture, err_capture; + std::string out_text, err_text; + bool capturing = false; }; +// The expectations that held, in the whole program: a case's share is the difference over its +// run. Counted without the lock, as a passing check is the common one. +std::atomic g_held{0}; + struct declared_suite { std::string name; std::source_location where; @@ -55,6 +69,8 @@ struct state { std::vector suites; std::size_t suites_run = 0; std::map counts; + // Of `counts`, the cases that declared cases: the console's tally is of the cases that check. + std::map container_counts; std::set clauses; bool any_failed = false; bool program_written = false; @@ -308,7 +324,7 @@ void write_program() { {"protocol", 1}, {"level", 3}, {"framework", "mcpplibs.mtest"}, - {"version", "0.5.2"}, + {"version", "0.6.0"}, {"isolation", {"process", "case"}}, {"guide", "https://github.com/mcpp-community/mtest/blob/main/lib/docs/writing-tests.md"}}); } @@ -378,11 +394,92 @@ std::string sanitized(std::string_view id) { return out; } +// SPEC-017 O8: a case's own output. What the streams hold is written first, so that what the +// case writes is all that the capture keeps. +void flush_streams() { + std::cout.flush(); + std::cerr.flush(); + std::fflush(nullptr); +} + +void begin_capture(running_case& c) { + flush_streams(); + if (auto o = os::stdio::capture::begin(os::stdio::stream::output)) + c.out_capture.emplace(std::move(*o)); + if (auto e = os::stdio::capture::begin(os::stdio::stream::error)) + c.err_capture.emplace(std::move(*e)); +} + +// The capture ended and its text kept; the text is written to the stream it was taken from, so +// a person at the terminal reads what the case wrote where it wrote it. +void end_capture(running_case& c) { + flush_streams(); + if (c.err_capture) { + auto t = c.err_capture->text(); + c.err_capture.reset(); + if (!t.empty()) { + (void)os::stdio::write_error(t); + c.err_text += t; + } + } + if (c.out_capture) { + auto t = c.out_capture->text(); + c.out_capture.reset(); + if (!t.empty()) { + (void)os::stdio::write_output(t); + (void)os::stdio::flush_output(); + c.out_text += t; + } + } +} + void say(std::string_view line) { std::println("{}", line); std::cout.flush(); } +// The statement at `line` of `file` and the lines it continues on, as one line: what a check +// without operands states. Empty when the source is not readable here. +std::string statement_at(const std::string& file, std::uint_least32_t line) { + std::ifstream in(os::fs::from_utf8(file), std::ios::binary); + if (!in || line == 0) + return {}; + std::string text, one; + for (std::uint_least32_t n = 1; n < line && std::getline(in, one); ++n) {} + int depth = 0; + for (int taken = 0; taken < 8 && std::getline(in, one); ++taken) { + if (one.ends_with('\r')) + one.pop_back(); + const auto first = one.find_first_not_of(" \t"); + if (first == std::string::npos) + continue; + if (!text.empty()) + text += ' '; + text += one.substr(first); + for (char c : one) + depth += c == '(' ? 1 : c == ')' ? -1 : 0; + if (depth <= 0 && text.find(';') != std::string::npos) + break; + } + while (!text.empty() && (text.back() == ' ' || text.back() == '\t')) + text.pop_back(); + if (text.size() > 240) + text = text.substr(0, 237) + "..."; + return text; +} + +// A file under the working directory by its name there; any other by its own. +std::string shown_path(const std::string& file) { + std::error_code ec; + const auto p = os::fs::from_utf8(file); + if (!p.is_absolute()) + return file; + const auto rel = std::filesystem::proximate(p, std::filesystem::current_path(ec), ec); + if (ec || rel.empty() || rel.begin()->string() == "..") + return file; + return os::fs::generic_utf8(rel); +} + void record_failure(std::string_view message, const std::source_location& where, const comparison* value, bool is_fatal) { auto& s = the(); @@ -394,18 +491,55 @@ void record_failure(std::string_view message, const std::source_location& where, else s.any_failed = true; const std::string file = where.file_name(); + const bool compared = value && !value->expected.empty(); std::string text = std::string(message); if (value && !value->expression.empty()) text = text.empty() ? value->expression : value->expression + ": " + text; if (text.empty()) text = "the expectation did not hold"; - say(std::format(" {}:{}: {}", file, where.line(), text)); + // As a person reads it: where, the statement, the two sides or that it did not hold, the + // message the check added, and whether the case went on. + const std::string source = value ? statement_at(file, where.line()) : std::string{}; + std::string block = std::format("{}:{}", shown_path(file), where.line()); + if (!source.empty()) + block += "\n " + source; + if (compared) { + block += + std::format("\n expected {}\n actual {}", value->expected, value->actual); + if (!message.empty()) + block += std::format("\n note {}", message); + } else if (value) { + block += source.empty() || !message.empty() || !value->expression.empty() + ? "\n " + text + : std::string("\n did not hold"); + } else { + block += "\n " + text; + } + if (is_fatal) { + block += "\n fatal: the rest of the case did not run"; + if (in_case) + s.stack.back().stopped = true; + } + if (in_case) + s.stack.back().failures.push_back(std::move(block)); + else { + std::string shown; + for (std::string_view rest = block; !rest.empty();) { + const auto nl = rest.find('\n'); + shown += " " + std::string(rest.substr(0, nl)) + "\n"; + rest = nl == std::string_view::npos ? std::string_view{} : rest.substr(nl + 1); + } + shown.pop_back(); + say(shown); + } event ev = {{"e", "failure"}, {"id", id}, {"file", file}, {"line", where.line()}, {"message", text}}; - if (value && !value->expected.empty()) { + if (!source.empty()) + ev["source"] = source; + if (compared) { ev["expected"] = value->expected; ev["actual"] = value->actual; } @@ -502,8 +636,10 @@ expectation::expectation(comparison value, bool is_fatal, std::source_location w : held_(value.held), fatal_(is_fatal), value_(std::move(value)), where_(where) {} expectation::~expectation() noexcept(false) { - if (held_) + if (held_) { + g_held.fetch_add(1, std::memory_order_relaxed); return; + } record_failure(message_, where_, &value_, fatal_); if (fatal_ && std::uncaught_exceptions() == 0) throw fatal_signal{}; @@ -581,19 +717,55 @@ void test::operator=(body b) { } if (!selected(id)) return; - auto finish = [&](std::string_view status, double ms, std::string_view reason) { + auto finish = [&](std::string_view status, double ms, std::string_view reason, + const running_case* done = nullptr) { s.counts[std::string(status)]++; + const bool container = done && done->declared > 0; + if (container) + s.container_counts[std::string(status)]++; if (status == "fail" || status == "xpass") s.any_failed = true; event ev = {{"e", "end"}, {"id", id}, {"status", status}, {"ms", ms}}; if (!reason.empty()) ev["reason"] = reason; + // SPEC-017 O3: of the checks the case ran, those that held and those that did not. + if (done && status != "skip") + ev["checks"] = {{"held", g_held.load(std::memory_order_relaxed) - done->held_before}, + {"failed", done->failures.size()}}; write_event(ev); const char* mark = status == "pass" ? " ok " : status == "skip" ? " skip" : status == "xfail" ? " xfail" : " FAIL"; - say(std::format("{} {}{}{}", mark, id, reason.empty() ? "" : " -- ", reason)); + // What it amounts to: of a case that declared cases, how many of them failed; of one that + // checks, how many of its checks failed, and whether a fatal one ended it. + std::string tally; + if (done && status != "skip") { + if (container && done->declared_failed > 0) + tally = std::format("{} of {} cases failed", done->declared_failed, done->declared); + const std::size_t failed = done->failures.size(); + if (failed > 0) { + const auto held = g_held.load(std::memory_order_relaxed) - done->held_before; + if (!tally.empty()) + tally += "; "; + tally += std::format("{} of {} checks failed", failed, failed + held); + if (done->stopped) + tally += ", stopped"; + } + } + say(std::format("{} {}{}{}{}", mark, id, reason.empty() ? "" : " -- ", reason, + tally.empty() ? "" : " (" + tally + ")")); + if (done) + for (const auto& block : done->failures) { + std::string shown; + for (std::string_view rest = block; !rest.empty();) { + const auto nl = rest.find('\n'); + shown += " " + std::string(rest.substr(0, nl)) + "\n"; + rest = nl == std::string_view::npos ? std::string_view{} : rest.substr(nl + 1); + } + shown.pop_back(); + say(shown); + } }; const bool own = reported(id); if (attrs.skip) { @@ -634,7 +806,15 @@ void test::operator=(body b) { write_event(begun); if (!s.stack.empty() && !s.stack.back().scope) ++s.stack.back().declared; + const bool capturing = s.protocol && !s.listing && own; + if (capturing && !s.stack.empty() && s.stack.back().capturing) + end_capture(s.stack.back()); // the declaring case's output so far is its own s.stack.push_back({id, false, attrs}); + s.stack.back().held_before = g_held.load(std::memory_order_relaxed); + if (capturing) { + s.stack.back().capturing = true; + begin_capture(s.stack.back()); + } const auto started = clock::now(); std::string status = "pass"; std::string reason; @@ -661,8 +841,16 @@ void test::operator=(body b) { const double ms = std::chrono::duration(clock::now() - started).count(); if (own) report_use(id, use, level); + if (capturing) + end_capture(s.stack.back()); running_case done = std::move(s.stack.back()); s.stack.pop_back(); + if (capturing) { + if (!done.out_text.empty()) + write_event({{"e", "output"}, {"id", id}, {"stream", "out"}, {"text", done.out_text}}); + if (!done.err_text.empty()) + write_event({{"e", "output"}, {"id", id}, {"stream", "err"}, {"text", done.err_text}}); + } if (done.dir) { std::error_code ec; std::filesystem::remove_all(*done.dir, ec); @@ -684,9 +872,14 @@ void test::operator=(body b) { reason = "expected to fail: " + *attrs.expected_failure; } } - if (status == "fail" && !s.stack.empty() && !s.stack.back().scope) + if (status == "fail" && !s.stack.empty() && !s.stack.back().scope) { s.stack.back().case_failed = true; - finish(status, ms, reason); + ++s.stack.back().declared_failed; + } + finish(status, ms, reason, &done); + // The declaring case's capture resumes, after this case's own lines. + if (capturing && !s.stack.empty() && s.stack.back().capturing) + begin_capture(s.stack.back()); } void Suite::add(std::string name, std::source_location where, attributes attrs, @@ -810,9 +1003,12 @@ int report() { const bool failed = s.any_failed || missing; write_event({{"e", "done"}, {"status", failed ? "fail" : "pass"}, {"counts", s.counts}}); if (!s.listing) { + // The cases that check; a case that declared cases is told by them. auto n = [&](const char* k) { auto it = s.counts.find(k); - return it == s.counts.end() ? 0 : it->second; + auto c = s.container_counts.find(k); + return (it == s.counts.end() ? 0 : it->second) - + (c == s.container_counts.end() ? 0 : c->second); }; say(std::format("{} passed, {} failed, {} skipped{}{}", n("pass") + n("xfail"), n("fail") + n("xpass"), n("skip"), diff --git a/lib/tests/layout.cpp b/lib/tests/layout.cpp new file mode 100644 index 0000000..585652f --- /dev/null +++ b/lib/tests/layout.cpp @@ -0,0 +1,45 @@ +// What a person reads of a failing case: its line first, then each failure under it -- where it +// is, the statement, the two sides or that it did not hold, and whether the case went on. A child +// runs the cases, as a program that is not under a runner does, and its output is read. +import std; +import mcpplibs.mtest; +import mcpplibs.mtest.e2e; + +using namespace mcpplibs::mtest; + +child failing{"failing", [](std::vector) { + "values"_test = [] { + const std::string got = "second"; + expect(eq(got, std::string("first"))); + expect(got == "first"); + expect(got.size() == 6u); + }; + "stops"_test = [] { + expect(fatal(false)); + expect(true); + }; + return 0; + }}; + +UTest layout{"layout", [] { + "a failure is shown under its case with the statement and both sides"_test = [] { + auto o = e2e::run({.argv = child_command("failing", {})}); + expect(fatal(o.started)); + const auto at = [&](std::string_view s) { return o.out.find(s); }; + const auto header = at("FAIL values (2 of 3 checks failed)"); + expect(fatal(header != std::string::npos)) << o.out; + expect(at("expect(eq(got, std::string(\"first\")));") > header) << o.out; + expect(at("expected \"first\"") > header) << o.out; + expect(at("actual \"second\"") > header) << o.out; + expect(at("expect(got == \"first\");") > header) << o.out; + expect(at("did not hold") > header) << o.out; + }; + "a fatal failure says the rest of its case did not run"_test = [] { + auto o = e2e::run({.argv = child_command("failing", {})}); + expect(fatal(o.started)); + const auto header = o.out.find("FAIL stops (1 of 1 checks failed, stopped)"); + expect(fatal(header != std::string::npos)) << o.out; + expect(o.out.find("fatal: the rest of the case did not run") > header) + << o.out; + }; + }}; diff --git a/lib/tests/output.cpp b/lib/tests/output.cpp new file mode 100644 index 0000000..d1aad6a --- /dev/null +++ b/lib/tests/output.cpp @@ -0,0 +1,101 @@ +// Under a runner, a case states what it wrote and how its checks went (mcpp test spec O3, O8): +// each case's output as `output` events, a case's own apart from the cases it declares, and its +// `end` with the checks that held and failed. A child runs the cases with the protocol's variables +// set, as a runner would, and its report is read. +import std; +import mcpplibs.mtest; +import mcpplibs.mtest.e2e; +import nlohmann.json; +import mcpplibs.os; + +using namespace mcpplibs::mtest; + +child talker{"talker", [](std::vector) { + "speaks"_test = [] { + std::println("said by speaks"); + std::println(std::cerr, "warned by speaks"); + expect(true); + expect(eq(1, 2)); + }; + "outer"_test = [] { + std::println("before inner"); + "inner"_test = [] { std::println("said by inner"); }; + std::println("after inner"); + }; + return 0; + }}; + +namespace { +// Where the streams cannot be rearranged (an ok image), a case's output is not kept and the +// program's output is the program's (spec O8's fallback). +bool streams_kept() { + auto c = mcpplibs::os::stdio::capture::begin(mcpplibs::os::stdio::stream::error); + if (!c) + return false; + (void)c->text(); + return true; +} + +std::vector run_talker(std::string& out) { + const auto report = temp_dir() / "report.ndjson"; + auto o = e2e::run({.argv = child_command("talker", {}), + .env = {{"MTEST_PROTOCOL", "1"}, + {"MTEST_MODE", "run"}, + {"MTEST_REPORT", report.string()}, + {"MTEST_TMPDIR", temp_dir().string()}}}); + out = o.out; + std::vector events; + std::ifstream in(report); + for (std::string line; std::getline(in, line);) + if (auto e = nlohmann::json::parse(line, nullptr, false); e.is_object()) + events.push_back(std::move(e)); + return events; +} + +std::string output_of(const std::vector& events, std::string_view id, + std::string_view stream) { + std::string text; + for (const auto& e : events) + if (e.value("e", "") == "output" && e.value("id", "") == id && + e.value("stream", "") == stream) + text += e.value("text", ""); + return text; +} +} // namespace + +UTest output{"output", [] { + "a case states what it wrote, on each stream"_test = [] { + if (!streams_kept()) + skip("this system's streams cannot be kept for a case"); + std::string out; + const auto events = run_talker(out); + expect( + eq(output_of(events, "speaks", "out"), std::string("said by speaks\n"))); + expect( + eq(output_of(events, "speaks", "err"), std::string("warned by speaks\n"))); + // A person at the terminal still reads it. + expect(contains(out, "said by speaks")) << out; + }; + "a declared case's output is its own, and the declaring case keeps the rest"_test = + [] { + if (!streams_kept()) + skip("this system's streams cannot be kept for a case"); + std::string out; + const auto events = run_talker(out); + expect(eq(output_of(events, "outer/inner", "out"), + std::string("said by inner\n"))); + expect(eq(output_of(events, "outer", "out"), + std::string("before inner\nafter inner\n"))); + }; + "a case's end states the checks that held and failed"_test = [] { + std::string out; + const auto events = run_talker(out); + const nlohmann::json* end = nullptr; + for (const auto& e : events) + if (e.value("e", "") == "end" && e.value("id", "") == "speaks") + end = &e; + expect(fatal(end != nullptr)); + expect(eq((*end)["checks"].value("held", -1), 1)); + expect(eq((*end)["checks"].value("failed", -1), 1)); + }; + }}; diff --git a/manifest/mcpp.toml b/manifest/mcpp.toml index 0181856..6921d76 100644 --- a/manifest/mcpp.toml +++ b/manifest/mcpp.toml @@ -8,7 +8,7 @@ [package] namespace = "mcpplibs.mtest" name = "manifest" -version = "0.5.2" +version = "0.6.0" description = "The static case manifest of mtest: the cases a source file declares, read without building it" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/mcpp.toml b/mcpp.toml index cf1b623..5e66d8b 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -28,4 +28,4 @@ toolchain = "llvm@23.1.3" toolchain = "llvm@23.1.3" [workspace.target.'cfg(os = "openkal")'] -platform = "openkal-platform@0.24.4" +platform = "openkal-platform@0.25.0" diff --git a/spec/protocol.md b/spec/protocol.md index 6d15d31..417a98f 100644 --- a/spec/protocol.md +++ b/spec/protocol.md @@ -1,173 +1,48 @@ # The mtest test protocol, version 1 -**Status:** version 1, 2026-11-08; P3.3, P3.4, P6.5 and the `fixture` event added 2026-11-08; P6.6, P7.3, P10, P11, P12 and the `usage` and `component` events added 2026-11-09 (P9.2). -**Scope:** how a test program and a runner talk. A runner is anything that starts test programs and reads their results: the mtest engine, mcpp (`mcpp test`), an IDE, a CI step. A test program is any program, in any language. +**Status:** the normative text is the mcpp test spec, SPEC-017 v2.0 +([mcpp docs/specs/test.md](https://github.com/mcpp-community/mcpp/blob/main/docs/specs/test.md)), +from mtest 0.6.0 (2026-11-10). The spec keeps this protocol's clause numbers, P1 to P12, so a test, +a conformance report or a document that names `P3.2` names the same clause there; it adds the base +capabilities B1–B6 that every runner and framework implements and the optional capabilities O1–O9 +that a framework or its tools declare. -This document is normative. The words *must*, *must not* and *may* have their usual meaning. Each clause has a number; tests and conformance reports name clauses by number (`P3.2`). +**Scope of this file:** what mtest implements of the spec, part by part. A runner or a client relies +on the spec; this list says which of its clauses mtest's parts answer, and since which release. -## P1. Levels +## The library (`mcpplibs.mtest`) -| Level | What the program does | What the runner obtains | +| Clause | What the library does | Since | |---|---|---| -| L0 | nothing of this protocol | one result per program, from its exit status | -| L1 | P3, P4, P5 | the case list, the selection of cases, a result, a location and a message per case | -| L2 | L1 and P6 | metrics: samples, units, bench verdicts, conformance clauses | -| L3 | L2 and P7 | resources and artifacts per case: scheduling and affected selection | - -P1.1 A program that does not implement the protocol is an L0 program. A runner must run it unchanged and must judge it by its exit status alone: zero passes, anything else fails. - -P1.2 A runner must not pass the protocol's variables (P2) to a program it does not know to implement the protocol, unless it accepts that an L0 program runs completely (P2.4). - -## P2. The handshake - -P2.1 A runner states the protocol in the program's environment. All variables are optional except `MTEST_PROTOCOL`. - -| Variable | Value | -|---|---| -| `MTEST_PROTOCOL` | the protocol version the runner speaks: `1` | -| `MTEST_MODE` | `list` or `run` (default `run`) | -| `MTEST_REPORT` | a file the program appends its events to (P5); absent: no events are written | -| `MTEST_SELECT` | a file of case identifiers, one per line (P4); absent: every case | -| `MTEST_ISOLATION` | `process` (several cases in this process, the default) or `case` (the runner starts the program once per case) | -| `MTEST_TMPDIR` | a directory the program may use for its cases, removed by the runner | -| `MTEST_SEED` | the seed of anything random in the cases, as a decimal number | -| `MTEST_MONITOR` | how closely the program measures its use of the machine: `off`, `basic`, `detailed`, `trace` (P10) | -| `MTEST_MONITOR_SAMPLE` | `trace`: the milliseconds between two samples (P10.3) | -| `MTEST_FIXTURE_` | the endpoint of the run fixture `` (P11.2), its name upper-cased with each character that is not a letter or a digit written `_` | -| `MTEST_SERVE` | the run fixture the program serves instead of running cases (P11.3) | - -P2.2 A program that implements the protocol must not pass these variables to a program it starts (each is removed from the started program's environment), so that a test program started by a test is not taken for the runner's. A test that starts a test program as a runner sets the variables it means for it. - -P2.3 A program states that it implements the protocol in two ways: -- by a host declaration: a build system's manifest names the protocol for the program (mcpp: the test entry's package declares `test-protocol = 1`; mcpp docs/08); -- by a marker: the bytes `mtest-protocol=1` appear in the program's file, followed by a byte that is not a digit. A runner may read the file to find it. A program built from mtest's libraries carries the marker. - -Either statement is a claim the program's report confirms: a program that writes no `program` event is L0 (P2.4), as a file of plain `main` linked with a library that carries the marker is. - -P2.4 A runner that knows neither may still set `MTEST_MODE=list` and `MTEST_REPORT`. The first event (P5.2) tells it the level. When no `program` event arrives, the program is L0, and in `list` mode it has run all of its tests once. - -P2.5 A framework may state where its guide to writing tests is: a file of its package that a writer of a test, a person or an agent, reads before writing one or after one failed. A host declaration names it (mcpp: the entry's `test-guide = ""`; `mcpp test --guide` lists each framework with its guide), and the `program` event may carry `guide`, a URL, for a runner that learns of the framework only when it runs the program. A runner that names a failure may name the guide beside it. The guide states how a file of cases is written, how an expectation is stated and what its failure reports, and how one case is run. - -## P3. Listing - -P3.1 With `MTEST_MODE=list` the program must write one `program` event, then one `case` event per case it would run, then one `done` event, and must exit with status 0 without running any case. - -P3.2 A case identifier is a string of UTF-8. It is stable: the same case has the same identifier in every run and on every system. A program forms it from the names it gives its suites and cases, joined by `/` (for example `fs/a file is read`). Identifiers in one program are distinct. - -P3.3 A case event may name the case's location (`file`, `line`), kind (P8), tags, the clauses it covers, its expected status, and (L3) its resources and artifacts, and how a runner that schedules cases runs it: `timeout`, the seconds it may run; `isolation`, `case` when it runs in a process of its own; `retries`, how many times a failing case runs again, alone; `diagnose`, an object of `env` (variables) and `timeout`, the run a runner makes once after the case fails, whose output it keeps and which judges nothing. A runner that starts a process for one case ends it at the case's `timeout`; one that runs a program whole (mcpp's `mcpp test`) may run every case in it and judges each as reported. A case that names no kind is of the kind the runner gives its program (mtest docs/layout.md). - -P3.4 A case that exists only when another case runs (a case declared inside a case) need not be listed; it appears in the report when it runs, and its `begin` event may name its kind, tags and clauses. Selecting it runs the case that declares it as its scope: that case is reported only when a statement of its own fails. - -## P4. Selection - -P4.1 With `MTEST_SELECT` the program must run exactly the cases whose identifiers are lines of the file, compared byte for byte after removing one trailing `\r`. An identifier that names no case is reported as a `missing` event (P5) and fails the run. - -P4.2 Patterns are the runner's: a runner that offers `fs/*` resolves it against the list (P3) and passes identifiers. The protocol has no pattern syntax, so no escaping either. - -## P5. Reporting - -P5.1 Events are JSON objects, one per line (NDJSON, UTF-8, `\n` after each). The program appends them to `MTEST_REPORT` and must not write anything else to that file. Each line is written whole: a program with several threads must not interleave lines. - -P5.2 The first event is `program`. Every event has the field `e`, its kind. - -| `e` | Fields | When | +| B2: P1–P5, P8, P9 | a program built with it speaks the protocol at level 3 when a runner asks (P2.1), lists (P3), selects (P4) and reports (P5) | 0.1.0 | +| B6 | its entry (`mtest/main`) states `test-protocol = 1`, `test-main`, `test-guide`, `test-capabilities` and `test-session` | 0.1.0; the last two 0.6.0 | +| O3 `check-details` | a failure's `expected`, `actual`, `source` (the statement) and `fatal`; a case's `checks` (`held`, `failed`) at its `end` | `expected`/`actual` 0.1.0; the rest 0.6.0 | +| O4 `metrics` | `metric` events; bounds judged at the case's end (P6.5); standard names with their precision (P6.6) | 0.1.0; P6.5 0.4.0; P6.6 0.5.0 | +| O5 `scheduling` | `resources` and `artifacts` (P7); `timeout`, `isolation`, `retries`, `diagnose` (P3.3) in the `case` events | 0.1.0; P3.3's 0.3.0 | +| O6 `fixtures` | `shared` and the `fixture` event; components and run fixtures (P11), served under `MTEST_SERVE` | 0.3.0; P11 0.5.0 | +| O7 `usage` | the program's `usage` at its end, and the cases' use at `MTEST_MONITOR` (P10) | 0.5.0 | +| O8 `output` | each case's output, kept while it runs under a runner and stated as `output` events; a case declared inside another keeps its own. Where the system cannot rearrange the program's streams (an ok image), no output is kept for a case, the spec's fallback | 0.6.0 | + +## The engine (`mtest`) + +| Clause | What the engine does | Since | |---|---|---| -| `program` | `protocol` (1), `level` (1–3), `framework`, `version`, `isolation` (the values of P2.1 the program supports), optionally `guide` (P2.5) | first | -| `case` | `id`, `name`, `file`, `line`, `kind`, `tags`, `clauses`, `expect`, `resources`, `artifacts`, optionally `uses` (P11.2) | in `list` mode, one per case | -| `component` | `name` and `pools` (`{name, capacity}`, P7.3), or `fixture` and `scope` (P11.1) | in `list` mode, one per component and run fixture | -| `begin` | `id`, `t` (milliseconds since the program started), optionally `kind`, `tags`, `clauses` (P3.4) | a case starts | -| `failure` | `id`, `file`, `line`, `message`, and optionally `expected`, `actual` (strings), `fatal` | an expectation does not hold | -| `output` | `id`, `stream` (`out` or `err`), `text` | output a case captured | -| `metric` | `id`, `name`, `value`, `unit`, optionally `samples` (array of numbers), `better` (`lower` or `higher`), `precision` (P6.6) | L2 | -| `usage` | `user_ms`, `sys_ms`, optionally `peak_bytes`, `read_bytes`, `written_bytes` | the program's own use at its end (P10.2) | -| `artifact` | `id`, `path`, `kind` | a file the case produced | -| `fixture` | `id`, `name`, `ms` | an object the case caused to be made for the cases that share it, and how long making it took | -| `end` | `id`, `status`, `ms`, optionally `reason` | a case ends | -| `missing` | `id` | P4.1 | -| `done` | `status`, `counts` (an object of status to number) | last | - -P5.3 A case's `status` is one of `pass`, `fail`, `skip` (with `reason`), `xfail` (failed as expected), `xpass` (passed although expected to fail; a failure of the run), `invalid` (a measurement that cannot be judged; P6.3). - -P5.4 Unknown fields and unknown event kinds must be ignored by a runner. A runner must accept a report that ends without `done` (the program crashed): the cases that have a `begin` without an `end` failed, with the program's exit status as their reason. - -P5.5 The program's exit status is zero exactly when no case failed or passed unexpectedly and no selected case is missing. - -## P6. Metrics (L2) - -P6.1 A metric belongs to a case and has a name unique within it, a value and a unit (`s`, `ms`, `B`, `1` for a ratio or a count). +| B2, runner side | starts programs of any language, by the protocol or by their exit status (P1.1) | 0.1.0 | +| B4 | keeps every run in the store (P12), and reads the runs a host (`mcpp test`) keeps | 0.1.0; the store's form specified 0.5.0 | +| O5 | schedules cases by their resources and history, by pools since 0.5.0; ends a case at its `timeout`, runs `retries` and `diagnose` since 0.3.0 | 0.1.0 | +| O6 | serves run fixtures once per run (P11.2, P11.3) | 0.5.0 | +| O7 | measures a program that reports no use from a process of its own (P10.4) | 0.5.0 | +| O9 `session` | `mtest serve` answers `test/plan`, `test/run`, `test/add`, `test/cancel`, `test/status` and `shutdown`, and states `test/build`, `test/case`, `test/program`, `test/run`; it builds nothing, and starts each program from the launch mcpp gives (B5) | 0.6.0 | -P6.2 A bench case reports its samples. The runner, not the program, decides whether a measurement is valid and whether it passes, by the policy the project declares (pairs, spread bound, limit, whether a measurement may be repeated). +## The adapters and the C ABI -P6.3 A program may itself report `invalid` when it knows its measurement cannot be judged (for example, the machine was too noisy by its own test). A runner must not count an `invalid` case as passed or failed, and its policy decides whether it may be repeated. - -P6.4 A conformance case names in `clauses` the clauses of the specification it covers. A runner forms the matrix of clause × implementation × system from them. - -P6.5 A case may bound its metrics: `bounds`, a list of `{metric, max, min}` (either limit may be absent), in the metric's unit, in its `case` event and, for a case not listed, its `begin` (P3.4). The program judges them as expectations at the case's end: a metric over its `max` or under its `min`, or a bounded metric the case did not report, is a `failure` of the case. A case that declared cases is judged by them. A runner reads the bounds to show a metric beside its limit (mtest's `analyze trend`); a measurement against a baseline stays the runner's (P6.2). - -P6.6 The use of the machine is reported as metrics of standard names, each with its `precision`: `exact` (the case's own, measured at its bounds), `process` (the program's, of which the case is a part), `approx` (estimated), `sampled` (P10.3). - -| Metric | Unit | What | +| Part | Level | Clauses | |---|---|---| -| `cpu.user`, `cpu.sys` | `ms` | processor time in user and system mode | -| `mem.peak` | `B` | the largest resident set | -| `mem.delta` | `B` | the growth of the largest resident set over the case | -| `mem.rss` | `B` | the resident set, sampled | -| `cpu.load` | `1` | the processors in use, sampled | -| `io.read`, `io.write` | `B` | bytes read and written | -| `net.sent`, `net.recv` | `B` | bytes sent and received | -| `alloc.count`, `alloc.bytes` | `1`, `B` | allocations and their bytes | -| `proc.spawned` | `1` | programs started | - -## P7. Resources and artifacts (L3) - -P7.1 `resources` names what a case needs exclusively or in limited number: strings such as `serial:`, `machine` (alone on the machine), `gpu`, `network`, `display`. A runner must not run two cases that hold the same `serial:` group, or a `machine` case with any other, at the same time. - -P7.2 `artifacts` names what a case reads: build targets (`target:`), files and directories relative to the project (`file:`), external programs (`program:`). A runner selects the cases affected by a change through them. - -P7.3 A resource may have a capacity: a pool a component brings (P11.1) states how many cases hold it at once. A runner must not run more holders of a pool than its capacity; a resource with no stated capacity has one. - -## P8. Kinds - -`unit`, `e2e`, `cts` and `bench`. A kind states what is tested; the set a case belongs to (smoke, regression) is the runner's and the project's (mtest docs/sets.md), not the program's. - -## P9. Versions - -P9.1 A runner that speaks version N sets `MTEST_PROTOCOL=N`. A program that implements only lower versions answers with the highest one it implements in its `program` event; the runner then uses that version or treats the program as L0. - -P9.2 Within a version, fields and event kinds are only added. A field changes meaning only in a new version. - -## P10. Use of the machine - -P10.1 A runner states the level in `MTEST_MONITOR`. `off`: nothing is measured. `basic`: each case's processor time (P6.6, `exact`), and the program's own use at its end. `detailed`: also each case's largest resident set and its growth, and the bytes it read and wrote. `trace`: also the processor load and the resident set over each case, sampled. A case may ask for a level higher than the runner's; the higher one applies to it. - -P10.2 A program at a level other than `off` writes one `usage` event before `done`: its processor time, and where the system tells them, its largest resident set and the bytes it read and wrote. - -P10.3 At `trace`, a program samples every `MTEST_MONITOR_SAMPLE` milliseconds (50 when absent) while a case runs and reports each series as one metric whose `samples` are the series and whose `value` is its largest sample, with precision `sampled`. - -P10.4 A runner measures a program that writes no `usage` event from a process of its own that starts the program, where the system accounts a process's children (Linux, macOS), and records the measurement as the engine's. - -## P11. Components and run fixtures - -P11.1 A component is a named part of tests that programs share: the pools of resources it brings (P7.3) and the run fixtures it serves. A program lists each in a `component` event (P5.2). - -P11.2 A run fixture is one object for a whole run, used by the cases of several programs: a sandboxed home, a server, an index. A case states the run fixtures it uses in `uses`. A runner serves a run fixture before the first case that uses it starts and ends it after the last ends, and gives its endpoint to every program of a case that uses it as `MTEST_FIXTURE_`. - -P11.3 A runner serves a run fixture by starting a program that lists it with `MTEST_SERVE` naming it, with its input a pipe and without `MTEST_PROTOCOL`. The program runs no case: it prepares the fixture, writes the line `mtest-ready ` to its output, and ends the fixture when its input ends. A runner may also serve a run fixture by a command its project names. - -P11.4 A run fixture that does not become ready is the machine's failure: the cases that use it end with status `infra` and do not run. - -## P12. The store - -P12.1 A runner may keep its runs in a store, which other runners and tools read and write in this form (mtest's `.mtest`; a host's runner, such as mcpp's `mcpp test`, writes the same): - -| Path | Content | -|---|---| -| `index.ndjson` | one line per run: the run's record without its programs and cases | -| `runs//run.json` | the run's record: `schema` (`mtest.run/1`), `id`, `started`, `finished`, `status`, `system`, `arch`, `machine`, `commit`, `set`, `selection`, `group`, `monitor`, `ms`, `jobs`, `counts` | -| `runs//programs.ndjson` | one line per process of a program: `name`, `argv`, `status`, `exit`, `ms`, `started_ms` (after the run began), `worker`, `select`, `resources`, `fixtures`, `build_started_ms` and `build_ms` when the runner built it, `usage` (P10.2, with `source`: `program` or `engine`) | -| `runs//cases.ndjson` | one line per case: `program`, `id`, `kind`, `status`, `ms`, `started_ms` (after the run began), `reason`, `failures`, `metrics`, `bounds`, `tags`, `clauses` | -| `runs//fixtures.ndjson` | one line per run fixture served: `name`, `program`, `started_ms`, `ready_ms`, `ended_ms`, `status` | +| `mcpplibs.mtest.gtest` (GoogleTest) | L1 | B2 | +| the Bash and Python adapters (`adapters/script`) | L3 | B2, O4, O5 ([docs/adapters.md](../docs/adapters.md)) | +| `mcpplibs.mtest.capi` (C ABI) | L3 | B2, O4, O5 | -P12.2 A run identifier sorts by time. Runs of one `group` (the jobs of a CI run) are parts of one whole, which a reader reports together. +## Conformance -P12.3 The times of a run's programs, cases, builds and fixtures form a directed acyclic graph of its work: a build before the processes of its program, a worker's process before its next, the holders of one exclusive resource in turn, a run fixture's serving before its users and its end after them. A reader's critical path is the graph's longest weighted path, weighted by the measured times. +`conformance/` checks a program against the clauses it states, and reports by clause number +([conformance.md](conformance.md)).