From 9c224a5b5d179b0705b7e97bd248e73b26d4d51b Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Tue, 10 Nov 2026 11:20:30 +0900 Subject: [PATCH 1/7] 0.5.3: a program only a runner starts is listed through that runner The listing pass (P3) starts a program as its run does: under --run-under and a [trace.*] tool's run_under, with the run's --env variables. A program of another system, run on a machine of its system or under an emulator, was started directly to list its cases and failed with "did not start". --- CHANGELOG.md | 8 ++++++++ README.md | 2 +- README.zh-CN.md | 2 +- adapters/gtest/mcpp.toml | 2 +- adapters/gtest/src/protocol.cpp | 2 +- capi/mcpp.toml | 2 +- capi/src/mtest.cpp | 2 +- conformance/mcpp.toml | 2 +- docs/commands.md | 2 +- engine/mcpp.toml | 2 +- engine/src/commands.cpp | 7 ++++++- engine/src/exec.cpp | 13 ++++++++++--- engine/src/exec.cppm | 4 +++- engine/src/main.cpp | 4 ++-- engine/src/select.cpp | 2 +- engine/src/select.cppm | 5 +++++ engine/src/service.cpp | 2 +- engine/tests/engine.cpp | 8 ++++++++ lib/docs/writing-tests.md | 2 +- lib/mcpp.toml | 2 +- lib/src/mtest.cpp | 2 +- manifest/mcpp.toml | 2 +- 22 files changed, 57 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 645a6c9..4fba670 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## [0.5.3] - 2026-11-10 + +- **A program only a runner starts is listed through that runner.** The listing pass (P3) starts a + program as its run does: under `--run-under` (and a `[trace.*]` tool's `run_under`), with the + run's `--env` variables. A program of another system -- started on a virtual machine of its system + or under an emulator -- was started directly to list its cases and failed with "did not start"; + its cases could not be run through mtest from the host that built it. + ## [0.5.2] - 2026-11-09 - **A copy of a test program that does something else is an object** (`child`). A test that starts diff --git a/README.md b/README.md index c36317c..b7193a1 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ int main() { ```toml [dev-dependencies] -mcpplibs.mtest = "0.5.2" +mcpplibs.mtest = "0.5.3" ``` A failure names its place and both sides (`expected 2, actual 3`). Under a runner the same program diff --git a/README.zh-CN.md b/README.zh-CN.md index 6680412..5c6447a 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -37,7 +37,7 @@ int main() { ```toml [dev-dependencies] -mcpplibs.mtest = "0.5.2" +mcpplibs.mtest = "0.5.3" ``` 失败时给出位置与两侧的值（`expected 2, actual 3`）。在运行器之下，同一程序按协议报告；单独运行时每个用例输出一行。 diff --git a/adapters/gtest/mcpp.toml b/adapters/gtest/mcpp.toml index d558a1b..a9996b1 100644 --- a/adapters/gtest/mcpp.toml +++ b/adapters/gtest/mcpp.toml @@ -13,7 +13,7 @@ [package] namespace = "mcpplibs.mtest" name = "gtest" -version = "0.5.2" +version = "0.5.3" description = "GoogleTest speaking the mtest protocol: a test main that lists, selects and reports GoogleTest's cases" license = "Apache-2.0" diff --git a/adapters/gtest/src/protocol.cpp b/adapters/gtest/src/protocol.cpp index d536913..90f0bca 100644 --- a/adapters/gtest/src/protocol.cpp +++ b/adapters/gtest/src/protocol.cpp @@ -128,7 +128,7 @@ int mcpplibs_mtest_gtest_main(int argc, char** argv) { {"protocol", 1}, {"level", 1}, {"framework", "googletest"}, - {"version", "0.5.2"}, + {"version", "0.5.3"}, {"isolation", {"process", "case"}}, {"guide", "https://github.com/mcpp-community/mtest/blob/main/adapters/gtest/docs/" "writing-tests.md"}}); diff --git a/capi/mcpp.toml b/capi/mcpp.toml index 889c00f..2149f80 100644 --- a/capi/mcpp.toml +++ b/capi/mcpp.toml @@ -4,7 +4,7 @@ [package] namespace = "mcpplibs.mtest" name = "capi" -version = "0.5.2" +version = "0.5.3" description = "mtest's C ABI: the protocol's events for any language with a C FFI" license = "Apache-2.0" diff --git a/capi/src/mtest.cpp b/capi/src/mtest.cpp index 559ec55..94bdce3 100644 --- a/capi/src/mtest.cpp +++ b/capi/src/mtest.cpp @@ -116,7 +116,7 @@ void mtest_init(const char* framework, int level) { {"protocol", 1}, {"level", std::clamp(level, 1, 3)}, {"framework", text(framework)}, - {"version", "0.5.2"}, + {"version", "0.5.3"}, {"isolation", {"process", "case"}}}); } diff --git a/conformance/mcpp.toml b/conformance/mcpp.toml index 71509a8..e698f1c 100644 --- a/conformance/mcpp.toml +++ b/conformance/mcpp.toml @@ -8,7 +8,7 @@ [package] namespace = "mcpplibs.mtest" name = "conformance" -version = "0.5.2" +version = "0.5.3" description = "The conformance suite of the mtest protocol, for every implementation of its program side" license = "Apache-2.0" diff --git a/docs/commands.md b/docs/commands.md index be93f64..ae66563 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -36,7 +36,7 @@ nothing; a built program's own listing is the authority (`manifest/src/manifest. | Command | Does | |---|---| -| `run` | runs the selection; `--jobs N`, `--seed N`, `--fail-fast`, `--retries N`, `--isolate` (a process per case), `--timeout 5m`, `--env K=V`, `--run-under CMD`, `--monitor LEVEL` (P10), `--group G` (runs reported together); `--junit`, `--html`, `--report-json FILE`, `--md FILE`; `--coverage` measures the coverage map. A program's cases are divided among workers that would otherwise idle (`[run] split`), and programs run together within the machine's memory (`[run] memory`) | +| `run` | runs the selection; `--jobs N`, `--seed N`, `--fail-fast`, `--retries N`, `--isolate` (a process per case), `--timeout 5m`, `--env K=V`, `--run-under CMD` (every start of a program, its listing included: a runner such as an emulator or a machine of the program's system), `--monitor LEVEL` (P10), `--group G` (runs reported together); `--junit`, `--html`, `--report-json FILE`, `--md FILE`; `--coverage` measures the coverage map. A program's cases are divided among workers that would otherwise idle (`[run] split`), and programs run together within the machine's memory (`[run] memory`) | | `rerun` | the cases that failed in the last run | | `smoke` | the `smoke` set | | `golden` | a run with `MTEST_GOLDEN=update`: golden files are written, not compared | diff --git a/engine/mcpp.toml b/engine/mcpp.toml index 977cb02..e0bff66 100644 --- a/engine/mcpp.toml +++ b/engine/mcpp.toml @@ -10,7 +10,7 @@ [package] namespace = "mtest" name = "mtest" -version = "0.5.2" +version = "0.5.3" description = "The mtest engine: run, schedule, store, explain and analyse tests of any language" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/engine/src/commands.cpp b/engine/src/commands.cpp index a6d9ab2..6126cc4 100644 --- a/engine/src/commands.cpp +++ b/engine/src/commands.cpp @@ -121,6 +121,11 @@ std::expected selection_of(const context& ctx, const r.exclude_tags = strings(p, "exclude_tag"); r.exclude_programs = strings(p, "exclude_program"); r.failed_only = p.value("failed", false); + r.run_under = strings(p, "run_under"); + if (auto e = p.find("env"); e != p.end() && e->is_object()) + for (auto& item_ : e->items()) + if (item_.value().is_string()) + r.env.emplace_back(item_.key(), item_.value().get()); if (auto since = text_of(p, "since"); !since.empty()) { auto paths = changed_since(ctx.cfg.root, since); if (!paths) @@ -1829,7 +1834,7 @@ json catalog() { plain({{"out", {{"type", "string"}}}, {"argv", {{"type", "array"}, {"items", {{"type", "string"}}}}}}), true); - return {{"schema", "mtest.catalog/1"}, {"version", "0.5.2"}, {"tools", tools}}; + return {{"schema", "mtest.catalog/1"}, {"version", "0.5.3"}, {"tools", tools}}; } } // namespace mcpplibs::mtest::engine diff --git a/engine/src/exec.cpp b/engine/src/exec.cpp index 0e5c8da..23daa44 100644 --- a/engine/src/exec.cpp +++ b/engine/src/exec.cpp @@ -189,7 +189,9 @@ report_reading read_report(std::string_view text, std::string_view program) { std::expected, std::string> list_program(const program_spec& spec, const std::filesystem::path& work, - const std::filesystem::path& root, std::vector* components) { + const std::filesystem::path& root, std::vector* components, + const std::vector& run_under, + const std::vector>& env) { if (detect_level(spec) < 1) { case_info c; c.id = spec.name; @@ -203,7 +205,10 @@ list_program(const program_spec& spec, const std::filesystem::path& work, const auto report = work / "list.ndjson"; std::error_code ec; std::filesystem::remove(report, ec); - const auto argv = with_loader(spec.argv); + // Started as its run starts it: under the run's prefix, with the run's variables. + auto argv = run_under; + for (auto& a : with_loader(spec.argv)) + argv.push_back(std::move(a)); os::process::spawn_options o; o.program = resolve_program(argv.front()); o.args.assign(argv.begin() + 1, argv.end()); @@ -211,6 +216,8 @@ list_program(const program_spec& spec, const std::filesystem::path& work, o.in = os::process::stdio::null; for (const auto& [k, v] : spec.env) o.env.emplace_back(k, v); + for (const auto& [k, v] : env) + o.env.emplace_back(k, v); o.env.emplace_back("MTEST_PROTOCOL", "1"); o.env.emplace_back("MTEST_MODE", "list"); o.env.emplace_back("MTEST_REPORT", os::fs::utf8(report)); @@ -220,7 +227,7 @@ list_program(const program_spec& spec, const std::filesystem::path& work, auto done = os::process::run(std::move(o), limits); if (!done) return std::unexpected( - std::format("{} did not start: {}", spec.argv.front(), done.error().message())); + std::format("{} did not start: {}", argv.front(), done.error().message())); auto text = os::fs::read(report); auto r = read_report(text ? std::string_view(*text) : std::string_view{}, spec.name); if (r.level == 0) { diff --git a/engine/src/exec.cppm b/engine/src/exec.cppm index fa68677..b6cdaab 100644 --- a/engine/src/exec.cppm +++ b/engine/src/exec.cppm @@ -34,7 +34,9 @@ struct run_outcome { // Lists the cases of a program at level 1 or more (P3); an L0 program has one case, its own name. std::expected, std::string> list_program(const program_spec& spec, const std::filesystem::path& work, - const std::filesystem::path& root, std::vector* components = nullptr); + const std::filesystem::path& root, std::vector* components = nullptr, + const std::vector& run_under = {}, + const std::vector>& env = {}); // `mtest measure --out FILE -- ARGV`: runs ARGV with this process's streams, writes what it used // (its children's account) to FILE as JSON, and ends with its status. diff --git a/engine/src/main.cpp b/engine/src/main.cpp index 73a13f4..e0cf804 100644 --- a/engine/src/main.cpp +++ b/engine/src/main.cpp @@ -98,7 +98,7 @@ const json* tool_named(const json& tools, std::string_view name) { cmdline::App application(const json& tools) { auto app = cmdline::App("mtest") - .version("0.5.2") + .version("0.5.3") .description("run tests of any language, and ask about their runs") .option(cmdline::Option("json").global().help("print the answer's data")); for (const auto& t : tools) @@ -253,7 +253,7 @@ int main(int argc, char** argv) { const std::string name(parsed->subcommand_name()); const auto& sub = parsed->subcommand()->get(); if (name == "version") { - std::println("mtest 0.5.2"); + std::println("mtest 0.5.3"); return 0; } const auto cwd = std::filesystem::current_path(); diff --git a/engine/src/select.cpp b/engine/src/select.cpp index 26cd145..97b2d82 100644 --- a/engine/src/select.cpp +++ b/engine/src/select.cpp @@ -561,7 +561,7 @@ std::expected make_plan(const config& c, const store& s, auto listed = list_program(p, s.dir / "work" / "list" / std::to_string(std::hash{}(p.name)), - c.root, &components); + c.root, &components, r.run_under, r.env); if (!listed) return std::unexpected(listed.error()); for (const auto& ci : components) { diff --git a/engine/src/select.cppm b/engine/src/select.cppm index 22272b6..69442b0 100644 --- a/engine/src/select.cppm +++ b/engine/src/select.cppm @@ -32,6 +32,11 @@ struct selection_request { // The change could not be read (a shallow clone without the base): a set that selects by // change selects every case, and says why. Empty: nothing went wrong. std::string unknown_change; + // How a program is started (`--run-under`, `--env`): its listing (P3) starts it as its run + // does, so that a program only a runner can start (an emulator, a machine of its system) is + // listed through that runner. + std::vector run_under; + std::vector> env; }; struct unit_case { diff --git a/engine/src/service.cpp b/engine/src/service.cpp index 6d61ddc..7f237fa 100644 --- a/engine/src/service.cpp +++ b/engine/src/service.cpp @@ -44,7 +44,7 @@ int serve(const std::filesystem::path& cwd) { {"result", {{"protocolVersion", params.value("protocolVersion", std::string("2025-06-18"))}, {"capabilities", {{"tools", {{"listChanged", false}}}}}, - {"serverInfo", {{"name", "mtest"}, {"version", "0.5.2"}}}, + {"serverInfo", {{"name", "mtest"}, {"version", "0.5.3"}}}, {"instructions", "Run tests and ask about runs: run, plan, explain, history, analyze, graph, " "gate, ci. " diff --git a/engine/tests/engine.cpp b/engine/tests/engine.cpp index 5da76f6..dd1da82 100644 --- a/engine/tests/engine.cpp +++ b/engine/tests/engine.cpp @@ -93,6 +93,14 @@ int main() { expect(eq(only["cases"].get(), 1)); expect(eq(only["status"].get(), std::string("pass"))); }; + "a program only a runner starts is listed and run through it (--run-under)"_test[kind::e2e] = [] { + // Not executable by itself, as a program of another system is not on this one: both the + // listing (P3) and the run start it under the runner. + auto root = project({{"t.cross", script({"a:pass", "b:pass"})}}); + auto j = json_of(mtest(root, {"run", "--json", "--run-under", "bash", "./t.cross"})); + expect(eq(j["cases"].get(), 2)); + expect(eq(j["status"].get(), std::string("pass"))); + }; "a case a crash ended fails with the program's status (P5.4)"_test[kind::e2e] = [] { auto root = project({{"t.sh", script({"dies:crash"}, "crash() { exit 9; }\n")}}); auto j = json_of(mtest(root, {"run", "--json", "bash t.sh"})); diff --git a/lib/docs/writing-tests.md b/lib/docs/writing-tests.md index a4a95d3..7adc974 100644 --- a/lib/docs/writing-tests.md +++ b/lib/docs/writing-tests.md @@ -53,7 +53,7 @@ E2E cli{"cli", resource{"serial:home"}, timeout{std::chrono::minutes{2}}, [] { / The entry enters the child before any case; the copy runs no case and reports nothing. The files of one directory are then one program (mcpp `[test] programs = "directory"`) however many of them start copies. -- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.5.2"`. +- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.5.3"`. - Identifiers are stable: a selection, a history and a shard name a case by them. ## Expectations diff --git a/lib/mcpp.toml b/lib/mcpp.toml index 3668ffc..dea2e97 100644 --- a/lib/mcpp.toml +++ b/lib/mcpp.toml @@ -17,7 +17,7 @@ [package] namespace = "mcpplibs" name = "mtest" -version = "0.5.2" +version = "0.5.3" description = "The C++ test library of mtest: cases, expectations, fixtures, bench, conformance and end-to-end helpers, with no macro and no header, speaking the mtest protocol" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/lib/src/mtest.cpp b/lib/src/mtest.cpp index 3e9443a..3b64f65 100644 --- a/lib/src/mtest.cpp +++ b/lib/src/mtest.cpp @@ -308,7 +308,7 @@ void write_program() { {"protocol", 1}, {"level", 3}, {"framework", "mcpplibs.mtest"}, - {"version", "0.5.2"}, + {"version", "0.5.3"}, {"isolation", {"process", "case"}}, {"guide", "https://github.com/mcpp-community/mtest/blob/main/lib/docs/writing-tests.md"}}); } diff --git a/manifest/mcpp.toml b/manifest/mcpp.toml index 0181856..75aec6b 100644 --- a/manifest/mcpp.toml +++ b/manifest/mcpp.toml @@ -8,7 +8,7 @@ [package] namespace = "mcpplibs.mtest" name = "manifest" -version = "0.5.2" +version = "0.5.3" description = "The static case manifest of mtest: the cases a source file declares, read without building it" license = "Apache-2.0" authors = ["mcpp-community"] From 934c683d8b50ecd1cc805cda1c6e200187fa20f0 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Tue, 10 Nov 2026 14:29:55 +0900 Subject: [PATCH 2/7] lib: a failing case's layout, its checks, and each case's own output under a runner A failing case's line counts its failed checks (and the cases it declared); each failure is shown under it with its place, the statement, the two sides or that it did not hold, the note and whether the case went on. The failure event states source and fatal, the end event checks {held, failed} (mcpp test spec O3). Under a runner each case's output, on each stream, is kept with os::stdio::capture, stated as output events and written on to the terminal; a case declared inside another keeps its own (O8). --- lib/src/mtest.cpp | 216 +++++++++++++++++++++++++++++++++++++++++-- lib/tests/layout.cpp | 45 +++++++++ lib/tests/output.cpp | 86 +++++++++++++++++ 3 files changed, 337 insertions(+), 10 deletions(-) create mode 100644 lib/tests/layout.cpp create mode 100644 lib/tests/output.cpp diff --git a/lib/src/mtest.cpp b/lib/src/mtest.cpp index 3b64f65..691e4db 100644 --- a/lib/src/mtest.cpp +++ b/lib/src/mtest.cpp @@ -30,9 +30,23 @@ struct running_case { bool case_failed = false; // a case it declared failed std::optional dir; std::map> metrics; // name -> value, unit - std::size_t declared = 0; // the cases it declared that ran + std::size_t declared = 0; // the cases it declared that ran + std::size_t declared_failed = 0; // of those, the ones that failed + // Its failures as a person reads them, printed under its line when it ends. + std::vector failures; + std::uint64_t held_before = 0; // the checks held in the program when it began + bool stopped = false; // a fatal expectation ended it + // SPEC-017 O8: what it writes, kept while it runs under a runner and stated as `output` + // events when it ends; a case it declares keeps its own, and this one's resumes after it. + std::optional out_capture, err_capture; + std::string out_text, err_text; + bool capturing = false; }; +// The expectations that held, in the whole program: a case's share is the difference over its +// run. Counted without the lock, as a passing check is the common one. +std::atomic g_held{0}; + struct declared_suite { std::string name; std::source_location where; @@ -55,6 +69,8 @@ struct state { std::vector suites; std::size_t suites_run = 0; std::map counts; + // Of `counts`, the cases that declared cases: the console's tally is of the cases that check. + std::map container_counts; std::set clauses; bool any_failed = false; bool program_written = false; @@ -308,7 +324,7 @@ void write_program() { {"protocol", 1}, {"level", 3}, {"framework", "mcpplibs.mtest"}, - {"version", "0.5.3"}, + {"version", "0.6.0"}, {"isolation", {"process", "case"}}, {"guide", "https://github.com/mcpp-community/mtest/blob/main/lib/docs/writing-tests.md"}}); } @@ -378,11 +394,92 @@ std::string sanitized(std::string_view id) { return out; } +// SPEC-017 O8: a case's own output. What the streams hold is written first, so that what the +// case writes is all that the capture keeps. +void flush_streams() { + std::cout.flush(); + std::cerr.flush(); + std::fflush(nullptr); +} + +void begin_capture(running_case& c) { + flush_streams(); + if (auto o = os::stdio::capture::begin(os::stdio::stream::output)) + c.out_capture.emplace(std::move(*o)); + if (auto e = os::stdio::capture::begin(os::stdio::stream::error)) + c.err_capture.emplace(std::move(*e)); +} + +// The capture ended and its text kept; the text is written to the stream it was taken from, so +// a person at the terminal reads what the case wrote where it wrote it. +void end_capture(running_case& c) { + flush_streams(); + if (c.err_capture) { + auto t = c.err_capture->text(); + c.err_capture.reset(); + if (!t.empty()) { + (void)os::stdio::write_error(t); + c.err_text += t; + } + } + if (c.out_capture) { + auto t = c.out_capture->text(); + c.out_capture.reset(); + if (!t.empty()) { + (void)os::stdio::write_output(t); + (void)os::stdio::flush_output(); + c.out_text += t; + } + } +} + void say(std::string_view line) { std::println("{}", line); std::cout.flush(); } +// The statement at `line` of `file` and the lines it continues on, as one line: what a check +// without operands states. Empty when the source is not readable here. +std::string statement_at(const std::string& file, std::uint_least32_t line) { + std::ifstream in(os::fs::from_utf8(file), std::ios::binary); + if (!in || line == 0) + return {}; + std::string text, one; + for (std::uint_least32_t n = 1; n < line && std::getline(in, one); ++n) {} + int depth = 0; + for (int taken = 0; taken < 8 && std::getline(in, one); ++taken) { + if (one.ends_with('\r')) + one.pop_back(); + const auto first = one.find_first_not_of(" \t"); + if (first == std::string::npos) + continue; + if (!text.empty()) + text += ' '; + text += one.substr(first); + for (char c : one) + depth += c == '(' ? 1 : c == ')' ? -1 : 0; + if (depth <= 0 && text.find(';') != std::string::npos) + break; + } + while (!text.empty() && (text.back() == ' ' || text.back() == '\t')) + text.pop_back(); + if (text.size() > 240) + text = text.substr(0, 237) + "..."; + return text; +} + +// A file under the working directory by its name there; any other by its own. +std::string shown_path(const std::string& file) { + std::error_code ec; + const auto p = os::fs::from_utf8(file); + if (!p.is_absolute()) + return file; + const auto rel = std::filesystem::proximate(p, std::filesystem::current_path(ec), ec); + if (ec || rel.empty() || rel.begin()->string() == "..") + return file; + return os::fs::generic_utf8(rel); +} + void record_failure(std::string_view message, const std::source_location& where, const comparison* value, bool is_fatal) { auto& s = the(); @@ -394,18 +491,55 @@ void record_failure(std::string_view message, const std::source_location& where, else s.any_failed = true; const std::string file = where.file_name(); + const bool compared = value && !value->expected.empty(); std::string text = std::string(message); if (value && !value->expression.empty()) text = text.empty() ? value->expression : value->expression + ": " + text; if (text.empty()) text = "the expectation did not hold"; - say(std::format(" {}:{}: {}", file, where.line(), text)); + // As a person reads it: where, the statement, the two sides or that it did not hold, the + // message the check added, and whether the case went on. + const std::string source = value ? statement_at(file, where.line()) : std::string{}; + std::string block = std::format("{}:{}", shown_path(file), where.line()); + if (!source.empty()) + block += "\n " + source; + if (compared) { + block += + std::format("\n expected {}\n actual {}", value->expected, value->actual); + if (!message.empty()) + block += std::format("\n note {}", message); + } else if (value) { + block += source.empty() || !message.empty() || !value->expression.empty() + ? "\n " + text + : std::string("\n did not hold"); + } else { + block += "\n " + text; + } + if (is_fatal) { + block += "\n fatal: the rest of the case did not run"; + if (in_case) + s.stack.back().stopped = true; + } + if (in_case) + s.stack.back().failures.push_back(std::move(block)); + else { + std::string shown; + for (std::string_view rest = block; !rest.empty();) { + const auto nl = rest.find('\n'); + shown += " " + std::string(rest.substr(0, nl)) + "\n"; + rest = nl == std::string_view::npos ? std::string_view{} : rest.substr(nl + 1); + } + shown.pop_back(); + say(shown); + } event ev = {{"e", "failure"}, {"id", id}, {"file", file}, {"line", where.line()}, {"message", text}}; - if (value && !value->expected.empty()) { + if (!source.empty()) + ev["source"] = source; + if (compared) { ev["expected"] = value->expected; ev["actual"] = value->actual; } @@ -502,8 +636,10 @@ expectation::expectation(comparison value, bool is_fatal, std::source_location w : held_(value.held), fatal_(is_fatal), value_(std::move(value)), where_(where) {} expectation::~expectation() noexcept(false) { - if (held_) + if (held_) { + g_held.fetch_add(1, std::memory_order_relaxed); return; + } record_failure(message_, where_, &value_, fatal_); if (fatal_ && std::uncaught_exceptions() == 0) throw fatal_signal{}; @@ -581,19 +717,55 @@ void test::operator=(body b) { } if (!selected(id)) return; - auto finish = [&](std::string_view status, double ms, std::string_view reason) { + auto finish = [&](std::string_view status, double ms, std::string_view reason, + const running_case* done = nullptr) { s.counts[std::string(status)]++; + const bool container = done && done->declared > 0; + if (container) + s.container_counts[std::string(status)]++; if (status == "fail" || status == "xpass") s.any_failed = true; event ev = {{"e", "end"}, {"id", id}, {"status", status}, {"ms", ms}}; if (!reason.empty()) ev["reason"] = reason; + // SPEC-017 O3: of the checks the case ran, those that held and those that did not. + if (done && status != "skip") + ev["checks"] = {{"held", g_held.load(std::memory_order_relaxed) - done->held_before}, + {"failed", done->failures.size()}}; write_event(ev); const char* mark = status == "pass" ? " ok " : status == "skip" ? " skip" : status == "xfail" ? " xfail" : " FAIL"; - say(std::format("{} {}{}{}", mark, id, reason.empty() ? "" : " -- ", reason)); + // What it amounts to: of a case that declared cases, how many of them failed; of one that + // checks, how many of its checks failed, and whether a fatal one ended it. + std::string tally; + if (done && status != "skip") { + if (container && done->declared_failed > 0) + tally = std::format("{} of {} cases failed", done->declared_failed, done->declared); + const std::size_t failed = done->failures.size(); + if (failed > 0) { + const auto held = g_held.load(std::memory_order_relaxed) - done->held_before; + if (!tally.empty()) + tally += "; "; + tally += std::format("{} of {} checks failed", failed, failed + held); + if (done->stopped) + tally += ", stopped"; + } + } + say(std::format("{} {}{}{}{}", mark, id, reason.empty() ? "" : " -- ", reason, + tally.empty() ? "" : " (" + tally + ")")); + if (done) + for (const auto& block : done->failures) { + std::string shown; + for (std::string_view rest = block; !rest.empty();) { + const auto nl = rest.find('\n'); + shown += " " + std::string(rest.substr(0, nl)) + "\n"; + rest = nl == std::string_view::npos ? std::string_view{} : rest.substr(nl + 1); + } + shown.pop_back(); + say(shown); + } }; const bool own = reported(id); if (attrs.skip) { @@ -634,7 +806,15 @@ void test::operator=(body b) { write_event(begun); if (!s.stack.empty() && !s.stack.back().scope) ++s.stack.back().declared; + const bool capturing = s.protocol && !s.listing && own; + if (capturing && !s.stack.empty() && s.stack.back().capturing) + end_capture(s.stack.back()); // the declaring case's output so far is its own s.stack.push_back({id, false, attrs}); + s.stack.back().held_before = g_held.load(std::memory_order_relaxed); + if (capturing) { + s.stack.back().capturing = true; + begin_capture(s.stack.back()); + } const auto started = clock::now(); std::string status = "pass"; std::string reason; @@ -661,8 +841,16 @@ void test::operator=(body b) { const double ms = std::chrono::duration(clock::now() - started).count(); if (own) report_use(id, use, level); + if (capturing) + end_capture(s.stack.back()); running_case done = std::move(s.stack.back()); s.stack.pop_back(); + if (capturing) { + if (!done.out_text.empty()) + write_event({{"e", "output"}, {"id", id}, {"stream", "out"}, {"text", done.out_text}}); + if (!done.err_text.empty()) + write_event({{"e", "output"}, {"id", id}, {"stream", "err"}, {"text", done.err_text}}); + } if (done.dir) { std::error_code ec; std::filesystem::remove_all(*done.dir, ec); @@ -684,9 +872,14 @@ void test::operator=(body b) { reason = "expected to fail: " + *attrs.expected_failure; } } - if (status == "fail" && !s.stack.empty() && !s.stack.back().scope) + if (status == "fail" && !s.stack.empty() && !s.stack.back().scope) { s.stack.back().case_failed = true; - finish(status, ms, reason); + ++s.stack.back().declared_failed; + } + finish(status, ms, reason, &done); + // The declaring case's capture resumes, after this case's own lines. + if (capturing && !s.stack.empty() && s.stack.back().capturing) + begin_capture(s.stack.back()); } void Suite::add(std::string name, std::source_location where, attributes attrs, @@ -810,9 +1003,12 @@ int report() { const bool failed = s.any_failed || missing; write_event({{"e", "done"}, {"status", failed ? "fail" : "pass"}, {"counts", s.counts}}); if (!s.listing) { + // The cases that check; a case that declared cases is told by them. auto n = [&](const char* k) { auto it = s.counts.find(k); - return it == s.counts.end() ? 0 : it->second; + auto c = s.container_counts.find(k); + return (it == s.counts.end() ? 0 : it->second) - + (c == s.container_counts.end() ? 0 : c->second); }; say(std::format("{} passed, {} failed, {} skipped{}{}", n("pass") + n("xfail"), n("fail") + n("xpass"), n("skip"), diff --git a/lib/tests/layout.cpp b/lib/tests/layout.cpp new file mode 100644 index 0000000..585652f --- /dev/null +++ b/lib/tests/layout.cpp @@ -0,0 +1,45 @@ +// What a person reads of a failing case: its line first, then each failure under it -- where it +// is, the statement, the two sides or that it did not hold, and whether the case went on. A child +// runs the cases, as a program that is not under a runner does, and its output is read. +import std; +import mcpplibs.mtest; +import mcpplibs.mtest.e2e; + +using namespace mcpplibs::mtest; + +child failing{"failing", [](std::vector) { + "values"_test = [] { + const std::string got = "second"; + expect(eq(got, std::string("first"))); + expect(got == "first"); + expect(got.size() == 6u); + }; + "stops"_test = [] { + expect(fatal(false)); + expect(true); + }; + return 0; + }}; + +UTest layout{"layout", [] { + "a failure is shown under its case with the statement and both sides"_test = [] { + auto o = e2e::run({.argv = child_command("failing", {})}); + expect(fatal(o.started)); + const auto at = [&](std::string_view s) { return o.out.find(s); }; + const auto header = at("FAIL values (2 of 3 checks failed)"); + expect(fatal(header != std::string::npos)) << o.out; + expect(at("expect(eq(got, std::string(\"first\")));") > header) << o.out; + expect(at("expected \"first\"") > header) << o.out; + expect(at("actual \"second\"") > header) << o.out; + expect(at("expect(got == \"first\");") > header) << o.out; + expect(at("did not hold") > header) << o.out; + }; + "a fatal failure says the rest of its case did not run"_test = [] { + auto o = e2e::run({.argv = child_command("failing", {})}); + expect(fatal(o.started)); + const auto header = o.out.find("FAIL stops (1 of 1 checks failed, stopped)"); + expect(fatal(header != std::string::npos)) << o.out; + expect(o.out.find("fatal: the rest of the case did not run") > header) + << o.out; + }; + }}; diff --git a/lib/tests/output.cpp b/lib/tests/output.cpp new file mode 100644 index 0000000..880bdf5 --- /dev/null +++ b/lib/tests/output.cpp @@ -0,0 +1,86 @@ +// Under a runner, a case states what it wrote and how its checks went (mcpp test spec O3, O8): +// each case's output as `output` events, a case's own apart from the cases it declares, and its +// `end` with the checks that held and failed. A child runs the cases with the protocol's variables +// set, as a runner would, and its report is read. +import std; +import mcpplibs.mtest; +import mcpplibs.mtest.e2e; +import nlohmann.json; + +using namespace mcpplibs::mtest; + +child talker{"talker", [](std::vector) { + "speaks"_test = [] { + std::println("said by speaks"); + std::println(std::cerr, "warned by speaks"); + expect(true); + expect(eq(1, 2)); + }; + "outer"_test = [] { + std::println("before inner"); + "inner"_test = [] { std::println("said by inner"); }; + std::println("after inner"); + }; + return 0; + }}; + +namespace { +std::vector run_talker(std::string& out) { + const auto report = temp_dir() / "report.ndjson"; + auto o = e2e::run({.argv = child_command("talker", {}), + .env = {{"MTEST_PROTOCOL", "1"}, + {"MTEST_MODE", "run"}, + {"MTEST_REPORT", report.string()}, + {"MTEST_TMPDIR", temp_dir().string()}}}); + out = o.out; + std::vector events; + std::ifstream in(report); + for (std::string line; std::getline(in, line);) + if (auto e = nlohmann::json::parse(line, nullptr, false); e.is_object()) + events.push_back(std::move(e)); + return events; +} + +std::string output_of(const std::vector& events, std::string_view id, + std::string_view stream) { + std::string text; + for (const auto& e : events) + if (e.value("e", "") == "output" && e.value("id", "") == id && + e.value("stream", "") == stream) + text += e.value("text", ""); + return text; +} +} // namespace + +UTest output{"output", [] { + "a case states what it wrote, on each stream"_test = [] { + std::string out; + const auto events = run_talker(out); + expect( + eq(output_of(events, "speaks", "out"), std::string("said by speaks\n"))); + expect( + eq(output_of(events, "speaks", "err"), std::string("warned by speaks\n"))); + // A person at the terminal still reads it. + expect(contains(out, "said by speaks")) << out; + }; + "a declared case's output is its own, and the declaring case keeps the rest"_test = + [] { + std::string out; + const auto events = run_talker(out); + expect(eq(output_of(events, "outer/inner", "out"), + std::string("said by inner\n"))); + expect(eq(output_of(events, "outer", "out"), + std::string("before inner\nafter inner\n"))); + }; + "a case's end states the checks that held and failed"_test = [] { + std::string out; + const auto events = run_talker(out); + const nlohmann::json* end = nullptr; + for (const auto& e : events) + if (e.value("e", "") == "end" && e.value("id", "") == "speaks") + end = &e; + expect(fatal(end != nullptr)); + expect(eq((*end)["checks"].value("held", -1), 1)); + expect(eq((*end)["checks"].value("failed", -1), 1)); + }; + }}; From fd883a29945c63a29f552a3dae2f0d3f50f3a754 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Tue, 10 Nov 2026 14:29:55 +0900 Subject: [PATCH 3/7] engine: the test session of the mcpp test spec (mtest serve, O9) test/plan, test/run (answered at once), test/add, test/cancel and test/status beside the commands; test/build, test/case, test/program and test/run as they happen. Builds are mcpp's (mcpp test --no-run, B5): one at a time, each program started from its launch. Runs share the session's workers; a case added joins its run; cancelling ends the process tree (own_unit), a case that had begun ends cancelled and the cases that had not begun run in another process. The end of the input lets runs end; shutdown cancels them. --- docs/service.md | 32 ++ engine/src/service.cpp | 51 +++- engine/src/session.cpp | 647 ++++++++++++++++++++++++++++++++++++++++ engine/src/session.cppm | 39 +++ engine/tests/engine.cpp | 69 ++++- 5 files changed, 828 insertions(+), 10 deletions(-) create mode 100644 engine/src/session.cpp create mode 100644 engine/src/session.cppm diff --git a/docs/service.md b/docs/service.md index 055e7ba..f8a27a5 100644 --- a/docs/service.md +++ b/docs/service.md @@ -12,3 +12,35 @@ Each request runs on its own thread, so `status` and `cancel` are answered while a run runs. An MCP client is configured with the command `mtest serve` in the project's directory. + +## The test session (0.6.0) + +`mtest serve` is also the test session of the mcpp test spec (mcpp `docs/specs/test.md`, O9): a client +that holds it runs an mcpp project's cases as a person asks for them, adds cases to a run in progress +and cancels a run, a program or a case. The library's entry states it (`test-session = ["mtest", +"serve"]`), and `mcpp test --guide` reports it. + +| Method | Params | Result | +|---|---|---| +| `test/plan` | `project`, optional `filter`, `options` | `programs`: the records of `mcpp test --cases` | +| `test/run` | `project`, `selection` (`[{program, cases}]`; empty: every program), `options` | `run`, at once | +| `test/add` | `run`, `selection` | `run`: the cases join the run in progress | +| `test/cancel` | `run`, optional `program` and `case` | what was cancelled | +| `test/status` | optional `run` | each run's status, counts and cases | +| `shutdown` | | every run cancelled | + +`options`: `mcpp` (a program, or a program and its first arguments; else `MTEST_MCPP`, else `mcpp`), +`args` (more arguments of `mcpp test`: `--profile`, `--target`), `isolation` (`case`: a process per +case). + +- **Notifications.** `test/build` (`begin`, then `end` with the programs built), `test/case` (`begin`, + `output`, `failure`, `end`, with the fields of the program's events), `test/program` (a process's + status, exit and time) and `test/run` (the run's status and counts). +- **Builds are mcpp's.** The session asks mcpp for the programs it runs (`mcpp test --no-run + --message-format json`, mcpp 2026.11.10.1 or later), which builds them when they need it, and starts + each from the launch the record carries. One build runs at a time; the processes share the + session's workers. +- **Cancelling** ends the process and everything it started. A case that had begun ends `cancelled`; + the cases its process was to run and that had not begun run in another process, unless the run or + the program was cancelled. A run of cancelled cases and no failure ends `cancelled`. +- **The end of the input** lets the runs in progress end; `shutdown` cancels them. diff --git a/engine/src/service.cpp b/engine/src/service.cpp index 7f237fa..34d4d82 100644 --- a/engine/src/service.cpp +++ b/engine/src/service.cpp @@ -3,6 +3,7 @@ module mcpplibs.mtest.engine.service; import std; import mcpplibs.mtest.engine.model; import mcpplibs.mtest.engine.commands; +import mcpplibs.mtest.engine.session; namespace mcpplibs::mtest::engine { @@ -25,6 +26,17 @@ json error_reply(const json& id, int code, std::string message) { } // namespace int serve(const std::filesystem::path& cwd) { + // The test session (mcpp test spec O9), made on its first method. + std::unique_ptr tests; + std::mutex tests_lock; + auto the_session = [&]() -> session& { + std::lock_guard g(tests_lock); + if (!tests) + tests = std::make_unique([](std::string_view method, const json& params) { + send({{"jsonrpc", "2.0"}, {"method", std::string(method)}, {"params", params}}); + }); + return *tests; + }; std::vector requests; for (std::string line; std::getline(std::cin, line);) { if (line.empty() || line == "\r") @@ -44,19 +56,45 @@ int serve(const std::filesystem::path& cwd) { {"result", {{"protocolVersion", params.value("protocolVersion", std::string("2025-06-18"))}, {"capabilities", {{"tools", {{"listChanged", false}}}}}, - {"serverInfo", {{"name", "mtest"}, {"version", "0.5.3"}}}, + {"serverInfo", {{"name", "mtest"}, {"version", "0.6.0"}}}, {"instructions", "Run tests and ask about runs: run, plan, explain, history, analyze, graph, " - "gate, ci. " + "gate, ci. A test session (test/plan, test/run, test/add, test/cancel, " + "test/status) runs an mcpp project's cases as they are asked for. " "Each result carries a `conclusion`; a failure's `again` repeats it."}}}}); continue; } if (method == "notifications/initialized" || method == "initialized") continue; if (method == "shutdown") { + { + std::lock_guard g(tests_lock); + if (tests) + tests->shutdown(); + } send({{"jsonrpc", "2.0"}, {"id", id}, {"result", nullptr}}); continue; } + if (method.starts_with("test/")) { + const auto verb = method.substr(5); + requests.emplace_back([&, id, verb, params, notification] { + auto& t = the_session(); + json r = verb == "plan" ? t.plan(params) + : verb == "run" ? t.run(params) + : verb == "add" ? t.add(params) + : verb == "cancel" ? t.cancel(params) + : verb == "status" + ? t.status(params) + : json{{"error", std::format("no method test/{}", verb)}}; + if (notification) + return; + if (r.contains("error")) + send(error_reply(id, -32602, r["error"].get())); + else + send({{"jsonrpc", "2.0"}, {"id", id}, {"result", r}}); + }); + continue; + } if (method == "exit") break; if (method == "tools/list") { @@ -96,6 +134,15 @@ int serve(const std::filesystem::path& cwd) { } }); } + // The input ended: what was asked for runs to its end (`shutdown` is how a client cancels). + requests.clear(); + { + std::lock_guard g(tests_lock); + if (tests) { + tests->drain(); + tests->shutdown(); + } + } return 0; } diff --git a/engine/src/session.cpp b/engine/src/session.cpp new file mode 100644 index 0000000..6b2196e --- /dev/null +++ b/engine/src/session.cpp @@ -0,0 +1,647 @@ +module mcpplibs.mtest.engine.session; + +import std; +import mcpplibs.os; +import mcpplibs.mtest.engine.model; + +namespace mcpplibs::mtest::engine { + +namespace os = mcpplibs::os; + +namespace { + +// How mcpp starts one test program (B5 `launch`), and the protocol it implements. +struct launch { + std::vector argv; + std::vector> env; + std::filesystem::path cwd; + int protocol = 0; +}; + +// One process of a run: a program, the cases it runs (none: all of them), and what it did. +struct job { + std::size_t serial = 0; + std::string run; + std::string program; + std::vector cases; + launch how; + std::set begun, ended, cancelled_cases; + bool cancelled = false; // the whole process is cancelled + bool requeue = false; // its cases that did not begin run in another process + os::process::process* proc = nullptr; +}; + +// A run's verdict: a failure fails it; else a cancelled case cancels it. +std::string verdict(bool cancelled, const std::map& counts) { + if (counts.contains("fail") || counts.contains("build-fail")) + return "fail"; + return cancelled || counts.contains("cancelled") ? "cancelled" : "pass"; +} + +struct run_state { + std::filesystem::path project; + std::vector mcpp; // the program, and arguments before `test` + std::vector args; // more arguments of `mcpp test` (profile, target) + bool isolate = false; // a process per case + std::size_t outstanding = 0; // builds and processes not yet ended + bool cancelled = false, ended = false; + std::map counts; + std::map cases; // "/" -> status +}; + +std::vector records_of(std::string_view text) { + std::vector out; + for (std::size_t at = 0; at < text.size();) { + auto nl = text.find('\n', at); + auto line = + text.substr(at, nl == std::string_view::npos ? std::string_view::npos : nl - at); + at = nl == std::string_view::npos ? text.size() : nl + 1; + if (line.empty() || line.front() != '{') + continue; + if (auto j = json::parse(line, nullptr, false); j.is_object()) + out.push_back(std::move(j)); + } + return out; +} + +} // namespace + +struct session::impl { + notify_fn notify; + std::filesystem::path tmp; + std::mutex m; + std::condition_variable cv; + bool stopping = false; + std::deque> pending; + std::vector> running; + std::map runs; + std::size_t next_run = 1, next_job = 1; + std::mutex build_m; // one build at a time: the build state is mcpp's (B5) + std::vector workers; + std::vector planners; + + // `mcpp test` with `args` in `project`, its records. + std::pair> call_mcpp(const run_state& r, std::vector args) { + os::process::spawn_options o; + o.program = r.mcpp.front(); + o.args.assign(r.mcpp.begin() + 1, r.mcpp.end()); + o.args.insert(o.args.end(), args.begin(), args.end()); + o.args.insert(o.args.end(), r.args.begin(), r.args.end()); + o.cwd = r.project; + o.in = os::process::stdio::null; + auto out = os::process::run(o); + if (!out) + return {-1, + {json{{"error", std::format("{} did not start: {}", r.mcpp.front(), + out.error().message())}}}}; + return {out->status.shell_code(), records_of(out->out)}; + } + + void case_end(const job& j, const std::string& id, std::string_view status, const json& extra) { + json p = {{"run", j.run}, + {"program", j.program}, + {"id", id}, + {"event", "end"}, + {"status", status}}; + for (auto it = extra.begin(); it != extra.end(); ++it) + p[it.key()] = *it; + { + std::lock_guard g(m); + auto& r = runs[j.run]; + r.counts[std::string(status)]++; + r.cases[j.program + "/" + id] = std::string(status); + } + notify("test/case", p); + } + + // The run ends when nothing of it is outstanding; m held. + void maybe_end(const std::string& id) { + auto& r = runs[id]; + if (r.outstanding != 0 || r.ended) + return; + r.ended = true; + cv.notify_all(); + notify("test/run", + {{"run", id}, {"status", verdict(r.cancelled, r.counts)}, {"counts", r.counts}}); + } + + void enqueue(const std::string& run, const std::string& program, const launch& how, + const std::vector& cases) { + auto& r = runs[run]; + auto add = [&](std::vector one) { + auto j = std::make_shared(); + j->serial = next_job++; + j->run = run; + j->program = program; + j->cases = std::move(one); + j->how = how; + pending.push_back(j); + ++r.outstanding; + }; + if (r.isolate && !cases.empty()) + for (const auto& c : cases) + add({c}); + else + add(cases); + cv.notify_all(); + } + + // B5: the selected programs built when they need to be, and their launches; then their jobs. + void + build_and_enqueue(const std::string& run, + std::vector>> selection) { + run_state r; + { + std::lock_guard g(m); + r = runs[run]; + } + std::map launches; + { + std::lock_guard b(build_m); + notify("test/build", {{"run", run}, {"event", "begin"}}); + std::vector args = {"test", "--no-run", "--message-format", "json"}; + if (selection.size() == 1) + args.push_back(selection.front().first); + auto [status, records] = call_mcpp(r, args); + json built = json::array(); + for (const auto& rec : records) { + const auto name = rec.value("test", std::string{}); + if (name.empty()) + continue; + if (rec.value("status", std::string{}) == "built" && rec.contains("launch")) { + const auto& l = rec["launch"]; + launch one; + one.argv = l.value("argv", std::vector{}); + if (l.contains("env") && l["env"].is_object()) + for (auto it = l["env"].begin(); it != l["env"].end(); ++it) + one.env.emplace_back(it.key(), it->get()); + one.cwd = l.value("cwd", std::string{}); + one.protocol = rec.value("protocol", json(nullptr)).is_number() + ? rec["protocol"].get() + : 0; + launches[name] = std::move(one); + built.push_back(name); + } else if (rec.value("status", std::string{}) == "compile_fail") { + std::lock_guard g(m); + runs[run].counts["build-fail"]++; + notify("test/program", + {{"run", run}, + {"program", name}, + {"status", "build-fail"}, + {"output", rec.value("compile_output", std::string{})}}); + } + } + const bool ok = status == 0 || !launches.empty(); + json end = {{"run", run}, {"event", "end"}, {"ok", ok}, {"programs", built}}; + if (!ok && !records.empty() && records.front().contains("error")) + end["error"] = records.front()["error"]; + else if (!ok) + end["error"] = std:: + format("mcpp test --no-run exited {} and named no launch (it gives one from " + "2026.11.10.1, feature `test-launch-records`)", + status); + notify("test/build", end); + } + std::lock_guard g(m); + auto& rs = runs[run]; + if (!rs.cancelled) { + if (selection.empty()) + for (const auto& [name, how] : launches) + enqueue(run, name, how, {}); + for (const auto& [program, cases] : selection) + if (auto it = launches.find(program); it != launches.end()) + enqueue(run, program, it->second, cases); + } + --rs.outstanding; + maybe_end(run); + } + + void execute(job& j) { + const auto dir = tmp / j.run / std::format("{}", j.serial); + std::error_code ec; + std::filesystem::create_directories(dir / "tmp", ec); + const auto report = dir / "report.ndjson"; + os::process::spawn_options o; + if (j.how.argv.empty()) + return; + o.program = j.how.argv.front(); + o.args.assign(j.how.argv.begin() + 1, j.how.argv.end()); + o.cwd = j.how.cwd; + o.in = os::process::stdio::null; + o.out = os::process::stdio::pipe; + o.err = os::process::stdio::pipe; + o.own_unit = true; // cancelling ends it and everything it started + for (const auto& [k, v] : j.how.env) + o.env.emplace_back(k, v); + if (j.how.protocol) { + o.env.emplace_back("MTEST_PROTOCOL", "1"); + o.env.emplace_back("MTEST_MODE", "run"); + o.env.emplace_back("MTEST_REPORT", report.string()); + o.env.emplace_back("MTEST_TMPDIR", (dir / "tmp").string()); + if (!j.cases.empty()) { + std::string lines; + for (const auto& c : j.cases) + lines += c + "\n"; + (void)os::fs::write(dir / "select.txt", lines); + o.env.emplace_back("MTEST_SELECT", (dir / "select.txt").string()); + } + } + const auto started = std::chrono::steady_clock::now(); + auto p = os::process::process::spawn(o); + if (!p) { + notify("test/program", + {{"run", j.run}, + {"program", j.program}, + {"status", "not-run"}, + {"reason", std::format("{} did not start: {}", o.program.string(), + p.error().message())}}); + return; + } + { + std::lock_guard g(m); + j.proc = &*p; + if (j.cancelled) + p->kill(); + } + std::string out_text; + std::jthread out_reader([&] { + for (;;) { + auto chunk = p->read_output(); + if (!chunk || chunk->empty()) + break; + out_text += *chunk; + } + }); + std::jthread err_reader([&] { + for (;;) { + auto chunk = p->read_error(); + if (!chunk || chunk->empty()) + break; + } + }); + std::uintmax_t offset = 0; + auto poll = [&] { + std::ifstream in(report, std::ios::binary); + if (!in) + return; + in.seekg(static_cast(offset)); + std::string text{std::istreambuf_iterator(in), {}}; + const auto last = text.rfind('\n'); + if (last == std::string::npos) + return; + offset += last + 1; + for (const auto& e : records_of(std::string_view(text).substr(0, last + 1))) { + const auto kind = e.value("e", std::string{}); + const auto id = e.value("id", std::string{}); + if (kind == "begin") { + { + std::lock_guard g(m); + j.begun.insert(id); + } + notify("test/case", {{"run", j.run}, + {"program", j.program}, + {"id", id}, + {"event", "begin"}, + {"t_ms", e.value("t", 0.0)}}); + } else if (kind == "end") { + { + std::lock_guard g(m); + j.ended.insert(id); + } + json extra = {{"ms", e.value("ms", 0.0)}}; + for (const char* k : {"reason", "checks"}) + if (e.contains(k)) + extra[k] = e[k]; + case_end(j, id, e.value("status", std::string("fail")), extra); + } else if (kind == "output" || kind == "failure") { + json n = e; + n.erase("e"); + n["run"] = j.run; + n["program"] = j.program; + n["event"] = kind; + notify("test/case", n); + } + } + }; + std::optional st; + for (;;) { + auto w = p->wait_for(std::chrono::milliseconds(50)); + poll(); + if (!w) + break; + if (*w) { + st = **w; + break; + } + } + out_reader.join(); + err_reader.join(); + poll(); + { + std::lock_guard g(m); + j.proc = nullptr; + } + const int code = st ? st->shell_code() : -1; + // P5.4: a case that began and did not end failed, or was cancelled. + std::set begun; + { + std::lock_guard g(m); + begun = j.begun; + } + for (const auto& id : begun) + if (!j.ended.contains(id)) { + const bool cancelled = j.cancelled || j.cancelled_cases.contains(id) || j.requeue; + case_end(j, id, cancelled ? "cancelled" : "fail", + {{"reason", cancelled ? std::string("cancelled") + : std::format("the program ended with status {} " + "before the case ended", + code)}}); + std::lock_guard g(m); + j.ended.insert(id); + } + const double ms = + std::chrono::duration(std::chrono::steady_clock::now() - started) + .count(); + std::string status = j.cancelled ? "cancelled" : code == 0 ? "pass" : "fail"; + json prog = {{"run", j.run}, + {"program", j.program}, + {"status", status}, + {"exit", code}, + {"ms", ms}}; + if (!j.how.protocol || (code != 0 && j.begun.empty())) + prog["output"] = out_text; + if (!j.how.protocol) { + // A program of no protocol is one case, judged by its exit status (P1.1). + case_end(j, j.program, + j.cancelled ? "cancelled" + : code == 0 ? "pass" + : "fail", + {{"ms", ms}}); + } + notify("test/program", prog); + std::filesystem::remove_all(dir, ec); + } + + // After a job: the cases it was to run that did not begin, cancelled with it or run again. + void settle(const std::shared_ptr& j) { + std::lock_guard g(m); + std::vector rest; + for (const auto& c : j->cases) + if (!j->begun.contains(c) && !j->cancelled_cases.contains(c)) + rest.push_back(c); + auto& r = runs[j->run]; + if (j->requeue && !j->cancelled && !r.cancelled && !rest.empty()) + enqueue(j->run, j->program, j->how, rest); + else if (j->cancelled || r.cancelled) + for (const auto& c : rest) { + r.counts["cancelled"]++; + r.cases[j->program + "/" + c] = "cancelled"; + notify("test/case", {{"run", j->run}, + {"program", j->program}, + {"id", c}, + {"event", "end"}, + {"status", "cancelled"}, + {"reason", "cancelled"}}); + } + std::erase(running, j); + --r.outstanding; + maybe_end(j->run); + } + + void work() { + for (;;) { + std::shared_ptr j; + { + std::unique_lock l(m); + cv.wait(l, [&] { return stopping || !pending.empty(); }); + if (pending.empty()) + return; + j = pending.front(); + pending.pop_front(); + running.push_back(j); + } + execute(*j); + settle(j); + } + } +}; + +session::session(notify_fn notify) : d_(std::make_unique()) { + d_->notify = std::move(notify); + const auto self = os::process::self(); + d_->tmp = std::filesystem::temp_directory_path() / + std::format("mtest-session-{}-{}", self ? self->id : 0, self ? self->started_ns : 0); + unsigned n = std::max(2u, std::thread::hardware_concurrency()); + for (unsigned i = 0; i < n; ++i) + d_->workers.emplace_back([this] { d_->work(); }); +} + +session::~session() { + shutdown(); +} + +namespace { +run_state state_of(const json& params) { + run_state r; + r.project = params.value("project", std::filesystem::current_path().string()); + const auto options = params.value("options", json::object()); + // `mcpp`: a program, or a program and the arguments before `test`; else MTEST_MCPP, else mcpp. + if (const auto m = options.value("mcpp", json()); m.is_string()) + r.mcpp = {m.get()}; + else if (m.is_array()) + r.mcpp = m.get>(); + if (r.mcpp.empty() || r.mcpp.front().empty()) + r.mcpp = {os::env::get("MTEST_MCPP").value_or("mcpp")}; + r.args = options.value("args", std::vector{}); + r.isolate = options.value("isolation", std::string{}) == "case"; + return r; +} + +std::vector>> selection_of(const json& params) { + std::vector>> out; + for (const auto& s : params.value("selection", json::array())) + out.emplace_back(s.value("program", std::string{}), + s.value("cases", std::vector{})); + return out; +} +} // namespace + +json session::plan(const json& params) { + auto r = state_of(params); + std::vector args = {"test", "--cases", "--message-format", "json"}; + if (auto f = params.value("filter", std::string{}); !f.empty()) + args.push_back(f); + std::lock_guard b(d_->build_m); + auto [status, records] = d_->call_mcpp(r, args); + json programs = json::array(); + for (const auto& rec : records) + if (rec.contains("test")) + programs.push_back(rec); + if (programs.empty() && status != 0) + return {{"error", std::format("mcpp test --cases exited {}", status)}}; + return {{"programs", programs}}; +} + +json session::run(const json& params) { + auto r = state_of(params); + auto selection = selection_of(params); + std::string id; + { + std::lock_guard g(d_->m); + if (d_->stopping) + return {{"error", "the session is shutting down"}}; + id = std::format("r{}", d_->next_run++); + r.outstanding = 1; // its build + d_->runs[id] = std::move(r); + } + { + std::lock_guard g(d_->m); + d_->planners.emplace_back([this, id, selection] { d_->build_and_enqueue(id, selection); }); + } + return {{"run", id}}; +} + +json session::add(const json& params) { + const auto id = params.value("run", std::string{}); + auto selection = selection_of(params); + { + std::lock_guard g(d_->m); + auto it = d_->runs.find(id); + if (it == d_->runs.end()) + return {{"error", std::format("no run {}", id)}}; + if (it->second.ended || it->second.cancelled) + return {{"error", std::format("run {} has ended", id)}}; + ++it->second.outstanding; // its build + d_->planners.emplace_back([this, id, selection] { d_->build_and_enqueue(id, selection); }); + } + return {{"run", id}}; +} + +json session::cancel(const json& params) { + const auto id = params.value("run", std::string{}); + const auto program = params.value("program", std::string{}); + const auto one = params.value("case", std::string{}); + std::lock_guard g(d_->m); + auto it = d_->runs.find(id); + if (it == d_->runs.end()) + return {{"error", std::format("no run {}", id)}}; + auto& r = it->second; + if (program.empty() && one.empty()) + r.cancelled = true; + auto matches = [&](const job& j) { + return j.run == id && (program.empty() || j.program == program); + }; + // What has not started: removed, or the case taken out of its selection. + for (auto p = d_->pending.begin(); p != d_->pending.end();) { + auto& j = **p; + if (!matches(j)) { + ++p; + continue; + } + std::vector dropped; + if (one.empty()) + dropped = j.cases.empty() ? std::vector{} : j.cases; + else if (std::erase(j.cases, one) > 0) + dropped = {one}; + else { + ++p; + continue; + } + for (const auto& c : dropped) { + r.counts["cancelled"]++; + r.cases[j.program + "/" + c] = "cancelled"; + d_->notify("test/case", {{"run", id}, + {"program", j.program}, + {"id", c}, + {"event", "end"}, + {"status", "cancelled"}, + {"reason", "cancelled"}}); + } + if (one.empty() || j.cases.empty()) { + d_->notify("test/program", + {{"run", id}, {"program", j.program}, {"status", "cancelled"}}); + p = d_->pending.erase(p); + --r.outstanding; + } else { + ++p; + } + } + // What runs: its process tree ended. A case shared with others ends the process; the cases it + // was to run that did not begin run in another. + for (const auto& jp : d_->running) { + auto& j = *jp; + if (!matches(j)) + continue; + if (one.empty()) { + j.cancelled = true; + } else { + const bool selected = + j.cases.empty() || std::ranges::find(j.cases, one) != j.cases.end(); + if (!selected || j.ended.contains(one)) + continue; + j.cancelled_cases.insert(one); + if (j.cases.size() == 1) + j.cancelled = true; + else + j.requeue = true; + } + if (j.proc) + j.proc->kill(); + } + d_->maybe_end(id); + return {{"run", id}, + {"cancelled", one.empty() ? (program.empty() ? "run" : "program") : "case"}}; +} + +json session::status(const json& params) { + std::lock_guard g(d_->m); + json runs = json::object(); + const auto only = params.value("run", std::string{}); + for (const auto& [id, r] : d_->runs) { + if (!only.empty() && id != only) + continue; + runs[id] = {{"status", r.ended ? verdict(r.cancelled, r.counts) : std::string("running")}, + {"counts", r.counts}, + {"cases", r.cases}}; + } + return {{"runs", runs}}; +} + +void session::drain() { + std::unique_lock l(d_->m); + d_->cv.wait(l, [&] { + return d_->stopping || + std::ranges::all_of(d_->runs, [](const auto& r) { return r.second.ended; }); + }); +} + +void session::shutdown() { + if (!d_) + return; + { + std::lock_guard g(d_->m); + if (d_->stopping) + return; + d_->stopping = true; + for (auto& [id, r] : d_->runs) + r.cancelled = true; + for (const auto& j : d_->running) { + j->cancelled = true; + if (j->proc) + j->proc->kill(); + } + d_->pending.clear(); + } + d_->cv.notify_all(); + std::vector planners; + { + std::lock_guard g(d_->m); + planners = std::move(d_->planners); + } + planners.clear(); + d_->workers.clear(); + std::error_code ec; + std::filesystem::remove_all(d_->tmp, ec); +} + +} // namespace mcpplibs::mtest::engine diff --git a/engine/src/session.cppm b/engine/src/session.cppm new file mode 100644 index 0000000..6c00f1f --- /dev/null +++ b/engine/src/session.cppm @@ -0,0 +1,39 @@ +// The test session of the mcpp test spec (docs/specs/test.md O9): a client holds one session and +// runs, adds and cancels cases while it does. The session builds nothing: it asks mcpp for the +// programs' launches (B5, `mcpp test --no-run --message-format json`), starts each program with the +// protocol's variables, and states what the program reports as it happens. `mtest serve` answers +// the session's methods beside its commands. +export module mcpplibs.mtest.engine.session; + +import std; +import mcpplibs.mtest.engine.model; + +export namespace mcpplibs::mtest::engine { + +class session { +public: + // A notification: its method (`test/build`, `test/case`, `test/program`, `test/run`) and its + // parameters. Called from the session's threads. + using notify_fn = std::function; + + explicit session(notify_fn notify); + ~session(); + session(const session&) = delete; + session& operator=(const session&) = delete; + + // Each answers a request's parameters with its result, or an `error` member naming why not. + json plan(const json& params); + json run(const json& params); + json add(const json& params); + json cancel(const json& params); + json status(const json& params); + // Every run ended as it does: what the session's client asked for is done. + void drain(); + // Every run cancelled, the threads ended. + void shutdown(); +private: + struct impl; + std::unique_ptr d_; +}; + +} // namespace mcpplibs::mtest::engine diff --git a/engine/tests/engine.cpp b/engine/tests/engine.cpp index dd1da82..5f45555 100644 --- a/engine/tests/engine.cpp +++ b/engine/tests/engine.cpp @@ -93,14 +93,15 @@ int main() { expect(eq(only["cases"].get(), 1)); expect(eq(only["status"].get(), std::string("pass"))); }; - "a program only a runner starts is listed and run through it (--run-under)"_test[kind::e2e] = [] { - // Not executable by itself, as a program of another system is not on this one: both the - // listing (P3) and the run start it under the runner. - auto root = project({{"t.cross", script({"a:pass", "b:pass"})}}); - auto j = json_of(mtest(root, {"run", "--json", "--run-under", "bash", "./t.cross"})); - expect(eq(j["cases"].get(), 2)); - expect(eq(j["status"].get(), std::string("pass"))); - }; + "a program only a runner starts is listed and run through it (--run-under)"_test[kind::e2e] = + [] { + // Not executable by itself, as a program of another system is not on this one: both the + // listing (P3) and the run start it under the runner. + auto root = project({{"t.cross", script({"a:pass", "b:pass"})}}); + auto j = json_of(mtest(root, {"run", "--json", "--run-under", "bash", "./t.cross"})); + expect(eq(j["cases"].get(), 2)); + expect(eq(j["status"].get(), std::string("pass"))); + }; "a case a crash ended fails with the program's status (P5.4)"_test[kind::e2e] = [] { auto root = project({{"t.sh", script({"dies:crash"}, "crash() { exit 9; }\n")}}); auto j = json_of(mtest(root, {"run", "--json", "bash t.sh"})); @@ -244,6 +245,58 @@ int main() { expect(!replies[3]["result"]["isError"].get()); expect(replies[4]["result"]["data"].contains("conclusion")); }; + "a test session runs the cases asked for, from the launches mcpp gives (spec O9)"_test + [kind::e2e] = [] { + // mcpp's part, B5: the program built, and how it starts. + auto root = + project({{"t.sh", script({"a:pass", "b:fail"})}, + {"mcpp.sh", "echo '{\"test\":\"t\",\"status\":\"built\",\"protocol\":1," + "\"launch\":{\"argv\":[\"bash\",\"t.sh\"],\"env\":{},\"cwd\":" + "\"\",\"via\":\"\"}}'\n"}}); + const json run = {{"jsonrpc", "2.0"}, + {"id", 1}, + {"method", "test/run"}, + {"params", + {{"project", root.generic_string()}, + {"options", {{"mcpp", {"bash", "mcpp.sh"}}, {"isolation", "case"}}}, + {"selection", {{{"program", "t"}, {"cases", {"a", "b"}}}}}}}}; + auto r = mtest(root, {"serve"}, run.dump() + "\n"); + std::vector notes; + json reply; + std::string_view rest = r.out; + while (!rest.empty()) { + auto nl = rest.find('\n'); + auto j = json::parse(rest.substr(0, nl), nullptr, false); + rest = nl == std::string_view::npos ? std::string_view{} : rest.substr(nl + 1); + if (j.is_discarded()) + continue; + if (j.contains("id")) + reply = j; + else + notes.push_back(j); + } + expect(fatal(reply.contains("result"))) << r.out << r.err; + const auto id = reply["result"]["run"].get(); + auto count = [&](std::string_view method, std::string_view key, + std::string_view value) { + return std::ranges::count_if(notes, [&](const json& n) { + return n.value("method", "") == method && + n["params"].value(std::string(key), "") == value; + }); + }; + expect(eq(count("test/build", "event", "end"), 1)) << r.out; + // A process per case: two programs, each case's end. + expect(eq(count("test/program", "program", "t"), 2)) << r.out; + expect(eq(count("test/case", "event", "end"), 2)) << r.out; + const auto end = std::ranges::find_if(notes, [](const json& n) { + return n.value("method", "") == "test/run"; + }); + expect(fatal(end != notes.end())) << r.out; + expect(eq((*end)["params"]["run"].get(), id)); + expect(eq((*end)["params"]["status"].get(), std::string("fail"))); + expect(eq((*end)["params"]["counts"].value("pass", 0), 1)); + expect(eq((*end)["params"]["counts"].value("fail", 0), 1)); + }; "every answer carries a conclusion"_test[kind::e2e] = [] { auto root = project({{"t.sh", script({"a:pass"})}}); (void)mtest(root, {"run", "bash t.sh"}); From f37d577b63497d3a2f07d756274336e6a3e43887 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Tue, 10 Nov 2026 14:30:02 +0900 Subject: [PATCH 4/7] 0.6.0: mtest implements the mcpp test spec, which is the protocol's normative text spec/protocol.md lists what the library, the engine, the adapters and the C ABI implement of SPEC-017 v2.0, whose clauses keep the protocol's numbers. The library's entry states test-capabilities and test-session. The engine and the conformance suite ask for mcpplibs.os ^0.4.1, the openkal targets build over openkal-platform 0.25.0, and CI runs mcpp 2026.11.9.1, the floor of that platform. --- .github/versions.env | 2 +- CHANGELOG.md | 25 +++- README.md | 2 +- README.zh-CN.md | 2 +- adapters/gtest/mcpp.toml | 2 +- adapters/gtest/src/protocol.cpp | 2 +- capi/mcpp.toml | 2 +- capi/src/mtest.cpp | 2 +- conformance/mcpp.toml | 4 +- engine/mcpp.toml | 4 +- engine/src/commands.cpp | 2 +- engine/src/main.cpp | 4 +- lib/docs/writing-tests.md | 2 +- lib/mcpp.toml | 6 +- manifest/mcpp.toml | 2 +- mcpp.toml | 2 +- spec/protocol.md | 197 ++++++-------------------------- 17 files changed, 81 insertions(+), 181 deletions(-) diff --git a/.github/versions.env b/.github/versions.env index f2f6c0e..d2b40d3 100644 --- a/.github/versions.env +++ b/.github/versions.env @@ -1,4 +1,4 @@ # The versions CI builds and tests with; .github/actions/setup-mcpp exports them. -MCPP_VERSION=2026.11.8.1 +MCPP_VERSION=2026.11.9.1 # The xlings that mcpp release bundles. XLINGS_VERSION=v2026.10.26.1 diff --git a/CHANGELOG.md b/CHANGELOG.md index 4fba670..bda0118 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,28 @@ # Changelog -## [0.5.3] - 2026-11-10 - +## [0.6.0] - 2026-11-10 + +The mtest protocol's normative text is the mcpp test spec (mcpp `docs/specs/test.md`, SPEC-017 v2.0), +which keeps its clause numbers; `spec/protocol.md` lists what each part of mtest implements of it. + +- **A test session** (spec O9): `mtest serve` answers `test/plan`, `test/run`, `test/add`, + `test/cancel` and `test/status`, and states `test/build`, `test/case`, `test/program` and + `test/run` as they happen. A run is answered at once and its programs run on the session's + workers beside other runs; a case added joins the run in progress; cancelling ends the process and + everything it started, and the cases that had not begun run in another process. The session builds + nothing: it starts each program from the launch mcpp gives (`mcpp test --no-run`, spec B5). +- **A case's own output** (spec O8). Under a runner the library keeps what each case writes, on each + stream, states it as `output` events and writes it on to the terminal; a case declared inside + another keeps its own. +- **A case's checks** (spec O3): its `end` states the checks that held and failed (`checks`); a + failure states its statement (`source`) and whether it ended the case (`fatal`). +- **A failing case reads as a person reads it.** Under its line, each failure: its place, the + statement, the two sides or that it did not hold, the message the check added, and whether the case + went on; the line counts the checks that failed, and a case that declared cases counts them. +- The library's entry states the capabilities it implements and its session (`test-capabilities`, + `test-session`; mcpp 2026.11.10.1 reads them, an earlier mcpp ignores them). +- The engine and the conformance suite ask for mcpplibs.os `^0.4.1`, as the library does, and build + `-openkal` over openkal-platform 0.25.0. - **A program only a runner starts is listed through that runner.** The listing pass (P3) starts a program as its run does: under `--run-under` (and a `[trace.*]` tool's `run_under`), with the run's `--env` variables. A program of another system -- started on a virtual machine of its system diff --git a/README.md b/README.md index b7193a1..c734bba 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ int main() { ```toml [dev-dependencies] -mcpplibs.mtest = "0.5.3" +mcpplibs.mtest = "0.6.0" ``` A failure names its place and both sides (`expected 2, actual 3`). Under a runner the same program diff --git a/README.zh-CN.md b/README.zh-CN.md index 5c6447a..04bc531 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -37,7 +37,7 @@ int main() { ```toml [dev-dependencies] -mcpplibs.mtest = "0.5.3" +mcpplibs.mtest = "0.6.0" ``` 失败时给出位置与两侧的值（`expected 2, actual 3`）。在运行器之下，同一程序按协议报告；单独运行时每个用例输出一行。 diff --git a/adapters/gtest/mcpp.toml b/adapters/gtest/mcpp.toml index a9996b1..1746fa3 100644 --- a/adapters/gtest/mcpp.toml +++ b/adapters/gtest/mcpp.toml @@ -13,7 +13,7 @@ [package] namespace = "mcpplibs.mtest" name = "gtest" -version = "0.5.3" +version = "0.6.0" description = "GoogleTest speaking the mtest protocol: a test main that lists, selects and reports GoogleTest's cases" license = "Apache-2.0" diff --git a/adapters/gtest/src/protocol.cpp b/adapters/gtest/src/protocol.cpp index 90f0bca..482fbc1 100644 --- a/adapters/gtest/src/protocol.cpp +++ b/adapters/gtest/src/protocol.cpp @@ -128,7 +128,7 @@ int mcpplibs_mtest_gtest_main(int argc, char** argv) { {"protocol", 1}, {"level", 1}, {"framework", "googletest"}, - {"version", "0.5.3"}, + {"version", "0.6.0"}, {"isolation", {"process", "case"}}, {"guide", "https://github.com/mcpp-community/mtest/blob/main/adapters/gtest/docs/" "writing-tests.md"}}); diff --git a/capi/mcpp.toml b/capi/mcpp.toml index 2149f80..c0e566a 100644 --- a/capi/mcpp.toml +++ b/capi/mcpp.toml @@ -4,7 +4,7 @@ [package] namespace = "mcpplibs.mtest" name = "capi" -version = "0.5.3" +version = "0.6.0" description = "mtest's C ABI: the protocol's events for any language with a C FFI" license = "Apache-2.0" diff --git a/capi/src/mtest.cpp b/capi/src/mtest.cpp index 94bdce3..b81633b 100644 --- a/capi/src/mtest.cpp +++ b/capi/src/mtest.cpp @@ -116,7 +116,7 @@ void mtest_init(const char* framework, int level) { {"protocol", 1}, {"level", std::clamp(level, 1, 3)}, {"framework", text(framework)}, - {"version", "0.5.3"}, + {"version", "0.6.0"}, {"isolation", {"process", "case"}}}); } diff --git a/conformance/mcpp.toml b/conformance/mcpp.toml index e698f1c..4dac10e 100644 --- a/conformance/mcpp.toml +++ b/conformance/mcpp.toml @@ -8,7 +8,7 @@ [package] namespace = "mcpplibs.mtest" name = "conformance" -version = "0.5.3" +version = "0.6.0" description = "The conformance suite of the mtest protocol, for every implementation of its program side" license = "Apache-2.0" @@ -33,5 +33,5 @@ main = "src/fixture.cpp" [dependencies] mcpplibs.mtest = { path = "../lib" } -mcpplibs.os = "0.4.1" +mcpplibs.os = "^0.4.1" nlohmann.json = "3.12.0" diff --git a/engine/mcpp.toml b/engine/mcpp.toml index e0bff66..0bf81dd 100644 --- a/engine/mcpp.toml +++ b/engine/mcpp.toml @@ -10,7 +10,7 @@ [package] namespace = "mtest" name = "mtest" -version = "0.5.3" +version = "0.6.0" description = "The mtest engine: run, schedule, store, explain and analyse tests of any language" license = "Apache-2.0" authors = ["mcpp-community"] @@ -30,7 +30,7 @@ main = "src/main.cpp" [dependencies] mcpplibs.mtest.manifest = { path = "../manifest" } # the static case manifest -mcpplibs.os = "0.4.1" +mcpplibs.os = "^0.4.1" mcpplibs.cmdline = "0.0.2" nlohmann.json = "3.12.0" marzer.tomlplusplus = "3.4.0" diff --git a/engine/src/commands.cpp b/engine/src/commands.cpp index 6126cc4..20fbdf3 100644 --- a/engine/src/commands.cpp +++ b/engine/src/commands.cpp @@ -1834,7 +1834,7 @@ json catalog() { plain({{"out", {{"type", "string"}}}, {"argv", {{"type", "array"}, {"items", {{"type", "string"}}}}}}), true); - return {{"schema", "mtest.catalog/1"}, {"version", "0.5.3"}, {"tools", tools}}; + return {{"schema", "mtest.catalog/1"}, {"version", "0.6.0"}, {"tools", tools}}; } } // namespace mcpplibs::mtest::engine diff --git a/engine/src/main.cpp b/engine/src/main.cpp index e0cf804..6675713 100644 --- a/engine/src/main.cpp +++ b/engine/src/main.cpp @@ -98,7 +98,7 @@ const json* tool_named(const json& tools, std::string_view name) { cmdline::App application(const json& tools) { auto app = cmdline::App("mtest") - .version("0.5.3") + .version("0.6.0") .description("run tests of any language, and ask about their runs") .option(cmdline::Option("json").global().help("print the answer's data")); for (const auto& t : tools) @@ -253,7 +253,7 @@ int main(int argc, char** argv) { const std::string name(parsed->subcommand_name()); const auto& sub = parsed->subcommand()->get(); if (name == "version") { - std::println("mtest 0.5.3"); + std::println("mtest 0.6.0"); return 0; } const auto cwd = std::filesystem::current_path(); diff --git a/lib/docs/writing-tests.md b/lib/docs/writing-tests.md index 7adc974..f386727 100644 --- a/lib/docs/writing-tests.md +++ b/lib/docs/writing-tests.md @@ -53,7 +53,7 @@ E2E cli{"cli", resource{"serial:home"}, timeout{std::chrono::minutes{2}}, [] { / The entry enters the child before any case; the copy runs no case and reports nothing. The files of one directory are then one program (mcpp `[test] programs = "directory"`) however many of them start copies. -- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.5.3"`. +- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.6.0"`. - Identifiers are stable: a selection, a history and a shard name a case by them. ## Expectations diff --git a/lib/mcpp.toml b/lib/mcpp.toml index dea2e97..8300a35 100644 --- a/lib/mcpp.toml +++ b/lib/mcpp.toml @@ -17,7 +17,7 @@ [package] namespace = "mcpplibs" name = "mtest" -version = "0.5.3" +version = "0.6.0" description = "The C++ test library of mtest: cases, expectations, fixtures, bench, conformance and end-to-end helpers, with no macro and no header, speaking the mtest protocol" license = "Apache-2.0" authors = ["mcpp-community"] @@ -44,6 +44,10 @@ main = "src/entry/main.cpp" test-protocol = 1 test-main = true test-guide = "docs/writing-tests.md" +# What the library implements of the mcpp test spec's optional capabilities (mcpp +# docs/specs/test.md, B6), and the session a client starts (O9, `mtest serve`). +test-capabilities = ["check-details", "metrics", "scheduling", "fixtures", "usage", "output", "session"] +test-session = ["mtest", "serve"] [dependencies] # A range, not one release: the library is in the test graph of every package diff --git a/manifest/mcpp.toml b/manifest/mcpp.toml index 75aec6b..6921d76 100644 --- a/manifest/mcpp.toml +++ b/manifest/mcpp.toml @@ -8,7 +8,7 @@ [package] namespace = "mcpplibs.mtest" name = "manifest" -version = "0.5.3" +version = "0.6.0" description = "The static case manifest of mtest: the cases a source file declares, read without building it" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/mcpp.toml b/mcpp.toml index cf1b623..5e66d8b 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -28,4 +28,4 @@ toolchain = "llvm@23.1.3" toolchain = "llvm@23.1.3" [workspace.target.'cfg(os = "openkal")'] -platform = "openkal-platform@0.24.4" +platform = "openkal-platform@0.25.0" diff --git a/spec/protocol.md b/spec/protocol.md index 6d15d31..11d6edf 100644 --- a/spec/protocol.md +++ b/spec/protocol.md @@ -1,173 +1,48 @@ # The mtest test protocol, version 1 -**Status:** version 1, 2026-11-08; P3.3, P3.4, P6.5 and the `fixture` event added 2026-11-08; P6.6, P7.3, P10, P11, P12 and the `usage` and `component` events added 2026-11-09 (P9.2). -**Scope:** how a test program and a runner talk. A runner is anything that starts test programs and reads their results: the mtest engine, mcpp (`mcpp test`), an IDE, a CI step. A test program is any program, in any language. +**Status:** the normative text is the mcpp test spec, SPEC-017 v2.0 +([mcpp docs/specs/test.md](https://github.com/mcpp-community/mcpp/blob/main/docs/specs/test.md)), +from mtest 0.6.0 (2026-11-10). The spec keeps this protocol's clause numbers, P1 to P12, so a test, +a conformance report or a document that names `P3.2` names the same clause there; it adds the base +capabilities B1–B6 that every runner and framework implements and the optional capabilities O1–O9 +that a framework or its tools declare. -This document is normative. The words *must*, *must not* and *may* have their usual meaning. Each clause has a number; tests and conformance reports name clauses by number (`P3.2`). +**Scope of this file:** what mtest implements of the spec, part by part. A runner or a client relies +on the spec; this list says which of its clauses mtest's parts answer, and since which release. -## P1. Levels +## The library (`mcpplibs.mtest`) -| Level | What the program does | What the runner obtains | +| Clause | What the library does | Since | |---|---|---| -| L0 | nothing of this protocol | one result per program, from its exit status | -| L1 | P3, P4, P5 | the case list, the selection of cases, a result, a location and a message per case | -| L2 | L1 and P6 | metrics: samples, units, bench verdicts, conformance clauses | -| L3 | L2 and P7 | resources and artifacts per case: scheduling and affected selection | - -P1.1 A program that does not implement the protocol is an L0 program. A runner must run it unchanged and must judge it by its exit status alone: zero passes, anything else fails. - -P1.2 A runner must not pass the protocol's variables (P2) to a program it does not know to implement the protocol, unless it accepts that an L0 program runs completely (P2.4). - -## P2. The handshake - -P2.1 A runner states the protocol in the program's environment. All variables are optional except `MTEST_PROTOCOL`. - -| Variable | Value | -|---|---| -| `MTEST_PROTOCOL` | the protocol version the runner speaks: `1` | -| `MTEST_MODE` | `list` or `run` (default `run`) | -| `MTEST_REPORT` | a file the program appends its events to (P5); absent: no events are written | -| `MTEST_SELECT` | a file of case identifiers, one per line (P4); absent: every case | -| `MTEST_ISOLATION` | `process` (several cases in this process, the default) or `case` (the runner starts the program once per case) | -| `MTEST_TMPDIR` | a directory the program may use for its cases, removed by the runner | -| `MTEST_SEED` | the seed of anything random in the cases, as a decimal number | -| `MTEST_MONITOR` | how closely the program measures its use of the machine: `off`, `basic`, `detailed`, `trace` (P10) | -| `MTEST_MONITOR_SAMPLE` | `trace`: the milliseconds between two samples (P10.3) | -| `MTEST_FIXTURE_` | the endpoint of the run fixture `` (P11.2), its name upper-cased with each character that is not a letter or a digit written `_` | -| `MTEST_SERVE` | the run fixture the program serves instead of running cases (P11.3) | - -P2.2 A program that implements the protocol must not pass these variables to a program it starts (each is removed from the started program's environment), so that a test program started by a test is not taken for the runner's. A test that starts a test program as a runner sets the variables it means for it. - -P2.3 A program states that it implements the protocol in two ways: -- by a host declaration: a build system's manifest names the protocol for the program (mcpp: the test entry's package declares `test-protocol = 1`; mcpp docs/08); -- by a marker: the bytes `mtest-protocol=1` appear in the program's file, followed by a byte that is not a digit. A runner may read the file to find it. A program built from mtest's libraries carries the marker. - -Either statement is a claim the program's report confirms: a program that writes no `program` event is L0 (P2.4), as a file of plain `main` linked with a library that carries the marker is. - -P2.4 A runner that knows neither may still set `MTEST_MODE=list` and `MTEST_REPORT`. The first event (P5.2) tells it the level. When no `program` event arrives, the program is L0, and in `list` mode it has run all of its tests once. - -P2.5 A framework may state where its guide to writing tests is: a file of its package that a writer of a test, a person or an agent, reads before writing one or after one failed. A host declaration names it (mcpp: the entry's `test-guide = ""`; `mcpp test --guide` lists each framework with its guide), and the `program` event may carry `guide`, a URL, for a runner that learns of the framework only when it runs the program. A runner that names a failure may name the guide beside it. The guide states how a file of cases is written, how an expectation is stated and what its failure reports, and how one case is run. - -## P3. Listing - -P3.1 With `MTEST_MODE=list` the program must write one `program` event, then one `case` event per case it would run, then one `done` event, and must exit with status 0 without running any case. - -P3.2 A case identifier is a string of UTF-8. It is stable: the same case has the same identifier in every run and on every system. A program forms it from the names it gives its suites and cases, joined by `/` (for example `fs/a file is read`). Identifiers in one program are distinct. - -P3.3 A case event may name the case's location (`file`, `line`), kind (P8), tags, the clauses it covers, its expected status, and (L3) its resources and artifacts, and how a runner that schedules cases runs it: `timeout`, the seconds it may run; `isolation`, `case` when it runs in a process of its own; `retries`, how many times a failing case runs again, alone; `diagnose`, an object of `env` (variables) and `timeout`, the run a runner makes once after the case fails, whose output it keeps and which judges nothing. A runner that starts a process for one case ends it at the case's `timeout`; one that runs a program whole (mcpp's `mcpp test`) may run every case in it and judges each as reported. A case that names no kind is of the kind the runner gives its program (mtest docs/layout.md). - -P3.4 A case that exists only when another case runs (a case declared inside a case) need not be listed; it appears in the report when it runs, and its `begin` event may name its kind, tags and clauses. Selecting it runs the case that declares it as its scope: that case is reported only when a statement of its own fails. - -## P4. Selection - -P4.1 With `MTEST_SELECT` the program must run exactly the cases whose identifiers are lines of the file, compared byte for byte after removing one trailing `\r`. An identifier that names no case is reported as a `missing` event (P5) and fails the run. - -P4.2 Patterns are the runner's: a runner that offers `fs/*` resolves it against the list (P3) and passes identifiers. The protocol has no pattern syntax, so no escaping either. - -## P5. Reporting - -P5.1 Events are JSON objects, one per line (NDJSON, UTF-8, `\n` after each). The program appends them to `MTEST_REPORT` and must not write anything else to that file. Each line is written whole: a program with several threads must not interleave lines. - -P5.2 The first event is `program`. Every event has the field `e`, its kind. - -| `e` | Fields | When | +| B2: P1–P5, P8, P9 | a program built with it speaks the protocol at level 3 when a runner asks (P2.1), lists (P3), selects (P4) and reports (P5) | 0.1.0 | +| B6 | its entry (`mtest/main`) states `test-protocol = 1`, `test-main`, `test-guide`, `test-capabilities` and `test-session` | 0.1.0; the last two 0.6.0 | +| O3 `check-details` | a failure's `expected`, `actual`, `source` (the statement) and `fatal`; a case's `checks` (`held`, `failed`) at its `end` | `expected`/`actual` 0.1.0; the rest 0.6.0 | +| O4 `metrics` | `metric` events; bounds judged at the case's end (P6.5); standard names with their precision (P6.6) | 0.1.0; P6.5 0.4.0; P6.6 0.5.0 | +| O5 `scheduling` | `resources` and `artifacts` (P7); `timeout`, `isolation`, `retries`, `diagnose` (P3.3) in the `case` events | 0.1.0; P3.3's 0.3.0 | +| O6 `fixtures` | `shared` and the `fixture` event; components and run fixtures (P11), served under `MTEST_SERVE` | 0.3.0; P11 0.5.0 | +| O7 `usage` | the program's `usage` at its end, and the cases' use at `MTEST_MONITOR` (P10) | 0.5.0 | +| O8 `output` | each case's output, kept while it runs under a runner and stated as `output` events; a case declared inside another keeps its own | 0.6.0 | + +## The engine (`mtest`) + +| Clause | What the engine does | Since | |---|---|---| -| `program` | `protocol` (1), `level` (1–3), `framework`, `version`, `isolation` (the values of P2.1 the program supports), optionally `guide` (P2.5) | first | -| `case` | `id`, `name`, `file`, `line`, `kind`, `tags`, `clauses`, `expect`, `resources`, `artifacts`, optionally `uses` (P11.2) | in `list` mode, one per case | -| `component` | `name` and `pools` (`{name, capacity}`, P7.3), or `fixture` and `scope` (P11.1) | in `list` mode, one per component and run fixture | -| `begin` | `id`, `t` (milliseconds since the program started), optionally `kind`, `tags`, `clauses` (P3.4) | a case starts | -| `failure` | `id`, `file`, `line`, `message`, and optionally `expected`, `actual` (strings), `fatal` | an expectation does not hold | -| `output` | `id`, `stream` (`out` or `err`), `text` | output a case captured | -| `metric` | `id`, `name`, `value`, `unit`, optionally `samples` (array of numbers), `better` (`lower` or `higher`), `precision` (P6.6) | L2 | -| `usage` | `user_ms`, `sys_ms`, optionally `peak_bytes`, `read_bytes`, `written_bytes` | the program's own use at its end (P10.2) | -| `artifact` | `id`, `path`, `kind` | a file the case produced | -| `fixture` | `id`, `name`, `ms` | an object the case caused to be made for the cases that share it, and how long making it took | -| `end` | `id`, `status`, `ms`, optionally `reason` | a case ends | -| `missing` | `id` | P4.1 | -| `done` | `status`, `counts` (an object of status to number) | last | - -P5.3 A case's `status` is one of `pass`, `fail`, `skip` (with `reason`), `xfail` (failed as expected), `xpass` (passed although expected to fail; a failure of the run), `invalid` (a measurement that cannot be judged; P6.3). - -P5.4 Unknown fields and unknown event kinds must be ignored by a runner. A runner must accept a report that ends without `done` (the program crashed): the cases that have a `begin` without an `end` failed, with the program's exit status as their reason. - -P5.5 The program's exit status is zero exactly when no case failed or passed unexpectedly and no selected case is missing. - -## P6. Metrics (L2) - -P6.1 A metric belongs to a case and has a name unique within it, a value and a unit (`s`, `ms`, `B`, `1` for a ratio or a count). +| B2, runner side | starts programs of any language, by the protocol or by their exit status (P1.1) | 0.1.0 | +| B4 | keeps every run in the store (P12), and reads the runs a host (`mcpp test`) keeps | 0.1.0; the store's form specified 0.5.0 | +| O5 | schedules cases by their resources and history, by pools since 0.5.0; ends a case at its `timeout`, runs `retries` and `diagnose` since 0.3.0 | 0.1.0 | +| O6 | serves run fixtures once per run (P11.2, P11.3) | 0.5.0 | +| O7 | measures a program that reports no use from a process of its own (P10.4) | 0.5.0 | +| O9 `session` | `mtest serve` answers `test/plan`, `test/run`, `test/add`, `test/cancel`, `test/status` and `shutdown`, and states `test/build`, `test/case`, `test/program`, `test/run`; it builds nothing, and starts each program from the launch mcpp gives (B5) | 0.6.0 | -P6.2 A bench case reports its samples. The runner, not the program, decides whether a measurement is valid and whether it passes, by the policy the project declares (pairs, spread bound, limit, whether a measurement may be repeated). +## The adapters and the C ABI -P6.3 A program may itself report `invalid` when it knows its measurement cannot be judged (for example, the machine was too noisy by its own test). A runner must not count an `invalid` case as passed or failed, and its policy decides whether it may be repeated. - -P6.4 A conformance case names in `clauses` the clauses of the specification it covers. A runner forms the matrix of clause × implementation × system from them. - -P6.5 A case may bound its metrics: `bounds`, a list of `{metric, max, min}` (either limit may be absent), in the metric's unit, in its `case` event and, for a case not listed, its `begin` (P3.4). The program judges them as expectations at the case's end: a metric over its `max` or under its `min`, or a bounded metric the case did not report, is a `failure` of the case. A case that declared cases is judged by them. A runner reads the bounds to show a metric beside its limit (mtest's `analyze trend`); a measurement against a baseline stays the runner's (P6.2). - -P6.6 The use of the machine is reported as metrics of standard names, each with its `precision`: `exact` (the case's own, measured at its bounds), `process` (the program's, of which the case is a part), `approx` (estimated), `sampled` (P10.3). - -| Metric | Unit | What | +| Part | Level | Clauses | |---|---|---| -| `cpu.user`, `cpu.sys` | `ms` | processor time in user and system mode | -| `mem.peak` | `B` | the largest resident set | -| `mem.delta` | `B` | the growth of the largest resident set over the case | -| `mem.rss` | `B` | the resident set, sampled | -| `cpu.load` | `1` | the processors in use, sampled | -| `io.read`, `io.write` | `B` | bytes read and written | -| `net.sent`, `net.recv` | `B` | bytes sent and received | -| `alloc.count`, `alloc.bytes` | `1`, `B` | allocations and their bytes | -| `proc.spawned` | `1` | programs started | - -## P7. Resources and artifacts (L3) - -P7.1 `resources` names what a case needs exclusively or in limited number: strings such as `serial:`, `machine` (alone on the machine), `gpu`, `network`, `display`. A runner must not run two cases that hold the same `serial:` group, or a `machine` case with any other, at the same time. - -P7.2 `artifacts` names what a case reads: build targets (`target:`), files and directories relative to the project (`file:`), external programs (`program:`). A runner selects the cases affected by a change through them. - -P7.3 A resource may have a capacity: a pool a component brings (P11.1) states how many cases hold it at once. A runner must not run more holders of a pool than its capacity; a resource with no stated capacity has one. - -## P8. Kinds - -`unit`, `e2e`, `cts` and `bench`. A kind states what is tested; the set a case belongs to (smoke, regression) is the runner's and the project's (mtest docs/sets.md), not the program's. - -## P9. Versions - -P9.1 A runner that speaks version N sets `MTEST_PROTOCOL=N`. A program that implements only lower versions answers with the highest one it implements in its `program` event; the runner then uses that version or treats the program as L0. - -P9.2 Within a version, fields and event kinds are only added. A field changes meaning only in a new version. - -## P10. Use of the machine - -P10.1 A runner states the level in `MTEST_MONITOR`. `off`: nothing is measured. `basic`: each case's processor time (P6.6, `exact`), and the program's own use at its end. `detailed`: also each case's largest resident set and its growth, and the bytes it read and wrote. `trace`: also the processor load and the resident set over each case, sampled. A case may ask for a level higher than the runner's; the higher one applies to it. - -P10.2 A program at a level other than `off` writes one `usage` event before `done`: its processor time, and where the system tells them, its largest resident set and the bytes it read and wrote. - -P10.3 At `trace`, a program samples every `MTEST_MONITOR_SAMPLE` milliseconds (50 when absent) while a case runs and reports each series as one metric whose `samples` are the series and whose `value` is its largest sample, with precision `sampled`. - -P10.4 A runner measures a program that writes no `usage` event from a process of its own that starts the program, where the system accounts a process's children (Linux, macOS), and records the measurement as the engine's. - -## P11. Components and run fixtures - -P11.1 A component is a named part of tests that programs share: the pools of resources it brings (P7.3) and the run fixtures it serves. A program lists each in a `component` event (P5.2). - -P11.2 A run fixture is one object for a whole run, used by the cases of several programs: a sandboxed home, a server, an index. A case states the run fixtures it uses in `uses`. A runner serves a run fixture before the first case that uses it starts and ends it after the last ends, and gives its endpoint to every program of a case that uses it as `MTEST_FIXTURE_`. - -P11.3 A runner serves a run fixture by starting a program that lists it with `MTEST_SERVE` naming it, with its input a pipe and without `MTEST_PROTOCOL`. The program runs no case: it prepares the fixture, writes the line `mtest-ready ` to its output, and ends the fixture when its input ends. A runner may also serve a run fixture by a command its project names. - -P11.4 A run fixture that does not become ready is the machine's failure: the cases that use it end with status `infra` and do not run. - -## P12. The store - -P12.1 A runner may keep its runs in a store, which other runners and tools read and write in this form (mtest's `.mtest`; a host's runner, such as mcpp's `mcpp test`, writes the same): - -| Path | Content | -|---|---| -| `index.ndjson` | one line per run: the run's record without its programs and cases | -| `runs//run.json` | the run's record: `schema` (`mtest.run/1`), `id`, `started`, `finished`, `status`, `system`, `arch`, `machine`, `commit`, `set`, `selection`, `group`, `monitor`, `ms`, `jobs`, `counts` | -| `runs//programs.ndjson` | one line per process of a program: `name`, `argv`, `status`, `exit`, `ms`, `started_ms` (after the run began), `worker`, `select`, `resources`, `fixtures`, `build_started_ms` and `build_ms` when the runner built it, `usage` (P10.2, with `source`: `program` or `engine`) | -| `runs//cases.ndjson` | one line per case: `program`, `id`, `kind`, `status`, `ms`, `started_ms` (after the run began), `reason`, `failures`, `metrics`, `bounds`, `tags`, `clauses` | -| `runs//fixtures.ndjson` | one line per run fixture served: `name`, `program`, `started_ms`, `ready_ms`, `ended_ms`, `status` | +| `mcpplibs.mtest.gtest` (GoogleTest) | L1 | B2 | +| the Bash and Python adapters (`adapters/script`) | L3 | B2, O4, O5 ([docs/adapters.md](../docs/adapters.md)) | +| `mcpplibs.mtest.capi` (C ABI) | L3 | B2, O4, O5 | -P12.2 A run identifier sorts by time. Runs of one `group` (the jobs of a CI run) are parts of one whole, which a reader reports together. +## Conformance -P12.3 The times of a run's programs, cases, builds and fixtures form a directed acyclic graph of its work: a build before the processes of its program, a worker's process before its next, the holders of one exclusive resource in turn, a run fixture's serving before its users and its end after them. A reader's critical path is the graph's longest weighted path, weighted by the measured times. +`conformance/` checks a program against the clauses it states, and reports by clause number +([conformance.md](conformance.md)). From c2e68e675bb51c6bfa39ff98116f67cd2dbf653c Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Tue, 10 Nov 2026 14:43:38 +0900 Subject: [PATCH 5/7] lib tests: a case's output is checked where the streams can be kept An ok image cannot rearrange its standard streams (os::stdio::capture answers not_supported); there a case's output is the program's, the spec's fallback, and the two cases that read it are skipped with that reason. --- lib/tests/output.cpp | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/lib/tests/output.cpp b/lib/tests/output.cpp index 880bdf5..d1aad6a 100644 --- a/lib/tests/output.cpp +++ b/lib/tests/output.cpp @@ -6,6 +6,7 @@ import std; import mcpplibs.mtest; import mcpplibs.mtest.e2e; import nlohmann.json; +import mcpplibs.os; using namespace mcpplibs::mtest; @@ -25,6 +26,16 @@ child talker{"talker", [](std::vector) { }}; namespace { +// Where the streams cannot be rearranged (an ok image), a case's output is not kept and the +// program's output is the program's (spec O8's fallback). +bool streams_kept() { + auto c = mcpplibs::os::stdio::capture::begin(mcpplibs::os::stdio::stream::error); + if (!c) + return false; + (void)c->text(); + return true; +} + std::vector run_talker(std::string& out) { const auto report = temp_dir() / "report.ndjson"; auto o = e2e::run({.argv = child_command("talker", {}), @@ -54,6 +65,8 @@ std::string output_of(const std::vector& events, std::string_vie UTest output{"output", [] { "a case states what it wrote, on each stream"_test = [] { + if (!streams_kept()) + skip("this system's streams cannot be kept for a case"); std::string out; const auto events = run_talker(out); expect( @@ -65,6 +78,8 @@ UTest output{"output", [] { }; "a declared case's output is its own, and the declaring case keeps the rest"_test = [] { + if (!streams_kept()) + skip("this system's streams cannot be kept for a case"); std::string out; const auto events = run_talker(out); expect(eq(output_of(events, "outer/inner", "out"), From e2328e55ee263351af6669b704943100c8e7c680 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Tue, 10 Nov 2026 14:43:47 +0900 Subject: [PATCH 6/7] spec/protocol.md: where a case's output is not kept --- spec/protocol.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/protocol.md b/spec/protocol.md index 11d6edf..417a98f 100644 --- a/spec/protocol.md +++ b/spec/protocol.md @@ -21,7 +21,7 @@ on the spec; this list says which of its clauses mtest's parts answer, and since | O5 `scheduling` | `resources` and `artifacts` (P7); `timeout`, `isolation`, `retries`, `diagnose` (P3.3) in the `case` events | 0.1.0; P3.3's 0.3.0 | | O6 `fixtures` | `shared` and the `fixture` event; components and run fixtures (P11), served under `MTEST_SERVE` | 0.3.0; P11 0.5.0 | | O7 `usage` | the program's `usage` at its end, and the cases' use at `MTEST_MONITOR` (P10) | 0.5.0 | -| O8 `output` | each case's output, kept while it runs under a runner and stated as `output` events; a case declared inside another keeps its own | 0.6.0 | +| O8 `output` | each case's output, kept while it runs under a runner and stated as `output` events; a case declared inside another keeps its own. Where the system cannot rearrange the program's streams (an ok image), no output is kept for a case, the spec's fallback | 0.6.0 | ## The engine (`mtest`) From 8ad8fb03505369cbf636c3bbcb594994f559132c Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Tue, 10 Nov 2026 16:53:52 +0900 Subject: [PATCH 7/7] CI on mcpp 2026.11.10.1, which reads the entry's test-capabilities and test-session --- .github/versions.env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/versions.env b/.github/versions.env index d2b40d3..d44f49a 100644 --- a/.github/versions.env +++ b/.github/versions.env @@ -1,4 +1,4 @@ # The versions CI builds and tests with; .github/actions/setup-mcpp exports them. -MCPP_VERSION=2026.11.9.1 +MCPP_VERSION=2026.11.10.1 # The xlings that mcpp release bundles. XLINGS_VERSION=v2026.10.26.1