From af3df4aaf263ac881f6a100224d940f7f9fb188d Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Sat, 14 Nov 2026 04:04:52 +0900 Subject: [PATCH] 0.8.0: where a case runs, stated by the case; capabilities answered by the program - only{} and except{} state the platforms and architectures of mcpplibs.os a case or a suite runs on; elsewhere it is skipped with one reason. The script adapters and the C ABI state the same; the engine skips a listed case outside its places without starting it. - mtest re-exports mcpplibs.os's platform, architecture, can_run_on, host and on

: code for one platform is selected when it is compiled, and an ok image decides when it is loaded. - capability declares what a machine can do in code, by the tests of [capabilities] and a function; the program answers it once per process under any runner, and a case lacking it is skipped. The engine takes the program's answers and records them. - A case event may carry reason, only, except and capabilities (P3.3; schema, conformance). - mtest ci coverage requires a case to pass on each of its places a collected run covered. --- .agents/docs/2026-11-08-mtest-design.md | 14 ++ CHANGELOG.md | 29 +++ README.md | 8 +- README.zh-CN.md | 4 +- adapters/gtest/mcpp.toml | 2 +- adapters/gtest/src/protocol.cpp | 2 +- adapters/script/mtest.sh | 47 ++++- adapters/script/mtest_report.py | 39 +++- adapters/script/tests/fixture.py | 5 + adapters/script/tests/fixture.sh | 1 + capi/fixture/main.c | 4 + capi/include/mtest.h | 6 + capi/mcpp.toml | 2 +- capi/src/mtest.cpp | 14 +- conformance/mcpp.toml | 2 +- conformance/src/fixture.cpp | 1 + conformance/tests/protocol.cpp | 21 +- docs/adapters.md | 8 + docs/commands.md | 2 +- docs/config.md | 6 +- engine/mcpp.toml | 2 +- engine/src/cicheck.cpp | 61 ++++++ engine/src/commands.cpp | 2 +- engine/src/main.cpp | 4 +- engine/src/model.cpp | 17 +- engine/src/model.cppm | 6 + engine/src/schedule.cpp | 10 + engine/src/select.cpp | 42 +++- engine/src/service.cpp | 2 +- engine/tests/engine.cpp | 83 ++++++++ lib/docs/writing-tests.md | 42 +++- lib/mcpp.toml | 2 +- lib/src/mtest.cpp | 245 +++++++++++++++++++++++- lib/src/mtest.cppm | 102 +++++++++- lib/tests/places.cpp | 164 ++++++++++++++++ manifest/mcpp.toml | 2 +- spec/protocol.md | 2 + spec/schema/event-v1.json | 25 +++ 38 files changed, 995 insertions(+), 35 deletions(-) create mode 100644 lib/tests/places.cpp diff --git a/.agents/docs/2026-11-08-mtest-design.md b/.agents/docs/2026-11-08-mtest-design.md index 8bc6ccc..3d684ba 100644 --- a/.agents/docs/2026-11-08-mtest-design.md +++ b/.agents/docs/2026-11-08-mtest-design.md @@ -311,3 +311,17 @@ mtest 不止是开发者写测试的接口：围绕测试结果的数据、分 - **M3** 已决定：首批项目 PR CI 墙钟中位数减半，发布 CI 关键路径缩短 30%，以迁移前测量的正式基线为准。 - **M4** 已决定：其他语言以工具支持与薄的 FFI 接口（C ABI 报告库）接入，不提供其他语言的完整测试库。 - **M5** 已决定：本地存储为按运行分文件的 NDJSON 加索引，可直接作为 CI 产物。 + +## 14. 用例的运行场所与机器能力（0.8.0） + +**问题。** 0.7.0 的 `[capabilities]` 只由引擎在选择时求值。实现协议的其他运行器（`mcpp test`）与单独运行的程序不求值，使用方因而在用例上同时写标签 `requires:<词>`、在用例体内再次判断，并另维护一份探测表；用例必须通过的平台另写在 `[ci.coverage] hosts` 的表中，与用例分离。 + +**决定。** + +1. **场所。** `only{...}` 与 `except{...}` 以 mcpplibs.os 的平台与架构陈述用例在何处运行：`only` 的平台之间、架构之间为"或"，二者之间为"与"；套件与用例的 `only` 取交集，`except` 累加。场所之外，用例跳过，原因为 `only on <场所> (this host: <系统> <架构>)` 或 `not on <场所> (...)`。库、Bash 与 Python 适配器由程序自行判定；引擎依据清单中的 `only`、`except` 同样判定，以覆盖只列出场所的实现（C ABI）。 +2. **编译期选择。** 不新增宏或常量：mtest 重新导出 mcpplibs.os 的 `platform`、`architecture`、`can_run_on`、`host`、`on

`。`on

([&](auto){...})` 的函数体在不能运行于 `P` 的构建中被丢弃，承担 `if constexpr` 的作用；openkal 镜像在加载时得知 `host()`，与第 3.1 节一致。 +3. **能力。** `capability` 以 `[capabilities]` 的同一组检验（系统、架构、PATH 上的程序、glob、命令、环境变量、`all`/`none`）加一个函数在代码中声明一次，由程序在每个进程中求值一次，并作为属性使用（`"x"_test[gcc]`，同时带标签 `requires:gcc`）。缺失时用例跳过，原因与引擎相同：`missing capability: <名> (<未成立的检验>)`。在模块接口中声明的能力先于导入它的单元初始化，因此命名空间作用域的套件可以使用。 +4. **协议。** `case` 事件新增可选字段 `only`、`except`、`capabilities`（名到布尔）与 `reason`（`expect` 为 `skip` 时的原因）。引擎对程序已回答的能力词不再要求 `[capabilities]` 声明，并把答案记入运行记录；`[capabilities]` 继续服务于脚本程序。 +5. **覆盖。** `mtest ci coverage` 要求陈述了 `only` 的用例在所收集运行覆盖到的每个场所上通过，且至少有一个运行位于其场所；场所由用例自身陈述，不再需要表格。 + +**兼容。** 新字段均为可选；未陈述场所与能力的程序、清单与配置的含义不变。 diff --git a/CHANGELOG.md b/CHANGELOG.md index 44b236c..2da2b64 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,34 @@ # Changelog +## [0.8.0] - 2026-11-14 + +- **Where a case runs is stated by the case** (`only{}`, `except{}`). A case or a suite names the + platforms and architectures of `mcpplibs.os` it runs on (`only{platform::windows}`, + `only{platform::linux, architecture::x86_64}`) or not (`except{platform::macos}`); a suite's and + its case's narrow together. Elsewhere it is skipped with one reason, `only on windows (this host: + linux x86_64)`. The Bash adapter (`--only`, `--except`), the Python adapter (`only=`, + `excepted=`) and the C ABI (`mtest_case_only`, `mtest_case_except`) state the same; the engine + skips a listed case outside its places without starting it. +- **Code for one platform is selected when it is compiled.** mtest re-exports mcpplibs.os's + `platform`, `architecture`, `can_run_on`, `host` and `on

`: `on([&](auto) { + ... })` is a body that a build for another platform discards, and an ok image decides when it is + loaded. +- **What a machine can do is declared in code and answered by the program** (`capability`). The + tests of `[capabilities]` -- a system, an architecture, programs on PATH, a file a glob matches, a + command, environment values, other capabilities that must or must not hold -- and a function for + what they cannot state, answered once per process. A capability is an attribute + (`"links"_test[gcc]`, with the tag `requires:gcc`); a case whose machine lacks it is skipped, + `missing capability: gcc (no g++ on PATH)`, under any runner (`mtest`, `mcpp test`, the program + alone). The engine takes the program's answers: a word the program answers needs no + `[capabilities]` entry, and the run records it. +- **The listing says why a case is skipped, and where it runs** (P3.3): a `case` event may carry + `reason` (with `expect: skip`), `only`, `except` and `capabilities`; the schema states them. The + conformance suite checks them of every level-3 implementation (`placed`). +- **`mtest ci coverage` checks each case on its places**: a case that states `only` must pass on + each of its places that a collected run covered, and some run must have been on one -- what + `[ci.coverage] hosts` states of a program, stated by the case. +- Additive: a program, a listing and a configuration that state none of these mean what they did. + ## [0.7.2] - 2026-11-13 - **Each call measured by itself** (`bench::each`, `mcpplibs.mtest.bench`). `bench::measure` times diff --git a/README.md b/README.md index 3ed4083..0dd14b5 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ int main() { ```toml [dev-dependencies] -mcpplibs.mtest = "0.7.2" +mcpplibs.mtest = "0.8.0" ``` A failure names its place and both sides (`expected 2, actual 3`). Under a runner the same program @@ -62,6 +62,12 @@ E2E cli{"cli", resource{"serial:home"}, timeout{std::chrono::minutes{2}}, [] { }}; ``` +Where a case runs and what it needs of the machine are attributes too: `only{platform::windows}`, +`except{architecture::aarch64}`, and a `capability` declared once in code +(`const capability gcc{"gcc", {.program = {"g++"}}};`, then `"links"_test[gcc]`). The program +skips a case elsewhere, saying why, under any runner; `on([&](auto) { ... })` +is code a build for another platform discards. + ## Tests, with the engine ```bash diff --git a/README.zh-CN.md b/README.zh-CN.md index f8c6c31..3330b3b 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -37,7 +37,7 @@ int main() { ```toml [dev-dependencies] -mcpplibs.mtest = "0.7.2" +mcpplibs.mtest = "0.8.0" ``` 失败时给出位置与两侧的值（`expected 2, actual 3`）。在运行器之下，同一程序按协议报告；单独运行时每个用例输出一行。 @@ -57,6 +57,8 @@ E2E cli{"cli", resource{"serial:home"}, timeout{std::chrono::minutes{2}}, [] { }}; ``` +用例在何处运行、需要机器具备什么，同样是属性：`only{platform::windows}`、`except{architecture::aarch64}`，以及在代码中声明一次的 `capability`（`const capability gcc{"gcc", {.program = {"g++"}}};`，再写 `"links"_test[gcc]`）。无论由哪个运行器启动，程序都在场所之外跳过用例并说明原因；`on([&](auto) { ... })` 中的代码在其他平台的构建中被丢弃。 + ## 用引擎运行测试 ```bash diff --git a/adapters/gtest/mcpp.toml b/adapters/gtest/mcpp.toml index ac4ef33..7835dca 100644 --- a/adapters/gtest/mcpp.toml +++ b/adapters/gtest/mcpp.toml @@ -13,7 +13,7 @@ [package] namespace = "mcpplibs.mtest" name = "gtest" -version = "0.7.2" +version = "0.8.0" description = "GoogleTest speaking the mtest protocol: a test main that lists, selects and reports GoogleTest's cases" license = "Apache-2.0" diff --git a/adapters/gtest/src/protocol.cpp b/adapters/gtest/src/protocol.cpp index 33b25f7..19d541c 100644 --- a/adapters/gtest/src/protocol.cpp +++ b/adapters/gtest/src/protocol.cpp @@ -128,7 +128,7 @@ int mcpplibs_mtest_gtest_main(int argc, char** argv) { {"protocol", 1}, {"level", 1}, {"framework", "googletest"}, - {"version", "0.7.2"}, + {"version", "0.8.0"}, {"isolation", {"process", "case"}}, {"guide", "https://github.com/mcpp-community/mtest/blob/main/adapters/gtest/docs/" "writing-tests.md"}}); diff --git a/adapters/script/mtest.sh b/adapters/script/mtest.sh index 6bf0a1a..597bf36 100644 --- a/adapters/script/mtest.sh +++ b/adapters/script/mtest.sh @@ -54,12 +54,48 @@ _mtest_selected() { # mtest_case ID FUNCTION [--tag T] [--kind K] [--clause C] [--resource R] [--artifact A] [--timeout SECONDS] # [--isolate] [--retries N] [--diagnose-env NAME=VALUE] [--diagnose-timeout SECONDS] # [--at-most METRIC=VALUE] [--at-least METRIC=VALUE] -# [--expect-fail REASON] [--skip REASON] +# [--expect-fail REASON] [--skip REASON] [--only PLACE] [--except PLACE] +# A PLACE is a platform (linux, macos, windows) or an architecture (x86_64, aarch64, riscv64): with +# --only the case runs only on one of the platforms named, when any is, and on one of the +# architectures, when any is; with --except it does not run on any named. Elsewhere it is skipped. # A case that states no kind is of the program's (its directory's, docs/layout.md). +# This host's platform and architecture, in mcpplibs.os's names. +_mtest_host_platform() { + case "$(uname -s)" in + Linux*) echo linux ;; Darwin*) echo macos ;; MINGW*|MSYS*|CYGWIN*|Windows*) echo windows ;; + *) uname -s | tr '[:upper:]' '[:lower:]' ;; + esac +} +_mtest_host_arch() { + case "$(uname -m)" in + x86_64|amd64|AMD64) echo x86_64 ;; aarch64|arm64|ARM64) echo aarch64 ;; *) uname -m ;; + esac +} +_mtest_is_arch() { case "$1" in x86_64|aarch64|riscv64|wasm32) return 0 ;; esac; return 1; } + +# _mtest_place_skip N ONLY... EXCEPT...: why the case does not run here, or nothing. +_mtest_place_skip() { + local n="$1"; shift + local system arch here words="" w platforms=0 archs=0 platform_ok=0 arch_ok=0 i=0 + system="$(_mtest_host_platform)"; arch="$(_mtest_host_arch)"; here="$system $arch" + while [ $i -lt "$n" ]; do + w="$1"; shift; i=$((i + 1)) + words="${words:+$words, }$w" + if _mtest_is_arch "$w"; then archs=1; [ "$w" = "$arch" ] && arch_ok=1 + else platforms=1; [ "$w" = "$system" ] && platform_ok=1; fi + done + if { [ $platforms -eq 1 ] && [ $platform_ok -eq 0 ]; } || { [ $archs -eq 1 ] && [ $arch_ok -eq 0 ]; }; then + echo "only on $words (this host: $here)"; return 0 + fi + for w in "$@"; do + if [ "$w" = "$system" ] || [ "$w" = "$arch" ]; then echo "not on $w (this host: $here)"; return 0; fi + done +} + mtest_case() { local id="$1" fn="$2"; shift 2 local tags=() clauses=() resources=() artifacts=() kind="" limit="" xfail="" skip="" - local isolate="" retries="" diagnose=() diagnose_limit="" bounds=() + local isolate="" retries="" diagnose=() diagnose_limit="" bounds=() only=() except=() while [ $# -gt 0 ]; do case "$1" in --tag) tags+=("$2"); shift 2 ;; @@ -76,10 +112,13 @@ mtest_case() { --at-least) bounds+=("min:$2"); shift 2 ;; --expect-fail) xfail="$2"; shift 2 ;; --skip) skip="$2"; shift 2 ;; + --only) only+=("$2"); shift 2 ;; + --except) except+=("$2"); shift 2 ;; *) echo "mtest_case: unknown option $1" >&2; shift ;; esac done _mtest_seen+=("$id") + [ -z "$skip" ] && skip="$(_mtest_place_skip "${#only[@]}" ${only[@]+"${only[@]}"} ${except[@]+"${except[@]}"})" if [ "$_mtest_mode" = list ]; then local ev ev="\"e\":\"case\",\"id\":$(_mtest_json "$id"),\"name\":$(_mtest_json "$id"),\"file\":$(_mtest_json "${BASH_SOURCE[1]:-$0}"),\"line\":${BASH_LINENO[0]:-0},\"tags\":$(_mtest_list "${tags[@]}"),\"clauses\":$(_mtest_list "${clauses[@]}"),\"resources\":$(_mtest_list "${resources[@]}"),\"artifacts\":$(_mtest_list "${artifacts[@]}")" @@ -103,7 +142,9 @@ mtest_case() { done ev+=",\"bounds\":[$list]" fi - [ -n "$skip" ] && ev+=',"expect":"skip"' + [ ${#only[@]} -gt 0 ] && ev+=",\"only\":$(_mtest_list "${only[@]}")" + [ ${#except[@]} -gt 0 ] && ev+=",\"except\":$(_mtest_list "${except[@]}")" + [ -n "$skip" ] && ev+=",\"expect\":\"skip\",\"reason\":$(_mtest_json "$skip")" [ -n "$xfail" ] && [ -z "$skip" ] && ev+=',"expect":"fail"' _mtest_event "$ev" return 0 diff --git a/adapters/script/mtest_report.py b/adapters/script/mtest_report.py index 1b24342..70284d4 100644 --- a/adapters/script/mtest_report.py +++ b/adapters/script/mtest_report.py @@ -17,6 +17,7 @@ def reads(): import inspect import json import os +import platform import sys import time @@ -128,13 +129,42 @@ def _check_bounds(at_most, at_least): _failure(f"metric {name} = {reported[name]:g}, under its bound {limit:g} (at_least)", depth=3) +_ARCHITECTURES = ("x86_64", "aarch64", "riscv64", "wasm32") + + +def _host(): + """This host's platform and architecture, in mcpplibs.os's names.""" + system = {"Linux": "linux", "Darwin": "macos", "Windows": "windows"}.get(platform.system(), platform.system().lower()) + machine = platform.machine() + arch = {"amd64": "x86_64", "AMD64": "x86_64", "arm64": "aarch64", "ARM64": "aarch64"}.get(machine, machine) + return system, arch + + +def _place_skip(only, excepted): + """Why a case with these places does not run here, or None.""" + system, arch = _host() + here = f"{system} {arch}" + platforms = [w for w in only if w not in _ARCHITECTURES] + archs = [w for w in only if w in _ARCHITECTURES] + if (platforms and system not in platforms) or (archs and arch not in archs): + return f"only on {', '.join(only)} (this host: {here})" + for w in excepted: + if w in (system, arch): + return f"not on {w} (this host: {here})" + return None + + def case(id, tags=(), kind=None, clauses=(), resources=(), artifacts=(), expect_fail=None, skip_reason=None, - timeout=None, isolate=False, retries=None, diagnose=None, at_most=None, at_least=None): + timeout=None, isolate=False, retries=None, diagnose=None, at_most=None, at_least=None, only=(), excepted=()): """Declares a case: lists it, runs it, or passes over it, as the runner asks. A case that states no kind is of its program's (its directory's, docs/layout.md). `timeout` (seconds), `isolate`, `retries` and `diagnose` ({"env": {...}, "timeout": seconds}) are how a runner runs it (P3.3); - `at_most` and `at_least` ({metric: value}) bound its metrics (P6.5).""" + `at_most` and `at_least` ({metric: value}) bound its metrics (P6.5). `only` and `excepted` name + platforms (linux, macos, windows) and architectures (x86_64, aarch64, riscv64): the case runs only + on one of the platforms and one of the architectures `only` names, when it names any, and on none + `excepted` names; elsewhere it is skipped.""" at_most, at_least = dict(at_most or {}), dict(at_least or {}) + skip_reason = skip_reason or _place_skip(list(only), list(excepted)) def declare(fn): global _failed, _current _seen.append(id) @@ -154,8 +184,13 @@ def declare(fn): fields["diagnose"] = diagnose if at_most or at_least: fields["bounds"] = _bounds_json(at_most, at_least) + if only: + fields["only"] = list(only) + if excepted: + fields["except"] = list(excepted) if skip_reason: fields["expect"] = "skip" + fields["reason"] = skip_reason elif expect_fail: fields["expect"] = "fail" _event(**fields) diff --git a/adapters/script/tests/fixture.py b/adapters/script/tests/fixture.py index bf39df3..f21ec53 100755 --- a/adapters/script/tests/fixture.py +++ b/adapters/script/tests/fixture.py @@ -48,6 +48,11 @@ def holds(): pass +@mt.case("placed", only=["wasm32"]) +def placed(): + pass + + @mt.case("bounded", at_most={"elapsed": 0.1}) def bounded(): mt.metric("elapsed", 0.5, "s") diff --git a/adapters/script/tests/fixture.sh b/adapters/script/tests/fixture.sh index 71de070..3d97d22 100755 --- a/adapters/script/tests/fixture.sh +++ b/adapters/script/tests/fixture.sh @@ -16,6 +16,7 @@ mtest_case "group/inner" nothing mtest_case "measures" measures --kind bench mtest_case "covers" nothing --kind cts --clause P3.1 mtest_case "holds" nothing --resource serial:db --artifact file:data/x.txt +mtest_case "placed" nothing --only wasm32 slow() { mtest_metric elapsed 0.5 s; } mtest_case "bounded" slow --at-most elapsed=0.1 mtest_report diff --git a/capi/fixture/main.c b/capi/fixture/main.c index 6d97938..2cce2ad 100644 --- a/capi/fixture/main.c +++ b/capi/fixture/main.c @@ -37,6 +37,8 @@ int main(void) { mtest_case("holds", __FILE__, __LINE__, "unit"); mtest_case_resource("serial:db"); mtest_case_artifact("file:data/x.txt"); + mtest_case("placed", __FILE__, __LINE__, "unit"); + mtest_case_only("wasm32"); mtest_case("bounded", __FILE__, __LINE__, NULL); mtest_bound("bounded", "elapsed", NAN, 0.1); return mtest_done(); @@ -49,6 +51,8 @@ int main(void) { run("measures", "pass", NULL, 0); run("covers", "pass", NULL, 0); run("holds", "pass", NULL, 0); + /* The C ABI lists places and leaves the decision to its program: this one runs nowhere else. */ + run("placed", "skip", "only on wasm32", 0); run("bounded", "pass", NULL, 0); /* its bound fails it */ return mtest_done(); } diff --git a/capi/include/mtest.h b/capi/include/mtest.h index beaec93..ada8b0c 100644 --- a/capi/include/mtest.h +++ b/capi/include/mtest.h @@ -33,6 +33,12 @@ void mtest_case_tag(const char* tag); void mtest_case_clause(const char* clause); void mtest_case_resource(const char* resource); void mtest_case_artifact(const char* artifact); +/* Where the case last listed runs (0.8.0): a platform (linux, macos, windows) or an architecture + (x86_64, aarch64, riscv64). With `only` it runs only on one of the platforms named, when any is, + and on one of the architectures, when any is; with `except` on none named. The engine skips it + elsewhere, saying where; a program that runs alone decides for itself. */ +void mtest_case_only(const char* place); +void mtest_case_except(const char* place); /* How long the case last listed may run, in seconds (P3.3). */ void mtest_case_timeout(double seconds); /* The case last listed runs in a process of its own (P3.3). */ diff --git a/capi/mcpp.toml b/capi/mcpp.toml index 38c5cd3..dfab92f 100644 --- a/capi/mcpp.toml +++ b/capi/mcpp.toml @@ -4,7 +4,7 @@ [package] namespace = "mcpplibs.mtest" name = "capi" -version = "0.7.2" +version = "0.8.0" description = "mtest's C ABI: the protocol's events for any language with a C FFI" license = "Apache-2.0" diff --git a/capi/src/mtest.cpp b/capi/src/mtest.cpp index 0f9589d..ba8afb3 100644 --- a/capi/src/mtest.cpp +++ b/capi/src/mtest.cpp @@ -23,7 +23,7 @@ std::string text(const char* value) { struct pending_case { event body; - std::vector tags, clauses, resources, artifacts; + std::vector tags, clauses, resources, artifacts, only, except; bool expect_fail = false; double timeout = 0; bool isolated = false; @@ -78,6 +78,10 @@ struct state { p.body["clauses"] = p.clauses; p.body["resources"] = p.resources; p.body["artifacts"] = p.artifacts; + if (!p.only.empty()) + p.body["only"] = p.only; + if (!p.except.empty()) + p.body["except"] = p.except; if (p.expect_fail) p.body["expect"] = "fail"; if (p.timeout > 0) @@ -116,7 +120,7 @@ void mtest_init(const char* framework, int level) { {"protocol", 1}, {"level", std::clamp(level, 1, 3)}, {"framework", text(framework)}, - {"version", "0.7.2"}, + {"version", "0.8.0"}, {"isolation", {"process", "case"}}}); } @@ -161,6 +165,12 @@ void mtest_case_resource(const char* value) { void mtest_case_artifact(const char* value) { attribute(&pending_case::artifacts, value); } +void mtest_case_only(const char* value) { + attribute(&pending_case::only, value); +} +void mtest_case_except(const char* value) { + attribute(&pending_case::except, value); +} void mtest_case_timeout(double seconds) { auto& s = the(); diff --git a/conformance/mcpp.toml b/conformance/mcpp.toml index de3206e..efa275e 100644 --- a/conformance/mcpp.toml +++ b/conformance/mcpp.toml @@ -8,7 +8,7 @@ [package] namespace = "mcpplibs.mtest" name = "conformance" -version = "0.7.2" +version = "0.8.0" description = "The conformance suite of the mtest protocol, for every implementation of its program side" license = "Apache-2.0" diff --git a/conformance/src/fixture.cpp b/conformance/src/fixture.cpp index 3a27ec2..3f6d1b0 100644 --- a/conformance/src/fixture.cpp +++ b/conformance/src/fixture.cpp @@ -17,6 +17,7 @@ int main() { }; "covers"_test[kind::cts][clause{"P3.1"}] = [] {}; "holds"_test[resource{"serial:db"}][artifact{"file:data/x.txt"}] = [] {}; + "placed"_test[only{architecture::wasm32}] = [] {}; "bounded"_test[at_most{"elapsed", 0.1}] = [] { metric("elapsed", 0.5, "s"); }; return report(); } diff --git a/conformance/tests/protocol.cpp b/conformance/tests/protocol.cpp index 27a623e..f0a7514 100644 --- a/conformance/tests/protocol.cpp +++ b/conformance/tests/protocol.cpp @@ -115,8 +115,10 @@ std::vector fixture_ids(int level) { "measures", "covers", "holds"}) if (level >= 3 || s != "expected to fail") out.push_back(id(s)); - if (level >= 3) + if (level >= 3) { out.push_back(id("bounded")); + out.push_back(id("placed")); + } return out; } @@ -205,6 +207,23 @@ int main() { auto xf = find(r, "case", id("expected to fail")); expect(fatal(xf.has_value())); expect(eq(xf->value("expect", ""), std::string("fail"))); + // Where it runs (0.8.0): the places it runs only on, and a case it lists skipped says why. + auto placed = find(r, "case", id("placed")); + expect(fatal(placed.has_value())); + expect(placed->value("only", json::array()) == json::array({"wasm32"})) << placed->dump(); + if (placed->value("expect", "") == "skip") + expect(contains(placed->value("reason", ""), "only on wasm32")) << placed->dump(); + }; + "P3.3 a case outside its places is not run"_test[kind::cts][clause{"P3.3"}] = [] { + const auto r = run("run"); + if (level(r) < 3) + skip("the implementation is below level 3"); + if (os::ARCH == os::architecture::wasm32) + skip("this host is the fixture's place"); + auto end = find(r, "end", id("placed")); + expect(fatal(end.has_value())); + expect(eq(end->value("status", ""), std::string("skip"))); + expect(contains(end->value("reason", ""), "only on wasm32")) << end->dump(); }; "P4.1 a selection runs exactly the selected cases"_test[kind::cts][clause{"P4.1"}] = [] { const auto r = run("run", std::vector{id("passes"), id("group/inner")}); diff --git a/docs/adapters.md b/docs/adapters.md index 99a4840..9003a84 100644 --- a/docs/adapters.md +++ b/docs/adapters.md @@ -25,5 +25,13 @@ Every level-3 implementation states the same attributes of a case, in its own wo | diagnosis | `diagnose{}` | `--diagnose-env K=V`, `--diagnose-timeout S` | `diagnose={"env": {...}}` | `mtest_case_diagnose(k, v, s)` | | bounds (P6.5) | `at_most{}`, `at_least{}` | `--at-most M=V`, `--at-least M=V` | `at_most={M: V}`, `at_least=` | `mtest_bound(id, metric, min, max)` | | expected verdict | `expect_failure{}`, `skip_if{}` | `--expect-fail`, `--skip` | `expect_fail=`, `skip_reason=` | `mtest_case_expect_fail()` | +| places (0.8.0) | `only{}`, `except{}` | `--only P`, `--except P` | `only=[...]`, `excepted=[...]` | `mtest_case_only(p)`, `mtest_case_except(p)` | +| capabilities (0.8.0) | a `capability` | `# requires:` and `[capabilities]` | `tags=["requires:..."]` and `[capabilities]` | `mtest_case_tag("requires:...")` and `[capabilities]` | + +A place is a platform (`linux`, `macos`, `windows`) or an architecture (`x86_64`, `aarch64`, +`riscv64`). The library and the script adapters skip a case outside its places themselves, so a +program run alone agrees with a runner; the engine skips it too, for a program that only lists them +(the C ABI). A capability in C++ is answered by the program; the other implementations name the +word, and the engine answers it from `[capabilities]`. A case that states no kind is of its program's (its directory's, docs/layout.md). diff --git a/docs/commands.md b/docs/commands.md index 2f3d75a..1d3d23c 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -75,7 +75,7 @@ A run is named by its identifier, `last`, or `last-N`. | `ci annotate`, `ci collect --from DIR` | a run's failures as GitHub annotations and its Markdown report as the job's summary; a CI store taken in | | `ci report [--group G]` | every collected run of a group (default: `GITHUB_RUN_ID`) as one Markdown report, in the job's summary and `--out` | | `ci timeline --repo OWNER/REPO --run-id N` | a workflow run's jobs as a schedule with the critical path (each job waited for the job that ended last before it began), as Markdown with a gantt; `--summary` writes it to the job's summary | -| `ci coverage [--group G]` | every program of mtest.toml judged against the collected runs of a group (default `GITHUB_RUN_ID`; none: every stored run): `ran` on some host, `unaffected` (a narrowed run left it out), `excused` or `owed` (`[ci.coverage]`), else `uncovered`, which fails, as does a listed entry that ran or names no program, and a program that did not pass on a host `[ci.coverage] hosts` requires; the capability words no run's machine provided are listed | +| `ci coverage [--group G]` | every program of mtest.toml judged against the collected runs of a group (default `GITHUB_RUN_ID`; none: every stored run): `ran` on some host, `unaffected` (a narrowed run left it out), `excused` or `owed` (`[ci.coverage]`), else `uncovered`, which fails, as does a listed entry that ran or names no program, a program that did not pass on a host `[ci.coverage] hosts` requires, and a case that states its places (`only`) and did not pass on one of them that a run covered, or on none; the capability words no run's machine provided are listed | | `ci plan [--hosts H]... [--format github]` | the selection divided among the hosts (`[ci.plan]`) by each host's history: the cases its capability answers admit, their measured time there, and the shards that fit `[ci.plan] shard`; `github` writes `=[1,...,N]` and `_work=true\|false` to `GITHUB_OUTPUT`, for a job matrix | | `capabilities` | how this machine answers each word of `[capabilities]`, and why | | `measure --out FILE -- ARGV` | runs ARGV and writes what it used (its children's account) as JSON: how the engine measures a program that reports no use of its own (P10.4) | diff --git a/docs/config.md b/docs/config.md index 53d363e..72460d4 100644 --- a/docs/config.md +++ b/docs/config.md @@ -105,7 +105,9 @@ max_age = "7d" # an older map is not read: every case i # case's tag `requires:gcc`). Every test an entry states must hold; an entry that states none holds # everywhere. A case that requires a word this machine lacks is skipped, with the reason # `missing capability: ()`; a word no entry declares is an error, for no -# machine could answer it. `mtest capabilities` prints the answers; each run records them. +# machine could answer it. `mtest capabilities` prints the answers; each run records them. A word a +# program answers itself (the C++ library's `capability`, declared in code with these tests) needs +# no entry: the program lists its answer and the reason of the skip, and the run records it too. [capabilities] tag = "requires:" # the tag prefix whose words are capabilities (the default) gcc = { program = "g++" } # found on PATH (a list: each) @@ -123,6 +125,8 @@ no-msvc = { os = ["windows"], none = ["msvc"] } exceptions = "tests/coverage-exceptions.tsv" # \t debt = "tests/coverage-debt.tsv" # \t: skips everywhere hosts = "tests/host-requirements.tsv" # \t\t: must pass on each host +# A case that states its places (`only{platform::windows}`, `--only windows`) must also pass on each +# of them that a collected run was on, and some run must have been on one: no table names it. # `mtest ci plan`: a selection divided among CI's hosts by each host's history. [ci.plan] diff --git a/engine/mcpp.toml b/engine/mcpp.toml index 66c0ffe..c080e32 100644 --- a/engine/mcpp.toml +++ b/engine/mcpp.toml @@ -10,7 +10,7 @@ [package] namespace = "mtest" name = "mtest" -version = "0.7.2" +version = "0.8.0" description = "The mtest engine: run, schedule, store, explain and analyse tests of any language" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/engine/src/cicheck.cpp b/engine/src/cicheck.cpp index c864103..83e4f42 100644 --- a/engine/src/cicheck.cpp +++ b/engine/src/cicheck.cpp @@ -76,6 +76,65 @@ std::vector words_of(const config& c, const std::vector& only, std::string_view system, std::string_view arch) { + bool any_platform = false, any_arch = false, platform_ok = false, arch_ok = false; + for (const auto& w : only) { + if (is_architecture(w)) { + any_arch = true; + arch_ok = arch_ok || w == arch; + } else { + any_platform = true; + platform_ok = platform_ok || w == system; + } + } + return (!any_platform || platform_ok) && (!any_arch || arch_ok); +} + +// A case stated to run only on some places must pass on each of them that a run covered, and on +// at least one: what [ci.coverage] hosts states of a program, stated by the case itself. +std::vector placed_problems(const std::vector& runs, + const std::function& narrowed) { + std::map> places; // program::case -> its `only` + for (const auto& r : runs) + for (const auto& cr : r.cases) + if (!cr.only.empty()) + places.try_emplace(cr.program + "::" + cr.id, cr.only); + std::vector out; + for (const auto& [key, only] : places) { + std::map passed; // " " covered -> passed there + for (const auto& r : runs) { + if (!covers(only, r.system, r.arch)) + continue; + const auto at = std::format("{} {}", r.system, r.arch); + const auto result = std::ranges::find_if(r.cases, [&](const case_result& cr) { + return cr.program + "::" + cr.id == key; + }); + if (result == r.cases.end() && narrowed(r)) + continue; // absent by the run's selection + const bool ok = + result != r.cases.end() && (result->status == "pass" || result->status == "xfail" || + result->status == "flaky"); + passed[at] = passed[at] || ok; + } + std::string words; + for (const auto& w : only) + words += (words.empty() ? "" : ", ") + w; + if (passed.empty()) + out.push_back(std::format("{} runs only on {} and no run was there", key, words)); + for (const auto& [at, ok] : passed) + if (!ok) + out.push_back( + std::format("{} runs only on {} and did not pass on {}", key, words, at)); + } + return out; +} + } // namespace ci_answer ci_coverage(const config& c, const std::vector& programs, @@ -176,6 +235,8 @@ ci_answer ci_coverage(const config& c, const std::vector& programs host)); } } + for (auto& pr : placed_problems(runs, [&](const run_record& r) { return narrowed(c, r); })) + problems.push_back(std::move(pr)); std::vector unprovided; for (const auto& [w, _] : c.capabilities.words) if (!word_somewhere[w]) diff --git a/engine/src/commands.cpp b/engine/src/commands.cpp index e235403..ea59fcc 100644 --- a/engine/src/commands.cpp +++ b/engine/src/commands.cpp @@ -1958,7 +1958,7 @@ json catalog() { plain({{"out", {{"type", "string"}}}, {"argv", {{"type", "array"}, {"items", {{"type", "string"}}}}}}), true); - return {{"schema", "mtest.catalog/1"}, {"version", "0.7.2"}, {"tools", tools}}; + return {{"schema", "mtest.catalog/1"}, {"version", "0.8.0"}, {"tools", tools}}; } } // namespace mcpplibs::mtest::engine diff --git a/engine/src/main.cpp b/engine/src/main.cpp index 5a029ba..fa222fe 100644 --- a/engine/src/main.cpp +++ b/engine/src/main.cpp @@ -98,7 +98,7 @@ const json* tool_named(const json& tools, std::string_view name) { cmdline::App application(const json& tools) { auto app = cmdline::App("mtest") - .version("0.7.2") + .version("0.8.0") .description("run tests of any language, and ask about their runs") .option(cmdline::Option("json").global().help("print the answer's data")); for (const auto& t : tools) @@ -253,7 +253,7 @@ int main(int argc, char** argv) { const std::string name(parsed->subcommand_name()); const auto& sub = parsed->subcommand()->get(); if (name == "version") { - std::println("mtest 0.7.2"); + std::println("mtest 0.8.0"); return 0; } const auto cwd = std::filesystem::current_path(); diff --git a/engine/src/model.cpp b/engine/src/model.cpp index 236bd0e..1ad9ccc 100644 --- a/engine/src/model.cpp +++ b/engine/src/model.cpp @@ -58,7 +58,9 @@ json to_json(const case_info& c) { {"line", c.line}, {"kind", c.kind}, {"tags", c.tags}, {"clauses", c.clauses}, {"resources", c.resources}, {"artifacts", c.artifacts}, {"expect", c.expect}, {"timeout", c.timeout}, {"isolated", c.isolated}, - {"retries", c.retries}, {"uses", c.uses}, {"bounds", to_json(c.bounds)}}; + {"retries", c.retries}, {"uses", c.uses}, {"bounds", to_json(c.bounds)}, + {"only", c.only}, {"except", c.except}, {"capabilities", c.capabilities}, + {"reason", c.reason}}; } json to_json(const failure_info& f) { json j = {{"file", f.file}, {"line", f.line}, {"message", f.message}}; @@ -102,6 +104,10 @@ json to_json(const case_result& c) { j["artifacts"] = c.artifacts; if (!c.tags.empty()) j["tags"] = c.tags; + if (!c.only.empty()) + j["only"] = c.only; + if (!c.except.empty()) + j["except"] = c.except; if (!c.clauses.empty()) j["clauses"] = c.clauses; if (c.quarantined) @@ -210,6 +216,13 @@ case_info case_info_from(const json& j) { c.resources = strings(j, "resources"); c.artifacts = strings(j, "artifacts"); c.uses = strings(j, "uses"); + c.only = strings(j, "only"); + c.except = strings(j, "except"); + if (auto a = j.find("capabilities"); a != j.end() && a->is_object()) + for (auto it = a->begin(); it != a->end(); ++it) + if (it.value().is_boolean()) + c.capabilities[it.key()] = it.value().get(); + c.reason = str(j, "reason"); return c; } case_result case_result_from(const json& j) { @@ -241,6 +254,8 @@ case_result case_result_from(const json& j) { c.artifacts = strings(j, "artifacts"); c.tags = strings(j, "tags"); c.clauses = strings(j, "clauses"); + c.only = strings(j, "only"); + c.except = strings(j, "except"); c.quarantined = j.value("quarantined", false); c.attempts = j.value("attempts", 1); if (auto it = j.find("fixtures"); it != j.end() && it->is_array()) diff --git a/engine/src/model.cppm b/engine/src/model.cppm index 503e82a..a030309 100644 --- a/engine/src/model.cppm +++ b/engine/src/model.cppm @@ -72,6 +72,11 @@ struct case_info { std::vector stated_bounds; // mtest.toml's, which the engine judges (the program judges its own) std::vector uses; // run fixtures (P11) + // Where it runs (0.8.0): the platforms and architectures it runs only on, and those it does + // not run on; the capabilities its program answered, and why the program lists it skipped. + std::vector only, except; + std::map capabilities; + std::string reason; }; // A shared object a case caused to be made (P5.2 `fixture`), and how long making it took. @@ -102,6 +107,7 @@ struct case_result { std::vector failures; std::vector metrics; std::vector artifacts, tags, clauses; + std::vector only, except; // where it runs, as listed (0.8.0) std::vector fixtures; std::vector bounds; // as listed, for `analyze trend` bool quarantined = false; diff --git a/engine/src/schedule.cpp b/engine/src/schedule.cpp index 72d3b33..678d39d 100644 --- a/engine/src/schedule.cpp +++ b/engine/src/schedule.cpp @@ -415,6 +415,11 @@ run_record execute(const config& c, const store& s, const plan& p, const execute rec.host = host_name(); for (const auto& [word, a] : machine_capabilities(c)) rec.capabilities[word] = a.present; + // And each capability a program declares in code and answered when it was listed (0.8.0). + for (const auto& u : p.units) + for (const auto& uc : u.cases) + for (const auto& [word, present] : uc.info.capabilities) + rec.capabilities.try_emplace(word, present); const int hw = int(std::max(1u, std::thread::hardware_concurrency())); rec.jobs = o.jobs > 0 ? o.jobs : c.jobs > 0 ? c.jobs : hw; const int retries = o.retries >= 0 ? o.retries : c.retries; @@ -437,6 +442,9 @@ run_record execute(const config& c, const store& s, const plan& p, const execute cr.kind = uc.info.kind; cr.status = "skip"; cr.reason = uc.info.skip; + cr.tags = uc.info.tags; + cr.only = uc.info.only; + cr.except = uc.info.except; skipped.push_back(std::move(cr)); return true; }; @@ -691,6 +699,8 @@ run_record execute(const config& c, const store& s, const plan& p, const execute cr.kind = ci->info.kind; cr.tags = ci->info.tags; cr.clauses = ci->info.clauses; + cr.only = ci->info.only; + cr.except = ci->info.except; cr.bounds = ci->info.bounds; judge_bounds(cr, ci->info.stated_bounds); expected_to_fail(cr, ci->info.expect_fail); diff --git a/engine/src/select.cpp b/engine/src/select.cpp index 32cd236..fd311cc 100644 --- a/engine/src/select.cpp +++ b/engine/src/select.cpp @@ -327,6 +327,36 @@ void apply_attributes(const config& c, const program_spec& program, case_info& c ci.kind = program.kind.empty() ? "unit" : program.kind; } +// Why a case whose places are `only` and `except` does not run on this machine (0.8.0), in the +// library's words; empty when it runs here. +std::string place_skip(const std::vector& only, + const std::vector& except) { + const std::string system(os::name(os::host())); + const std::string arch(os::name(os::ARCH)); + const auto here = std::format("{} {}", system, arch); + const auto architecture = [](std::string_view w) { + return w == "x86_64" || w == "aarch64" || w == "riscv64" || w == "wasm32"; + }; + bool platforms = false, architectures = false, platform_ok = false, arch_ok = false; + std::string words; + for (const auto& w : only) { + words += (words.empty() ? "" : ", ") + w; + if (architecture(w)) { + architectures = true; + arch_ok = arch_ok || w == arch; + } else { + platforms = true; + platform_ok = platform_ok || w == system; + } + } + if ((platforms && !platform_ok) || (architectures && !arch_ok)) + return std::format("only on {} (this host: {})", words, here); + for (const auto& w : except) + if (w == system || w == arch) + return std::format("not on {} (this host: {})", w, here); + return {}; +} + std::string layout_kind(std::string_view path) { std::string_view previous; for (const auto part : std::views::split(path, '/')) { @@ -578,14 +608,22 @@ std::expected make_plan(const config& c, const store& s, u.expected_peak = pk->second; for (auto ci : *listed) { apply_attributes(c, u.program, ci); + // A case its program lists skipped, saying why (0.8.0: a place it does not run on, a + // capability it answered absent), is recorded so and not started. + if (ci.expect == "skip" && ci.skip.empty() && !ci.reason.empty()) + ci.skip = ci.reason; + // A place it does not run on, whether or not its program decided so itself. + if (ci.skip.empty()) + ci.skip = place_skip(ci.only, ci.except); // A required capability this machine lacks makes the case a skip that says which; one - // `[capabilities]` does not declare is an error, for no machine could answer it. + // that neither the program answers nor `[capabilities]` declares is an error, for no + // machine could answer it. if (!c.capabilities.words.empty()) { auto tags = ci.tags; tags.insert(tags.end(), p.tags.begin(), p.tags.end()); for (const auto& t : tags) { auto word = capability_of_tag(c, t); - if (!word) + if (!word || ci.capabilities.contains(*word)) continue; if (!c.capabilities.words.contains(*word)) return std::unexpected( diff --git a/engine/src/service.cpp b/engine/src/service.cpp index 8e3030d..95f997f 100644 --- a/engine/src/service.cpp +++ b/engine/src/service.cpp @@ -56,7 +56,7 @@ int serve(const std::filesystem::path& cwd) { {"result", {{"protocolVersion", params.value("protocolVersion", std::string("2025-06-18"))}, {"capabilities", {{"tools", {{"listChanged", false}}}}}, - {"serverInfo", {{"name", "mtest"}, {"version", "0.7.2"}}}, + {"serverInfo", {{"name", "mtest"}, {"version", "0.8.0"}}}, {"instructions", "Run tests and ask about runs: run, plan, explain, history, analyze, graph, " "gate, ci. A test session (test/plan, test/run, test/add, test/cancel, " diff --git a/engine/tests/engine.cpp b/engine/tests/engine.cpp index 2055a0e..335219c 100644 --- a/engine/tests/engine.cpp +++ b/engine/tests/engine.cpp @@ -807,6 +807,89 @@ printf ')" + header + expect(eq(caps["capabilities"]["c"]["present"].get(), true)); expect(eq(caps["capabilities"]["d"]["present"].get(), false)); }; + // --- 0.8.0: where a case runs, stated by the case; capabilities its program answers + "a case runs only where its places hold, and the engine says where"_test[kind::e2e] = [] { + const std::string head = + "#!/usr/bin/env bash\n# mtest-protocol=1\n. '" + adapter() + "'\nok() { :; }\n"; + const std::string here(mcpplibs::os::name(mcpplibs::os::host())); + auto root = + project({{"t.sh", head + "mtest_case here ok --only " + here + + "\nmtest_case elsewhere ok --except " + here + + "\nmtest_case 'on another architecture' ok --only riscv64\n" + "mtest_report\n"}}); + auto j = json_of(mtest(root, {"run", "--json", "--report-json", "r.json", "bash t.sh"})); + expect(eq(j["status"].get(), std::string("pass"))) << j.dump(); + auto r = json::parse(*mcpplibs::os::fs::read(root / "r.json")); + std::map by_id; + for (const auto& c : r["cases"]) + by_id[c["id"].get()] = c; + expect(eq(by_id["here"]["status"].get(), std::string("pass"))) << r.dump(); + expect(eq(by_id["here"].value("only", json::array()), json::array({here}))) + << "the record keeps its places"; + expect(eq(by_id["elsewhere"]["status"].get(), std::string("skip"))); + expect(contains(by_id["elsewhere"].value("reason", ""), "not on " + here)); + if (mcpplibs::os::ARCH != mcpplibs::os::architecture::riscv64) + expect( + contains(by_id["on another architecture"].value("reason", ""), "only on riscv64")); + }; + "a capability its program answers needs no entry in [capabilities]"_test[kind::e2e] = [&] { + // A program that answers `requires:` words itself (the C++ library's `capability`) lists + // them under `capabilities`, and the reason of a skip; the engine takes both. + const std::string listing = + R"js({"e":"program","framework":"fake","level":3} +{"e":"case","id":"has","name":"has","tags":["requires:mine"],"capabilities":{"mine":true}} +{"e":"case","id":"lacks","name":"lacks","tags":["requires:theirs"],"capabilities":{"theirs":false},"expect":"skip","reason":"missing capability: theirs (no theirs on PATH)"} +{"e":"done"} +)js"; + const std::string ran = + R"({"e":"program","framework":"fake","level":3} +{"e":"begin","id":"has","t":0} +{"e":"end","id":"has","status":"pass","ms":1} +{"e":"done"} +)"; + auto root = + project({{"listing.ndjson", listing}, + {"ran.ndjson", ran}, + {"1_fake.sh", "# mtest-protocol=1\nif [ \"$MTEST_MODE\" = list ]; then cat " + "listing.ndjson > \"$MTEST_REPORT\"; " + "else cat ran.ndjson > \"$MTEST_REPORT\"; fi\n"}, + {"mtest.toml", + "[discover.s]\nglobs = [\"[0-9]*.sh\"]\ncommand = [\"bash\", \"{path}\"]\n" + "[capabilities]\nother = {}\n"}}); + auto j = json_of(mtest(root, {"run", "--json", "--report-json", "r.json"})); + expect(eq(j["status"].get(), std::string("pass"))) << j.dump(); + auto r = json::parse(*mcpplibs::os::fs::read(root / "r.json")); + std::map by_id; + for (const auto& c : r["cases"]) + by_id[c["id"].get()] = c; + expect(eq(by_id["has"]["status"].get(), std::string("pass"))) << r.dump(); + expect(eq(by_id["lacks"]["status"].get(), std::string("skip"))); + expect(contains(by_id["lacks"].value("reason", ""), "missing capability: theirs")); + expect(eq(r["capabilities"].value("mine", false), true)) + << "the run records the program's answers"; + expect(eq(r["capabilities"].value("theirs", true), false)); + }; + "a case stated for a place must pass there"_test[kind::e2e] = [] { + const std::string head = "#!/usr/bin/env bash\n# mtest-protocol=1\n. '" + adapter() + "'\n"; + const std::string here(mcpplibs::os::name(mcpplibs::os::host())); + auto root = project( + {{"t.sh", head + "bad() { mtest_eq 1 2; }\nok() { :; }\nmtest_case fails bad --only " + + here + "\nmtest_case passes ok --only " + here + + "\nmtest_case nowhere ok --only riscv64 --only " + here + + "\nmtest_report\n"}, + {"mtest.toml", + "[discover.s]\nglobs = [\"t.sh\"]\ncommand = [\"bash\", \"{path}\"]\n"}}); + (void)mtest(root, {"run", "--json", "--group", "g8"}); + auto j = json_of(mtest(root, {"ci", "coverage", "--group", "g8", "--json"})); + const auto problems = j["problems"].dump(); + expect(contains(problems, "t.sh::fails runs only on " + here + " and did not pass on")) + << j.dump(); + expect(!contains(problems, "t.sh::passes")) << j.dump(); + if (mcpplibs::os::ARCH != mcpplibs::os::architecture::riscv64) + expect(contains(problems, "t.sh::nowhere runs only on riscv64, " + here + + " and no run was there")) + << j.dump(); + }; "every program ran somewhere or is accounted for"_test[kind::e2e] = [&] { auto root = project({{"1_ok.sh", "exit 0\n"}, {"2_skips.sh", "echo 'SKIP: nothing here'; exit 77\n"}, diff --git a/lib/docs/writing-tests.md b/lib/docs/writing-tests.md index 1a5fd80..95df479 100644 --- a/lib/docs/writing-tests.md +++ b/lib/docs/writing-tests.md @@ -53,7 +53,7 @@ E2E cli{"cli", resource{"serial:home"}, timeout{std::chrono::minutes{2}}, [] { / The entry enters the child before any case; the copy runs no case and reports nothing. The files of one directory are then one program (mcpp `[test] programs = "directory"`) however many of them start copies. -- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.7.2"`. +- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.8.0"`. - Identifiers are stable: a selection, a history and a shard name a case by them. ## Expectations @@ -73,7 +73,8 @@ skipped, `fail("why")` as failed. ## Attributes Written after an object's name, before its body, or in brackets after a case's name. A case has -the attributes of every scope that declares it: tags, clauses, resources and artifacts add up, +the attributes of every scope that declares it: tags, clauses, resources, artifacts, the places it +does not run on and the capabilities it requires add up, the places it runs only on narrow, isolation holds for everything inside; the others are the innermost one stated. | Attribute | States | The runner | @@ -90,9 +91,46 @@ isolation holds for everything inside; the others are the innermost one stated. | `at_most{"ready", 15}`, `at_least{"recall", 0.9}` | the limits of a metric it reports | the case fails when its metric passes a limit or is not reported; `mtest analyze trend` shows each metric beside its limits | | `skip_if{cond, "why"}`, `expect_failure{"why"}`, `known_gap{"why", "YYYY-MM-DD"}` | its expected verdict | reports skip, xfail, xpass | | `monitor{monitor_level::detailed}` | how closely its use of the machine is measured | measures at least that closely: its processor time (`basic`), its memory and I/O (`detailed`), both over time (`trace`), as metrics with their precision (protocol P10) | +| `only{platform::windows}`, `only{platform::linux, architecture::x86_64}`, `except{platform::macos}` | where it runs: on one of the platforms and one of the architectures `only` names, when it names any, and on none `except` names | skips it elsewhere, saying where (`only on windows (this host: linux x86_64)`); `mtest ci coverage` requires it to pass on each named place a run covered | +| a `capability`, `[gcc]` | what it requires of the machine, and the tag `requires:gcc` | skips it on a machine that lacks it, saying why (`missing capability: gcc (no g++ on PATH)`) | | `uses{"index"}` | a run fixture it uses | serves the fixture once for the run before the first case that uses it, gives its endpoint (`fixture_endpoint("index")`), ends it after the last | `Bench` holds `machine`: a measurement means something only on a machine nothing else uses. + +### Places and capabilities + +The program decides where a case runs, whichever runner starts it (`mtest`, `mcpp test`, the +program alone), and lists the decision with its reason. Places are the platforms and architectures +of `mcpplibs.os`. A capability is declared once, in code, by the tests of mtest.toml's +`[capabilities]` (a system, an architecture, programs on PATH, a file a glob matches, a command +that succeeds, environment values, other capabilities that must or must not hold) or a function +for what those cannot state; it is answered once per process. Declared in a module interface, it +is made before the units that import it, so a suite at namespace scope may state it: + +```cpp +// tests/support/machine.cppm +export module app.test.machine; +import mcpplibs.mtest; +using namespace mcpplibs::mtest; +export const capability gcc{"gcc", {.os = {platform::linux}, .program = {"g++"}}}; +export const capability pack{"pack", {.all = {gcc}, .program = {"patchelf"}}}; +export const capability msvc{"msvc", {.os = {platform::windows}, .test = has_visual_studio}}; + +// tests/e2e/linking.cpp +import app.test.machine; + +E2E linking{"linking", [] { + "links statically"_test[gcc] = [] { ... }; + "reads the registry"_test[only{platform::windows}] = [] { + on([&](auto) { expect(eq(registry_value(), 1)); }); + }; +}}; +``` + +What may be decided when the program is compiled is: `can_run_on(platform)` is a constant, and +`host()` is one on a native build (an ok image knows it when it is loaded). Code that names what +exists for one platform only is written in `on

([&](auto) { ... })`, whose body a build for +another platform discards. How the tests of a project are laid out, from the directory to the line, is in [docs/layout.md](../../docs/layout.md). diff --git a/lib/mcpp.toml b/lib/mcpp.toml index c65c099..82b5659 100644 --- a/lib/mcpp.toml +++ b/lib/mcpp.toml @@ -17,7 +17,7 @@ [package] namespace = "mcpplibs" name = "mtest" -version = "0.7.2" +version = "0.8.0" description = "The C++ test library of mtest: cases, expectations, fixtures, bench, conformance and end-to-end helpers, with no macro and no header, speaking the mtest protocol" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/lib/src/mtest.cpp b/lib/src/mtest.cpp index bc05b0f..fc21cbb 100644 --- a/lib/src/mtest.cpp +++ b/lib/src/mtest.cpp @@ -324,7 +324,7 @@ void write_program() { {"protocol", 1}, {"level", 3}, {"framework", "mcpplibs.mtest"}, - {"version", "0.7.2"}, + {"version", "0.8.0"}, {"isolation", {"process", "case"}}, {"guide", "https://github.com/mcpp-community/mtest/blob/main/lib/docs/writing-tests.md"}}); } @@ -673,7 +673,8 @@ void test::operator=(body b) { std::unique_lock g(s.lock); write_program(); const std::string id = full_id(name_); - const attributes attrs = within(inherited(), attrs_); + attributes attrs = within(inherited(), attrs_); + attrs.skip = skip_reason(attrs); s.seen.insert(id); s.clauses.insert(attrs.clauses.begin(), attrs.clauses.end()); const std::string file = b.where.file_name(); @@ -690,6 +691,20 @@ void test::operator=(body b) { ev["clauses"] = attrs.clauses; ev["resources"] = attrs.resources; ev["artifacts"] = attrs.artifacts; + if (attrs.only_platforms || attrs.only_architectures) { + auto words = attrs.only_platforms.value_or(std::vector{}); + for (const auto& w : attrs.only_architectures.value_or(std::vector{})) + words.push_back(w); + ev["only"] = words; + } + if (!attrs.excepted.empty()) + ev["except"] = attrs.excepted; + if (!attrs.capabilities.empty()) { + event answered = event::object(); + for (const auto& c : attrs.capabilities) + answered[c.name()] = c.answer().present; + ev["capabilities"] = answered; + } if (attrs.limit) ev["timeout"] = std::chrono::duration(*attrs.limit).count(); if (!attrs.bounds.empty()) @@ -708,9 +723,10 @@ void test::operator=(body b) { d["timeout"] = std::chrono::duration(attrs.diagnosis->limit).count(); ev["diagnose"] = d; } - if (attrs.skip) + if (attrs.skip) { ev["expect"] = "skip"; - else if (attrs.expected_failure) + ev["reason"] = *attrs.skip; + } else if (attrs.expected_failure) ev["expect"] = "fail"; write_event(ev); return; @@ -1128,6 +1144,220 @@ void apply(attributes& a, known_gap g) { a.gap_due = g.due; } +// ---- where a case runs, and what the machine can do (0.8.0) ------------------ + +namespace { + +bool holds_word(const std::vector& words, std::string_view w) { + return std::ranges::find(words, w) != words.end(); +} + +std::string joined(const std::vector& words) { + std::string out; + for (const auto& w : words) + out += (out.empty() ? "" : ", ") + w; + return out; +} + +std::string this_host() { + return std::format("{} {}", os::name(os::host()), os::name(os::ARCH)); +} + +// `${NAME}` replaced by the environment's value (empty when unset). +std::string expand(std::string_view text) { + std::string out; + for (std::size_t i = 0; i < text.size(); ++i) { + if (text[i] == '$' && i + 1 < text.size() && text[i + 1] == '{') { + const auto end = text.find('}', i + 2); + if (end != std::string_view::npos) { + out += os::env::get(text.substr(i + 2, end - i - 2)).value_or(""); + i = end; + continue; + } + } + out += text[i]; + } + return out; +} + +// Whether `name` matches `pattern`, where `*` is any run of characters and `?` any one. +bool glob_match(std::string_view pattern, std::string_view name) { + if (pattern.empty()) + return name.empty(); + if (pattern.front() == '*') + for (std::size_t skip = 0; skip <= name.size(); ++skip) + if (glob_match(pattern.substr(1), name.substr(skip))) + return true; + if (pattern.front() == '*') + return false; + return !name.empty() && (pattern.front() == '?' || pattern.front() == name.front()) && + glob_match(pattern.substr(1), name.substr(1)); +} + +// Whether some file matches `pattern`: a path whose components may hold `*` and `?`. +bool some_file(const std::string& pattern) { + const auto p = os::fs::from_utf8(pattern); + std::vector at{p.root_path()}; + for (const auto& part : p.relative_path()) { + const auto word = os::fs::utf8(part); + std::vector next; + for (const auto& dir : at) { + std::error_code ec; + if (!word.contains('*') && !word.contains('?')) { + if (std::filesystem::exists(dir / part, ec)) + next.push_back(dir / part); + continue; + } + for (const auto& e : std::filesystem::directory_iterator(dir, ec)) + if (glob_match(word, os::fs::utf8(e.path().filename()))) + next.push_back(e.path()); + } + if (next.empty()) + return false; + at = std::move(next); + } + return !at.empty(); +} + +capability_answer evaluate(const probe& t) { + const std::string system(os::name(os::host())); + const std::string arch(os::name(os::ARCH)); + if (!t.os.empty() && std::ranges::find(t.os, os::host()) == t.os.end()) + return {false, std::format("the system is {}", system)}; + if (!t.arch.empty() && std::ranges::find(t.arch, os::ARCH) == t.arch.end()) + return {false, std::format("the architecture is {}", arch)}; + for (const auto& [k, want] : t.env) { + const auto have = os::env::get(k); + if (want == "*" ? (!have || have->empty()) : have != want) + return {false, std::format("{} is {}", k, have ? std::format("`{}`", *have) : "unset")}; + } + for (const auto& c : t.all) + if (const auto& a = c.answer(); !a.present) + return {false, std::format("`{}` is absent ({})", c.name(), a.why)}; + for (const auto& c : t.none) + if (c.answer().present) + return {false, std::format("`{}` is present", c.name())}; + for (const auto& program : t.program) + if (!os::env::find_executable(expand(program))) + return {false, std::format("no {} on PATH", program)}; + if (!t.path.empty() && + std::ranges::none_of(t.path, [](const auto& g) { return some_file(expand(g)); })) + return {false, std::format("no file matches {}", expand(t.path.front()))}; + if (!t.command.empty()) { + os::process::spawn_options o; + o.program = os::fs::from_utf8(expand(t.command.front())); + for (std::size_t i = 1; i < t.command.size(); ++i) + o.args.push_back(expand(t.command[i])); + o.in = os::process::stdio::null; + o.out = os::process::stdio::null; + o.err = os::process::stdio::null; + os::process::bounds limits; + limits.hard = std::chrono::seconds{30}; + auto r = os::process::run(std::move(o), limits); + if (!r) + return {false, std::format("{} did not start", t.command.front())}; + if (r->timed_out || !r->status.success()) + return {false, + std::format("{} ended with {}", t.command.front(), + r->timed_out ? std::string("a timeout") + : std::format("status {}", r->status.shell_code()))}; + } + if (t.test && !t.test()) + return {false, "its test does not hold"}; + return {true, "every test holds"}; +} + +// Only the words both name, when both name some. +void narrow(std::optional>& to, const std::vector& words) { + if (!to) + to = words; + else + std::erase_if(*to, [&](const std::string& w) { return !holds_word(words, w); }); +} + +void restrict_to(attributes& a, const std::vector& places) { + std::vector platforms, architectures; + for (const auto& p : places) + (p.architecture_word ? architectures : platforms).push_back(p.word); + if (!platforms.empty()) + narrow(a.only_platforms, platforms); + if (!architectures.empty()) + narrow(a.only_architectures, architectures); +} + +void add_capability(attributes& a, const capability& c) { + const auto tag = "requires:" + c.name(); + if (!holds_word(a.tags, tag)) + a.tags.push_back(tag); + if (std::ranges::none_of(a.capabilities, + [&](const capability& x) { return x.name() == c.name(); })) + a.capabilities.push_back(c); +} + +} // namespace + +struct capability::state { + std::string name; + probe tests; + std::once_flag once; + capability_answer answer; +}; + +capability::capability(std::string name, probe tests) : state_(std::make_shared()) { + state_->name = std::move(name); + state_->tests = std::move(tests); +} + +const std::string& capability::name() const noexcept { + return state_->name; +} + +const capability_answer& capability::answer() const { + std::call_once(state_->once, [this] { state_->answer = evaluate(state_->tests); }); + return state_->answer; +} + +void apply(attributes& a, only o) { + restrict_to(a, o.places); +} + +void apply(attributes& a, except e) { + for (const auto& p : e.places) + if (!holds_word(a.excepted, p.word)) + a.excepted.push_back(p.word); +} + +void apply(attributes& a, capability c) { + add_capability(a, c); +} + +std::optional skip_reason(const attributes& a) { + if (a.skip) + return a.skip; + const std::string system(os::name(os::host())); + const std::string arch(os::name(os::ARCH)); + const auto outside = [](const auto& only, const std::string& w) { + return only && !holds_word(*only, w); + }; + if (outside(a.only_platforms, system) || outside(a.only_architectures, arch)) { + std::vector words = a.only_platforms.value_or(std::vector{}); + for (const auto& w : a.only_architectures.value_or(std::vector{})) + words.push_back(w); + return std::format("only on {} (this host: {})", + words.empty() + ? std::string("no place: its `only` statements exclude one another") + : joined(words), + this_host()); + } + for (const auto& w : a.excepted) + if (w == system || w == arch) + return std::format("not on {} (this host: {})", w, this_host()); + for (const auto& c : a.capabilities) + if (const auto& answer = c.answer(); !answer.present) + return std::format("missing capability: {} ({})", c.name(), answer.why); + return std::nullopt; +} + attributes within(const attributes& outer, const attributes& inner) { attributes a = outer; auto add = [](std::vector& to, const std::vector& from) { @@ -1159,6 +1389,13 @@ attributes within(const attributes& outer, const attributes& inner) { a.retries = inner.retries; if (inner.diagnosis) a.diagnosis = inner.diagnosis; + if (inner.only_platforms) + narrow(a.only_platforms, *inner.only_platforms); + if (inner.only_architectures) + narrow(a.only_architectures, *inner.only_architectures); + add(a.excepted, inner.excepted); + for (const auto& c : inner.capabilities) + add_capability(a, c); return a; } diff --git a/lib/src/mtest.cppm b/lib/src/mtest.cppm index 6a5cc13..bc9112d 100644 --- a/lib/src/mtest.cppm +++ b/lib/src/mtest.cppm @@ -8,6 +8,7 @@ export module mcpplibs.mtest; import std; +import mcpplibs.os; export namespace mcpplibs::mtest { @@ -35,6 +36,75 @@ struct metric_bound { std::optional max, min; }; +// ---- where a case runs ------------------------------------------------------ + +// The platforms and architectures of mcpplibs.os, and its two questions (0.8.0): whether this build +// can run on a platform, a constant, and the platform it runs on now, `host()`, a constant on a +// native build and known when an ok image is loaded. Code that compiles only for one platform is +// written in `on

([&](auto) { ... })`, whose body a build for another platform discards: +// +// "reads the registry"_test[only{platform::windows}] = [] { +// on([&](auto) { expect(eq(registry_value(), 1)); }); +// }; +using platform = mcpplibs::os::platform; +using architecture = mcpplibs::os::architecture; +using mcpplibs::os::can_run_on; +using mcpplibs::os::host; +using mcpplibs::os::on; + +// A platform or an architecture, by mcpplibs.os's name for it ("windows", "aarch64"). +struct place { + std::string word; + bool architecture_word = false; + place(platform p) : word(mcpplibs::os::name(p)) {} + place(architecture a) : word(mcpplibs::os::name(a)), architecture_word(true) {} +}; + +// WHAT THIS MACHINE CAN DO (0.8.0), declared once in code beside the tests that need it, by the +// tests of mtest.toml's [capabilities] (docs/config.md), and answered by the program itself, once +// per process, whichever runner starts it: +// +// export const capability gcc{"gcc", {.os = {platform::linux}, .program = {"g++"}}}; +// export const capability pack{"pack", {.all = {gcc}, .program = {"patchelf"}}}; +// "links statically"_test[gcc] = [] { ... }; +// +// A case that requires a capability this machine lacks is skipped, with the reason +// `missing capability: ()`; it carries the tag `requires:`. A +// capability declared in a module interface is made before the units that import it, so suites at +// namespace scope may state it. +class capability; + +struct probe { + std::vector os; // the system is one of them + std::vector arch; // the architecture is one of them + std::vector program; // each is found on PATH + std::vector path; // some file matches one (`*`, `?`, `${NAME}`) + std::vector command; // runs and exits 0 within 30 s + std::vector> env; // equals; "*": set and not empty + std::vector all; // each is present + std::vector none; // none is present + std::function test; // what the words above cannot state +}; + +struct capability_answer { + bool present = false; + std::string why; // the test that decided it, in a person's words +}; + +class capability { +public: + capability(std::string name, probe tests); + const std::string& name() const noexcept; + // Asked once per process; every copy shares the answer. + const capability_answer& answer() const; + explicit operator bool() const { + return answer().present; + } +private: + struct state; + std::shared_ptr state_; +}; + struct attributes { std::optional kind; std::vector tags; @@ -52,6 +122,11 @@ struct attributes { std::vector bounds; // what its metrics may be (at_most, at_least) std::optional monitoring; // how closely its use is measured (monitor) std::vector uses; // the run fixtures it uses (P11) + // Where it runs (only, except): the platforms and the architectures it is restricted to, none + // stated meaning every one, and the words of either on which it does not run. + std::optional> only_platforms, only_architectures; + std::vector excepted; + std::vector capabilities; // what it requires of the machine }; // Each is written in brackets after the case's name, "x"_test[tag{"slow"}][resource{"machine"}], @@ -115,9 +190,24 @@ struct uses { std::string fixture; }; -// Each attribute applied to a set of them. Tags, clauses, resources, artifacts and bounds add up, -// and a case within an isolated suite is isolated; a kind, a skip, an expected failure, a known -// gap, a time limit, retries and a diagnosis are the innermost one stated. +// The case runs only on the platforms and architectures named: on one of the platforms, when any is +// named, and on one of the architectures, when any is. Stated on a suite and on its case, both +// hold. Elsewhere it is skipped: `only on windows (this host: linux x86_64)`. +struct only { + std::vector places; + only(std::initializer_list p) : places(p) {} +}; +// The case does not run on any platform or architecture named: `not on macos (this host: ...)`. +struct except { + std::vector places; + except(std::initializer_list p) : places(p) {} +}; + +// Each attribute applied to a set of them. Tags, clauses, resources, artifacts, bounds, the +// platforms not run on and the capabilities required add up, the platforms and architectures a +// case runs only on narrow, and a case within an isolated suite is isolated; a kind, a skip, an +// expected failure, a known gap, a time limit, retries and a diagnosis are the innermost one +// stated. inline void apply(attributes& a, mtest::kind k) { a.kind = k; } @@ -165,6 +255,12 @@ inline void apply(attributes& a, uses u) { a.uses.push_back(std::move(u.fixture)); } void apply(attributes& a, known_gap g); +void apply(attributes& a, only o); +void apply(attributes& a, except e); +void apply(attributes& a, capability c); +// Why a case with these attributes does not run on this machine: its own skip, else a platform or +// an architecture it does not run on, else a capability the machine lacks. +std::optional skip_reason(const attributes& a); template concept attribute = requires(attributes& set, A a) { apply(set, std::move(a)); }; diff --git a/lib/tests/places.cpp b/lib/tests/places.cpp new file mode 100644 index 0000000..fb74aa9 --- /dev/null +++ b/lib/tests/places.cpp @@ -0,0 +1,164 @@ +// Where a case runs and what the machine can do (0.8.0): `only` and `except` skip a case elsewhere +// with one reason, a suite's and its case's narrow together, and a capability declared in code is +// answered by the program, once, for whichever runner lists and runs it. The program runs itself +// again under a runner with the cases declared, and reads the events. +import std; +import nlohmann.json; +import mcpplibs.mtest; +import mcpplibs.mtest.e2e; +import mcpplibs.os; + +using namespace mcpplibs::mtest; +using json = nlohmann::json; + +namespace { + +// What a machine has or lacks, declared by the vocabulary of [capabilities]. +const capability always{"always", {}}; +const capability never{"never", {.env = {{"MTEST_PLACES_NEVER_SET", "*"}}}}; +const capability on_top_of_never{"on-top-of-never", {.all = {never}}}; +const capability without_never{"without-never", {.none = {never}}}; +const capability refused{"refused", {.test = [] { return false; }}}; + +// The questions of mcpplibs.os, answered at compile time where the build has one platform. +static_assert(can_run_on(platform::linux) || can_run_on(platform::macos) || + can_run_on(platform::windows) || can_run_on(platform::emscripten)); + +const std::string this_host = + std::format("{} {}", mcpplibs::os::name(host()), mcpplibs::os::name(mcpplibs::os::ARCH)); +constexpr architecture other_architecture = + mcpplibs::os::ARCH == architecture::x86_64 ? architecture::aarch64 : architecture::x86_64; + +// What the copy started with --declared declares. +void declare_cases() { + "here"_test[only{host()}] = [] {}; + "elsewhere"_test[except{host()}] = [] { + expect(false) << "never runs on its excepted platform"; + }; + "on another architecture"_test[only{other_architecture}] = [] { expect(false); }; + "on this architecture, any platform"_test[only{mcpplibs::os::ARCH}] = [] {}; + Suite narrowed{"narrowed", + only{platform::linux, platform::macos, platform::windows, platform::openkal}, + [] { + "to this host"_test[only{host()}] = [] {}; + "to a platform its suite excludes"_test[only{platform::emscripten}] = [] { + expect(false); + }; + }}; + (void)narrowed; + "with always"_test[always] = [] {}; + "with never"_test[never] = [] { expect(false) << "never runs where `never` is absent"; }; + "on top of never"_test[on_top_of_never] = [] { expect(false); }; + "without never"_test[without_never] = [] {}; + "refused by its test"_test[refused] = [] { expect(false); }; +} + +std::vector events_of(std::string_view mode) { + const auto report = temp_dir() / std::format("{}.ndjson", mode); + auto argv = self_command(); + argv.push_back("--declared"); + e2e::command c{.argv = argv}; + c.env = {{"MTEST_PROTOCOL", "1"}, + {"MTEST_MODE", std::string(mode)}, + {"MTEST_REPORT", report.string()}, + {"MTEST_PLACES_NEVER_SET", std::nullopt}}; + auto r = e2e::run(std::move(c)); + expect(fatal(r.started)); + std::vector out; + std::ifstream in(report); + for (std::string line; std::getline(in, line);) + if (!line.empty()) + out.push_back(json::parse(line)); + return out; +} + +std::optional event(const std::vector& events, std::string_view kind, + std::string_view id) { + for (const auto& e : events) + if (e.value("e", "") == kind && e.value("id", "") == id) + return e; + return std::nullopt; +} + +std::string status(const std::vector& events, std::string_view id) { + auto e = event(events, "end", id); + return e ? e->value("status", "") : "(no end)"; +} + +std::string reason(const std::vector& events, std::string_view id) { + auto e = event(events, "end", id); + return e ? e->value("reason", "") : "(no end)"; +} + +} // namespace + +int main(int argc, char** argv) { + if (argc > 1 && std::string_view(argv[1]) == "--declared") { + declare_cases(); + return run(argc, argv); + } + "a case runs where its places hold and is skipped elsewhere, saying where"_test = [] { + const auto events = events_of("run"); + expect(eq(status(events, "here"), std::string("pass"))); + expect(eq(status(events, "on this architecture, any platform"), std::string("pass"))); + expect(eq(status(events, "elsewhere"), std::string("skip"))); + expect(eq(reason(events, "elsewhere"), + std::format("not on {} (this host: {})", mcpplibs::os::name(host()), this_host))); + expect(eq(reason(events, "on another architecture"), + std::format("only on {} (this host: {})", mcpplibs::os::name(other_architecture), + this_host))); + }; + "a suite's places and its case's narrow together"_test = [] { + const auto events = events_of("run"); + expect(eq(status(events, "narrowed/to this host"), std::string("pass"))); + expect( + eq(status(events, "narrowed/to a platform its suite excludes"), std::string("skip"))); + expect(contains(reason(events, "narrowed/to a platform its suite excludes"), + "its `only` statements exclude one another")); + }; + "a capability is answered by its tests, and one absent skips naming the test"_test = [] { + const auto events = events_of("run"); + expect(eq(status(events, "with always"), std::string("pass"))); + expect(eq(status(events, "without never"), std::string("pass"))); + expect(eq(reason(events, "with never"), + std::string("missing capability: never (MTEST_PLACES_NEVER_SET is unset)"))); + expect(eq(reason(events, "on top of never"), + std::string("missing capability: on-top-of-never (`never` is absent " + "(MTEST_PLACES_NEVER_SET is unset))"))); + expect(eq(reason(events, "refused by its test"), + std::string("missing capability: refused (its test does not hold)"))); + }; + "a listing states the places, the answers and the reason of a skip"_test = [] { + const auto events = events_of("list"); + auto here = event(events, "case", "here"); + expect(fatal(here.has_value())); + expect(eq(here->value("only", json::array()), json::array({mcpplibs::os::name(host())}))); + expect(!here->contains("expect")); + auto elsewhere = event(events, "case", "elsewhere"); + expect(fatal(elsewhere.has_value())); + expect(eq(elsewhere->value("except", json::array()), + json::array({mcpplibs::os::name(host())}))); + expect(eq(elsewhere->value("expect", ""), std::string("skip"))); + expect(contains(elsewhere->value("reason", ""), "not on")); + auto never_case = event(events, "case", "with never"); + expect(fatal(never_case.has_value())); + expect(eq(never_case->value("capabilities", json::object()), json{{"never", false}})); + expect(std::ranges::any_of(never_case->value("tags", json::array()), [](const json& t) { + return t.get() == "requires:never"; + })) << "a required capability is also its tag"; + auto always_case = event(events, "case", "with always"); + expect(fatal(always_case.has_value())); + expect(eq(always_case->value("capabilities", json::object()), json{{"always", true}})); + }; + "code for one platform runs only there"_test = [] { + int ran = 0; + on([&](auto) { ran += host() == platform::linux; }); + on([&](auto) { ran += host() == platform::macos; }); + on([&](auto) { ran += host() == platform::windows; }); + expect(eq(ran, host() == platform::linux || host() == platform::macos || + host() == platform::windows + ? 1 + : 0)); + }; + return report(); +} diff --git a/manifest/mcpp.toml b/manifest/mcpp.toml index 9f07ce9..2a6381c 100644 --- a/manifest/mcpp.toml +++ b/manifest/mcpp.toml @@ -8,7 +8,7 @@ [package] namespace = "mcpplibs.mtest" name = "manifest" -version = "0.7.2" +version = "0.8.0" description = "The static case manifest of mtest: the cases a source file declares, read without building it" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/spec/protocol.md b/spec/protocol.md index 2655b27..6ebef39 100644 --- a/spec/protocol.md +++ b/spec/protocol.md @@ -21,6 +21,7 @@ on the spec; this list says which of its clauses mtest's parts answer, and since | O5 `scheduling` | `resources` and `artifacts` (P7); `timeout`, `isolation`, `retries`, `diagnose` (P3.3) in the `case` events | 0.1.0; P3.3's 0.3.0 | | O6 `fixtures` | `shared` and the `fixture` event; components and run fixtures (P11), served under `MTEST_SERVE` | 0.3.0; P11 0.5.0 | | O7 `usage` | the program's `usage` at its end, and the cases' use at `MTEST_MONITOR` (P10) | 0.5.0 | +| P3.3, places and capabilities | a `case` event states `only` and `except` (the platforms and architectures of `only{}`, `except{}`), `capabilities` (each `capability` the case requires, as the program answered it), and the `reason` of a case it lists skipped; the program skips a case outside its places or lacking a capability, whichever runner starts it | 0.8.0 | | O8 `output` | each case's output, kept while it runs under a runner and stated as `output` events; a case declared inside another keeps its own. Where the system cannot rearrange the program's streams (an ok image), no output is kept for a case, the spec's fallback | 0.6.0 | ## The engine (`mtest`) @@ -32,6 +33,7 @@ on the spec; this list says which of its clauses mtest's parts answer, and since | O5 | schedules cases by their resources and history, by pools since 0.5.0; ends a case at its `timeout`, runs `retries` and `diagnose` since 0.3.0 | 0.1.0 | | O6 | serves run fixtures once per run (P11.2, P11.3) | 0.5.0 | | O7 | measures a program that reports no use from a process of its own (P10.4) | 0.5.0 | +| P3.3, places | skips a listed case outside its `only` and `except`, and one its program lists skipped with a `reason`, without starting it; takes the program's answers for the capabilities it names (no `[capabilities]` entry needed) and records them in the run; `mtest ci coverage` requires a case to pass on each of its places that a collected run covered | 0.8.0 | | O9 `session` | `mtest serve` answers `test/plan`, `test/run`, `test/add`, `test/cancel`, `test/status` and `shutdown`, and states `test/build`, `test/case`, `test/program`, `test/run`; it builds nothing, and starts each program from the launch mcpp gives (B5) | 0.6.0 | ## The adapters and the C ABI diff --git a/spec/schema/event-v1.json b/spec/schema/event-v1.json index 90708b1..6ead5df 100644 --- a/spec/schema/event-v1.json +++ b/spec/schema/event-v1.json @@ -168,6 +168,31 @@ "type": "string" }, "description": "P11.2: the run fixtures the case uses" + }, + "reason": { + "type": "string", + "description": "why the program lists the case skipped (expect skip)" + }, + "only": { + "type": "array", + "items": { + "type": "string" + }, + "description": "0.8.0: the platforms and architectures the case runs only on" + }, + "except": { + "type": "array", + "items": { + "type": "string" + }, + "description": "0.8.0: the platforms and architectures the case does not run on" + }, + "capabilities": { + "type": "object", + "additionalProperties": { + "type": "boolean" + }, + "description": "0.8.0: the capabilities the case requires, as its program answered them" } }, "required": [