0.8.3: each call measured by itself, as a loop (bench::laps)

CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## [0.8.3] - 2026-11-14
+
+- **Each call measured by itself, as a loop** (`bench::laps`). `bench::each` takes the call as a
+  function; a body that keeps its state in the function it runs in writes the same measurement as
+  a loop: `bench::laps run{"draw"}; while (run.next()) { ... }`. The time from one `next` to the
+  following one is a call's, `run` is the call's `lap` (`pause`, `resume`, `items`, `part`, `time`,
+  `value`, `say`, `last`), and when `next` returns false the metrics have been reported as `each`
+  reports them; `result()` is where the calls lay. `each` is that loop around its function: one
+  way of measuring, in two forms.
+- **Run alone, a program shows what it measured.** A measurement's line in the program's output --
+  `draw: 2.100 ms (p95 5.300 ms, p99 7.700 ms, max 8.400 ms; mean 2.900 ms; 139 calls; 0 over
+  8.333 ms)`, then a line for each part and quantity -- where no runner keeps the run's record.
+  Under a runner nothing is added to the case's output: the record has the metrics.
+- **What the lap keeps is not the call's.** The allocations `part`, `time` and `value` make to keep
+  what a call says are left out of the call's count: a frame that allocates nothing reports none
+  while it states its parts (`lib/tests/allocations.cpp`, a program that replaces `operator new`).
+- **A program run alone selects as the runner does.** Its arguments name cases by identifier, by
+  the name they lie under (`suite/group`, which was run and then reported as no case), or by a
+  pattern with `*` (the cases it matches, as `mcpp test --case` takes one); a selection no
+  case answers fails the run. The interface said "by prefix" and the program took whole
+  identifiers: the statement and the behaviour are now one (`lib/tests/alone.cpp`).
+- Additive: `each` measures and reports as it did.
+
 ## [0.8.2] - 2026-11-14
 
 - 0.8.1 as released: its tag was made and its release was not (a test of `bench::each` held that

README.md

@@ -38,7 +38,7 @@ int main() {
 
 ```toml
 [dev-dependencies]
-mcpplibs.mtest = "0.8.2"
+mcpplibs.mtest = "0.8.3"
 ```
 
 A failure names its place and both sides (`expected 2, actual 3`). Under a runner the same program

README.zh-CN.md

@@ -37,7 +37,7 @@ int main() {
 
 ```toml
 [dev-dependencies]
-mcpplibs.mtest = "0.8.2"
+mcpplibs.mtest = "0.8.3"
 ```
 
 失败时给出位置与两侧的值（`expected 2, actual 3`）。在运行器之下，同一程序按协议报告；单独运行时每个用例输出一行。

adapters/gtest/mcpp.toml

@@ -13,7 +13,7 @@
 [package]
 namespace   = "mcpplibs.mtest"
 name        = "gtest"
-version     = "0.8.2"
+version     = "0.8.3"
 description = "GoogleTest speaking the mtest protocol: a test main that lists, selects and reports GoogleTest's cases"
 license     = "Apache-2.0"
 

adapters/gtest/src/protocol.cpp

@@ -128,7 +128,7 @@ int mcpplibs_mtest_gtest_main(int argc, char** argv) {
                   {"protocol", 1},
                   {"level", 1},
                   {"framework", "googletest"},
-                  {"version", "0.8.2"},
+                  {"version", "0.8.3"},
                   {"isolation", {"process", "case"}},
                   {"guide", "https://github.com/mcpp-community/mtest/blob/main/adapters/gtest/docs/"
                             "writing-tests.md"}});

capi/mcpp.toml

@@ -4,7 +4,7 @@
 [package]
 namespace   = "mcpplibs.mtest"
 name        = "capi"
-version     = "0.8.2"
+version     = "0.8.3"
 description = "mtest's C ABI: the protocol's events for any language with a C FFI"
 license     = "Apache-2.0"
 

capi/src/mtest.cpp

@@ -120,7 +120,7 @@ void mtest_init(const char* framework, int level) {
              {"protocol", 1},
              {"level", std::clamp(level, 1, 3)},
              {"framework", text(framework)},
-             {"version", "0.8.2"},
+             {"version", "0.8.3"},
              {"isolation", {"process", "case"}}});
 }
 

conformance/mcpp.toml

@@ -8,7 +8,7 @@
 [package]
 namespace   = "mcpplibs.mtest"
 name        = "conformance"
-version     = "0.8.2"
+version     = "0.8.3"
 description = "The conformance suite of the mtest protocol, for every implementation of its program side"
 license     = "Apache-2.0"
 

engine/mcpp.toml

@@ -10,7 +10,7 @@
 [package]
 namespace   = "mtest"
 name        = "mtest"
-version     = "0.8.2"
+version     = "0.8.3"
 description = "The mtest engine: run, schedule, store, explain and analyse tests of any language"
 license     = "Apache-2.0"
 authors     = ["mcpp-community"]

engine/src/commands.cpp

@@ -1958,7 +1958,7 @@ json catalog() {
         plain({{"out", {{"type", "string"}}},
                {"argv", {{"type", "array"}, {"items", {{"type", "string"}}}}}}),
         true);
-    return {{"schema", "mtest.catalog/1"}, {"version", "0.8.2"}, {"tools", tools}};
+    return {{"schema", "mtest.catalog/1"}, {"version", "0.8.3"}, {"tools", tools}};
 }
 
 }  // namespace mcpplibs::mtest::engine

engine/src/main.cpp

@@ -98,7 +98,7 @@ const json* tool_named(const json& tools, std::string_view name) {
 
 cmdline::App application(const json& tools) {
     auto app = cmdline::App("mtest")
-                   .version("0.8.2")
+                   .version("0.8.3")
                    .description("run tests of any language, and ask about their runs")
                    .option(cmdline::Option("json").global().help("print the answer's data"));
     for (const auto& t : tools)
@@ -253,7 +253,7 @@ int main(int argc, char** argv) {
     const std::string name(parsed->subcommand_name());
     const auto& sub = parsed->subcommand()->get();
     if (name == "version") {
-        std::println("mtest 0.8.2");
+        std::println("mtest 0.8.3");
         return 0;
     }
     const auto cwd = std::filesystem::current_path();

engine/src/service.cpp

@@ -56,7 +56,7 @@ int serve(const std::filesystem::path& cwd) {
                   {"result",
                    {{"protocolVersion", params.value("protocolVersion", std::string("2025-06-18"))},
                     {"capabilities", {{"tools", {{"listChanged", false}}}}},
-                    {"serverInfo", {{"name", "mtest"}, {"version", "0.8.2"}}},
+                    {"serverInfo", {{"name", "mtest"}, {"version", "0.8.3"}}},
                     {"instructions",
                      "Run tests and ask about runs: run, plan, explain, history, analyze, graph, "
                      "gate, ci. A test session (test/plan, test/run, test/add, test/cancel, "

lib/docs/writing-tests.md

@@ -53,7 +53,7 @@ E2E cli{"cli", resource{"serial:home"}, timeout{std::chrono::minutes{2}}, [] { /
   The entry enters the child before any case; the copy runs no case and reports nothing. The files of
   one directory are then one program (mcpp `[test] programs = "directory"`) however many of them start
   copies.
-- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.8.2"`.
+- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.8.3"`.
 - Identifiers are stable: a selection, a history and a shard name a case by them.
 
 ## Expectations
@@ -158,7 +158,9 @@ How the tests of a project are laid out, from the directory to the line, is in
   `operator new` and calls `bench::allocated(n)` from it gets each call's allocations too:
   `draw.allocations`, `draw.allocations.max`, `draw.allocated` (bytes). With `{.limit = 8333us}`,
   `draw.over` counts the calls that took more (and they are the ones stated one by one);
-  `l.last()` ends the run at this call.
+  `l.last()` ends the run at this call. The same as a loop, for a body that keeps its state where it
+  runs: `bench::laps run{"draw"}; while (run.next()) { ... }` -- `run` is the call's lap, and
+  `run.result()` where the calls lay.
 
 ## Programs under test
 
@@ -272,6 +274,7 @@ its precision, so a bound is written as for any metric: `at_most{"mem.peak", 64.
 | `mcpp test --cases` | each program's cases, running none |
 | `mcpp test --message-format json` | a record per program with each case's result, place and both sides |
 | `mcpp test <test> --print-launch` | how a test starts, for a debugger |
+| `<the built program> [--list] [identifier \| name \| 'pattern*' ...]` | the program alone: the cases named — one by its identifier, those under a name (`suite/group`), those a pattern matches; a measurement's numbers are printed |
 | `mtest run --isolate --jobs N --shard I/N` | the cases scheduled by their resources and history, a part of them |
 
 A failure reads `case fields/a field holds its text failed at tests/fields.cpp:9: ...`; the JSON

lib/mcpp.toml

@@ -17,7 +17,7 @@
 [package]
 namespace   = "mcpplibs"
 name        = "mtest"
-version     = "0.8.2"
+version     = "0.8.3"
 description = "The C++ test library of mtest: cases, expectations, fixtures, bench, conformance and end-to-end helpers, with no macro and no header, speaking the mtest protocol"
 license     = "Apache-2.0"
 authors     = ["mcpp-community"]

lib/src/bench.cpp

@@ -12,6 +12,17 @@ double median_of(std::vector<double> v) {
     const auto n = v.size();
     return n % 2 ? v[n / 2] : (v[n / 2 - 1] + v[n / 2]) / 2;
 }
+
+// A time for a reader: 9.100 ms, 412.0 us, 1.250 s.
+std::string readable(double seconds) {
+    if (seconds >= 1)
+        return std::format("{:.3f} s", seconds);
+    if (seconds >= 1e-3)
+        return std::format("{:.3f} ms", seconds * 1e3);
+    if (seconds >= 1e-6)
+        return std::format("{:.1f} us", seconds * 1e6);
+    return std::format("{:.0f} ns", seconds * 1e9);
+}
 }  // namespace
 
 summary summarize(std::vector<double> samples, std::size_t iterations) {
@@ -32,6 +43,10 @@ summary summarize(std::vector<double> samples, std::size_t iterations) {
 
 void record(std::string_view name, const summary& s) {
     metric(name, s.median, "s", s.samples, "lower");
+    // Run alone, a program shows what it measured; under a runner the run's record has it.
+    if (!under_runner())
+        std::println("{}: {} (from {} to {}; {} samples of {} calls)", name, readable(s.median),
+                     readable(s.min), readable(s.max), s.samples.size(), s.iterations);
 }
 
 double percentile(std::span<const double> sorted, double q) {
@@ -110,16 +125,6 @@ quantity over(std::string name, const Rows& rows, std::size_t at, Pick pick) {
     return q;
 }
 
-// A time for a reader: 9.100 ms, 412.0 us, 1.250 s.
-std::string readable(double seconds) {
-    if (seconds >= 1)
-        return std::format("{:.3f} s", seconds);
-    if (seconds >= 1e-3)
-        return std::format("{:.3f} ms", seconds * 1e3);
-    if (seconds >= 1e-6)
-        return std::format("{:.1f} us", seconds * 1e6);
-    return std::format("{:.0f} ns", seconds * 1e9);
-}
 }  // namespace
 
 namespace {
@@ -138,14 +143,24 @@ allocation_count allocations() noexcept {
     return {allocation_calls, allocation_bytes};
 }
 
+// What the lap allocates to keep what a call says is not the call's: a frame that allocates nothing
+// still reports none while it states its parts.
 void lap::part(std::string_view name, clock::duration spent) {
+    const allocation_count before = allocations();
     double& at = cell(now_.parts, column(part_names_, name));
     const double seconds = std::chrono::duration<double>(spent).count();
     at = std::isnan(at) ? seconds : at + seconds;
+    const allocation_count after = allocations();
+    out_count_ += after.count - before.count;
+    out_bytes_ += after.bytes - before.bytes;
 }
 
 void lap::value(std::string_view name, double v) {
+    const allocation_count before = allocations();
     cell(now_.values, column(value_names_, name)) = v;
+    const allocation_count after = allocations();
+    out_count_ += after.count - before.count;
+    out_bytes_ += after.bytes - before.bytes;
 }
 
 void lap_access::keep(lap& l) {
@@ -213,6 +228,47 @@ distribution distribute(std::vector<double> samples, const lap& l, std::size_t s
     return d;
 }
 
+laps::laps(std::string_view name, each_options o) : name_(name), o_(o) {
+    samples_.reserve(std::min<std::size_t>(o_.most, 4096));
+}
+
+bool laps::next() {
+    const auto now = clock::now();
+    if (done_)
+        return false;  // ended: it stays so, however often it is asked
+    if (running_) {
+        // The call that was running ends here: what `next` does after is no call's.
+        const auto spent = std::max(clock::duration{}, now - began_ - lap_access::end(*this));
+        if (warmed_ < o_.warmup) {
+            ++warmed_;
+            lap_access::drop(*this);
+        } else {
+            total_ += spent;
+            samples_.push_back(std::chrono::duration<double>(spent).count());
+            lap_access::keep(*this);
+        }
+        running_ = false;
+    }
+    const bool warming = warmed_ < o_.warmup;
+    const bool more =
+        !lap_access::last(*this) && (warming || (samples_.size() < o_.most &&
+                                                 (samples_.size() < o_.least || total_ < o_.time)));
+    if (!more) {
+        if (!done_) {
+            done_ = true;
+            result_ = distribute(std::move(samples_), *this, o_.slowest,
+                                 std::chrono::duration<double>(o_.limit).count());
+            record(name_, result_);
+        }
+        return false;
+    }
+    lap_access::warmup(*this, warming);
+    lap_access::begin(*this);
+    running_ = true;
+    began_ = clock::now();
+    return true;
+}
+
 // The calls differ by their nature, so the samples are not stated with the metrics: the verdict on
 // samples that spread (P6.2) is for a measurement meant to be one number.
 void record(std::string_view name, const distribution& d) {
@@ -241,6 +297,20 @@ void record(std::string_view name, const distribution& d) {
         metric(base + ".over.share",
                d.samples.empty() ? 0 : double(d.over) / double(d.samples.size()), "");
     }
+    if (!under_runner()) {
+        std::string line = std::format("{}: {} (p95 {}, p99 {}, max {}; mean {}; {} calls", name,
+                                       readable(d.p50), readable(d.p95), readable(d.p99),
+                                       readable(d.max), readable(d.mean), d.samples.size());
+        if (d.limit > 0)
+            line += std::format("; {} over {}", d.over, readable(d.limit));
+        line += ")";
+        for (const quantity& q : d.parts)
+            line += std::format("\n    {}: {} (p99 {}, max {})", q.name, readable(q.p50),
+                                readable(q.p99), readable(q.max));
+        for (const quantity& q : d.values)
+            line += std::format("\n    {}: {:g} (max {:g})", q.name, q.mean, q.max);
+        std::println("{}", line);
+    }
     for (const slow_call& c : d.slowest) {
         std::string line = std::format("{}: call {} took {}", name, c.index, readable(c.seconds));
         const char* between = ": ";

lib/src/bench.cppm

@@ -265,41 +265,49 @@ distribution distribute(std::vector<double> samples, const lap& l, std::size_t s
 // `.max`; the slowest calls a line each in the case's output.
 void record(std::string_view name, const distribution& d);
 
+// THE SAME, AS A LOOP: for a body that keeps its state in the function it runs in.
+//   bench::laps run{"draw", {.limit = std::chrono::microseconds{8333}}};
+//   while (run.next()) {
+//       run.pause(); scene.advance(); run.resume();
+//       scene.draw();
+//       run.value("nodes", scene.nodes());
+//   }
+// The time from one `next` that returned true to the following `next` is a call's; `run` is the
+// call's `lap`. `next` returns false once what the options ask has been measured (or a call said
+// `last`), having reported the metrics as `each` does; `result()` is then where the calls lay.
+// `each` is this loop around its function: one way of measuring, in two forms.
+class laps : public lap {
+public:
+    explicit laps(std::string_view name, each_options o = {});
+    laps(const laps&) = delete;
+    laps& operator=(const laps&) = delete;
+    [[nodiscard]] bool next();
+    // Where the calls lay: after `next` has returned false.
+    [[nodiscard]] const distribution& result() const noexcept {
+        return result_;
+    }
+private:
+    std::string name_;
+    each_options o_;
+    std::vector<double> samples_;
+    clock::duration total_{};
+    clock::time_point began_{};
+    std::size_t warmed_ = 0;
+    bool running_ = false, done_ = false;
+    distribution result_;
+};
+
 template <class F>
 requires std::invocable<F&, lap&> || std::invocable<F&>
 distribution each(std::string_view name, F&& f, each_options o = {}) {
-    using clock = lap::clock;
-    lap l;
-    const auto call = [&] {
-        lap_access::begin(l);
-        const auto t0 = clock::now();
+    laps run{name, o};
+    while (run.next()) {
         if constexpr (std::invocable<F&, lap&>)
-            std::invoke(f, l);
+            std::invoke(f, static_cast<lap&>(run));
         else
             std::invoke(f);
-        const auto t1 = clock::now();
-        return std::max(clock::duration{}, t1 - t0 - lap_access::end(l));
-    };
-    lap_access::warmup(l, true);
-    for (std::size_t i = 0; i < o.warmup && !lap_access::last(l); ++i) {
-        call();
-        lap_access::drop(l);
-    }
-    lap_access::warmup(l, false);
-    std::vector<double> samples;
-    samples.reserve(std::min<std::size_t>(o.most, 4096));
-    clock::duration total{};
-    while (!lap_access::last(l) && samples.size() < o.most &&
-           (samples.size() < o.least || total < o.time)) {
-        const auto spent = call();
-        total += spent;
-        samples.push_back(std::chrono::duration<double>(spent).count());
-        lap_access::keep(l);
     }
-    auto d = distribute(std::move(samples), l, o.slowest,
-                        std::chrono::duration<double>(o.limit).count());
-    record(name, d);
-    return d;
+    return run.result();
 }
 
 }  // namespace mcpplibs::mtest::bench