Skip to content

Commit 899013f

Browse files
authored
1.5.0: tests by the mcpp test spec, test sessions, tests beside each other (#11)
* Tests by the mcpp test spec: no framework is known, tests run beside each other - The backend knows no test framework (mcpp test spec, docs/specs/test.md): a program's cases are what it lists (B3), one listing of the project kept until it changes; cases run by identifier (P4) whatever framework wrote them, and a program of no protocol runs whole. spec/test-frameworks.json, the framework table, its static scanning and report readers, the setting mcpp.test.macros and the dependency on mtest's static manifest are removed; scan reads where a source starts and nothing else. - A case's outcome carries its checks (held, failed), each failure's source and fatal (O3) and its own output (O8); VS Code shows one message per failed check, a diff where it states both sides, the statement and whether the case went on, the case's own output, and the failed checks on the case's line. - A test runs at once in a lane of its own: two tests of a project, or a test beside a build, do not wait for each other, and cancelling one leaves the other (mcpp releases its build directory before tests run, mcpp#915). mcpp.runtime.concurrency holds only clean and pack. - One set of arguments for every test command (profile, target, mcpp.task.testArgs), so a listing and a run plan one configuration; discovery after events is debounced. - Debugging some cases of a protocol program states the protocol's variables and the selection. * Cases run through the test session a framework states (mcpp test spec O9) - Where a project's test framework states a test session (test-session, B6) and its mcpp gives launches (test-launch-records, B5), test/runCases runs the cases through it: the session is started once per project and spoken to in JSON-RPC on its standard streams; a run is a job of its own whose cases arrive as test/caseEvent with their checks, failures and output, and whose end is job/finished. Cancelling the job cancels the run. - A run is known before the session's first notification of it: its answer is taken on the session's reading thread. - The session ends with its input, so it ends with the backend; a lifetime bound by the system is not asked, as on Linux it follows the thread that started it. - The setting mcpp.test.session (auto, off) runs every case through mcpp test when off. - docs/architecture.md states a test run by the mcpp test spec. * Cases stay as their program last listed them; CLion and Neovim by the test spec - test/cases answers the project's last listing without `list`, and lists again with it only when the project changed since: the cases shown stay until a listing replaces them, and writing a file builds nothing. - CLion runs some cases of a test by their identifiers whatever framework wrote them: the run configuration names a test and its cases (its framework field is removed), and the gutter places a case's run on the first element of the line its program lists it at, in the file it names. Cases are listed when first shown and after a build or a test, one listing at a time per program. - Neovim lists the cases when a file is read, when the session starts and after a run; writing a file shows the last listing. - protocol.schema.json states test/cases, test/runCases, test/caseResult and test/caseEvent by the mcpp test spec: a case's checks (O3), its output (O8), a failure's source and fatal; source/entries and launch parameters name no framework. - mcpp.lock no longer holds mcpplibs.mtest.manifest, which no target depends on. * 1.5.0: tests by the mcpp test spec, test sessions, tests beside each other * docs/settings.md: mcpp.test.session in place of mcpp.test.macros
1 parent 2e91ca8 commit 899013f

48 files changed

Lines changed: 1040 additions & 1246 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/docs/2026-11-10-tests-from-mcpp.md‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Tests as mcpp states them: programs, parts, cases, live results, metrics and history
22

3-
Status, 2026-11-10: implemented in mcpp-ide 1.3.0 (mcpp-ide#8), with the client side of mcpp#917; the backend's own tests are mtest's.
3+
Status, 2026-11-10: implemented in mcpp-ide 1.3.0 (mcpp-ide#8), with the client side of mcpp#917; the backend's own tests are mtest's. mcpp-ide 1.5.0 reads tests by the mcpp test spec (last section): the static reading of cases with mtest's manifest is withdrawn.
44

55
## Scope and principle
66

@@ -183,3 +183,32 @@ insights are empty and say so.
183183
- Neovim: the smoke test covers live results, the virtual text and `:Mcpp test.insights`.
184184
- CLion: the comparison failure and the nested suites, in the plugin's unit tests.
185185
- The insights page is rendered by a Chromium test, as the graph page is.
186+
187+
## 1.5.0: the mcpp test spec
188+
189+
mcpp 2026.11.10.1 states what an editor may rely on of any test framework as the mcpp test spec
190+
(mcpp docs/specs/test.md, SPEC-017 v2.0): a base every framework implements (B1–B6) and optional
191+
capabilities a framework states of itself (O1–O9). From 1.5.0 the backend reads tests by that spec and
192+
knows no framework by name.
193+
194+
- **Cases.** A program's cases are its listing (B3, `mcpp test --cases`), one listing per project. The
195+
last listing is kept: `test/cases` with `list` lists again when the project changed since, and
196+
without it answers the kept listing. Writing a file builds nothing; the cases shown stay until a
197+
listing after a build or a test replaces them. No case is read from a source.
198+
- **Selection.** Cases run by identifier (P4, `--case`). A configuration names a test and its cases;
199+
no framework is named.
200+
- **Outcomes.** A case carries its checks (O3: held, failed; each failure's statement and whether the
201+
case went on) and its own output (O8).
202+
- **Frameworks.** What a framework offers beyond the base is what `mcpp test --guide` states of it
203+
(B6): its name, shown beside a program, and its capabilities.
204+
- **Sessions.** A framework that states a test session (`test-session`, O9), with an mcpp that gives
205+
launches (`test-launch-records`, B5), runs a selection of cases without a new `mcpp test`: the backend
206+
starts the session once per project and speaks JSON-RPC 2.0 to it on its standard streams. A run is
207+
answered at once; its build, its cases and its end arrive as notifications and become the job's
208+
`test/caseEvent` and `job/finished`. A run is registered when its answer is read, before any
209+
notification of it. The session ends at the end of its input, hence with the backend; a lifetime
210+
bound by the system is not used, since on Linux it follows the thread that started the process.
211+
`mcpp.test.session = "off"` runs every case through `mcpp test`.
212+
- **Concurrency.** A test runs in a lane of its own; two tests, or a test and a build, do not wait for
213+
each other. `mcpp.runtime.concurrency` holds only clean and pack.
214+

‎CHANGELOG.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,39 @@
22

33
The changelog is kept by series (`X.Y`); each section lists its components' releases.
44

5+
## 1.5
6+
7+
### 1.5.0
8+
9+
Every component is 1.5.0: the backend, mcpp-vscode, mcpp-clion and mcpp-nvim (from 1.4.0).
10+
11+
- **Tests by the mcpp test spec** (mcpp docs/specs/test.md, SPEC-017 v2.0). The backend knows no test
12+
framework by name. A program's cases are its listing (`mcpp test --cases`, B3), one listing per
13+
project, kept: a listing is asked when cases are first shown and after a build or a test, and writing
14+
a file shows the last listing and builds nothing. Cases run by identifier (`--case`, P4), whatever
15+
framework or language wrote them. A framework's name and capabilities are what `mcpp test --guide`
16+
states of it (B6). The framework table (`spec/test-frameworks.json`), the static reading of cases with
17+
mtest's manifest and the setting `mcpp.test.macros` are removed.
18+
- **Checks and output.** A case carries the checks it ran and those that failed, each failure's
19+
statement and whether the case went on (O3), and its own output (O8). VS Code shows one message per
20+
failed check, a diff where both sides are stated, and the failed checks on the case's line.
21+
- **Test sessions** (O9). Where a project's framework states a test session and its mcpp gives launches
22+
(`test-launch-records`, mcpp 2026.11.10.1), a run of some cases goes through the session, started once
23+
per project: its cases arrive as they begin and end, and a run after the first does not start
24+
`mcpp test` again. `mcpp.test.session` (`auto`, `off`) chooses.
25+
- **Tests run beside each other.** A test runs at once in a lane of its own: two tests of a project, or
26+
a test beside a build, do not wait for each other, and cancelling one leaves the other (mcpp
27+
2026.11.10.1 releases its build directory before tests run). `mcpp.runtime.concurrency` holds only
28+
clean and pack.
29+
- **One configuration for every test command.** A listing, a run and a debug launch take the profile,
30+
the target and `mcpp.task.testArgs` alike. Discovery after events is debounced. Debugging some cases of
31+
a protocol program states the protocol's variables and the selection.
32+
- **CLion.** A run configuration names a test and its cases; its framework field is removed. The gutter
33+
places a case's run on the first element of the line its program lists it at.
34+
- **Neovim.** Signs list the cases when a file is read, when the session starts and after a run.
35+
- **Protocol.** `protocol.schema.json` states `test/cases`, `test/runCases`, `test/caseResult` and
36+
`test/caseEvent` by the spec; `framework` of `test/runCases` is accepted and not read.
37+
538
## 1.4
639

740
### 1.4.0

‎build.mcpp‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ int main() {
4545
const std::filesystem::path root = env_or_empty("MCPP_MANIFEST_DIR");
4646
const std::filesystem::path out = env_or_empty("MCPP_OUT_DIR");
4747
const std::filesystem::path spec = root / "spec";
48-
const std::array<std::pair<const char*, const char*>, 9> files{{
48+
const std::array<std::pair<const char*, const char*>, 8> files{{
4949
{"kSettingsJson", "settings.json"},
5050
{"kPresetsJson", "presets.json"},
5151
{"kCommandsJson", "commands.json"},
@@ -54,7 +54,6 @@ int main() {
5454
{"kManifestZhCnJson", "i18n/manifest.zh-cn.json"},
5555
{"kTomlSchemaJson", "toml-schema.json"},
5656
{"kBuildScriptApiJson", "buildscript-api.json"},
57-
{"kTestFrameworksJson", "test-frameworks.json"},
5857
}};
5958
std::string module =
6059
"// Generated by mcpp-ide's build.mcpp from spec/ (not part of the repository).\n"

‎docs/architecture.md‎

Lines changed: 28 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -75,29 +75,37 @@ mcppls what a compiler answers better.
7575

7676
## A test run, end to end
7777

78-
The editors show tests as `mcpp test` states them (design 2026-11-10). mcpp builds and runs the
79-
programs; mtest's protocol is how a program's cases report to mcpp; mtest's store (spec P12) is the form
80-
in which mcpp keeps a run (`target/mtest`, or `[test] store`). The backend reads those three; it runs no
81-
other test runner.
78+
The editors show tests as `mcpp test` states them, by the mcpp test spec (mcpp docs/specs/test.md). mcpp
79+
builds and runs the programs and states their cases and results; a framework is whatever library a
80+
program is written with, and the backend knows none by name. What a framework offers beyond the base,
81+
it states of itself (`mcpp test --guide`, B6).
8282

8383
1. `test/list` runs `mcpp test --list` and answers each program with its files, its `main` and its parts.
8484
A protocol program of several files (`[test] programs = "directory"`) runs as one process per file,
85-
`<program>/<stem>`. mcpp states the parts with `test-parts` (mcpp#917); before it, the backend derives
86-
them as mcpp 2026.11.9.1 forms them. The programs are kept per project.
87-
2. `test/cases` reads the cases of every file of a program with mtest's static manifest (their kinds,
88-
tags and the bounds on their metrics), and with `list` the program's own listing (`--cases`).
89-
3. A test job runs `mcpp test --message-format json`. Each record becomes `test/result`, with the
90-
program it belongs to, the part it is and its cases, each status mapped (`xfail` passes, `xpass`
91-
fails, `invalid` is errored). With `test-case-stream` (mcpp#917) each case is sent as it begins and
92-
ends (`test/caseEvent`).
93-
4. VS Code marks the Testing view's items, a failure with both sides as a diff, and ends a case's line
94-
with its duration and metrics. CLion fills an ID-based test tree: a suite per program and per file, a
95-
comparison failure when both sides are stated. Neovim writes `mcpp://tests`, the quickfix list and
96-
the signs.
97-
5. `test/insights` reads the store's last runs (20 by default) and answers the last run, its failures
98-
and since when, the slowest and the flaky cases, each metric against its bound, and the schedule
99-
with its critical path. VS Code and CLion draw it with one page (`webui/tests.html`); Neovim prints
100-
it.
85+
`<program>/<stem>`. mcpp states the parts with `test-parts`; before it, the backend derives them as
86+
mcpp 2026.11.9.1 forms them. The programs are kept per project.
87+
2. `test/cases` answers a program's cases as the program lists them (`mcpp test --cases`, B3): one
88+
listing of the project, kept until the project changes.
89+
3. A test job runs `mcpp test --message-format json` with one set of arguments for every test command
90+
(the profile, the target, `mcpp.task.testArgs`). Each record becomes `test/result`, with the program
91+
it belongs to, the part it is and its cases, each status mapped (`xfail` passes, `xpass` fails,
92+
`invalid` is errored); a case carries its checks (O3) and its own output (O8). With
93+
`test-case-stream` each case is sent as it begins and ends (`test/caseEvent`). A test job runs in a
94+
lane of its own: two tests, or a test and a build, do not wait for each other.
95+
4. Where a framework states a test session (`test-session`, O9) and mcpp gives launches
96+
(`test-launch-records`, B5), `test/runCases` runs the cases through it: the session is started once
97+
per project, speaks JSON-RPC on its standard streams, answers a run at once and states its build,
98+
its cases and its end as notifications, which become the job's `test/caseEvent` and `job/finished`.
99+
The session ends with its input, so it ends with the backend. `mcpp.test.session = "off"` runs
100+
every case through `mcpp test`.
101+
5. VS Code marks the Testing view's items, one message per failed check (a diff where both sides are
102+
stated), and ends a case's line with its duration, its failed checks and its metrics. CLion fills an
103+
ID-based test tree: a suite per program and per file, a comparison failure when both sides are
104+
stated. Neovim writes `mcpp://tests`, the quickfix list and the signs.
105+
6. `test/insights` reads the last runs mcpp keeps (`target/mtest`, or `[test] store`; 20 by default)
106+
and answers the last run, its failures and since when, the slowest and the flaky cases, each
107+
metric against its bound, and the schedule with its critical path. VS Code and CLion draw it with
108+
one page (`webui/tests.html`); Neovim prints it.
101109

102110
Every step reads what the project's mcpp states and nothing more: an older mcpp gives programs and
103111
their results, and a view that needs a later feature is left out (`capabilities/explain`).

‎mcpp.lock‎

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -89,13 +89,6 @@ source = "index+mcpplibs@0.4.2"
8989
hash = "fnv1a:c06e4ca4f73f228d"
9090
targets = ["aarch64-openkal", "x86_64-linux-gnu", "x86_64-openkal"]
9191

92-
[package."mcpplibs.mtest.manifest"]
93-
namespace = "mcpplibs.mtest"
94-
version = "0.5.2"
95-
source = "index+mcpplibs.mtest@0.5.2"
96-
hash = "fnv1a:0caef37dd958cfef"
97-
targets = ["aarch64-openkal", "x86_64-linux-gnu", "x86_64-openkal"]
98-
9992
[package."nlohmann.json"]
10093
namespace = "nlohmann"
10194
version = "3.12.0"

‎mcpp.toml‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
# M2/M3), where the system it runs upon is answered when the image is loaded.
1010
[package]
1111
name = "mcpp-ide"
12-
version = "1.4.0"
12+
version = "1.5.0"
1313
description = "The IDE backend for mcpp: sessions, jobs, records and view models over mcpp's control contract"
1414
license = "Apache-2.0"
1515
authors = ["mcpp-community"]
@@ -80,8 +80,6 @@ main = "src/bin/dev.cpp"
8080
# runtime and loader remain the platform's.
8181
mcpplibs.os = "0.4.2"
8282
nlohmann.json = "3.12.0"
83-
# mtest's static case manifest: the cases a test file declares, read as mtest reads them.
84-
mcpplibs.mtest.manifest = "0.5.2"
8583

8684
# The tests are mtest's (mtest docs/layout.md): tests/unit, tests/e2e, tests/cts and tests/bench, each
8785
# directory one program whose files declare their cases; what every test shares is testsupport/, a

‎mide-plugins/clion/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ An individual packaged plugin can be checked without rebuilding the backend:
9999

100100
```bash
101101
xvfb-run -a gradle --no-daemon -p integration-tests test \
102-
-PpluginZip=/absolute/path/mcpp-clion-1.4.0.zip \
102+
-PpluginZip=/absolute/path/mcpp-clion-1.5.0.zip \
103103
-PmcpplsZip=/absolute/path/mcppls-clion-0.21.0.zip \
104104
-PideVersion=2026.1.5.2
105105
```

‎mide-plugins/clion/gradle.properties‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# where it installs (2026.1 and later); the verifier checks both ends.
33
platformType = CL
44
platformVersion = 2026.2.3
5-
pluginVersion = 1.4.0
5+
pluginVersion = 1.5.0
66
# The mcppls CLion plugin this one depends on (<depends>io.github.sunrisepeak.mcppls</depends>):
77
# its release's one package for every system, fetched for the sandbox, the tests and the verifier.
88
mcpplsPluginVersion = 0.21.0

‎mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/McppSession.kt‎

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -63,14 +63,14 @@ class McppSession(private val project: Project) : Disposable {
6363
if (key == "mcpp.ide.backendPath") restart() else pushSettings()
6464
})
6565
on("manifest/changed") { forgetSources() }
66-
// A build ended: a test program can list the cases its macros generate (§7.3).
66+
// A build or a test ended: the programs are listed again (mcpp test spec B3).
6767
on("job/finished") { p ->
6868
val o = p.asJsonObject
6969
if (o["status"]?.asString == "ok" && o["kind"]?.asString in setOf("build", "test")) {
7070
cases.keys.toList().forEach { key -> key.split('\u0000').takeIf { it.size == 2 }?.let { refreshCases(it[0], it[1], list = true) } }
7171
}
7272
}
73-
// A C++ source saved: what was read from it (main, a framework's cases) is read again.
73+
// A C++ source saved: what was read from it (main) is read again.
7474
project.messageBus.connect(this).subscribe(com.intellij.openapi.vfs.VirtualFileManager.VFS_CHANGES,
7575
object : com.intellij.openapi.vfs.newvfs.BulkFileListener {
7676
override fun after(events: MutableList<out com.intellij.openapi.vfs.newvfs.events.VFileEvent>) {
@@ -297,6 +297,8 @@ class McppSession(private val project: Project) : Disposable {
297297
// when unknown, and the editor's markers recomputed when the answer arrives.
298298
private val sources = java.util.concurrent.ConcurrentHashMap<String, JsonObject>()
299299
private val cases = java.util.concurrent.ConcurrentHashMap<String, JsonObject>()
300+
// The programs whose cases are being asked for: one listing at a time each.
301+
private val asking = java.util.concurrent.ConcurrentHashMap.newKeySet<String>()
300302

301303
fun entriesFor(path: String): JsonObject? {
302304
val known = sources[path]
@@ -322,30 +324,33 @@ class McppSession(private val project: Project) : Disposable {
322324

323325
fun casesFor(root: String, test: String): JsonObject? {
324326
val known = cases["$root\u0000$test"]
325-
if (known == null) refreshCases(root, test, list = false)
327+
if (known == null) refreshCases(root, test, list = true)
326328
return known
327329
}
328330

329331
fun refreshCases(root: String, test: String, list: Boolean) {
332+
val key = "$root\u0000$test"
333+
if (!asking.add(key)) return
330334
request("test/cases", JsonObject().apply {
331335
addProperty("root", root)
332336
addProperty("test", test)
333337
addProperty("list", list && TrustedProjects.isProjectTrusted(project))
334-
}).thenAccept { r ->
335-
cases["$root\u0000$test"] = r.asJsonObject
338+
}).whenComplete { r, error ->
339+
asking.remove(key)
340+
if (error != null) return@whenComplete
341+
cases[key] = r.asJsonObject
336342
restartMarkers()
337343
}
338344
}
339345

340-
/** A source changed: what was read from it is read again on its next use. */
346+
/** A source changed: what was read from it is read again on its next use. Its program's cases
347+
* stay as they were listed until a build or a test lists them again. */
341348
fun forgetSource(path: String) {
342349
sources.remove(path)
343-
cases.keys.removeIf { it.split('\u0000').getOrNull(1)?.let { test -> path.replace('\\', '/').endsWith("/tests/$test.cpp") } == true }
344350
}
345351

346352
private fun forgetSources() {
347353
sources.clear()
348-
cases.clear()
349354
}
350355

351356
private fun restartMarkers() = com.intellij.openapi.application.ApplicationManager.getApplication().invokeLater {

0 commit comments

Comments
 (0)