Repository navigation
2026.11.13.1: toolchain payloads as data, build directories by lineage, UEFI through openkal, test feedback (mcpp#927), end-to-end cases in C++ with places and capabilities (mtest 0.8.0), canaries - #930
Merged
Conversation
tests/support (mcpp.test.support) is the support of the end-to-end cases (design 2026-11-12 §5.9): a project as a table of files, mcpp run with the shared home or a fresh one, the readings of a file's lines, a JSON document and an executable's header (ELF, PE, Mach-O) made by this process. A case file imports mcpplibs.mtest itself, which begins it at mtest's entry. tests/e2e/basics holds 02 (new, build, run) as the first case. mtest 0.7.1 is the dev-dependency and the engine CI pins; the unit suite holds under it (169 programs, 2627 cases).
tests/e2e/MIGRATION.tsv gives each script its subsystem (25 directory-style names), its class (sink, merge, keep; design 2026-11-12 §5.9) and its planned destination: e2e/<subsystem> for a merged case, unit/<file> for a sunk one, keep for a script that stays. Assigned from each script's name, header and the commands it runs. merge 707, sink 12, keep 33. tests/e2e/keep.tsv is the baseline of the keep class (emulators, a physical device, upgrades and machine-state homes, CI-only and measurement legs); it may only shrink.
The 38 scripts MIGRATION.tsv assigns to `test` (mcpp test: discovery, selection,
reports, the JSON stream, timeouts, workspaces, --no-run and archives, the
launch description, the mtest protocol, the store, the fast path) are 39 cases
of the program e2e/test (tests/e2e/test/*.cpp, 12 files, one process each).
Each case is named by its script's sentence and keeps its number; its
`# requires:` words are its tags, and a case skips with "missing capability:
<word>" where the machine lacks one. The scripts are removed and MIGRATION.tsv
names each one's case.
Scripts: 15 16 17 18 72 78 90 94 152 153 154 155 157 158 159 160 164 178 213
673 745 854 855 856 865 989 993 1026 1034 1035 1037 1038 1050 1052 1065 1068
1069 1070.
Old run (the scripts, against this branch's mcpp, Linux x86_64, gcc and llvm in
~/.mcpp): 37 passed, 1 skipped (94 requires macos; started directly it exits 0
on Linux, its otool leg being conditional). Sequential time 227 s.
New run (`mcpp test e2e/test`): 38 passed, 0 failed, 1 skipped (94, missing
capability: macos). 72 s of runs, 12 programs at once.
tests/support gains: has(word) and needs({...}) (run_all.sh's capability
probes; an unknown word fails the case), on_path, glob (a shell's `*` per
component), has_line/no_line (grep -E per line), lacks, contains_nocase,
count_lines, ninja_log_outputs, mcpp_command (a mcpp for an e2e::session) and
project::scaffold (`mcpp new` in the case's directory).
Every assertion is kept. Where a reading changed:
- 159: `grep -qv '"test":"01-b/0"'` held for any line without the record; the
case asserts what it meant, that no line has it (stronger).
- 989: the evidence the script printed after a failure (file, xattr,
codesign) is a diagnostic, not an assertion; not carried over.
- 1070 O8: the script compared the output after the shell had stripped its
trailing newline; the case accepts the text with or without it.
- 993 is two cases: A-E, and the named-runner leg, which keeps
`requires:python3` because its runner is a Python program.
- python3 (993 A-E, 1026, 1035, 1037, 1050, 1065, 1070) read JSON for the
scripts; the cases read it with nlohmann::json and no longer require it.
- 90: its `requires: gcc` stood on line 8, which run_all.sh does not read, so it
ran on every host; the case does not require gcc either.
- 1052 sourced _toolchain_env.sh without using it; dropped.
- 178 and 1068 drive concurrent mcpp processes through e2e::session instead of
`&`, a fifo and kill. 1026 still unpacks with tar, 1038 still uses git, 94
still reads otool.
The 66 scripts MIGRATION.tsv assigns to `build-program` (build.mcpp: its directives and protocol, its environment and cache, dependencies' programs, host tools and host modules, build plugins and the build programs' graph, accelerator requests and the probe channel, payload overrides) are 66 cases of the program e2e/build-program (tests/e2e/build-program/*.cpp, 6 files: accel, cross, deps, graph, hosttools, protocol). Each case is named by its script's sentence and keeps its number; its `# requires:` words are its tags. Scripts: 58 89 92 110 111 112 125 139 143 144 145 168 181 186 187 189 193 194 309 310 311 312 334 600 605 612 617 618 622 624 625 626 632 633 651 665 683 693 698 712 720 775 779 800 802 804 823 824 826 831 859 873 874 877 902 903 904 905 906 908 910 958 997 1011 1036 1051. 168 and 804 keep their scripts: ci-aarch64-fresh-install.yml runs them by path. The other 64 are removed. Old run (the scripts, against this branch's mcpp, Linux x86_64): 65 passed, 1 failed (779). New run (`mcpp test e2e/build-program`): 65 passed, 1 failed (779), 0 skipped. 779 fails on this host in both runs for one reason outside mcpp: it builds its plugin with the `gcc` PATH names, here an xlings subos gcc whose shared objects record NEEDED libc.so (the linker script), which the loader cannot load; /usr/bin/gcc records none. Under the script the plugin built and `mcpp run` failed loading it; under `mcpp test` PATH names another subos gcc that does not find crti.o. tests/support gains: home kinds `linked` (the case's home with the shared registry linked, as `ln -s ~/.mcpp/registry`), `inherited` (_inherit_toolchain.sh) and `payloads` (the same without config and subos), made under the system's temporary directory (glibc refuses an install directory over 255 bytes, which a case's own directory exceeds); case_home, shared_home, newest_payload, payload_root, family_version (and llvm_version, gcc_version, musl_gcc_version, mingw_cross_version: _toolchain_env.sh), repository, listing, newest_file, write_executable, run, has_line_nocase, last_line, last_starting, count_matching. Every assertion is kept. Where a reading changed: - `mcpp run 2>&1 | tail -1` is read as the last line of the program's standard output (the shell's interleaving put the program's line last). - 168: `file`'s "statically linked" and readelf's interpreter are read from the ELF header: no PT_INTERP and no DT_NEEDED. - 802: readelf's NEEDED is read from the header; 804: the .comment strings are read from the file's bytes (`clang version <llvm version>`, `GCC: (...) 16.1.0`), the llvm version being the one _toolchain_env.sh chooses (23.1.3 here, the literal the script grepped). - 310: `strings -a` and `nm -C` become the file's bytes and `nm -C` when PATH has it. - 720, 877, 903: python3 read JSON for the scripts; the cases read it with nlohmann::json and no longer require python3. 1011 required python3 and did not use it; the case does not require it. - 600, 605, 612, 626, 1036 and the other `mcpp new` fixtures keep `mcpp new`; fixtures the scripts wrote whole are written whole. - 698's Windows long-path reading runs powershell where the script did.
The 49 merge scripts MIGRATION.tsv assigns to `deps` (path, git and index
dependencies, how a dependency is named and resolved, mcpp.lock, tool
packages a graph declares twice, floors a dependency states, patches and
`embed`, entry targets, the advice an edge states) are 49 cases of the
program e2e/deps (tests/e2e/deps/*.cpp, 12 files: advice, edges, entry,
floors, git, graph, identity, lock, names, path, resolve, tools). Each case is
named by its script's sentence and keeps its number; its `# requires:` words
are its tags.
Scripts: 09 12 24 27 31 32 33 61 62 63 79 161 162 163 165 196 203 326 603 627
628 629 661 679 682 713 816 821 822 844 939 943 944 945 946 948 980 990 1001
1002 1018 1027 1032 1046 1048 1060 1066 1071 1074. No workflow runs any of
them by path; all 49 are removed. 23_remove_update.sh is a sink row and stays
for the sink step.
Old run (the scripts, against this branch's mcpp, Linux x86_64): 49 passed.
New run (`mcpp test e2e/deps`): 49 passed, 0 failed, 0 skipped.
Every assertion is kept. Where a reading changed:
- python3 read JSON or rewrote a lock for 679, 682, 844, 1002, 1027 and 1071;
the cases do it in C++ (nlohmann::json, the FNV-1a digest, the lock
entry's lines) and 844, 939 (which did not use it) and 1071 no longer
require python3. 844's os.utime is std::filesystem::last_write_time.
- 628 read `mcpp why toolchain --format json` with jq only when jq was on
PATH; the case always reads it.
- `ls -1 <store> | wc -l` (627) counts the store's own entries;
`touch -d '30 days ago'` (945) sets the time 30 days back.
- 822 took its own release from `mcpp --version | awk '{print $2}'`, the case
from the second word of that line.
- 816's `LC_ALL=C grep $'\r'` and `$'\x1b'` are byte searches of the output.
- 713 skips when git is missing, as the script did (exit 77); the other git
cases skip the same way.
- Diagnostics the scripts printed on failure (`<(find ...)`, `diff`) are not
kept; the failing comparison shows both sides instead.
- Homes the scripts made with `MCPP_HOME=$TMP/home` (627, 628, 629) are fresh
case homes; `_inherit_toolchain.sh` homes are `home::inherited`.
docs/specs and docs/06 still name 163, 603, 627, 628 and 679 as the tests of
their rules; this change does not touch docs/.
The 37 scripts MIGRATION.tsv assigns to `actions` (actions a build program
declares and their roles, checks and their stamps, `prepare`, generated files,
device sources and the actions that consume them, rule packages, project build
hooks, what an action writes and reads, action progress) are 37 cases of the
program e2e/actions (tests/e2e/actions/*.cpp, 10 files: checks, commands,
declared, devices, generated, hooks, outputs, prepare, progress, rules). Each
case is named by its script's sentence and keeps its number; its `# requires:`
words are its tags.
Scripts: 51 107 124 142 188 313 314 315 317 606 607 608 609 610 611 613 619
623 630 631 648 659 660 780 790 791 792 793 798 799 808 827 896 897 982 1013
1015. No workflow runs any of them by path; all 37 are removed.
Old run (the scripts, against this branch's mcpp, Linux x86_64): 36 passed,
1 failed (790). New run (`mcpp test e2e/actions`): 36 passed, 1 failed (790),
0 skipped. 790 fails on this host in both runs for the reason 779 does (batch
2): its install.sh links a shared library with the `gcc` PATH names, an xlings
subos gcc whose objects record NEEDED libc.so, a linker script the loader
cannot load ("invalid ELF header" when the program runs, under the script);
under `mcpp test` PATH names a subos gcc that does not find crti.o.
tests/support gains archive_members and archive_symbols: an `ar` archive's
members and its symbol index, read by this process (System V/GNU and BSD).
Every assertion is kept. Where a reading changed:
- 608's `ar t` and `nm` are read from the archive (its members and its
symbol index); the script read nm only when PATH had it.
- 606 read `mcpp why toolchain --format json` with jq; 808, 897 and 982 read
JSON with python3 without requiring it. The cases use nlohmann::json.
- 896's mtime in nanoseconds (python3) and 142/188/660's `stat %Y` are
std::filesystem::last_write_time; 827's `find -newer <ref>` compares the
destination's time with the one it had.
- 1013 and 1015 keep their Python drivers: the actions they declare run
Python workers, and 1013 drives the build through a terminal and a process
group. Each case writes its script's driver unchanged and runs python3 on
it; both keep requiring python3.
- 317's Ctrl-C leg sends SIGINT through `sh -c` as the script did (Unix
only, as before).
- 659's `type -P true` is the `true` PATH names; 799's `pwd -P` is the
canonical path.
- `mcpp run 2>&1 | grep '^X=' | tail -1` is the last such line of the
program's standard output.
The 37 scripts MIGRATION.tsv assigns to `features` (feature activation, the --strict schema gate, feature defines and their propagation, capabilities and their providers, feature-deps, target-conditional configuration, feature-gated sources, default features and forwarding, layer predicates) are 37 cases of the program e2e/features (tests/e2e/features/*.cpp, 5 files: activation, conditional, defaults, layers, sources). Each case is named by its script's sentence and keeps its number; its `# requires:` words are its tags. Scripts: 67 71 80 81 82 83 85 86 100 106 108 119 126 127 128 138 146 149 195 247 308 328 601 615 637 654 677 710 711 714 739 813 895 1007 1012 1019 1045. No workflow runs any of them by path; all 37 are removed. Old run (the scripts, against this branch's mcpp, Linux x86_64): 37 passed. New run (`mcpp test e2e/features`): 37 passed, 0 failed, 0 skipped. Every assertion is kept. Where a reading changed: - python3 and jq readings of JSON are nlohmann::json. - 108 reads the host architecture from the compiler's predefined macros instead of `uname -m`. - 739's shell glob over the whole failed build (`*"sdk.h"*"No such file"*`) is the same search: "sdk.h", then one of the three phrases after it. - 813's `grep -rl` over the home's std-module records reads the records in sorted order, so "the first record holding the flag" is one record on every run (the script took find's order). - sed's line edits (1012, 711, 813) are the same edits on the manifest text; a fixture line an edit needs and does not find fails the case.
The 37 scripts MIGRATION.tsv assigns to `pack` (the packing modes and the closures they carry, library packages and the gate a prebuilt package keeps, formats a package provides, the report and the axes of a package, packs over workspace members) are 37 cases of the program e2e/pack (tests/e2e/pack/*.cpp, 4 files: formats, library, members, modes). Each case is named by its script's sentence and keeps its number; its `# requires:` words are its tags. Scripts: 30 215 216 242 243 244 245 249 250 251 252 253 254 261 264 265 638 649 652 662 669 689 722 723 801 828 829 841 867 868 869 870 871 954 1008 1043 1061. No workflow runs any of them by path; all 37 are removed. Old run (the scripts, against this branch's mcpp, Linux x86_64): 36 passed, 1 failed (669). New run (`mcpp test e2e/pack`): 36 passed, 1 failed (669), 0 skipped. 669 fails on this host in both runs at the step 779 and 790 fail at (batches 2 and 4): it links a prebuilt library with the `g++` PATH names. Under the script that is an xlings subos g++ whose library records NEEDED libc.so, so the pack refuses it; under `mcpp test` the subos g++ does not find crti.o. tests/support gains readers for what the scripts asked of binutils, tar, zip and python3: ELF sections and symbol tables (defined or all), DT_SONAME and whether a search path is DT_RPATH or DT_RUNPATH; a tar archive's members and their extraction (by the machine's tar); a zip's central directory; SHA-256. Every assertion is kept. Where a reading changed: - `file`, readelf, nm and the scripts' python3 ELF readers (read_tag, has_symtab, elf_facts) are read from the files by these readers; 243's `ar t` is read from the archive (the script read it when ar worked). - python3 readings of JSON (723, 841, 869, 870, 954) are nlohmann::json; 954's tarfile/zipfile/hashlib readings are tar's member list, the zip central directory and SHA-256. - 871 compares compile intervals on the .ninja_log's absolute clock (each edge ends at its output's recorded mtime and begins its duration earlier), allowing 50 ms for the write that precedes a compiler's exit. `mcpp test` runs ninja twice (the program, then the tests) and each run appends with its own relative clock, so the script's comparison of the relative columns paired edges of different runs; on this host its awk compared them as strings and found nothing either way. Two compiles run at once here overlap by about 300 ms. - `find | head -1` is the first path in sorted order. - sed edits of a manifest (244's abi tag, 828, 1043, 1061) are the same edits on its text; 244's runs per line, as sed's does. - `cp -r` copies (638) keep links as links.
The 45 scripts MIGRATION.tsv assigns to `build` (a new project built and run, incremental rebuilds, libraries, convention mode, C and assembly sources, profiles, package-owned and per-target flags, include directories, response files and link inputs, the C driver, link flags as written, paths with spaces or no UTF-8 spelling, serial and concurrent plans) are 45 cases of the program e2e/build (tests/e2e/build/*.cpp, 4 files: basics, flags, link, sources). Each case is named by its script's sentence and keeps its number; its `# requires:` words are its tags. Scripts: 02 04 07 08 25 26 50 54 68 70 73 87 104 105 109 115 116 117 123 141 148 179 190 191 238 246 316 620 765 776 777 795 861 882 894 907 909 929 938 991 1003 1006 1047 1054 1062. No workflow runs any of them by path; all 45 are removed. Old run (the scripts, against this branch's mcpp, Linux x86_64): 45 passed. New run (`mcpp test e2e/build`): 45 passed, 0 failed, 0 skipped (907 passed on its rerun after a fixture edit in the case was corrected). Every assertion is kept. Where a reading changed: - `file`, readelf, nm (238's `nm -D --defined-only`), `ar t` (07) and `strings -a` (929) are read from the files by tests/support's readers; 929's `strings` is the printable runs of four or more bytes, as strings prints them. - python3 readings of JSON (765, 894) are nlohmann::json; 148's python3 fixture generators are C++ that writes the same files. - `sleep 1; touch f` (04, 104) moves the file's time two seconds on. - 938 starts the program from a thread of the case while it relinks, as the script did from a background subshell. - `find | head -1` is the first path in sorted order.
A layer key answered by membership in the resolved kernel-abi provider's [kernel-abi] provides-interfaces, as accelerator is a set. A provider that states no list provides every interface; a graph without the layer provides none. A package selects its sources by what the implementation beneath it provides and names no implementation. The provider lookup is one function, shared with the requires-interfaces check.
Which package a family's toolchain is for a target on a host is the [[toolchain.payload]] table of data/builtin.toml (name, aliases, family, serves, hosts with exclusion, frontend directory and names, builds-for, glibc-fixup, family-version), and [[toolchain.offer]] what a host is offered. mcpp.toolchain.payloads reads both directions: to_xim_package, identify_xim_payload, available_toolchain_indexes and the payload names normalize_spec accepts. No payload is named in compat, and registry names none. xim:<name>@<version> accepts any package: it is installed and resolved as path:<root> with the table its .mcpp-toolchain.json states (schema 2: family, prefix, sysroot, tools, each inside the payload); the family is the descriptor's or its C++ driver's. check_payload_names.py counts the table's payload names in the sources against a baseline, as check_system_names.py counts systems; the refactor lowers the system-name baseline.
… package as a toolchain
…ge, and its size (section 17)
…bug in clean --report; debug = "lines" A configuration's build directory is named by its lineage (compiler family, target, standard library family, standard, configuration flags). The evolving parts (mcpp, compiler build, standard library version, runtime contract) are recorded inside it. An upgrade keeps every path the commands name, so ninja compiles only what the release changed. Before, the directory took the whole fingerprint's name, and its takeover changed every member's commands. A compiler build that differs empties the directory in place. Finished states the time spent waiting for a build directory. clean --report gives split debug information its own column. [profile] debug = "lines" keeps the line tables alone (-g1). Design 2026-11-12 section 17.
A build directory's graph names mcpp as engine/epoch-<N>/mcpp, a hard link (else a copy) the planner makes the running engine under the directory's lock. mcpp's own path holds its version under xlings, and every command that ran it -- the capped compile, BMI comparison, link wrapping, staging, module bindings -- changed with each release. Measured on a two-member workspace, an engine that differs in its version alone: 18 of 153 objects and BMIs compiled again before, none after. The link's path carries kCacheEpoch. Fast-path admission accepts a link to this engine; a self-contained home is stated for the programs mcpp starts, since it cannot be derived from the link's place. In-flight compiler records are dropped when another engine wrote them.
…that compiles nothing e2e 1000 gains G-I: the directory named by its lineage and recording the fingerprint, the graph running mcpp through engine/epoch-<N>/mcpp, the same mcpp from another path compiling nothing, and a compiler change emptying the directory in place. 878 states that the program a graph runs is the engine that wrote it. 960 and the accel cases read the new configuration-change warning. unit/test_graph_engine covers the link.
The 17 scripts MIGRATION.tsv assigns to `cli` and that migrate here (--help and --version, the error paths of a manifest and a source, usage errors kept off standard output, the exit-code contract, --jobs and the `--` boundary, --toolchain on run, test and pack, package templates and a transactional scaffold, a build's output and status streams, a terminal's status row, the control contract, and mcpp under signals) are 17 cases of the program e2e/cli (tests/e2e/cli/*.cpp, 5 files: commands, output, signals, templates, terminal). Each case is named by its script's sentence and keeps its number; its `# requires:` words are its tags, and 936's `# resources: machine` its resource. Scripts: 01 05 48 69 202 204 231 329 340 671 842 843 864 866 936 941 1016. No workflow runs any of them by path; all 17 are removed. 845 (--play-game restores the terminal) is kept as a script: its case finds the prepare action alive three seconds after a Ctrl-C ended mcpp, which criterion G3 forbids, while the script's own G3 check compares a directory name with pgrep's list of pids and cannot fail. MIGRATION.tsv records it; it waits for the engine owner. Old run (the 18 scripts, against this branch's mcpp, Linux x86_64): 18 passed. New run (`mcpp test e2e/cli`): 17 passed, 0 failed, 0 skipped. tests/support gains `process`: a program held by its pid, signalled alone, whose standard streams are a pseudo-terminal, one pipe for both streams, a file, /dev/null or a pipe whose reader has ended, and which starts with every signal at its default; `processes()` reads /proc; `strip_escapes` removes CSI sequences. Every assertion is kept. Where a reading changed: - `2>&1 | cat` and `2>&1 | tee` (864, 866) are both standard streams of mcpp on one pipe of `process`; the shell's `&`, `kill` and `wait` (340, 936 H, 1016) and Python's subprocess with `restore_signals` (941) are `process`; 941 B's `returncode == -SIGPIPE` is the signal `process` reports. - Python's pseudo-terminals (843, 866 D) are `process` with a terminal; 843's screen model is C++ with the same rules (carriage return, line feed, CSI A, J, K), over the code points of the UTF-8 the terminal received. - 842 R5's Python timer reads the pipe in 50 ms steps and takes the time at which the `error: build failed` line first appears and at which mcpp ends. - python3 readings of JSON (866 B, 936) are nlohmann::json; 936's `normcase(normpath(...))` comparisons are `lexically_normal` path comparisons, which do not fold case on Windows. - 69's `sed | grep -oE '"targets":[...]'` is a regular expression over the log after `\"` is unescaped; 01's awk reading of mcpp.toml is a regular expression. - `sleep 1; touch mark; find -newer mark` (231) is the clock's time taken after a second's wait, compared with each object's write time. - 05's build with no manifest runs in a directory outside the repository, and 202 runs in a project of its own: a case's directory lies inside the repository, whose mcpp.toml mcpp would find, and `mcpp pack --pkg-format bogus` builds the project it finds before it refuses the value (the script, run from tests/e2e, built the repository).
The 30 scripts MIGRATION.tsv assigns to `modules` (several modules and partitions built and run, the p1689 scanner and ninja's dyndep, module units in .cpp files, transitive and indirect imports, the std module compiled once, what the scanner reads, `[build] module_extensions` and the lib root that follows it, BMIs and their cache and staging, the split schedule, and modules across programs, members and homes) are 30 cases of the program e2e/modules (tests/e2e/modules/*.cpp, 5 files: basics, bmi, extensions, programs, scanner). Each case is named by its script's sentence and keeps its number; its `# requires:` words are its tags, and the `# resources: machine` of 170, 234 and 1039 their resource. Scripts: 03 19 20 21 49 113 118 167 170 171 217 218 234 236 260 263 634 635 636 639 807 847 849 857 858 898 924 940 1039 1064. No workflow runs any of them by path; all 30 are removed. Old run (the scripts, against this branch's mcpp, Linux x86_64): 29 passed, 1 failed. 218 fails on its reading alone: it looks for the graph in `target/*/<fingerprint>/build.ninja`, and the build directory is named by the configuration's lineage since this branch's engine (2da705c); with that one line reading the lineage instead, the script passes. New run (`mcpp test e2e/modules`): 30 passed, 0 failed, 0 skipped. tests/support gains `another_home` (a further home of a case, for a script that exported MCPP_HOME again) and `file_identity` (`stat -c '%i %Y %s'`). Every assertion is kept. Where a reading changed: - 218 finds the graph in the directory `--print-fingerprint` names as `Lineage:`; its assertion that module_extensions leaves the fingerprint unchanged is kept. - python3 readings (924's module maps, 1039's newest std-module record, 49's entry.json) are C++ and nlohmann::json; 234's inline replay check is C++ (write times and tests/support's sha256), and its sections 6, 8 and 9 run tests/e2e/helpers' Python checks as the script did. - 1039's two background builds are two threads of the case, each running mcpp. 1039 is a case of bmi beside 170 and 171: it removes the shared home's std entry they stage from, and its `machine` resource keeps them apart only within one program (in another program, 171 found the entry gone); 858's `export MCPP_HOME` to a second and a third home is `another_home`. - 171's `stat -c '%i %Y %s'` is `file_identity`, `chmod 444` is a permission change, `cmp` a comparison of the bytes, and its Windows holder the same PowerShell command run as an `e2e::session`; 170's `cmp` likewise. - `touch` of an existing file (170, 171, 234, 858) sets its write time to now; `sleep 1; touch marker; find -newer marker` (218, 847) is the clock's time taken before a second's wait, compared with the file's write time. - `ls -d target/x86_64-linux-*/` and `./target/*/*/bin/app` (20, 21, 849) are globs; `[ -d "$LLVM_ROOT" ]` (118, 217) is the llvm payload's directory; 635's `head -c 3 | od` compares the first three bytes; 639's `printf '%b'` sources are C++ string literals. - The commands that name no project (858's cache commands, 171's `mcpp stage`) run in a directory outside the repository, as the scripts' did: a case's directory lies inside the repository, whose mcpp.toml mcpp would find.
The planned refusal says no toolchain is published for the row; a build that names one on the command line has one, as a build whose [target.<row>] section names one does. MCPP_TOOLCHAIN, a standing preference, does not opt in. A platform's conformance run (openkal-platform's i686-linux-musl row) named its toolchain with --toolchain and was refused.
…bject format and suffix --toolchain is stated as MCPP_TOOLCHAIN, so the two are one statement and both opt a planned row in. The carrier of a system a platform defines took its section directive and program suffix from the system's name: a PE row such as x86_64-uefi was assembled as ELF and named without .efi. Both now come from the row (Triple::object_format, artifact_naming).
A case made its files in mtest::temp_dir(), which `mcpp test` places under the repository's target/ (test-protocol/<run>/<case>). mcpp run there in a directory with no manifest of its own found the repository's mcpp.toml above it and built the repository: one such run took over the released engine's build directory. tests/support gains `case_dir()`: the directory of the case under the system's temporary directory (`mcpp-e2e-case-XXXXXXXX`), made on first use. `project` and `project::scaffold` make their files there, and the cases of the earlier subsystems use it in place of mtest::temp_dir() (222 uses in 54 files; no case depended on the old location). The case's homes and its directory are filed under the runner's directory of the case, which mtest removes when the case ends: a directory whose case has ended is removed at the next use of the support, and the rest when the program ends.
The 22 scripts MIGRATION.tsv assigns to `cache` (the global build cache
across projects and keyed by what an entry was compiled against, its modes
and maintenance commands, an entry's layout and a hit's std BMI, the fast
path and what it compares, a configuration's build directory over time,
`clean --stale` and `clean --report`, the sweeps by age and by budget, and
what a build writes once) are 22 cases of the program e2e/cache
(tests/e2e/cache/*.cpp, 3 files: entries, fastpath, directories). Each case
is named by its script's sentence and keeps its number; its `# requires:`
words are its tags. No script declared a resource.
Scripts: 172 173 174 184 212 324 609 645 832 878 879 935 959 998 999 1000
1005 1022 1028 1029 1030 1031. No workflow runs any of them by path; all 22
are removed.
Old run (the scripts, against this branch's mcpp, Linux x86_64): 20 passed,
2 failed. 174 fails on an engine defect of this branch: `mcpp build
--cache=off` empties the build directory after the plan made its link to
the engine (`engine/epoch-<N>/mcpp`, design 2026-11-12 §17), and every step
then fails with "not found" (a project, `mcpp build`, then `mcpp build
--cache=off`). 832 failed once with a SIGSEGV of mcpp in nlohmann's lexer
while resolving the toolchain in the shared home, during a concurrent run;
it passed when run again.
New run (`mcpp test e2e/cache`): 22 passed (21 ok, 174 an expected
failure), 0 failed, 0 skipped. 174 carries
`known_gap{"build --cache=off removes the engine link the graph runs",
"2026-12-15"}`: every assertion is kept, the run reports it as xfail, and
from 2026-12-16 it is judged as any case.
Every assertion is kept. Where a reading changed:
- python3 readings (174's `cache list --json` and entry.json, 935's result
record, 998's programs.json) are nlohmann::json; 174's strip of the
`[build] cache` override writes the manifest without it; 1029's os.utime
walk sets the write times of the entry's files and directories.
- `touch -d '<n> days ago'` and `touch -t 200001010000` (609, 1000, 1005,
998) set write times; 1000's `find -mtime -1` compares the write time with
a day ago; 174's `touch -d @1000` sets the time 1000 s after the epoch.
- `sleep` is a wait of the case; 935's two background builds are a thread
of the case and the case itself, each running mcpp.
- 878's and 1000's engine at another path is the program under test
copied there and started by its path with the same environment; 878's
`grep -v '^engine='` and awk rewrite of `$mcpp`, and 879's awk ageing of
`depSources=`, are line rewrites of the record and the graph.
- `ar t` (998) is `archive_members`; `stat -c %i` (998) is the inode of
`file_identity`; `readelf -S` (998, 999), `readelf --debug-dump` (173) and
`gdb` (999) run as the scripts ran them, where the machine has them; git
(1028) likewise.
- 1022's `cp -a` is a copy of the tree that keeps its links and its write
times; 1031's `cp -R` is a recursive copy.
- 645's "NOT MEASURED" exit 0 when an unchanged second build is not
replayed is a skip naming the recorded request; its machine-default leg
without a second installed version ends the case as passed, as the
script's exit 0 did. 324's NOTE (the warm build not replayed) printed and
asserted nothing, and is left out.
- 1029's cache commands, which name no project, run in the case's
directory, outside the repository.
…UEFI converges into openkal (section 19)
…interfaces (section 19)
- tests/support: as_text reads "\r\n" as "\n" on Windows (mcpp, run, the archive helpers, project::read), as a Windows shell's grep did; a bare "\r" is kept. - tests/support: tar is the system's bsdtar on Windows (Git's GNU tar reads C: as a host); local_server runs on Winsock; host_cc is the first host C compiler that links a shared object (PATH may lead to a subos gcc that cannot link); a case's mcpp runs without the runner's MCPP_VERBOSE. - Cases: .exe names of executables (25, 234, 609, 849), the MSYS form of a cwd (799), the toolchain's Resolved line past a sysroot's (645, 671), sessions for concurrent writers (919), the host C compiler (779, 790, 884, 200 and the runtime plugin cases), tar_extract (1026), 920 on Windows. - 864 and 327 state unix-shell, as their scripts' requirements did.
- The C++ e2e programs run in each host's unit job; a change to tests/support runs them all (40 minutes measured on xcode-27). - 721 is a case of e2e/macos and passes in the macOS unit job; the e2e step that ran its removed script is removed. - tests/canary/run.sh: an empty array under set -u (bash 3.2 on macOS).
Windows refuses a rename over a file another program holds open, and every mcpp reads config.toml: replace_file now retries from 5 ms, doubling to 200 ms, for two seconds, where it gave up after four tries in 650 ms. Config/ConfigValueIsReplacedWhole's reader pauses between reads on Windows, as the readers of a home do, and is a jthread: a failed write ended the process (exit 137) with the reader still joinable.
The cases run as concurrent programs, and 97 and five msvc cases set the shared home's default (default_kept restored it afterwards): 760, 825, 814, 256, 820 and 1033 read the machine's default in the meantime and resolved mingw's gcc. s::home::registry is a home of the case whose config.toml is its own and whose [xlings] home is the shared registry; those cases set the default there, and default_kept is removed. The two local registry_home helpers become it.
842 R5 reads the build through e2e::session (s::process is POSIX); 849 finds ninja.exe; 681 writes its index paths with forward slashes, which a TOML basic string does not read as escapes.
… command The tail of thirty lines held only linker errors. A log of up to 300 lines is printed whole, a longer one as its head and tail.
…beside it
The e2e cases run as programs of one mcpp test, and a case stating
resource{"machine"} ran beside the others: 234 and 1039 failed on the Linux
unit job as another program changed the shared cache, and passed alone.
The listing carries each case's resources (P7.1). A program the run divides,
or one that runs without a runner, is listed; the cases of a file that hold
machine are a part of their own (named <program>/<file> (machine) beside the
file's other cases), queued after every other test and run under the gate
whole, while every other test shares it. The run's record states machine among
the part's resources.
A resource stated on a case inside a suite is not listed, so the runner cannot
keep it. The machine cases of seven files move into a suite of their own
(<suite>-machine, resource{"machine"}); the four machine.cpp suites state it
for all their cases.
…dings - A directory an mcpp before 2026.11.13.1 wrote records no base line in its lineage and no runtime line in its toolchain: its records are read on the lines they state, so the upgrade takes it over (it was left beside a new one: 753 MB of mcpp's own build). e2e 1000 A2. - A cold build (--no-cache) keeps the directory's records and lock, so the next build still tells a changed compiler or base. - An engine link that cannot be made is reported (diag::degraded). - A portable home stated for the programs mcpp starts keeps its source, portable (self env reported environment). - write_replacing writes through a symbolic link and keeps the file's permissions; a refusal is retried only where it is a sharing violation (Windows); config.toml is read through read_whole, which waits out a replacement on Windows (Config/ConfigValueIsReplacedWhole saw 4 failed opens there). - mcpp test lists programs on the run's workers; a program it does not divide runs once, alone when a case holds machine. - A started program is polled from 1 ms, growing to 20 ms. - 681 reads config.toml as text (CRLF on Windows).
…ach call The library is pinned at 0.8.3 by mcpp, its modules, the test support, the canaries and the benchmark; mcpp new and mcpp test --guide name it (was 0.5.2). The engine (.github/versions.env) stays 0.8.0: 0.8.3's engine differs from it in its version string alone, and xim-pkgindex holds 0.8.0. bench/tests/engine.cpp times each command call by call (bench::each): the median keeps the metric's name, with .p95, .max and the two slowest calls; the edit of the interface is paused out of the rebuild's time; the mcpp under test is started by e2e::run, with no shell and no #ifdef.
…h --verbose The macOS cxx test cell links without libSystem (undefined memcpy, pthread_create, _Unwind_Resume) where the same row's build links; the commands of a verbose run name the flags that differ.
check_system_names counted two new is_windows branches (128 against the baseline 126). A refused rename is retried on every system, a final refusal costing the failing write the wait; read_whole retries an open refused while the file exists, on every system.
… store measured mcpp.build.test_schedule reads the run store: the tests start longest first; --shard I/N assigns each test, longest first, to the shard with the least measured work, so N machines with one store run every test once; --failed runs what the store's last run failed; [test] split = "auto" (and --split for one invocation) divides a part longer than its share into groups of its cases. A test that runs a program of the package is affected by the package's sources for --since and --changed, and an archive runs on a machine whose toolchain could not compile it (SPEC-017 O10, docs/08).
…e release builds its platforms at once ci-unit.yml builds every test with the bootstrap mcpp beside build.yml, with the gate that held the Linux build 18.6 minutes, and runs them in four shards assigned by main's measured history. Every unit job orders its tests by that history and a re-run runs what the attempt before failed (mcpp-unit). The macOS 27 image runs the SDK-dependent subset on a pull request; bench.yml measures the engine alone for a commit whose CI passed. release.yml builds the four platforms at once and creates the release with every archive in one job; the ecosystem canaries run in a release started with canaries. Measurements and decisions: .agents/docs/2026-11-14.
…e C++ e2e programs pass on the Linux shards The first run of the shards finished every C++ e2e program on Linux, which the 45-minute unit job had always cut short, and found: - a home whose registry is a link to the shared home's recorded its own store in the shared registry; such a home now borrows the registry, as one whose [xlings] home names it does, and the owner's index places stay; - mcpp test --archive handed MCPP_RUNTIME_FILES to every program; it now does so only through a runner; - MCPP_BOOT was the xlings shim, which finds no project in a case directory; - toolchain/layers 663 read the newest std metadata of the shared cache, and build-program/hosttools 831 held the fast path while installs beside it rewrote the runtime description (now a machine suite); - the cross and release jobs could take the engine copy a self-build keeps under target/<triple> for the product. A writer of config.toml on Windows retries every 20 ms at most.
… the provider's state turn, each decided in one place
The fixes the Linux shards called for, folded into the mechanisms they belong
to (design 2026-11-14 §5.2):
- A: whether a home borrows its registry (`[xlings] home` elsewhere, or
`registry` reached through a link) is decided once
(GlobalConfig::registryBorrowed); the index places, the [index.repos]
places and the upstreams are written only by the owner.
- B: PrepareIntent {Build, Describe, Configure, RunBuilt} replaces plan_only,
configuration_only and runs_built_tests; the driver skips the build programs
and host tools once where nothing is compiled, so the tests of an MSVC-ABI
archive run without Visual Studio.
- C: runner_handoff_env is the one source of MCPP_RUNTIME_FILES and MCPP_RUN_*
for a build's test run and an archive's; a program run directly receives
neither, and an archived file is named by its place beside the program.
- D: mcpp::home::Turn, a reentrant turn across processes and threads, in the
provider's real state directory (Provider::state_dir); every write of the
provider's state takes it. Readers of a home's files open them shared for
deletion on Windows.
- E: MCPP_BOOT is the released binary install_pinned_mcpp.sh installed;
built_artifact.sh takes a product from the build's report; a test states its
length with mtest timeout{} and the runner honours it (trees, sets); the
tests that assert a replay hold the machine; 778 reads the graph.
…ment; the tested mcpp is built once `mcpp test --no-programs` plans the package's programs for their names and builds none: the tests take each from the environment's MCPP_BIN_<NAME>, and one the environment does not name is refused before a test runs. With `--no-run --archive` the archive carries none of them and names their variables (`named_programs`); its run takes each from its own environment. The spelling of MCPP_BIN_<NAME> is one function (program_env_key). CI: the archive stage builds the tests only (when its bootstrap knows the option), the gate runs from the archive in the first shard, and the shards, the gate, the MinGW cases and the macOS and Windows unit jobs test the binary the build job made. Before, the archive stage built mcpp a second time (1118 s) and the shards tested that build. `mcpp build --message-format json` of an unchanged workspace restates the record of the build it replays, as a package's fast path does: it reported no artifact, and built_artifact.sh found none on a built tree. Design record 2026-11-14 §5.4 (F, G, H); docs/08 and docs/50 (en, zh); cases 1085 and 1086.
…'s runtime, read from its programs; a compiler that did not answer says so Which toolchain payloads run on the C runtime payload was a list (gcc, llvm). The emsdk payload's clang names the home's loader link (<home>/abi) as well; restored from a store cached without that link it did not start, and nothing pinned the runtime again (mtest 0.8.4's Web row). runtime_payload_kind now reads a placed payload's programs: one of bin/ whose PT_INTERP lies inside its own provider home, as the provider's layout gives it. No name and no system loader is written down; a Mach-O or PE program names no loader. A probe that fails with its errors states its last line of output, and clang's std probe keeps why it went unanswered (stdModuleUnanswered), which the refusal of `import std` gives. built_artifact.sh reads the artifact events with grep and sed -E: the hermetic container has no python. Design record 2026-11-14 §5.5 (I-L); docs/91 (en, zh); unit cases RuntimePayload/APayloadStartsThroughItsHome and RuntimePayload/APayloadOnTheSystemsLoaderDoesNot.
…ut at its limit mcpp's tests, its modules' tests, the canaries, the engine benchmark and what `mcpp new` scaffolds and `mcpp test --guide` suggests name mtest 0.8.5 (mcpp-community/mtest#22, #23; mcpp-index #777).
…cement; a case that asserts a replay holds the machine
…rm's runner built for this machine, and writes its records beside the archive's programs
…other program's hold explains, on every system
…ds' store, the runner fallback is the archive run's, an archive that names programs is schema 2
…; the lifetime helper waits for a killed controller's lock
…r line ending; Windows had appended a second [toolchain]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
2026.11.13.1: the second part of the ecosystem architecture design (
.agents/docs/2026-11-12-ecosystem-architecture-optimization-design.md, sections 5.7, 5.9, 15–19).Engine
[[toolchain.payload]]table of the built-in data names the package that serves a family for a target on a host. The table is read in both directions, so no payload name remains incompator the registry.xim:<name>@<version>accepts any package and reads its.mcpp-toolchain.json(schema 2:family,prefix,sysroot,tools). The payload names are counted against a baseline (check_payload_names.py).engine/epoch-<N>/mcpp. Measured: an upgrade of mcpp alone recompiled 18 of 153 units before and 0 after. A cold build makes the link again.Finishedstates the time spent waiting for the build directory.clean --reportgives split debug information its own column.debug = "lines"is new.cenvanswers by the row's object format rather than its system name.cfg(object-format = ...)selects by object format andcfg(interface = ...)by the interfaces an implementation provides.link, its base andrunnerover the platform's.--toolchainopts a planned row in.[test] programs = "package": one program per package, linked once and run as one process per file. A part is named by its path below the program's directory."directory"notes the files it leaves as programs of their own, and--strictrefuses a setting that forms no group.__action-stampno longer outlives Ctrl-C.config.toml. The in-place rewrite let one read a part of another's (invalid config.toml: expected '='; 233 cases on xcode-27).mcpp::home::write_replacingwrites beside the file and renames over it, and the config writer and the migration use it. A unit test reads while 200 rewrites run: it failed 1108 times on the old writer and passes now.Tests
tests/supportprovides the case support. Each subsystem is migrated in one commit, and each commit message lists its scripts and the counts of the old and new runs. Each host's unit job runs the cases withmcpp test."x"_test[s::windows][s::msvc]). The program skips it elsewhere and names the platform or the capability, whichever runner starts it.tests/supportdefines the vocabulary once;s::hasands::needsare removed. The 45 Windows scripts are cases oftests/e2e/windows(batch 24). Only the keep-class scripts remain.build.ymlwritesmcpp-tests-<host>(mcpp test e2e/windows --no-run --archive)..github/actions/e2e-casesruns named cases from it and requires each case'sokline. It serves the jobs with Visual Studio masked (182, including ci-fresh-install against the released mcpp), the MinGW and MSVC steps (97, 99, 180), and the Linux cross job (198, 923, 240, 248, 1058).MCPP_E2E_MCPPnames the mcpp under test. The CI engine is mtest 0.8.0. The unit steps give an e2e program 1200 s.prset. A pull request labelledci:full, a push to main, or a manual run runsfull.mtest ci plansets each host's e2e shard count from main's measured history; a host with no work starts no shard. Each build job runs the host's gate (e2e/basics) before its dependants start, and a failed gate cancels the run.tests/canary/c(C89 on POSIX) andtests/canary/cxx(modules,import std, exceptions, threads, a test) are built, linked statically, run, tested and packed for every identity intests/canary/matrix.tsv. Each cell is compared with the matrix in both directions. The workflowcanaries.ymlruns nightly, on dispatch, and on a pull request labelledci:canaries. The Linux rows are measured locally (68 cells, 0 differ).programs = "package"and the note of"directory".Documents