Skip to content

Commit 7e2b3d0

Browse files
committed
A link holds a permit of the build's compiler limit, and a POSIX link is published by one command
Under bmi_schedule = "on" with GCC ninja runs six jobs per compiler and the generation's semaphore bounded only the compilers, so every link a build reached at once ran at once: mcpp's 26 test programs relinked together at 1 to 1.8 GB each and the 16 GB CI runner ran out of memory (unit-linux-archive, on every commit of this branch). A link now takes a permit of the same semaphore; the graph still names no number (e2e 861). The permit's transaction judges its output when the held command exits, so the permit wraps the whole publication: a POSIX link is published through a temporary name by mcpp link-output --temporary rather than a shell sequence whose rename followed the linker, which recovery took for an uncommitted output and removed. The tests are built by the pinned bootstrap, which predates this: ci-unit builds the archive with bmi_schedule off for a bootstrap older than 2026.11.16.1, whose ordinary job count bounds links. Measured from main's restored test build at MCPP_JOBS=4: the bootstrap ran up to 7 links at once in 539 s, this engine at most 4 in 360 s.
1 parent 085fbd2 commit 7e2b3d0

10 files changed

Lines changed: 125 additions & 14 deletions

File tree

‎.agents/docs/2026-11-15-ecosystem-gap-supplement-design.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -621,3 +621,22 @@ profile 给出一次提示。需要评审后实现,不在本轮。
621621

622622
建议的设计:包的对象与 BMI 路径带上该包的配置身份(feature、标志的摘要),使不同配置并存、互不覆盖,配合
623623
`mcpp clean --stale` 回收。代价是磁盘占用,这与 #874 的“每个配置一个目录”相互权衡,需要评审后实现,不在本轮。
624+
625+
### 13.6 链接没有并发上限(CI 中定位,已修复)
626+
627+
- 现象:本 PR 的 `unit-linux-archive`(构建全部测试并归档)在每个提交上都失败。runner 被关闭,或与服务端失去通信,没有上传日志。main 上同一作业通过。
628+
- 先前误判为基础设施故障。按“红色作业先读日志”的规则比对 main 的历史后,确认失败与本 PR 有关。
629+
- 根因:GCC 且 `bmi_schedule = "on"` 时,ninja 的 `-j` 是编译上限的 6 倍,世代的信号量只约束编译器,链接不受约束。
630+
- 本 PR 改动了几乎所有模块都导入的 `toolchain-model` 构建程序,因此 26 个测试程序在同一时刻就绪并同时链接。每个 `ld` 占 1–1.8 GB,16 GB 的 runner 内存耗尽。
631+
- main 上的增量运行没有重链,所以没有暴露。
632+
- 设计:链接取同一信号量的许可(`$mcpp __compiler-cap --sem .mcpp-sched --output $out -- <linker>`),编译与链接合计不超过本机的并发数。
633+
- 图中仍不写数字,与作业数无关(e2e 861 的规则);归档只复制成员,不取许可。
634+
- 许可包住整个发布:被持有的命令结束时,产物已经就位。
635+
- 世代的事务以命令结束时的输出身份判定提交;若许可只包住链接器,其后的 `mv` 改变了输出,恢复时该输出被判为未提交而删除(实测:构建程序链接后消失)。
636+
- 因此 POSIX 的临时名发布由 shell 序列(`t=… && … && mv`)改为一个命令:`mcpp link-output --temporary` 以 `<dir>/.<file>` 链接(保留扩展名,N9),成功后改名覆盖产物,失败时删除临时文件。
637+
- 至此每个主机的链接都经 `link-output` 发布(Windows `--aside`,伴随文件 `--companions`,POSIX `--temporary`);原地链接的目标不经发布。
638+
- 未采用 ninja `pool`:pool 的深度要写进 `build.ninja`,图就随作业数变化。
639+
- CI:测试由固定的 bootstrap(2026.11.13.1)构建,它没有这项修复,而发布后不提升自身固定版本。
640+
- `ci-unit.yml` 按 bootstrap 的版本判断:早于 2026.11.16.1 时以 `MCPP_BMI_SCHEDULE=off` 构建归档,此时 ninja 的作业数(核数 + 2)约束链接;固定版本前移后自动恢复分离调度。
641+
- 测试:单元 `NinjaBackend/LinksHoldAPermitOfTheScheduleBound`;e2e 938(重链不暴露半个文件)、861(图与作业数无关)。
642+
- 实测(从 main 的测试构建恢复,`MCPP_JOBS=4`):bootstrap 同时运行的 `ld` 最多 7 个,超过上限 4,用时 539 s;本分支最多 4 个,用时 360 s。

‎.github/workflows/ci-unit.yml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,17 @@ jobs:
133133
set -euo pipefail
134134
export MCPP_VENDORED_XLINGS="$XLINGS_BIN"
135135
mkdir -p "$RUNNER_TEMP/unit"
136+
# LINKS BOUNDED BY THE BOOTSTRAP'S SCHEDULE. Before 2026.11.16.1 a link
137+
# under the detached schedule (bmi_schedule = "on") took no permit, and
138+
# ninja ran six per compiler: a change every test links relinked the 26
139+
# tests at once, at 1 to 1.8 GB each, and the runner ran out of memory.
140+
# Such a bootstrap builds the archive with the ordinary schedule, whose
141+
# job count bounds links; a later one holds a permit for each link.
142+
v=$("$MCPP" --version 2>/dev/null | awk '{print $2; exit}')
143+
if [ "$(printf '%s\n' 2026.11.16.1 "$v" | sort -V | head -1)" != 2026.11.16.1 ]; then
144+
export MCPP_BMI_SCHEDULE=off
145+
echo "the bootstrap $v holds no permit for a link: the archive is built with bmi_schedule off"
146+
fi
136147
# shellcheck disable=SC2086 -- the flag is one word or none
137148
# The profile `ci` (mcpp.toml): -O0 and no debug information. The
138149
# default profile of this package is release (-O2); the tests built

‎CHANGELOG.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,13 @@
8686
rebuilt a shared module into the group's directory: three compiles of one
8787
unchanged module in a run, and on Windows a failed rewrite of a module
8888
another process had mapped (#936).
89+
- Under `bmi_schedule = "on"` with GCC a link holds a permit of the build's
90+
compiler limit. Ninja runs six jobs per compiler there and only compilers
91+
were bounded, so links that became ready together all ran at once: mcpp's
92+
26 test programs relinked together at 1 to 1.8 GB each and a 16 GB machine
93+
ran out of memory. A POSIX link is published through a temporary name by
94+
`mcpp link-output --temporary`, one command, so the permit is held until the
95+
artifact is in place.
8996
- A `sources` entry without a pattern that names no existing file is an error
9097
naming its table, and a warning in a package fetched into a store. It was
9198
skipped, and the build failed at an importer of its module (#938).

‎docs/04-mcpp-toml.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1072,6 +1072,8 @@ with two behaviours has to be.
10721072
With GCC and `bmi_schedule = "on"`, ordinary compilation and detached code
10731073
generation share the same total `jobs` limit (2026.11.7.1+). Additional
10741074
scheduling and join tasks do not increase the number of compiler processes.
1075+
A link shares the same limit (2026.11.16.1+): links that become ready together
1076+
run as many at once as compiles do, not all at once.
10751077
Cancelling the build ends its ordinary and detached compilers. An immediately
10761078
following build waits for their cleanup before starting another compiler;
10771079
partial outputs are rebuilt. A missing object from a completed module is

‎docs/zh/04-mcpp-toml.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -988,6 +988,7 @@ bmi_schedule = "off" # auto (default, = off) | on | off
988988

989989
使用 GCC 且 `bmi_schedule = "on"` 时,普通编译与分离的代码生成共用同一个 `jobs` 总上限
990990
(2026.11.7.1+)。额外的调度和等待任务不增加编译器进程数量。
991+
链接共用同一上限(2026.11.16.1+):同时就绪的链接按编译的并发数运行,而不是全部同时运行。
991992
取消构建会结束本轮普通编译器和分离的编译器。紧接着的构建等待它们完成清理后
992993
才启动新的编译器;未完成的输出会重新构建。已完成模块的 object 缺失时,
993994
该次构建会重新生成它。输入和输出均不变的构建仍不进行编译。

‎src/build/ninja_backend.cppm‎

Lines changed: 20 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2402,23 +2402,32 @@ std::string emit_ninja_string(const BuildPlan& plan, std::string* placements,
24022402
} else {
24032403
if (!companions.empty()) return "$mcpp link-output --output $out" + companionOption + " -- " + cmd;
24042404
if (inPlaceTarget) return cmd;
2405-
const auto pos = cmd.find("-o $out");
2406-
if (pos == std::string::npos) return cmd;
2407-
// THE TEMPORARY NAME KEEPS THE FILE'S EXTENSION: `<dir>/.<file>`,
2408-
// not `<out>.tmp`. A linker that names a file after its output's
2409-
// extension writes a second file otherwise: DJGPP's driver, given
2410-
// `-o x.exe.tmp`, writes `x.exe.tmp` and `x.exe.exe` (measured with
2411-
// GCC 12.2.0 for i586-pc-msdosdjgpp; `-o x.exe` writes one file),
2412-
// and the package held two programs (design 2026-11-15 N9).
2413-
cmd.replace(pos, 7, "-o $$t");
2414-
return "t=$$(dirname $out)/.$$(basename $out) && " + cmd + " && mv -f $$t $out";
2405+
if (cmd.find("-o $out") == std::string::npos) return cmd;
2406+
// Through a temporary name, `<dir>/.<file>` (cmd_link_output): one
2407+
// command, so a permit around it is held until the artifact is in
2408+
// place (2026.11.16.1; before it a shell sequence that renamed the
2409+
// file after the linker's command had ended).
2410+
return "$mcpp link-output --output $out --temporary -- " + cmd;
24152411
}
24162412
};
24172413
{
24182414
constexpr bool useRsp = true;
2415+
// A LINK HOLDS A PERMIT AS A COMPILE DOES. Under the detached schedule
2416+
// ninja is given six slots per compiler and the generation's semaphore
2417+
// bounds the compilers, so every link a build reached at once ran at
2418+
// once: a change that every test program links relinked mcpp's 26 tests
2419+
// together, at 1 to 1.8 GB each, and a 16 GB machine ran out of memory.
2420+
// A link takes a permit of the same semaphore, so compiles and links
2421+
// together are what this machine runs at once; the graph still names no
2422+
// number (e2e 861). An archive copies its members and holds none.
2423+
const std::string linkPrefix = plan.scheduleCompilerCap > 0
2424+
? std::string("$mcpp __compiler-cap --sem .mcpp-sched --output $out -- ")
2425+
: std::string{};
24192426
auto link_rule = [&](std::string_view name, std::string cmd,
24202427
std::string_view desc) {
2421-
if (desc != "AR") cmd = publish_link(std::move(cmd));
2428+
// The permit wraps the publication: the attempt's output is the
2429+
// artifact in place when the command it holds exits.
2430+
if (desc != "AR") cmd = linkPrefix + publish_link(std::move(cmd));
24222431
append(std::format("rule {}\n", name));
24232432
if (useRsp) {
24242433
if (auto pos = cmd.find("$in"); pos != std::string::npos)

‎src/cli.cppm‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1382,6 +1382,7 @@ int run(int argc, char** argv) {
13821382
.option(cl::Option("output").takes_value().value_name("PATH").help("the artifact the command links"))
13831383
.option(cl::Option("companions").takes_value().value_name("SUFFIXES").help("the row's companions, removed before the link (`.wasm`)"))
13841384
.option(cl::Option("aside").help("move the previous artifact aside first (a Windows host)"))
1385+
.option(cl::Option("temporary").help("link under a temporary name in the artifact's directory, then rename it over the artifact (POSIX)"))
13851386
.action([&passthrough](const cl::ParsedArgs& p) {
13861387
return cmd_link_output(p, std::span<const std::string>(passthrough));
13871388
}))

‎src/cli/cmd_build.cppm‎

Lines changed: 34 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1840,8 +1840,13 @@ export int cmd_dyndep(const mcpplibs::cmdline::ParsedArgs& parsed) {
18401840
// nothing holds any more are deleted here. A process started while the linker
18411841
// writes fails to start (file not found, sharing violation) -- Windows never
18421842
// starts an image that is being written -- rather than running half of one.
1843-
// (POSIX links write `<artifact>.tmp` and rename it over the name in the
1844-
// edge's own shell command: a rename replaces a running program there.)
1843+
// On POSIX (`--temporary`) the linker writes `<dir>/.<artifact>` -- the
1844+
// argument after its `-o`, renamed; the name keeps its extension, as a driver
1845+
// that names a file after it writes a second one otherwise (design 2026-11-15
1846+
// N9) -- and the file is renamed over the artifact once the linker succeeds: a
1847+
// rename replaces a running program there, and a program started meanwhile
1848+
// never meets a partial file. The publication is one command, so an edge that
1849+
// holds a permit for the link holds it until the artifact is in place.
18451850
export int cmd_link_output(const mcpplibs::cmdline::ParsedArgs& parsed,
18461851
std::span<const std::string> command) {
18471852
namespace fs = std::filesystem;
@@ -1864,6 +1869,33 @@ export int cmd_link_output(const mcpplibs::cmdline::ParsedArgs& parsed,
18641869
// running program's file cannot be overwritten). An edge that reaches here
18651870
// for its companions alone does not: an aside file would be named like a
18661871
// companion (`app.js.old-1`).
1872+
if (parsed.is_flag_set("temporary")) {
1873+
std::vector<std::string> argv(command.begin(), command.end());
1874+
const auto temporary = out.parent_path() / ("." + out.filename().string());
1875+
auto named = std::ranges::adjacent_find(argv, [&](const std::string& a, const std::string& b) {
1876+
return a == "-o" && fs::path(b) == out;
1877+
});
1878+
if (named == argv.end()) {
1879+
std::println(libc::stderr(), "error: link-output --temporary: the linker command names no `-o {}`",
1880+
out.string());
1881+
return 2;
1882+
}
1883+
*std::next(named) = temporary.string();
1884+
const auto linked = mcpp::run::execute(argv, {.input = mcpp::run::Input::inherit,
1885+
.streams = mcpp::run::Streams::inherit});
1886+
if (linked.exit_code != 0) {
1887+
fs::remove(temporary, ec);
1888+
return linked.exit_code;
1889+
}
1890+
fs::rename(temporary, out, ec);
1891+
if (ec) {
1892+
std::println(libc::stderr(), "error: link-output: cannot rename {} over {}: {}",
1893+
temporary.string(), out.string(), ec.message());
1894+
fs::remove(temporary, ec);
1895+
return 1;
1896+
}
1897+
return 0;
1898+
}
18671899
if (!parsed.is_flag_set("aside"))
18681900
return mcpp::run::run_exec(std::vector<std::string>(command.begin(), command.end()));
18691901
const auto dir = out.parent_path().empty() ? fs::path(".") : out.parent_path();

‎tests/e2e/build/link.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -279,7 +279,7 @@ ldflags = ["-Wl,-rpath='$$ORIGIN/../x'"]
279279
bool published = false;
280280
for (std::size_t i = 0; i + 1 < lines.size(); ++i)
281281
if (lines[i].starts_with("rule cxx_link"))
282-
published = published || std::regex_search(lines[i + 1], std::regex{R"(-o \$\$t .*&& mv -f \$\$t \$out)"});
282+
published = published || lines[i + 1].contains("link-output --output $out --temporary -- ");
283283
expect(fatal(published)) << "A: the link rule does not publish through a temporary name";
284284
const auto exe = path_ending(p.at("target"), "/bin/app");
285285
expect(fatal(exe.has_value())) << "no bin/app";

‎tests/unit/test_ninja_backend.cpp‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,35 @@ UTest ordinary_compilers_share_the_detached_schedule_bound{"NinjaBackend/Ordinar
169169
<< "a join must not wait for a second permit";
170170
}};
171171

172+
UTest links_hold_a_permit_of_the_schedule_bound{"NinjaBackend/LinksHoldAPermitOfTheScheduleBound", [] {
173+
// A link takes a permit of the semaphore the compilers share, so links that
174+
// become ready together do not all run at once; an archive holds none, and a
175+
// graph without a cap wraps nothing.
176+
auto plan = minimal_plan();
177+
plan.toolchain.version = "16.1.0";
178+
plan.scheduleTag = "detach-codegen";
179+
plan.scheduleCompilerCap = 2;
180+
auto rule = [](const std::string& text, std::string_view name) {
181+
const auto start = text.find("rule " + std::string(name) + "\n");
182+
return start == std::string::npos ? std::string{} : text.substr(start, text.find("\n\n", start) - start);
183+
};
184+
const auto text = emit_ninja_string(plan);
185+
const std::string permit = "$mcpp __compiler-cap --sem .mcpp-sched --output $out -- ";
186+
for (const auto name : {"cxx_link", "cxx_shared"}) {
187+
const auto r = rule(text, name);
188+
expect(ne(r.find(permit), std::string::npos)) << name << "\n" << r;
189+
// The permit wraps the publication, so the artifact is in place when
190+
// the command it holds exits.
191+
if constexpr (!mcpplibs::os::is_windows)
192+
expect(ne(r.find(permit + "$mcpp link-output --output $out --temporary -- "), std::string::npos))
193+
<< name << "\n" << r;
194+
}
195+
expect(eq(rule(text, "cxx_archive").find("__compiler-cap"), std::string::npos)) << rule(text, "cxx_archive");
196+
expect(eq(text.find("--cap"), std::string::npos)) << "the graph names no number of compilers";
197+
plan.scheduleCompilerCap = 0;
198+
expect(eq(rule(emit_ninja_string(plan), "cxx_link").find("__compiler-cap"), std::string::npos));
199+
}};
200+
172201
UTest objective_c_source_uses_c_object_rule_and_c_flags{"NinjaBackend/ObjectiveCSourceUsesCObjectRuleAndCFlags", [] {
173202
auto plan = minimal_plan();
174203
plan.compileUnits.push_back({

0 commit comments

Comments
 (0)