Skip to content

Commit 32cb9fa

Browse files
a test archive is compressed as its name states, gzip passes zlib in pieces, and a program that cannot start is reported with the system's account
- tar_algorithm(name), the inverse of tar_extension, is the one rule by which unpack, --merge and the test archive read a name; `mcpp test --archive` refuses a name that states no tar archive before the build (exit 2). The archive was written gzip whatever its name and read by its name (e2e 964 E). - zlib counts in 32 bits: input and output pass in 1 GiB pieces, positions kept outside zlib, so a .tar.gz over 4 GiB is one member (unit PackCompress). - errno_of classifies invalid, no memory, directory, not supported and too large; RunResult::spawn_detail carries os's context and native value into the reports of `mcpp test` and `mcpp test --archive`, which said only "error 5". Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com>
1 parent b2102cf commit 32cb9fa

14 files changed

Lines changed: 190 additions & 60 deletions

‎CHANGELOG.md‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,9 @@
1919
(e2e 963). `[build] target` takes one target.
2020
- **Tests built once, run elsewhere.** `mcpp test --no-run --archive <file>`
2121
writes every leg's tests with `mcpp-tests.json`; `mcpp test --archive <file>`
22-
in a checkout runs what this machine reaches and builds nothing (e2e 964).
22+
in a checkout runs what this machine reaches and builds nothing. The archive
23+
is a tar compressed as its name states (`.tar.gz`, `.tgz`, `.tar.zst`,
24+
`.tar.xz`, `.tar`); another name is refused before the build (e2e 964).
2325
- **The build's contract, `${mcpp.contract}`.** Every configuration writes
2426
`mcpp-contract.json` beside its compile database: the compiler of each
2527
language with the words every unit is compiled with (target, sysroot,
@@ -99,6 +101,16 @@
99101
domain may search but not read. openkal-linux 0.18.3 opens every directory
100102
handle with `O_PATH`; mcpp is built on mcpplibs.os 0.2.3, which pins it, and
101103
openkal-platform 0.18.0 carries it to every ok image and hosted program.
104+
- **A gzip stream over 4 GiB.** zlib counts in 32 bits, and mcpp handed it the
105+
whole tree at once: a `.tar.gz` of more than 4 GiB was refused as "an input
106+
over 4 GiB", by `mcpp pack` and by a test archive alike. Input and output
107+
now pass in pieces of 1 GiB, so such a tree is one gzip member as any other.
108+
- **A program that cannot be started is reported with the system's account.**
109+
Every condition beyond not found, permission and not a program was reported
110+
as "error 5" (EIO), the system's own message dropped. The classification now
111+
covers invalid, no memory, directory, not supported and too large, and the
112+
report of `mcpp test` and `mcpp test --archive` carries what was being done
113+
and the kernel layer's value.
102114

103115
## [2026.10.29.1] - 2026-10-29
104116

‎docs/08-testing.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -213,7 +213,9 @@ each test's leg, arguments and working directory. Reading it builds nothing;
213213
the tests run in the checkout, through this machine's runner or system, and the
214214
report is `mcpp test`'s. A failed test exits 1, and so does an archive of which
215215
no test ran. Legs this machine does not reach are reported as not run
216-
([SPEC-015 §4](specs/target-sets.md)).
216+
([SPEC-015 §4](specs/target-sets.md)). The archive is a tar compressed as its
217+
name states: `.tar.gz` (or `.tgz`), `.tar.zst`, `.tar.xz` or `.tar`; another
218+
name is refused before anything is built.
217219

218220
## Tests that cannot run beside each other
219221

‎docs/specs/target-sets.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,8 @@
8585
## 4. 测试归档
8686

8787
- **写**:`mcpp test [--target <t>...] --no-run --archive <file>` 构建各腿的测试程序,与清单 `mcpp-tests.json`(每个测试的腿、
88-
参数、工作目录)一起写为一个 tar 归档。归档是引擎的格式,与平台的容器无关。
88+
参数、工作目录)一起写为一个 tar 归档。归档是引擎的格式,与平台的容器无关。压缩方式由文件名决定
89+
(`.tar.gz`/`.tgz`、`.tar.zst`、`.tar.xz`、`.tar`),读写同一规则;其他名称在构建前拒绝(退出码 2)。
8990
- **读**:`mcpp test --archive <file>` 在同一项目的检出中运行归档内本机能到达的测试,不构建;工作目录为项目检出;报告与
9091
`mcpp test` 相同。
9192
- 读取时:有测试失败则退出码 1;一个测试都没有运行则退出码 1 并写明;本机到达不了的腿报告为"未运行"。
@@ -111,7 +112,8 @@
111112
6. 集合的识别、补全、展开、去重与顺序,`planned` 的行不在内,候选选到不同平台被拒绝(unit `TargetSet`)。
112113
7. 腿的规划串行、构建一起、作业分摊与第一条失败的腿(unit `BuildLegs`)。
113114
8. `--no-run --archive` 写出的归档含两条腿与清单;在没有 `target/` 的检出中 `--archive` 运行本机的腿、不构建;失败的
114-
测试使退出码为 1;没有一条腿能到达的归档不运行任何测试且退出码为 1(e2e 964)。
115+
测试使退出码为 1;没有一条腿能到达的归档不运行任何测试且退出码为 1;`.tar.zst` 写为 zstd 帧并同样运行,不表明 tar
116+
归档的名称在构建前以退出码 2 拒绝(e2e 964);文件名与压缩方式互逆(unit `PackCompress`)。
115117

116118
## 变更记录
117119

‎docs/zh/08-testing.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,8 @@ mcpp test --archive tests.tar.gz # 在检出中运
186186
归档含各腿的测试程序与 `mcpp-tests.json`(每个测试的腿、参数与工作目录)。读取
187187
时不构建;测试在检出中经本机的运行器或系统运行,报告与 `mcpp test` 相同。有测试
188188
失败时退出码为 1,一个测试都没运行的归档也是 1。本机到达不了的腿报告为未运行
189-
([SPEC-015 §4](../specs/target-sets.md))。
189+
([SPEC-015 §4](../specs/target-sets.md))。归档是 tar,压缩方式由文件名决定:`.tar.gz`(或
190+
`.tgz`)、`.tar.zst`、`.tar.xz` 或 `.tar`;其他名称在构建前即被拒绝。
190191

191192
## 不能彼此并排运行的测试
192193

‎src/build/execute.cppm‎

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -128,7 +128,8 @@ std::string test_dest(std::string_view triple, std::string_view test, std::strin
128128
std::string program_dest(std::string_view triple, std::string_view file);
129129

130130
// Copies `copies` into a fresh tree under `stage`, writes the manifest beside
131-
// them and the tree as `archive` (gzip-compressed tar).
131+
// them and the tree as `archive`: a tar compressed as its name states
132+
// (`.tar.gz`/`.tgz`, `.tar.zst`, `.tar.xz`, `.tar`); another name is refused.
132133
std::expected<void, std::string> write(const Manifest& m, const std::vector<Copy>& copies,
133134
const std::filesystem::path& stage,
134135
const std::filesystem::path& archive);
@@ -215,8 +216,13 @@ std::expected<void, std::string> write(const Manifest& m, const std::vector<Copy
215216
const fs::path& stage, const fs::path& archive) {
216217
std::error_code ec;
217218
fs::remove_all(stage, ec);
219+
// The compression is the one the name states, as reading it takes it.
218220
auto stem = archive.filename().string();
219-
for (std::string_view ext : {".tar.gz", ".tgz", ".tar"})
221+
const auto algorithm = mcpp::pack::compress::tar_algorithm(stem);
222+
if (!algorithm)
223+
return std::unexpected(std::format("'{}' names no tar archive (.tar, .tar.gz, .tgz, .tar.zst, "
224+
".tar.xz)", stem));
225+
for (std::string_view ext : {".tgz", ".tar.gz", ".tar.zst", ".tar.xz", ".tar"})
220226
if (stem.ends_with(ext)) { stem.resize(stem.size() - ext.size()); break; }
221227
if (stem.empty()) stem = "mcpp-tests";
222228
const auto root = stage / stem;
@@ -237,9 +243,9 @@ std::expected<void, std::string> write(const Manifest& m, const std::vector<Copy
237243
if (!out) return std::unexpected(std::format("cannot write {}", kManifestName));
238244
}
239245
if (archive.has_parent_path()) fs::create_directories(archive.parent_path(), ec);
240-
auto choice = mcpp::pack::compress::parse("gzip");
241-
if (!choice) return std::unexpected(choice.error());
242-
return mcpp::pack::distribution::write_tar(root, archive, *choice);
246+
mcpp::pack::compress::Choice choice;
247+
choice.algorithm = *algorithm;
248+
return mcpp::pack::distribution::write_tar(root, archive, choice);
243249
}
244250

245251
std::expected<Opened, std::string> open(const fs::path& archive, const fs::path& into) {
@@ -4291,12 +4297,14 @@ static int test_run_member(TestBuild& tb, const TestOptions& testOpts,
42914297
bool timedOut = false;
42924298
int exitCode = 0;
42934299
int spawnErr = 0;
4300+
std::string spawnDetail;
42944301
std::string runOutput;
42954302
if (capture) {
42964303
auto rr = mcpp::run::capture_exec_deadline(
42974304
r.argv, r.env, deadline, &timedOut, {}, &spawnErr);
42984305
exitCode = rr.exit_code;
42994306
runOutput = std::move(rr.output);
4307+
spawnDetail = std::move(rr.spawn_detail);
43004308
} else {
43014309
mcpp::ui::status("Running", std::format("bin/{}", r.name));
43024310
exitCode = mcpp::run::run_exec_deadline(
@@ -4315,7 +4323,7 @@ static int test_run_member(TestBuild& tb, const TestOptions& testOpts,
43154323
using namespace mcpp::build::runner_lookup;
43164324
std::string reason;
43174325
if (!runnerTmpl.empty())
4318-
reason = spawn_failed_message(r.argv.front(), spawnErr);
4326+
reason = spawn_failed_message(r.argv.front(), spawnErr, spawnDetail);
43194327
else if (classify(spawnErr) == SpawnClass::Unloadable)
43204328
reason = std::format(
43214329
"this host cannot execute {} artifacts: {} (error {}); "
@@ -4324,7 +4332,7 @@ static int test_run_member(TestBuild& tb, const TestOptions& testOpts,
43244332
runnerChoice.tripleKey, errno_text(spawnErr), spawnErr,
43254333
runnerChoice.tripleKey);
43264334
else
4327-
reason = spawn_failed_message(r.argv.front(), spawnErr);
4335+
reason = spawn_failed_message(r.argv.front(), spawnErr, spawnDetail);
43284336
if (!hostCannotRun.exchange(true)) {
43294337
hostCannotRunReason = reason;
43304338
if (!json) mcpp::ui::warning(reason);
@@ -4765,7 +4773,8 @@ export int run_test_archive(std::span<const std::string> passthrough, BuildOverr
47654773
if (spawnErr != 0) {
47664774
++notRun;
47674775
reasons.push_back(std::format("{}: {}", leg.triple,
4768-
mcpp::build::runner_lookup::spawn_failed_message(argv.front(), spawnErr)));
4776+
mcpp::build::runner_lookup::spawn_failed_message(argv.front(), spawnErr,
4777+
rr.spawn_detail)));
47694778
} else if (ok) {
47704779
++passed;
47714780
} else {

‎src/build/runner_lookup.cppm‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -144,9 +144,12 @@ inline std::string not_found_message(std::string_view triple, std::string_view a
144144
argv0, triple, dirs);
145145
}
146146

147-
inline std::string spawn_failed_message(std::string_view program, int e) {
148-
return std::format("'{}' could not be started: {} (error {})",
149-
program, errno_text(e), e);
147+
// `detail` is the system's own account (`RunResult::spawn_detail`), which
148+
// names what the classification `e` does not.
149+
inline std::string spawn_failed_message(std::string_view program, int e,
150+
std::string_view detail = {}) {
151+
return std::format("'{}' could not be started: {} (error {}){}{}",
152+
program, errno_text(e), e, detail.empty() ? "" : "; ", detail);
150153
}
151154

152155
// The hosted sibling of mcpp::freestanding::no_runner_message. It reports what

‎src/cli/cmd_build.cppm‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ import mcpp.home.files;
4848
import mcpp.ui.terminal;
4949
import mcpp.project;
5050
import mcpp.manifest;
51+
import mcpp.pack.compress; // the compression a test archive's name states
5152
import mcpp.toolchain.fingerprint;
5253
import mcpp.ui;
5354
import mcpp.wire;
@@ -1226,6 +1227,14 @@ export int cmd_test(const mcpplibs::cmdline::ParsedArgs& parsed,
12261227
return 2;
12271228
}
12281229
}
1230+
// The archive is a tar compressed as its name states, written and read so:
1231+
// a name that states none is refused before anything is built.
1232+
if (!to.archive.empty()
1233+
&& !mcpp::pack::compress::tar_algorithm(to.archive.filename().string())) {
1234+
mcpp::ui::error(std::format("--archive {}: the name states no tar archive (.tar, .tar.gz, .tgz, "
1235+
".tar.zst, .tar.xz)", to.archive.string()));
1236+
return 2;
1237+
}
12291238
if (runArchive) {
12301239
if (!testLegs.empty() || !ov.target_triple.empty()) {
12311240
mcpp::ui::error("mcpp test --archive runs the legs the archive holds; --target "

‎src/pack/compress.cppm‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,11 @@ std::string_view name(Algorithm a);
3939
// `.tar`.
4040
std::string_view tar_extension(Algorithm a);
4141

42+
// The compression a tar archive's file name states, the inverse of
43+
// `tar_extension` (`.tgz` is `.tar.gz`); none for a name that is no tar
44+
// archive's. Writers and readers of an archive by name take it from here.
45+
std::optional<Algorithm> tar_algorithm(std::string_view filename);
46+
4247
// The whole of `in`, compressed to `out` under `c` (a gzip member, a zstd
4348
// frame, an xz stream; a copy for None).
4449
std::expected<void, std::string> file(const std::filesystem::path& in,
@@ -98,6 +103,13 @@ std::string_view tar_extension(Algorithm a) {
98103
return ".tar.gz";
99104
}
100105

106+
std::optional<Algorithm> tar_algorithm(std::string_view filename) {
107+
if (filename.ends_with(".tgz")) return Algorithm::Gzip;
108+
for (auto a : {Algorithm::Gzip, Algorithm::Zstd, Algorithm::Xz, Algorithm::None})
109+
if (filename.ends_with(tar_extension(a))) return a;
110+
return std::nullopt;
111+
}
112+
101113
std::expected<Choice, std::string> parse(std::string_view text) {
102114
Choice c;
103115
c.stated = true;

‎src/pack/compress_codecs.cpp‎

Lines changed: 43 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,11 @@ namespace mcpp::pack::compress::codec {
1414

1515
namespace {
1616

17+
// zlib's counters are `uInt` (32 bits) and `uLong` (32 bits on Windows): the
18+
// input is handed over and the output taken in pieces of at most `kZlibPiece`,
19+
// positions are kept here, so a tree over 4 GiB is one gzip member as any other.
20+
constexpr std::size_t kZlibPiece = std::size_t{1} << 30;
21+
1722
// zlib's deflate with `windowBits`: 15 + 16 is a gzip member, -15 is raw
1823
// DEFLATE. A gzip member's header is stated rather than left to zlib, whose own
1924
// header names the system zlib was built for (3 on Unix, 10 or 11 on Windows):
@@ -32,30 +37,39 @@ std::expected<std::string, std::string> zlib_deflate(std::string_view in, int le
3237
return std::unexpected(std::string("zlib: deflateSetHeader failed"));
3338
}
3439
std::string out;
35-
out.resize(deflateBound(&s, static_cast<uLong>(in.size())) + 64);
36-
s.next_in = reinterpret_cast<Bytef*>(const_cast<char*>(in.data()));
37-
s.avail_in = static_cast<uInt>(in.size());
38-
s.next_out = reinterpret_cast<Bytef*>(out.data());
39-
s.avail_out = static_cast<uInt>(out.size());
40-
const int rc = ::deflate(&s, Z_FINISH);
41-
const auto written = s.total_out;
40+
out.resize(std::min(in.size(), kZlibPiece) + in.size() / 1000 + 1024);
41+
std::size_t fed = 0, written = 0;
42+
int rc = Z_OK;
43+
while (rc != Z_STREAM_END) {
44+
if (s.avail_in == 0 && fed < in.size()) {
45+
const auto piece = std::min(in.size() - fed, kZlibPiece);
46+
s.next_in = reinterpret_cast<Bytef*>(const_cast<char*>(in.data() + fed));
47+
s.avail_in = static_cast<uInt>(piece);
48+
fed += piece;
49+
}
50+
if (written == out.size()) out.resize(out.size() + std::max(out.size() / 2, std::size_t{1024}));
51+
const auto room = std::min(out.size() - written, kZlibPiece);
52+
s.next_out = reinterpret_cast<Bytef*>(out.data() + written);
53+
s.avail_out = static_cast<uInt>(room);
54+
rc = ::deflate(&s, fed == in.size() ? Z_FINISH : Z_NO_FLUSH);
55+
written += room - s.avail_out;
56+
if (rc != Z_OK && rc != Z_STREAM_END && rc != Z_BUF_ERROR) {
57+
deflateEnd(&s);
58+
return std::unexpected(std::format("zlib: deflate returned {}", rc));
59+
}
60+
}
4261
deflateEnd(&s);
43-
if (rc != Z_STREAM_END) return std::unexpected(std::format("zlib: deflate returned {}", rc));
4462
out.resize(written);
4563
return out;
4664
}
4765

4866
} // namespace
4967

5068
std::expected<std::string, std::string> gzip(std::string_view in, int level) {
51-
if (in.size() > std::numeric_limits<uInt>::max())
52-
return std::unexpected(std::string("gzip: an input over 4 GiB"));
5369
return zlib_deflate(in, level, 15 + 16);
5470
}
5571

5672
std::expected<std::string, std::string> deflate(std::string_view in, int level) {
57-
if (in.size() > std::numeric_limits<uInt>::max())
58-
return std::unexpected(std::string("deflate: an input over 4 GiB"));
5973
return zlib_deflate(in, level, -15);
6074
}
6175

@@ -104,11 +118,10 @@ namespace {
104118

105119
// zlib's inflate until the stream ends: a gzip member (windowBits 15 + 32
106120
// detects the header), its output growing, or raw DEFLATE (-15) of a stated
107-
// `size`, its output never more than that.
121+
// `size`, its output never more than that. Input and output pass in pieces
122+
// (`kZlibPiece`), as for deflate.
108123
std::expected<std::string, std::string> zlib_inflate(std::string_view in, int windowBits,
109124
std::optional<std::size_t> size) {
110-
if (in.size() > std::numeric_limits<uInt>::max())
111-
return std::unexpected(std::string("zlib: an input over 4 GiB"));
112125
// DEFLATE expands at most 1032 times (a run of one byte): a stated size
113126
// beyond that is a lie, refused before anything is allocated, and a stream
114127
// that grows past it is refused as it grows.
@@ -122,11 +135,18 @@ std::expected<std::string, std::string> zlib_inflate(std::string_view in, int wi
122135
// One byte past a stated size: a stream that fills it holds more than it
123136
// said.
124137
out.resize(size ? *size + 1 : in.size() * 4 + 1024);
125-
s.next_in = reinterpret_cast<Bytef*>(const_cast<char*>(in.data()));
126-
s.avail_in = static_cast<uInt>(in.size());
138+
std::size_t fed = 0, written = 0;
127139
int rc = Z_OK;
128-
while (rc == Z_OK) {
129-
if (s.total_out >= out.size()) {
140+
while (rc == Z_OK || rc == Z_BUF_ERROR) {
141+
if (s.avail_in == 0 && fed < in.size()) {
142+
const auto piece = std::min(in.size() - fed, kZlibPiece);
143+
s.next_in = reinterpret_cast<Bytef*>(const_cast<char*>(in.data() + fed));
144+
s.avail_in = static_cast<uInt>(piece);
145+
fed += piece;
146+
} else if (rc == Z_BUF_ERROR && s.avail_in == 0 && written < out.size()) {
147+
break; // the input ended before the stream did
148+
}
149+
if (written >= out.size()) {
130150
if (size) {
131151
inflateEnd(&s);
132152
return std::unexpected(std::format("zlib: the stream holds more than its {} bytes",
@@ -138,12 +158,12 @@ std::expected<std::string, std::string> zlib_inflate(std::string_view in, int wi
138158
}
139159
out.resize(std::min(out.size() * 2, most + 1));
140160
}
141-
s.next_out = reinterpret_cast<Bytef*>(out.data() + s.total_out);
142-
s.avail_out = static_cast<uInt>(std::min<std::size_t>(out.size() - s.total_out,
143-
std::numeric_limits<uInt>::max()));
161+
const auto room = std::min(out.size() - written, kZlibPiece);
162+
s.next_out = reinterpret_cast<Bytef*>(out.data() + written);
163+
s.avail_out = static_cast<uInt>(room);
144164
rc = ::inflate(&s, Z_NO_FLUSH);
165+
written += room - s.avail_out;
145166
}
146-
const auto written = s.total_out;
147167
inflateEnd(&s);
148168
if (rc != Z_STREAM_END) return std::unexpected(std::format("zlib: inflate returned {}", rc));
149169
out.resize(written);

0 commit comments

Comments
 (0)