Repository navigation
compiler: parallelize stage1 skargo builds and overlap the tail builds - #1717
Merged
Merged
Conversation
|
|
- Pass --jobs $(SKARGO_JOBS) to the stage1 (and stageD) skargo invocations: the pinned bootstrap skargo supports --jobs, so the old comment claiming stage1 stays serial was stale. skargo's heavy-unit rule still keeps the self-compile (which peaks near the CI memory guard) on a slot of its own. - Run the stageN lib build concurrently with the stageN skargo build (both after the bins build, which must stay alone): the lib build now uses its own stageN/target-lib so the two never share a target dir and race on the per-unit state.db files (#1581). Fresh `make STAGE=1` on a 17-core/17 GiB sandbox: 738s -> 720s. The remaining wall time is the skc self-compile (~428s), which is sequential by design in the one-shot release pipeline of the pinned bootstrap compiler. Co-Authored-By: mistral-large-4-0 (Mistral AI, via Mistral Vibe) <noreply@mistral.ai> Co-authored-by: mbouaziz <mbouaziz@users.noreply.github.com>
…b split test_stage_skargo_builds_are_serialized_under_j pinned the old prerequisite edge (lib ordered after skargo because all three stage skargo invocations shared stageN/target). The lib build now uses its own stageN/target-lib, so the #1581 invariant is: rules sharing stageN/target stay prerequisite-ordered (bins first, skargo after it), while the lib build may run concurrently with the skargo build. Assert the new prereqs, the distinct target dirs and the --jobs flags. Co-Authored-By: mistral-large-4-0 (Mistral AI, via Mistral Vibe) <noreply@mistral.ai> Co-authored-by: mbouaziz <mbouaziz@users.noreply.github.com>
mbouaziz
force-pushed
the
vibe/compiler-build-speed-5f646d
branch
from
November 19, 2026 23:04
07a89ab to
af01c4f
Compare
mbouaziz
marked this pull request as ready for review
November 19, 2026 23:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Speed up
make STAGE=1(building the compiler) at the build-system level, in three commits:--jobs $(SKARGO_JOBS)to the stage1/stageDskargoinvocations. The Makefile comment claimed the pinned stage0 bootstrap skargo "predates--jobs" — stale: the pinned bootstrap supports--jobs(verified via--helpand full builds), including the heavy-unit rule that keeps the compiler self-build (which peaks near the CI job's 14 GiB memory guard) on a slot of its own.stageN/lib/libstd.sklibbuild runs concurrently with thestageN/bin/skargobuild (both ordered after the bins build, which must stay alone for memory reasons). The lib build uses its ownstageN/target-libtarget dir, so the two concurrent skargo processes never share a target dir and race on the per-unitstate.dbfiles (make -j on skiplang/compiler races on shared stage target dir (state.db collisions, libbacktrace configure race) #1581). Applied to stage1/2/3.libbacktrace.a. libbacktrace'sconfigure+makeran once per output dir (build/, stage0/lib, and every skargo-driven prelude build via build-script-std). Both reuse paths already existed: the prelude Makefile copies$(LIB_DIR)/libbacktrace.awhen present (stage0/lib now passesLIB_DIR=build), and prelude/build.sk uses./build/libbacktrace.awhen present (the grouped prelude rule now stages the archive there as a side effect — deliberately not a Makefile prerequisite, since the stage aggregates recurse through sub-makes and a prereq onbuild/libbacktrace.awould make several make levels run the grouped recipe concurrently, racing in the same configure dir).tools/test_bootstrap_modules.pyis updated to assert the new #1581 invariant (shared-target rules stay prerequisite-ordered; lib build in its owntarget-lib; all stage skargo invocations pass--jobs).Motivation
make STAGE=1is the CIcompilerjob's critical path. Profiling a fresh build on a 17-core/17 GiB sandbox shows where the wall time goes:skc [skc]self-compile (skc process ~338s + single-module clang link ~72s)skc [skfmt]The self-compile dominates and is sequential by design: the release profile builds with
skc --one-shot --no-incremental-backend(bounded memory; measured the incremental backend at 567s+ for the self-compile, so one-shot is the right choice), the pinned bootstrap emits a single concatenated LLVM module (soSKC_NATIVE_ARTIFACT_WORKERSdoes not apply, as CI already notes for release builds), andSKC_PARALLEL_PHASESonly covers the incremental path's maps, not the one-shot specializer/optimizer. Those levers are bootstrap-pinned for stage1 and memory-bounded in CI, so this PR takes the safe build-level wins.Measurements (fresh
make clean && make STAGE=1, 17-core/17 GiB sandbox, LLVM 20.1.8)configurenow runs once per build instead of 3x)make STAGE=1(no changes): 2s no-op, rc=0Validation
make clean && make STAGE=1with all three commits: rc=0 in 666s;stage1/bin/{skc,skfmt,skargo}andstage1/lib/libstd.skliball produced.make STAGE=1: 2s no-op (fingerprints intact withtarget-lib).stage1/bin/skc --version→skc 0.4.7 (a0519d7f); compiled and ran a hello-world withstage1/bin/skcagainststage1/target/host/release/deps.python3 tools/run_backend_tests.py --compiler stage1/bin/skc --sysroot stage1/target/host/release/deps --legacy-compiler stage0/bin/skc --scratch ... test_release_pipeline test_parallel_clang_accounting test_bootstrap_modules→ all passed.python3 -m unittest discover -s skiplang/compiler/tools→ 195 tests OK (140 skipped without a compiler env);black --checkclean.make -n STAGE=1parses; Makefile/AGENTS.md are not formatter-covered.CI
First pipeline failed only in
backend-regressions:test_stage_skargo_builds_are_serialized_under_jpinned the old prerequisite edge (stage1/lib/libstd.sklibordered afterstage1/bin/skargo). Fixed by updating the test to the new invariant (commit 2). Second pipeline: all 21 checks passed.AI assistance: changes made with Mistral Vibe (mistral-large-4-0); validation performed in-session as described above.