Repository navigation
Prebuilt engine binaries: build in CI for Linux/macOS/Windows on merge, fetch from the script, no Soufflé or compiler needed to run #454
Description
Activity
- added 3 commits that reference this issue
on Sep 13, 2026 - added a commit that references this issue
on Sep 18, 2026 Reopened. The work was merged as #478 and reverted in #903; it is back as an open pull request in #904 so it can be reviewed before it ships. Nothing is lost, and #904 is identical to what was on main.
Two fixes that landed on top of it are kept in #904 and stayed on main through the revert:
- the collation pin from URGENT: the engine id and cache key depend on the shell locale, so a published engine is refused on any machine with a UTF-8 collation #895, without which the engine id follows the user's locale and a published engine is refused for java and python on any machine with a UTF-8 collation
- bin, files and dependencies from npm install of the published package cannot run: parser dist not shipped, no bin entry, bin root walk fails through the node_modules/.bin symlink, no runtime dependencies #886, without which the published package installs and then cannot run
Measured state of the mechanism, engines built for five platforms and run from a real npm install on machines with no souffle and no compiler:
platform result darwin-arm64 four real projects, all pass darwin-x64 cross compiled on Apple Silicon, runs under Rosetta linux-x64 four real projects, all pass win32-x64 four real projects, all pass once #895 is in linux-arm64 install fails, see #901 Every platform's ENGINE_ID matched the generate step, and the smoke counts agreed across three operating systems and two architectures: java 22, typescript 45, python 85, javascript 34 relations.
Remaining before this can close:
- linux-arm64: npm install fails, no arm64 prebuilds in the 0.21.x tree-sitter generation and a nested duplicate that cannot build #901, linux-arm64 cannot install. No dependency in the 0.21.x tree-sitter generation ships an arm64 prebuild, and a nested duplicate cannot build even with a toolchain. Independent of the engine.
- The CI workflows are out of the tree (chore: take the CI workflows out of the tree for now #898), so nothing builds the binaries automatically yet.
- The npm org and token do not exist, and the platform packages have to be published before the main one or the install silently produces no engine.
- changed the title
[-]Prebuilt engine binaries: build in CI for Linux/macOS/Windows on merge, fetch from the script — no Soufflé or compiler needed to run[/-][+]Prebuilt engine binaries: build in CI for Linux/macOS/Windows on merge, fetch from the script, no Soufflé or compiler needed to run[/+]on Sep 18, 2026
Problem
Running the engine requires a Soufflé installation plus a C++ toolchain on the user's machine:
run-souffle.shgenerates C++ from the rules (souffle -g) and compiles it (c++ -O3 -march=native) on first use, ~70–120 s per language, and fails outright without both. Soufflé itself ships packages only for a handful of Linux distributions; on macOS it is a Homebrew build and on Windows a from-source CMake/vcpkg build. For a consumer that just wants a call graph, that is most of the setup.None of it is necessary at run time. The compiled engine is a single self-contained executable (measured: ~7.6 MB, linked against nothing but the system C++ runtime) that takes
-F <facts> -D <out>, it is already project-independent (relative.input/.output, every knob is a fact). Soufflé is a build-time dependency only, andsouffle -g's output is portable C++ that needs just Soufflé's headers and any C++17 compiler.Proposal
Build the engine binaries in CI on every merge to
mainand have the script fetch them, so a user needs Node and a shell, no Soufflé, no compiler.1. A canonical engine id. Today's cache key hashes the generated program text, which embeds absolute include paths, so it differs per checkout. Replace it with a content-only id per language: sha256 of the program with paths relativised to
src/, every included.dlin include order, the staging maps, and the pinned Soufflé version (a constant in the repo, since a machine without Soufflé must compute the same id CI did). Path-independent, changes iff the rules change.2. CI on merge (
.github/workflows/engine-binaries.yml):generateon Ubuntu: install the pinned Soufflé.deb, compute each language's id, skip languages whose release already exists, runsouffle -g→ the portable.cpp, plus the header tree.buildmatrix:ubuntu-24.04(x86_64),ubuntu-24.04-arm(arm64),macos-14(universal-arch arm64 -arch x86_64),windows-2025(MSVC,/std:c++17 /O2 /bigobj, the same headers Soufflé's own Windows CI compiles). Portable target flags, not-march=native.release: tagengine-<lang>-<id>with assetsaxiom-engine-<lang>-<os>-<arch>[.exe],sha256sum.txt, and the generated<lang>.cpp(a third path for anyone with only a compiler).3. The script.
run-souffle.shkeeps its compile branch whensouffleis on PATH; otherwise it resolves the platform (uname→ linux/darwin/windows × x86_64/arm64, Git Bash on Windows), looks in the local cache, downloads the asset for this id (viagh, orcurlwith a token, the repository is private), verifies the sha256, and runs it. Rules edited locally with no Soufflé installed → an explicit error naming the two ways out, never a silent stale binary.4. Guard. A preflight asserts the id is the same from two copies of the tree at different paths and differs after a one-character rule change.
Requirements after this: Node ≥ 22.5 (bundle stage) and bash + awk (Git Bash on Windows). Porting the staging driver from bash to TypeScript, one native
axiom-graphcommand everywhere, is the follow-up, not this issue.