Repository navigation
build: prebuilt engine binaries, CI builds Linux/macOS/Windows on every merge and commits them under engine/binaries/ - #455
Closed
swapnilpaliwal-sd wants to merge 12 commits into
Closed
swapnilpaliwal-sd wants to merge 12 commits into
swapnilpaliwal-sd wants to merge 12 commits into
Conversation
swapnilpaliwal-sd
added
build
swapnilpaliwal-sd
changed the title
swapnilpaliwal-sd
force-pushed
the
engine-prebuilt
branch
from
September 13, 2026 22:29
7a7a2b1 to
6b0979c
Compare
…rge, the script fetches them (#454) Running the engine required a Soufflé installation and a C++ toolchain: run-souffle.sh generated C++ from the rules and compiled it on first use, and failed outright without both. Neither is needed at run time. The compiled engine is one self-contained executable (linked against nothing but the C++ runtime) that is already project-independent, and `souffle -g`'s output is portable C++ that needs only Soufflé's headers to compile. The program is now a pure function of the repository: includes are relative to src/ (souffle -I src) and the .input list is derived from the staging maps rather than from a listing of the staged facts, so the same text comes out of every checkout and of CI with no client IR. Its sha256 together with every included file and the PINNED Soufflé version (src/pipeline/engine.conf — a machine without souffle cannot ask for one) is the engine id: path-independent, and different for any change to a rule, a map, the manifest or the pin. Two new modes expose it: --print-engine-id and --emit-program. .github/workflows/engine-binaries.yml, on every merge to main touching the rules: generate the C++ once on Ubuntu with the pinned .deb, compile it on ubuntu-24.04, ubuntu-24.04-arm, macos-14 (universal) and windows-2025 (MSVC) with the same flags the script uses locally and portable targets instead of -march=native, smoke-run each binary on empty inputs, and publish a release tagged engine-<lang>-<id> holding the binaries, sha256sum.txt and the generated .cpp. Languages whose id already has a release are skipped. run-souffle.sh keeps its compile branch when souffle is on PATH (warning if the version is not the pinned one). Otherwise it resolves the platform, fetches the asset for its id through gh or curl+GH_TOKEN (the repository is private), verifies the sha256, and caches the binary under its id. No release for the id fails with the two ways out named — never a silently stale binary. A missing basic tool is refused up front, because a missing grep inside a process substitution had silently produced an empty relation list and a wrong id. Guards, both without souffle or network: test/tools/engine-id-test.sh (same id from two paths; changed by a rule, a map, the manifest, the pin; one language's change does not move another's) and test/tools/engine-fetch-test.sh (a fake gh serves a fake engine: fetched, verified, cached, run through to the bundle; tampered checksum refused with nothing cached; absent release explained). Both are preflights of the Java suite.
…e header tree under that name
…ONG) instead of a vcpkg getopt
…d changed, release each language independently
…>/<platform>/ — a checkout carries its own engines The publish job places all four binaries of each rebuilt language under engine/binaries/<lang>/<platform>/, records the rule-set id in ENGINE_ID, and pushes one commit to main (rebased and retried if main moved during the build); a language missing a platform is left unchanged so the next merge retries it. The plan step now compares each language's id against the committed ENGINE_ID instead of a release. run-souffle.sh resolves the engine in order: the committed binary when its ENGINE_ID equals the checkout's id, a local compile when souffle is on PATH, then the release. The release stays as a secondary copy. engine-fetch-test.sh covers the committed path: used when the id matches, reported and skipped when it does not.
…drop the universal binary Hosted macOS minutes bill at 10x and a PR run of the matrix cost ~300 billable minutes; the four-platform compile has been validated, so builds now happen only on merge (and by workflow_dispatch). The macOS job runs on a self-hosted Apple Silicon runner and produces darwin-arm64 only; an Intel row can be added when a user needs it. The platform string on macOS is now darwin-<arch> like the others.
swapnilpaliwal-sd
force-pushed
the
engine-prebuilt
branch
from
September 14, 2026 01:23
eda78f4 to
4902b81
Compare
Merged
7 of 8 tasks
…s a dispatch-only workflow (dry run by default) Running the engine needed Soufflé and a C++ toolchain. The compiled engine is one self-contained executable per language and platform, so CI builds them (build-engines.yml: generate the portable C++ once on Ubuntu with the pinned Soufflé, compile on linux-x64, linux-arm64, win32-x64 and a self-hosted darwin-arm64, smoke-run each) and publish-npm.yml assembles one package per platform — @axiomcode/engine-<os>-<cpu>, every language's engine under <lang>/ with its ENGINE_ID — and publishes them. The workflow runs only on demand (version, dry_run=true by default, macos) or on a v* tag; a dry run packs and prints without uploading. This package lists the four as optionalDependencies, so `npm install` fetches exactly the one npm's os/cpu filter matches; nothing is committed to git. run-souffle.sh resolves the engine in order: the installed package when its ENGINE_ID equals the checkout's rule hash (edited rules never run a stale binary), else a local compile when souffle is present, else an error naming both ways out. The program the id hashes is a pure function of the repository (relative includes, inputs derived from the maps), exposed as --print-engine-id and --emit-program; the pinned Soufflé version lives in graph/pipeline/engine.conf. Guards, without souffle or network, as Java-suite preflights: engine-id-test.sh (same id from two paths; changed by a rule, a map, the manifest, the pin) and engine-package-test.sh (a hand-made engine package: used when its id matches, refused with both ways out when not, absence explained).
swapnilpaliwal-sd
force-pushed
the
main
branch
from
September 14, 2026 07:41
f1827c7 to
febf8b4
Compare
…e — LANGUAGES gains javascript in the generate, Linux, Windows and smoke steps, and in the engine-id preflight Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he committed-binaries workflow, the release fetch test and the binaries .gitattributes are superseded by build-engines.yml, publish-npm.yml and engine-package-test.sh
3 tasks done
Contributor
Author
|
Superseded: binaries are not committed to the repository; the engines ship on npm as @axiomcode/engine--, and that build/publish workflow now lives in #418. |
swapnilpaliwal-sd
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #454. Stacked on #453 (both touch
run-souffle.sh); base will be retargeted tomainonce #453 merges.What changes
Running the engine no longer needs Soufflé or a C++ compiler. The compiled engine is one self-contained executable (it links against nothing but the C++ runtime) and is project-independent; CI builds it for every platform on every merge and commits it to the repository, so a checkout carries its own engines.
The program is a pure function of the repository. Includes are relative to
src/(souffle -I src), and the.inputlist is derived from the staging maps rather than from a listing of the staged facts, so the same text comes out of every checkout and of CI with no client IR. Its sha256 with every included file and the pinned Soufflé version (src/pipeline/engine.conf) is the engine id: path-independent, changed by any rule/map/manifest/pin edit. Exposed asrun-souffle.sh --language L --print-engine-idand--emit-program FILE.CI (
.github/workflows/engine-binaries.yml) on every merge tomain:generate, per language, compare the id of the rules withengine/binaries/<lang>/ENGINE_ID; unchanged languages are skipped in seconds. For the rest: install the pinned Soufflé.deb,souffle -gonce, capture the headers.build,ubuntu-24.04,ubuntu-24.04-arm,macos-14(universal),windows-2025(MSVC with Soufflé's bundledgetopt_long), the same flags as the local compile, portable targets instead of-march=native, and a smoke run on empty inputs.publish, one commit tomainwith the rebuilt languages' four binaries each, plusENGINE_ID; rebased and retried ifmainmoved during the build. A language missing a platform is left unchanged so the next merge retries it, without blocking the others. The same files also go to a release taggedengine-<lang>-<id>as a secondary copy (with the generated.cpp).So a Python-only merge rebuilds Python on all four targets and commits those; Java and TypeScript are untouched.
The script resolves the engine in this order: (1) the committed binary, only if
ENGINE_IDequals the id of the checkout's rules, so edited rules can never silently run a stale binary; (2) a local compile whensouffleis on PATH; (3) the release for the id, fetched viagh/GH_TOKENand sha256-verified. None of the three → an error naming the ways out. Basic tools are checked up front (a missinggrepinside a process substitution had silently produced an empty relation list and a wrong id).A PR touching the build runs
generate+buildwithout publishing; the checks on this PR are that four-platform compile.Guards (no Soufflé, no network; Java-suite preflights):
engine-id-test.sh, same id from two paths; changed by a rule, a map, the manifest, the pin; one language's edit doesn't move another's.engine-fetch-test.sh, on a copy of the tree: a committed engine with a matching id is used with no download; one with a different id is reported and skipped; the release path fetches, verifies, caches and runs through to the bundle; a tampered checksum is refused with nothing cached; an absent release is explained.Verified locally: Java, TypeScript and Python cases solve through the restructured driver with goldens unchanged.
Cost, stated
Plain git, as decided: ~33 MB per language per rebuild is added to history permanently (~100 MB for all three). If that becomes a problem the layout is unchanged under Git LFS, or the binaries can move to a repository of their own with
AXIOM_ENGINE_REPOpointing at it.After this
Requirements are Node ≥ 22.5 and bash + awk (Git Bash on Windows). Porting the staging driver to TypeScript, one native command with no shell, is the follow-up.