build: engines ship on npm as @axiomcode/engine-<os>-<cpu>; publish is a dispatch-only workflow (dry run by default)

[swapnilpaliwal-sd]

Fixes #454. Supersedes #455 (binaries committed to the repository), engines ship on npm instead; nothing is committed.

How it works

• build-engines.yml (reusable + dispatchable): souffle -g once on Ubuntu with the pinned Soufflé → portable C++ per language → compiled on linux-x64, linux-arm64, win32-x64 (MSVC) and a self-hosted darwin-arm64, smoke-run on empty inputs, uploaded as engines-<platform> artifacts. Platform names are npm's (process.platform-process.arch).
• publish-npm.yml, runs only on demand (version, dry_run = true by default, macos) or on a v* tag. Assembles @axiomcode/engine-<os>-<cpu> per platform (every language's engine under <lang>/ with its ENGINE_ID, license, readme, os/cpu fields) and runs npm publish --access public, with --dry-run it packs and prints without uploading, so the pipeline can be exercised before any token exists.
• package.json lists the four as optionalDependencies: npm installs exactly the one whose os/cpu match the machine, skips the rest. Verified npm install succeeds today with them unpublished (optional).
• run-souffle.sh resolves the engine: (1) node_modules/@axiomcode/engine-<platform>/<lang>/ if its ENGINE_ID equals the checkout's rule hash; (2) local compile when souffle is present; else an error naming both ways out. The program the id hashes is a pure function of the repository (--print-engine-id, --emit-program); the Soufflé pin is graph/pipeline/engine.conf.

To publish (one-time setup, then one the CLI-argument library)

1. npmjs: create the axiomcode org; a granular token with publish on @axiomcode/* → repo secret NPM_TOKEN.
2. gh workflow run publish-npm.yml -f version=0.1.0 -f dry_run=true -f macos=false → inspect; then dry_run=false (or git tag v0.1.0 && git push --tags).
3. After a rule change that should ship: publish a new version and bump the four pins in package.json.

Verified

• Java 39/39, TypeScript 53/53, Python 15/15 on this branch (from a fresh npm install).
• Preflights (Java suite, no souffle/network): engine-id-test.sh (same id from two paths; changed by rule/map/manifest/pin) and engine-package-test.sh (a hand-made engine package is used when its id matches, refused with both ways out when not, absence explained).
• bin/axiomcode <mixed Java+TS+Python tree> <out> --library … → three graphs, Java library targets named.
• Package assembly + npm publish --dry-run locally: correct tarball contents.
• The four-platform compile itself was validated by build: prebuilt engine binaries, CI builds Linux/macOS/Windows on every merge and commits them under engine/binaries/ #455's CI runs (same steps).

[swapnilpaliwal-sd]

Merged into engine-prebuilt (#455) as its base, at the request of the repository owner: the npm mechanism replaces the committed-binaries workflow, the release fetch test and the binaries .gitattributes there (all superseded by build-engines.yml, publish-npm.yml and engine-package-test.sh). javascript is in the build matrix (36d0725d). Retargeted to engine-prebuilt; nothing here merges to main before the C# engine.

[swapnilpaliwal-sd]

Closing without merge: this branch is fully contained in engine-prebuilt (#455, merge aac70d76), which is the base for the CI gate (#418) and lands after the C# engine. Nothing from here goes to main on its own.

[swapnilpaliwal-sd]

Reopened: the engine build and npm publish stays its own open pull request until it merges to main. The same commits were also merged into #418's branch; whichever merges first carries the work, the other rebases.

[swapnilpaliwal-sd]

Not mergeable as it stands: package.json conflicts with main at 45fa172. Rebase and it can be re-checked. I have not touched the branch.

[swapnilpaliwal-sd]

Cross-checking this branch against a real install, on GCP rather than Actions. Results and two gaps.

Verified working. Built all engines from this branch for five platforms and ran them end to end. The engine id is path independent (same tree at two paths gives the same id), the generated C++ carries no build-machine paths, and every platform's ENGINE_ID matches the generate step. Smoke counts agree across three operating systems and two architectures: java 22, typescript 45, python 85, javascript 34 relations. engine-id-test.sh and engine-package-test.sh both pass, and a staged node_modules/@axiomcode/engine-darwin-arm64 resolves and runs with souffle absent, in 1s against 849s for a local compile. A corrupted ENGINE_ID is refused, as intended.

Gap 1: darwin-x64 is not built. The matrix covers darwin-arm64, linux-x64, linux-arm64 and win32-x64, so an Intel Mac gets no engine and falls back to needing souffle and a compiler. A missing optional dependency is not an install error, so it fails silently. This needs no Intel hardware and no extra runner: the Apple Silicon runner cross compiles with -arch x86_64 and Rosetta 2 runs the result, so the same job can build and smoke both. Measured here: the x86_64 java engine writes the same 22 relations and links only /usr/lib.

Gap 2: the build depends on Actions minutes. With none available this branch cannot produce a binary at all, which blocks a release for a reason unrelated to the tool. Worth a documented path that does not need runners. The shape of this design makes that cheap, since souffle -g runs once and every platform then only needs a C++17 compiler: the build VMs never install souffle, and Windows never needs a souffle that does not exist for it natively.

Notes from doing it:

• GCP cannot build macOS at all. Compute Engine has no macOS machine type or image, and Apple's licence permits macOS virtualisation only on Apple hardware. Both Mac engines have to come from a Mac.
• Ubuntu 20.04 is no longer published by ubuntu-os-cloud. 22.04 is the oldest available, so glibc 2.35 is the floor.
• On Windows Server, Invoke-WebRequest without -UseBasicParsing throws, because the IE engine has never completed first launch configuration. It cost a completed build its upload once, after all four engines had compiled and smoke tested.

#887 is required before this ships. It fixes four defects in front of the engine: the built parser is not in the tarball, there is no bin entry, the bin root walk fails through the node_modules/.bin symlink, and no runtime dependencies are declared. With those in place the engine resolves correctly from the package and the run still fails, because the parser half is not installable. On a clean VM with both changes, all four languages complete against real projects with nothing compiled.

[swapnilpaliwal-sd]

Resolved the conflict on this branch locally and did not merge, because the merge result
regresses the suite. Two separate findings.

1. The conflict hid a real drop. graph/pipeline/run-souffle.sh conflicted because
this branch extracts the program generation into write_program() while main added a
phase folder inside that same block. Taking either side wholesale is wrong:

write_program(): for d in projections containment resolution config-resolution \
                        expression-resolution call-edge-generation
main:            for d in ... call-edge-generation framework-behavior

framework-behavior/ is where persistence.dl and destinations.dl live. Resolving in
this branch's favour, which is what a careful merge looks like since the refactor is the
point of the PR, silently stops including them: no error, no conflict, the rules simply
never reach the program. Worth adding framework-behavior to write_program() on this
branch now so the next person to merge cannot get it wrong.

2. With that resolved and built, the suite regresses. On the merge result the bundle
stage fails for every language before any case runs:

✗ java: bundle failed:   SyntaxError: Unexpected token '{'
    at Loader.moduleStrategy (internal/modules/esm/translators.js:141:18)
✗ typescript: bundle failed:  (same)
engine-id: 8 failure(s)
aborting: the engine id is not a function of the rules alone

internal/modules/esm/translators.js is a Node 12/14 path; this machine has v14.15.0 at
/usr/local/bin/node and v22.23.2 on PATH, so something in this branch is reaching the
older interpreter where main does not.

Control: current main, same shell, same PATH, same Node v22.23.2, same JDK: 56
passed, 0 failed. So this is not the environment and not a pre-existing failure.

Also worth noting for validation: all four optionalDependencies resolve to
UNMET OPTIONAL DEPENDENCY @axiomcode/engine-*@0.1.0, so the packaging path this PR
exists to establish cannot actually be exercised until those are published. That is not
itself a blocker for optional deps, but it does mean the interesting half is untested.

I have not pushed anything to this branch. The conflict resolution above is the part I am
confident about; the interpreter question belongs to whoever owns the packaging design.

[swapnilpaliwal-sd]

Correcting my comment above and clearing this for merge. Pushed 2a7a31b7.

I was wrong about the bundle. I reported the bundle stage failing for every language
with a Node 12/14 ESM syntax error. That was my environment: /usr/local/bin/node is
v14.15.0 on this machine and had won the PATH race in that particular shell. Re-run with
v22.23.2 confirmed, the bundle is ok (3 languages, ...). Nothing in this PR touches it.
I should have run the control before posting, and did not.

The engine-id and engine-package failures were real, and are fixed here. Both
sandboxes rebuilt PATH from a whitelist of symlinked tools. That cannot work on macOS:
/usr/bin/shasum is a perl script and the system perl dispatches on the script's
CANONICAL path, so a symlink to it is refused, and so is a copy:

$ echo hi | shasum -a 256                 # works
98ea6e4f...
$ echo hi | PATH=$TMP/bin shasum -a 256   # symlink or copy, either way
perl version 5.30.3 can't run /var/folders/.../bin/shasum.  Try the alternative(s):
(Error: no alternatives found)

There is no sha256sum on a macOS box without coreutils, so the sandbox had no digest
tool at all. engine-id reported 8 failures and engine-package 4, none about the
engine. Both now hide souffle by dropping the one PATH entry that holds it, and assert
souffle is really gone; every other tool stays where the system expects it.

Verified with souffle present, which is the case that matters until the packages are
published.

java suite	56 passed, 0 failed (engine-id ok, engine-package ok)
cold run, empty engine cache	compiling souffle program (cache miss), 3m19s, 486 call edges
warm run, same engine id	3.0s, 486 call edges, identical
engine id	stable across both, rules + souffle 2.5

The cache lands souffle-engine-java-<engine-id>, so the compile happens once and the id
is a function of the rules and the souffle version as intended.

I also carried framework-behavior into write_program(). That was the conflict's real
content: this branch moved the phase list into the function while main added the folder
to it, so resolving in this branch's favour without noticing would have silently stopped
including persistence.dl and destinations.dl.

Understood that the UNMET OPTIONAL DEPENDENCY lines are expected until the engine
packages are published, so I am no longer treating that as a blocker.

[swapnilpaliwal-sd]

Blocker for this branch, filed as #895 with the fix in #897 against main.

The engine id is a function of the shell locale as well as the rules. The #include lines are built from globs, and bash orders a glob by LC_COLLATE: under a UTF-8 collation call-site.dl and callee-resolution.dl swap places against their byte order, so the program text and the id change with the environment.

Measured with engines built from this branch, installed as a real package:

language	macOS	Windows (MSYS2, en_US.UTF-8)
java	16a17ec0	d79f318c
typescript	e8df2d7c	e8df2d7c
python	934aa018	1302327180
javascript	c135763f	c135763f

End to end on Windows with the correct engine installed, java and python fail with engine-win32-x64 holds java at 16a17ec0..., these rules are d79f318c... - not using it, while typescript and javascript pass. The refusal is this branch's id check working correctly on wrong inputs. TypeScript and JavaScript agree only because no pair of their rule filenames collides.

CI runs under a C-ish locale and users commonly do not, so published engines would be refused for java and python on a large share of machines, with an error naming the rules rather than the locale. #897 pins the collation in the executor; write_program here needs the same guard, and engine-id-test.sh should assert invariance under at least two collations. Note it is invisible on macOS, whose collation matches C either way.

Separately, linux-arm64 cannot install at all, unrelated to the engine: the 0.21.x tree-sitter generation ships no linux-arm64 prebuilds, so npm falls back to node-gyp. tree-sitter-groovy also pins tree-sitter-java@0.23.4 exactly against the parser's ^0.21.0, forcing a nested second copy whose gyp config miscounts its relative path and fails even with a full toolchain. Prebuild coverage:

package	linux-arm64
tree-sitter@0.21.1	no
tree-sitter@0.22.4	yes
tree-sitter-java@0.21.0	no
tree-sitter-java@0.23.4	yes
tree-sitter-python@0.21.0	no
tree-sitter-python@0.23.6	yes

That one is a parser dependency upgrade and needs the suites run against the newer grammars, so it is not folded into either PR.