build: engines ship on npm as @axiomcode/engine-<os>-<cpu>

[swapnilpaliwal-sd]

Restores the work reverted in #903 so it can be reviewed before it ships. Reopens the problem tracked in #454.

Identical to what was on main, with the two fixes that landed on top of it kept.

What is in it

souffle -g turns the rules into portable C++ once, and every platform compiles that with its own C++17 compiler. Soufflé is a build-time dependency only, so a user needs neither Soufflé nor a compiler. The executor resolves, in order: the engine package npm installed for this machine when its ENGINE_ID matches the rules in the checkout, then a locally cached binary, then a local compile, and otherwise it errors naming both ways out.

Fixes kept from on top of it

• The collation pin (URGENT: the engine id and cache key depend on the shell locale, so a published engine is refused on any machine with a UTF-8 collation #895) stays inside write_program. Without it the engine id follows the user's locale, and a published engine is refused for java and python on any machine with a UTF-8 collation, which is the common configuration. Verified: java and python ids differ between macOS and MSYS2, and between LC_ALL=C and en_US.UTF-8 on glibc.
• bin, files and dependencies (npm install of the published package cannot run: parser dist not shipped, no bin entry, bin root walk fails through the node_modules/.bin symlink, no runtime dependencies #886) stay, so an install can run at all. optionalDependencies returns here, with the code that reads them.

Verified

Engines built for five platforms and run end to end from a real npm install, on a machine with no Soufflé and no compiler:

platform	languages run	result
darwin-arm64	4 real projects	all pass
darwin-x64	cross-compiled, run under Rosetta	passes
linux-x64	4 real projects	all pass
win32-x64	4 real projects	2 pass, 2 blocked by #895 before it was fixed
linux-arm64	install fails	see below

Every platform's ENGINE_ID matched the generate step, and smoke counts agreed across three operating systems and two architectures: java 22, typescript 45, python 85, javascript 34 relations.

Known gaps, neither fixed here

• linux-arm64 cannot install (linux-arm64: npm install fails, no arm64 prebuilds in the 0.21.x tree-sitter generation and a nested duplicate that cannot build #901). No dependency in the 0.21.x tree-sitter generation ships an arm64 prebuild, and tree-sitter-groovy pins tree-sitter-java@0.23.4 against the parser's ^0.21.0, forcing a nested copy that fails to build even with a full toolchain. This is unrelated to the engine: the install fails before any engine is resolved. The fix is a dependency upgrade that changes which grammar the parser uses, so it needs the suites run against it.
• Nothing builds the binaries while the CI workflows are out of the tree (chore: take the CI workflows out of the tree for now #898), so a release means building by hand on each platform. packaging/assemble-engine-package.sh still turns an engines-<platform>/ directory into a publishable package.
• Publish order matters. npm resolves optional dependencies at install time, so the platform packages have to be published before this one, or the install silently produces no engine.

[swapnilpaliwal-sd]

On hold: not to be merged. The repository owner has held the npm engine packaging, the bundle change and the CI gating from main for now.

Recorded here because several sessions are working this repo concurrently and can merge. Please do not merge this, and do not merge it on someone else's behalf. It is not a review finding and says nothing about the change's quality.

[swapnilpaliwal-sd]

Holding this until the engine work is complete, then merging it with the related PRs rather than piecemeal.

Correction: #909 is no longer held with this batch and has been merged on its own. It is independent of the engine work, it fixed a defect that was on main already, and it was a prerequisite for validating anything on arm64, since without it the install fails before an engine is ever resolved.

What #909 fixed. tree-sitter-groovy pins tree-sitter-java to an exact version while the parser asked for a range, so npm nested a second copy whose gyp config resolves node-addon-api by a relative path that is wrong once nested, and it could not build even with a full toolchain. Measured on a clean arm64 VM: the install previously failed with build-essential present and now succeeds, and the parser produces 39 csv files and 48128 rows on a real Java project, matching macOS exactly. IR is byte for byte identical to the previous grammar on real Java and Python projects.

What it did not fix. linux-arm64 still needs build-essential. No 0.21.x tree-sitter core ships an arm64 prebuild, and every published tree-sitter-java and tree-sitter-groovy pins the core to 0.21.x through its peer range, so no version set reaches one. Bumping the core to 0.22.4 resolves only with an npm overrides entry, and overrides apply at the root of an install, so a consumer still gets 0.21.1 hoisted and lands where they started.

So of the five platforms, four install with no compiler and no souffle; linux-arm64 installs with a compiler. Worth stating in the install docs rather than leaving it to a node-gyp trace that blames tree-sitter-c-sharp. #901 stays open for the upstream gap.