Repository navigation
No CI gate on main: the regression suites never run before a merge, and a suite that skips itself reports success #417
Description
Activity
Two findings from getting the suites onto a clean machine, both fixed in #418
1. The java torture families cannot be scored without the platform IR
The harness stages the JVM platform IR as a library on purpose: half the families call
java.util.List,Mapand the functional interfaces, and with the platform absent those
receivers cannot be typed by any rule. Measured on a run with it removed:with platform IR without missing-edge census 10 edges 23 edges recall vs possible — 0.847 The harness already warns that in this state "this score measures the staging, not the
rules". That IR is 1.8 GB and is built from a JDK source checkout, so it cannot live in
a repository, an Actions cache, or a runner.Running the families anyway would paint a red that reads as a regression in whatever
change happened to meet it — the most expensive kind of false signal, because the next
person debugs their own diff. CI therefore passes--no-torture, and the suite prints
that the families were excluded: an excluded family must never be mistakeable for one
that passed. They remain a local gate.2. Client->library coverage could vanish without anything noticing
Worth stating plainly, because the coverage is better than it looks:
front end client->client client->library typescript 23 cases 23 cases, each solved twice — the delta between the two goldens IS the mapping java 39 cases 6 cases shipping a lib-src/stub library, passed as--librarypython 12 cases + 2 whole-project fixtures torture (client + lib, with a monotonicity invariant) So client->library is already exercised broadly, and all of it runs in CI. Restricting to
the torture scripts would be a reduction — and for java specifically, torture is the one
part that cannot run in CI at all, per the finding above.The real risk was different: the client->library half rests on fewer fixtures, and
deleting one of its goldens breaks nothing. The case still runs, still passes, and quietly
stops making the claim, because a suite can only check the assertions it still has.
test/tools/lib-coverage.shnow pins the shape — both goldens per TypeScript lib case,
a golden per java stub case, and floors that cannot fall without being lowered in the same
commit. No parser, no solver, runs in seconds.Both torture harnesses also already assert library monotonicity — solving the same
client against an empty library must never produce MORE answers than solving it against
the real one. That is the property most worth having and it is easy to lose silently.Still open, not blocking this PR
- python has no ground truth in CI. Its frozen CPython oracle is authored by a harness
that is not in version control anywhere, so the python leg is goldens-only. Weaker than
the other two legs, and the workflow says so where it matters. 01-inheritance-overrideflaked once — failed inside a full run, then passed alone,
on re-run, and again under--oracle. A cross-language solver-cache collision was
probed and disproved. A flake erodes a merge gate faster than a missing test does, so
this deserves its own issue.- shipping prebuilt engine binaries. The engine is generated C++ compiled per machine,
and on a cache hitrun-souffle.shnever invokes the solver at all — so a prebuilt
engine would free consumers from needing either Souffle or a C++ toolchain. It does not
help this repo's CI, which must compile the rules as changed in the PR; a prebuilt
binary there would score rules that predate the change under review. Two notes if it is
taken up:-march=native(run-souffle.sh:285, :352) would have to go, and upstream ships
x86_64 Linux only, so macOS, Windows and arm64 builds would be ours to produce and
maintain.
- python has no ground truth in CI. Its frozen CPython oracle is authored by a harness
- added a commit that references this issue
on Sep 19, 2026
mainhas no automated gate. The three regression suites run when somebody remembersto run them, which is not often enough to be relied on.
What CI has to get right here
The suites are written for a developer machine, where a missing dependency is a good
reason to step aside: they print
SKIPand exit 77. On CI that is the one outcome thatmust never be tolerated, because a gate that opens when its input is missing is worse
than no gate, the green is read as evidence by whoever comes next. Two instances, both
of which would otherwise pass unnoticed:
test/java/torturereads class files withjava.lang.classfile, which is not finalbefore JDK 24. On an older JDK it exits 77 and the ten-family oracle does not run,
while the suite still reports success
Every shape of "did not actually run" therefore has to become a failure: exit 77, a
sub-harness that swallowed its own 77, and a suite reporting
passed 0.Scope
java / python / typescript suites in parallel
wrong answer satisfies as long as it was wrong last time too. Java and TypeScript run
with
--oracle, scored against javac/javap and the TypeScript compiler respectivelyred with no commit here to point at, and a parser regression cannot quietly become the
new expectation
mainthat opens an issue when the pin falls behindCODEOWNERSBranch protection
The ruleset for
main, pull request required, approving review,CIgreen, linearhistory, no force-push, no deletion, no bypass actors, is written as
.github/scripts/protect-main.shand applied separately from this change.